diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
new file mode 100644
index 000000000..8f8b3171d
--- /dev/null
+++ b/CONTRIBUTING.md
@@ -0,0 +1,407 @@
+# Contributing to Intel® AI Reference Models
+
+## Adding new sample
+
+> [!NOTE]
+> At the moment repository is being updated to a new repo structure to simplify
+> development and content layout. This section describes how to add new samples
+> in the new repo structure.
+
+To add a new sample:
+
+1. Create home directory for the new sample with sample scripts and documentation
+1. Add baremetal tests
+1. (optional) Add docker(s) for your sample
+1. (optional) Add tests for dockers
+1. Update project documentation with the new sample description
+
+Project documentation update must include an update of top level
+[README](./README.md) with the new sample name, supported precisions and a link
+to the new sample README.md.
+
+### Sample home directory
+
+Scripts and documentation of each sample must be placed in the separate sample
+home directory. Home directory must be named as follows:
+```
+mkdir -p models_v2/<framework>/<model-name>/<mode>/<platform>/
+```
+|              | Values                                        |
+| ------------ | --------------------------------------------- |
+| `framework`  | `pytorch` or `tensorflow`                     |
+| `model-name` | Actual model name, for example `efficientnet` |
+| `mode`       | `inference` or `training`                     |
+| `platform`   | `cpu` or `gpu`                                |
+
+Sample home directory must contain the following files:
+
+| File               |            | Purpose                                                                |
+| ------------------ | ---------- | ---------------------------------------------------------------------- |
+| `README.md`        | Required   | Readme describing the sample, original model source, and how to use it |
+| `setup.sh`         | Required   | Shell script to setup runtime environment on baremetal                 |
+| `run_model.sh`     | Required   | Shell script to run the sample                                         |
+| `tests.yaml`       | Required   | YAML file with test cases for CI/CD (see details below)                |
+| `*.*`              | Optional   | Other files and scripts used by the sample                             |
+
+If few samples share common utilities they can be placed at `models_v2/common`.
+Note that common folder may contain framework specific utilities if needed. Do
+not create common folder on framework folder level (i.e. do NOT create
+`models_v2/<framework>/common`). Utilties placed on `models_v2/common` must
+be useful across multiple samples and have solid API to avoid breaking multiple
+samples with modifications.
+
+If sample generates output files, one of the output files should be named
+`results.yaml` and have the following format:
+```
+results:
+- key: <key>
+  value: <value>
+  unit: <unit>
+```
+
+Common results keys:
+
+| Key        | Value | Unit                        |
+| ---------- | ----- | --------------------------- |
+| accuracy   | float | percents                    |
+| latency    | float | seconds, milliseconds, etc. |
+| throughput | float | images/s, records/s, etc.   |
+
+If environment variables are used in the sample scripts their names must be
+consistent across all samples available in the repository. The following table
+lists such environemnt variables.
+
+| Environment variable | Values            | Purpose                                   |
+| -------------------- | ----------------- | ----------------------------------------- |
+| `BATCH_SIZE`         |                   | Batch size                                |
+| `DATASET_DIR`        |                   | Path to data set                          |
+| `MULTI_TILE`         | `False` or `True` | Sets whether GPU is single or multi tile  |
+| `OUTPUT_DIR`         |                   | Path to output logs                       |
+| `PLATFORM`           |                   | Platform on which sample runs             |
+| `PRECISION`          | `bf16`, `fp16`, `fp32`, `int8` | Model precision              |
+| `NUM_ITERATIONS`     |                   | Number of iterations to perform in a test |
+
+### Baremetal tests
+
+Test cases to run should be specified in `tests.yaml` file stored in the sample
+home directory (see above). This YAML file must have the following format:
+```
+<test-case>:
+  cmd: <command>; command;
+  env:
+    <name>: <value>
+```
+
+`cmd` key sets commands to execute during test run. Tests must be executed by
+[test-runner.py script](https://github.com/intel/ai-containers/tree/main/test-runner).
+Before executing tests download and configure test runner as follows:
+```
+git clone https://github.com/intel/ai-containers.git
+python -m pip install -r ai-containers/test-runner/requirements.txt
+```
+
+Run tests by passing `tests.yaml` to the test runner:
+```
+pyton test_runner.py -f /path/to/tests.yaml -l /path/to/logs
+```
+
+The following environment variables must be set before running the test runner
+and can be used in `cmd` commands:
+
+| Environment variable          | Value                                                                           |
+| ----------------------------- | ------------------------------------------------------------------------------- |
+| `GITHUB_WORKSPACE`            | Path to the cloned copy of this repository                                      |
+| `FRAMEWORK_VERSION`           | Framework (pytorch or tesorflow) version to install (ex.: `2.14.0`)             |
+| `FRAMEWORK_EXTENSION_VERSION` | Version of Intel framework extension (IPEX or ITEX) to install (ex.: `2.14.0.1` |
+| `IS_LKG_DROP`                 | `true` or `false`: whether test runs under LKG environment                      |
+| `AIKIT_VERSION`               | Version of AI kit (valid only of `IS_LKG_DROP` is `true`)                       |
+
+If test specific files or scripts are needed (which is typically the case),
+they must be placed at the following locations:
+
+* Place framework specific scripts at `tests/cicd/<framework>/` folder where
+  `framework` is `pytorch`, `tensorflow`, `IPEX-XPU` or `ITEX-XPU`. The
+  following scripts are required:
+
+  | Script     |            | Purpose                      |
+  | ---------- | ---------- | ---------------------------- |
+  | `setup.sh` | Required   |  Script to install framework |
+
+* Place test specific scripts at `tests/cicd/<framework>/<model>-<mode>`
+  folder where `model` is model name being tested and `mode` is
+  `inference` or `training`. The following scripts are required:
+
+  | Script        |           | Purpose                     |
+  | ------------- | ----------| --------------------------- |
+  | `run_test.sh` | Required  | Script to execute test case |
+
+Test related scripts (`setup.sh` and `run_tests.sh` in particular) should exit
+with `0` to indicate passing result.
+
+### Containers home directory
+
+Optionally sample might provide containers for easy setup. Container
+Dockerfile(s) must be placed at the following directories:
+
+| Folder | Purpose |
+| ------ | ------- |
+| `docker/flex-gpu/<framework>-<model>-<mode>/` | Containers for [Intel® Data Center GPU Flex Series](https://ark.intel.com/content/www/us/en/ark/products/series/230021/intel-data-center-gpu-flex-series.html) |
+| `docker/max-gpu/<framework>-<model>-<mode>/`  | Containers for [Intel® Data Center GPU Max Series](https://ark.intel.com/content/www/us/en/ark/products/series/232874/intel-data-center-gpu-max-series.html)  |
+| `docker/pyt-cpu/<model>-<mode>/`              | Containers for pytorch running on CPU    |
+| `docker/tf-cpu/<model>-<mode>/`               | Containers for Tensorflow running on CPU |
+
+
+Each container folder must contain the following files:
+
+| File               |            | Purpose                                                      |
+| ------------------ | ---------- | ------------------------------------------------------------ |
+| `*Dockerfile`      | Required   | Dockerfile to build the sample                               |
+| `README.md`        | Optional   | Readme with container build and run instructions             |
+| `tests.yaml`       | Required   | YAML file with test cases for CI/CD (see details below)      |
+
+The Dockerfile should be buildable with `docker build` via command line. The
+Dockerfile may include optional and/or required command line arguments for custom
+container builds. In this case a README must describe the container build arguments
+that were used.
+
+New container must be added to the `docker-compose.yml` file located
+in the parent folder:
+
+* [docker/flex-gpu/docker-compose.yml](./docker/flex-gpu/docker-compose.yml)
+* [docker/max-gpu/docker-compose.yml](./docker/max-gpu/docker-compose.yml)
+* [docker/pyt-cpu/docker-compose.yml](./docker/pyt-cpu/docker-compose.yml)
+* [docker/tf-cpu/docker-compose.yml](./docker/tf-cpu/docker-compose.yml)
+
+Note that CI/CD is using `docker compose build` to build containers.
+
+### Container tests
+
+Test cases for containers should be specified in `tests.yaml` file stored in
+the containers home directory (see above). This YAML file must have the
+following format:
+```
+<test-case>:
+  img: <docker-image>
+  cmd: <command>; command;
+  ipc: host
+  device: /dev/dri
+  env:
+    <name>: <value>
+  volumes:
+    - src: <host-folder>
+      dst: <container-folder>
+```
+
+`cmd` key sets commands to execute during test run. Tests must be executed by
+[test-runner.py script](https://github.com/intel/ai-containers/tree/main/test-runner).
+Before executing tests download and configure test runner as follows:
+```
+git clone https://github.com/intel/ai-containers.git
+python -m pip install -r ai-containers/test-runner/requirements.txt
+```
+
+Run tests by passing `tests.yaml` to the test runner:
+```
+pyton test_runner.py -f /path/to/tests.yaml -l /path/to/logs
+```
+
+The following environment variables must be set before running the test runner
+and can be used in `cmd` commands:
+
+| Environment variable          | Value                                                                           |
+| ----------------------------- | ------------------------------------------------------------------------------- |
+| `REGISTRY`                    | Docker registry to fetch docker images from                                     |
+| `GITHUB_RUN_NUMBER`           | A unique number for each run of a particular workflow                           |
+
+## OLD: Adding scripts for a new TensorFlow model
+
+This section covers contribution guidelines for the old repo structure. Please,
+refer to this section only if you update not yet migrated content.
+
+### Code updates
+
+In order to add a new model to the zoo, there are a few things that are
+required:
+
+1. Setup the directory structure to allow the
+   [launch script](/docs/general/tensorflow/LaunchBenchmark.md) to find
+   your model. This involves creating folders for:
+   `/benchmarks/<use case>/<framework>/<model name>/<mode>/<precision>`.
+   Note that you will need to add `__init__.py` files in each new
+   directory that you add, in order for python to find the code.
+
+   ![Directory Structure](benchmarks_directory_structure.png)
+
+2. Next, in the leaf folder that was created in the previous step, you
+   will need to create `config.json` and `model_init.py` files:
+
+   ![Add model init](add_model_init_and_config.png)
+
+    The `config.json` file contains the best known KMP environment variable
+    settings to get optimal performance for the model. Below default settings are recommended for most of
+    the models in Model Zoo.
+
+    ```
+    {
+        "optimization_parameters": {
+            "KMP_AFFINITY": "granularity=fine,verbose,compact,1,0",
+            "KMP_BLOCKTIME": 1,
+            "KMP_SETTINGS": 1
+        }
+    }
+    ```
+
+   The `model_init.py` file is used to initialize the best known configuration for the
+   model, and then start executing inference or training. When the
+   [launch script](/docs/general/tensorflow/LaunchBenchmark.md) is run,
+   it will look for the appropriate `model_init.py` file to use
+   according to the model name, framework, mode, and precision that are
+   specified by the user.
+
+   The contents of the `model_init.py` file will vary by framework. For
+   TensorFlow models, we typically use the
+   [base model init class](/benchmarks/common/base_model_init.py) that
+   includes functions for doing common tasks such as setting up the best
+   known environment variables (like `KMP_BLOCKTIME`, `KMP_SETTINGS`,
+   `KMP_AFFINITY` by loading **config.json** and `OMP_NUM_THREADS`), num intra threads, and num
+   inter threads. The `model_init.py` file also sets up the string that
+   will ultimately be used to run inference or model training, which
+   normally includes the use of `numactl` and sending all of the
+   appropriate arguments to the model's script. Also, if your model
+   requires any non-standard arguments (arguments that are not part of
+   the [launch script flags](/docs/general/tensorflow/LaunchBenchmark.md#launch_benchmarkpy-flags)),
+   the `model_init.py` file is where you would define and parse those
+   args.
+
+3. [start.sh](/benchmarks/common/tensorflow/start.sh) is a shell script
+   that is called by the `launch_benchmarks.py` script in the docker
+   container. This script installs dependencies that are required by
+   the model, sets up the `PYTHONPATH` environment variable, and then
+   calls the [run_tf_benchmark.py](/benchmarks/common/tensorflow/run_tf_benchmark.py)
+   script with the appropriate args. That run script will end up calling
+   the `model_init.py` file that you have defined in the previous step.
+
+   To add support for a new model in the `start.sh` script, you will
+   need to add a function with the same name as your model. Note that
+   this function name should match the `<model name>` folder from the
+   first step where you setup the directories for your model. In this
+   function, add commands to install any third-party dependencies within
+   an `if [ ${NOINSTALL} != "True" ]; then` conditional block. The
+   purpose of the `--noinstall` flag or `NOINSTALL` env var is to be able 
+   to skip the installs for quicker iteration when running on bare metal 
+   or debugging. If your model requires the `PYTHONPATH` environment variable 
+   to be setup to find model code or dependencies, that should be done in the
+   model's function. Next, setup the command that will be run. The
+   standard launch script args are already added to the `CMD` variable,
+   so your model function will only need to add on more args if you have
+   model-specific args defined in your `model_init.py`. Lastly, call the
+   `run_model` function with the `PYTHONPATH` and the `CMD` string.
+
+   Below is a sample template of a `start.sh` model function that
+   installs dependencies from `requirements.txt` file, sets up the
+   `PYHTONPATH` to find model source files, adds on a custom steps flag
+   to the run command, and then runs the model:
+   ```bash
+   function <model_name>() {
+     if [ ${PRECISION} == "fp32" ]; then
+       if [ ${NOINSTALL} != "True" ]; then
+         pip install -r ${MOUNT_EXTERNAL_MODELS_SOURCE}/requirements.txt
+       fi
+
+       export PYTHONPATH=${PYTHONPATH}:${MOUNT_EXTERNAL_MODELS_SOURCE}
+       CMD="${CMD} $(add_steps_args)"
+       PYTHONPATH=${PYTHONPATH} CMD=${CMD} run_model
+     else
+       echo "PRECISION=${PRECISION} is not supported for ${MODEL_NAME}"
+       exit 1
+     fi
+   }
+   ```
+
+Optional step:
+* If there is CPU-optimized model code that has not been upstreamed to
+  the original repository, then it can be added to the
+  [models](/models) directory in the zoo repo. As with the first step
+  in the previous section, the directory structure should be setup like:
+  `/models/<use case>/<framework>/<model name>/<mode>/<precision>`.
+
+  ![Models Directory Structure](models_directory_structure.png)
+
+  If there are model files that can be shared by multiple modes or
+  precisions, they can be placed the higher-level directory. For
+  example, if a file could be shared by both `FP32` and `Int8`
+  precisions, then it could be placed in the directory at:
+  `/models/<use case>/<framework>/<model name>/<mode>` (omitting the
+  `<precision>` directory). Note that if this is being done, you need to
+  ensure that the license that is associated with the original model
+  repository is compatible with the license of the model zoo.
+
+### Debugging
+
+There are a couple of options for debugging and quicker iteration when
+developing new scripts:
+* Use the `--debug` flag in the launch_benchmark.py script, which will
+  give you a shell into the docker container. See the
+  [debugging section](/docs/general/tensorflow/LaunchBenchmark.md#debugging)
+  of the launch script documentation for more information on using this
+  flag.
+* Run the launch script on bare metal (without a docker container). The
+  launch script documentation also has a
+  [section](/docs/general/tensorflow/LaunchBenchmark.md#alpha-feature-running-on-bare-metal)
+  with instructions on how to do this. Note that when running without
+  docker, you are responsible for installing all dependencies on your
+  system before running the launch script. If you are using this option
+  during development, be sure to also test _with_ a docker container to
+  ensure that the `start.sh` script dependency installation is working
+  properly for your model.
+
+### Documentation updates
+
+1. Create a `README.md` file in the
+   `/benchmarks/<use case>/<framework>/<model name>` directory:
+
+   ![Add README file](add_readme.png)
+
+   This README file should describe all of the steps necessary to run
+   the model, including downloading and preprocessing the dataset,
+   downloading the pretrained model, cloning repositories, and running
+   the model script with the appropriate arguments. Most models
+   have best known settings for batch and online inference performance
+   testing as well as testing accuracy. The README file should specify
+   how to set these configs using the `launch_benchmark.py` script.
+
+2. Update the table in the [main `benchmarks` README](/benchmarks/README.md)
+   with a link to the model that you are adding. Note that the models
+   in this table are ordered alphabetically by use case, framework, and
+   model name. The model name should link to the original paper for the
+   model. The instructions column should link to the README
+   file that you created in the previous step.
+
+### Testing
+
+1. After you've completed the above steps, run the model according to
+   instructions in the README file for the new model. Ensure that the
+   performance and accuracy metrics are on par with what you would
+   expect.
+
+2. Add unit tests to cover the new model.
+   * For TensorFlow models, there is a
+     [parameterized test](/tests/unit/common/tensorflow/test_run_tf_benchmarks.py#L80)
+     that checks the flow running from `run_tf_benchmarks.py` to the
+     inference command that is executed by the `model_init.py` file. The
+     test ensures that the inference command has all of the expected
+     arguments.
+
+     To add a new parameterized instance of the test for your
+     new model, add a new JSON file `tf_<model_name>_args.json` to the [tf_models_args](/tests/unit/common/tensorflow/tf_model_args)
+     directory. Each file has a list of dictionaries, a dictionary has three
+     items: (1) `_comment` a comment describes the command,
+     (2) `input` the `run_tf_benchmarks.py` command with the appropriate
+     flags to run the model (3) `output` the expected inference or training
+     command that should get run by the `model_init.py` file.
+   * If any launch script or base class files were changed, then
+     additional unit tests should be added.
+   * Unit tests and style checks are run when you post a GitHub PR, and
+     the tests must be passing before the PR is merged.
+   * For information on how to run the unit tests and style checks
+     locally, see the [tests documentation](/tests/README.md).
diff --git a/Contribute.md b/Contribute.md
deleted file mode 100644
index 9d3a74850..000000000
--- a/Contribute.md
+++ /dev/null
@@ -1,191 +0,0 @@
-# Contributing to the Model Zoo for Intel® Architecture
-
-## Adding scripts for a new TensorFlow model
-
-### Code updates
-
-In order to add a new model to the zoo, there are a few things that are
-required:
-
-1. Setup the directory structure to allow the
-   [launch script](/docs/general/tensorflow/LaunchBenchmark.md) to find
-   your model. This involves creating folders for:
-   `/benchmarks/<use case>/<framework>/<model name>/<mode>/<precision>`.
-   Note that you will need to add `__init__.py` files in each new
-   directory that you add, in order for python to find the code.
-
-   ![Directory Structure](benchmarks_directory_structure.png)
-
-2. Next, in the leaf folder that was created in the previous step, you
-   will need to create `config.json` and `model_init.py` files:
-
-   ![Add model init](add_model_init_and_config.png)
-
-    The `config.json` file contains the best known KMP environment variable
-    settings to get optimal performance for the model. Below default settings are recommended for most of
-    the models in Model Zoo.
-
-    ```
-    {
-        "optimization_parameters": {
-            "KMP_AFFINITY": "granularity=fine,verbose,compact,1,0",
-            "KMP_BLOCKTIME": 1,
-            "KMP_SETTINGS": 1
-        }
-    }
-    ```
-
-   The `model_init.py` file is used to initialize the best known configuration for the
-   model, and then start executing inference or training. When the
-   [launch script](/docs/general/tensorflow/LaunchBenchmark.md) is run,
-   it will look for the appropriate `model_init.py` file to use
-   according to the model name, framework, mode, and precision that are
-   specified by the user.
-
-   The contents of the `model_init.py` file will vary by framework. For
-   TensorFlow models, we typically use the
-   [base model init class](/benchmarks/common/base_model_init.py) that
-   includes functions for doing common tasks such as setting up the best
-   known environment variables (like `KMP_BLOCKTIME`, `KMP_SETTINGS`,
-   `KMP_AFFINITY` by loading **config.json** and `OMP_NUM_THREADS`), num intra threads, and num
-   inter threads. The `model_init.py` file also sets up the string that
-   will ultimately be used to run inference or model training, which
-   normally includes the use of `numactl` and sending all of the
-   appropriate arguments to the model's script. Also, if your model
-   requires any non-standard arguments (arguments that are not part of
-   the [launch script flags](/docs/general/tensorflow/LaunchBenchmark.md#launch_benchmarkpy-flags)),
-   the `model_init.py` file is where you would define and parse those
-   args.
-
-3. [start.sh](/benchmarks/common/tensorflow/start.sh) is a shell script
-   that is called by the `launch_benchmarks.py` script in the docker
-   container. This script installs dependencies that are required by
-   the model, sets up the `PYTHONPATH` environment variable, and then
-   calls the [run_tf_benchmark.py](/benchmarks/common/tensorflow/run_tf_benchmark.py)
-   script with the appropriate args. That run script will end up calling
-   the `model_init.py` file that you have defined in the previous step.
-
-   To add support for a new model in the `start.sh` script, you will
-   need to add a function with the same name as your model. Note that
-   this function name should match the `<model name>` folder from the
-   first step where you setup the directories for your model. In this
-   function, add commands to install any third-party dependencies within
-   an `if [ ${NOINSTALL} != "True" ]; then` conditional block. The
-   purpose of the `--noinstall` flag or `NOINSTALL` env var is to be able 
-   to skip the installs for quicker iteration when running on bare metal 
-   or debugging. If your model requires the `PYTHONPATH` environment variable 
-   to be setup to find model code or dependencies, that should be done in the
-   model's function. Next, setup the command that will be run. The
-   standard launch script args are already added to the `CMD` variable,
-   so your model function will only need to add on more args if you have
-   model-specific args defined in your `model_init.py`. Lastly, call the
-   `run_model` function with the `PYTHONPATH` and the `CMD` string.
-
-   Below is a sample template of a `start.sh` model function that
-   installs dependencies from `requirements.txt` file, sets up the
-   `PYHTONPATH` to find model source files, adds on a custom steps flag
-   to the run command, and then runs the model:
-   ```bash
-   function <model_name>() {
-     if [ ${PRECISION} == "fp32" ]; then
-       if [ ${NOINSTALL} != "True" ]; then
-         pip install -r ${MOUNT_EXTERNAL_MODELS_SOURCE}/requirements.txt
-       fi
-
-       export PYTHONPATH=${PYTHONPATH}:${MOUNT_EXTERNAL_MODELS_SOURCE}
-       CMD="${CMD} $(add_steps_args)"
-       PYTHONPATH=${PYTHONPATH} CMD=${CMD} run_model
-     else
-       echo "PRECISION=${PRECISION} is not supported for ${MODEL_NAME}"
-       exit 1
-     fi
-   }
-   ```
-
-Optional step:
-* If there is CPU-optimized model code that has not been upstreamed to
-  the original repository, then it can be added to the
-  [models](/models) directory in the zoo repo. As with the first step
-  in the previous section, the directory structure should be setup like:
-  `/models/<use case>/<framework>/<model name>/<mode>/<precision>`.
-
-  ![Models Directory Structure](models_directory_structure.png)
-
-  If there are model files that can be shared by multiple modes or
-  precisions, they can be placed the higher-level directory. For
-  example, if a file could be shared by both `FP32` and `Int8`
-  precisions, then it could be placed in the directory at:
-  `/models/<use case>/<framework>/<model name>/<mode>` (omitting the
-  `<precision>` directory). Note that if this is being done, you need to
-  ensure that the license that is associated with the original model
-  repository is compatible with the license of the model zoo.
-
-### Debugging
-
-There are a couple of options for debugging and quicker iteration when
-developing new scripts:
-* Use the `--debug` flag in the launch_benchmark.py script, which will
-  give you a shell into the docker container. See the
-  [debugging section](/docs/general/tensorflow/LaunchBenchmark.md#debugging)
-  of the launch script documentation for more information on using this
-  flag.
-* Run the launch script on bare metal (without a docker container). The
-  launch script documentation also has a
-  [section](/docs/general/tensorflow/LaunchBenchmark.md#alpha-feature-running-on-bare-metal)
-  with instructions on how to do this. Note that when running without
-  docker, you are responsible for installing all dependencies on your
-  system before running the launch script. If you are using this option
-  during development, be sure to also test _with_ a docker container to
-  ensure that the `start.sh` script dependency installation is working
-  properly for your model.
-
-### Documentation updates
-
-1. Create a `README.md` file in the
-   `/benchmarks/<use case>/<framework>/<model name>` directory:
-
-   ![Add README file](add_readme.png)
-
-   This README file should describe all of the steps necessary to run
-   the model, including downloading and preprocessing the dataset,
-   downloading the pretrained model, cloning repositories, and running
-   the model script with the appropriate arguments. Most models
-   have best known settings for batch and online inference performance
-   testing as well as testing accuracy. The README file should specify
-   how to set these configs using the `launch_benchmark.py` script.
-
-2. Update the table in the [main `benchmarks` README](/benchmarks/README.md)
-   with a link to the model that you are adding. Note that the models
-   in this table are ordered alphabetically by use case, framework, and
-   model name. The model name should link to the original paper for the
-   model. The instructions column should link to the README
-   file that you created in the previous step.
-
-### Testing
-
-1. After you've completed the above steps, run the model according to
-   instructions in the README file for the new model. Ensure that the
-   performance and accuracy metrics are on par with what you would
-   expect.
-
-2. Add unit tests to cover the new model.
-   * For TensorFlow models, there is a
-     [parameterized test](/tests/unit/common/tensorflow/test_run_tf_benchmarks.py#L80)
-     that checks the flow running from `run_tf_benchmarks.py` to the
-     inference command that is executed by the `model_init.py` file. The
-     test ensures that the inference command has all of the expected
-     arguments.
-
-     To add a new parameterized instance of the test for your
-     new model, add a new JSON file `tf_<model_name>_args.json` to the [tf_models_args](/tests/unit/common/tensorflow/tf_model_args)
-     directory. Each file has a list of dictionaries, a dictionary has three
-     items: (1) `_comment` a comment describes the command,
-     (2) `input` the `run_tf_benchmarks.py` command with the appropriate
-     flags to run the model (3) `output` the expected inference or training
-     command that should get run by the `model_init.py` file.
-   * If any launch script or base class files were changed, then
-     additional unit tests should be added.
-   * Unit tests and style checks are run when you post a GitHub PR, and
-     the tests must be passing before the PR is merged.
-   * For information on how to run the unit tests and style checks
-     locally, see the [tests documentation](/tests/README.md).
diff --git a/Makefile b/Makefile
index aab2b068d..e19dd069e 100644
--- a/Makefile
+++ b/Makefile
@@ -53,6 +53,20 @@ unit_test:
 	@echo "Running unit tests..."
 	tox -e py3-py.test
 
+test_tl_tf_notebook: venv
+	@. $(ACTIVATE) && pip install -r docs/notebooks/transfer_learning/requirements.txt && \
+	bash docs/notebooks/transfer_learning/run_tl_notebooks.sh $(CURDIR)/docs/notebooks/transfer_learning/text_classification/tfhub_bert_text_classification/BERT_Binary_Text_Classification.ipynb remove_for_custom_dataset && \
+	bash docs/notebooks/transfer_learning/run_tl_notebooks.sh $(CURDIR)/docs/notebooks/transfer_learning/image_classification/tf_image_classification/Image_Classification_Transfer_Learning.ipynb remove_for_custom_dataset && \
+	bash docs/notebooks/transfer_learning/run_tl_notebooks.sh $(CURDIR)/docs/notebooks/transfer_learning/image_classification/huggingface_image_classification/HuggingFace_Image_Classification_Transfer_Learning.ipynb && \
+	bash docs/notebooks/transfer_learning/run_tl_notebooks.sh $(CURDIR)/docs/notebooks/transfer_learning/question_answering/BERT_Question_Answering.ipynb && \
+	bash docs/notebooks/transfer_learning/run_tl_notebooks.sh $(CURDIR)/docs/notebooks/transfer_learning/text_classification/tfhub_bert_text_classification/BERT_Multi_Text_Classification.ipynb remove_for_custom_dataset
+
+test_tl_pyt_notebook: venv
+	@. $(ACTIVATE) && pip install -r docs/notebooks/transfer_learning/requirements.txt && \
+	bash docs/notebooks/transfer_learning/run_tl_notebooks.sh $(CURDIR)/docs/notebooks/transfer_learning/image_classification/pytorch_image_classification/PyTorch_Image_Classification_Transfer_Learning.ipynb remove_for_custom_dataset && \
+	bash docs/notebooks/transfer_learning/run_tl_notebooks.sh $(CURDIR)/docs/notebooks/transfer_learning/object_detection/pytorch_object_detection/PyTorch_Object_Detection_Transfer_Learning.ipynb remove_for_custom_dataset && \
+	bash docs/notebooks/transfer_learning/run_tl_notebooks.sh $(CURDIR)/docs/notebooks/transfer_learning/text_classification/pytorch_text_classification/PyTorch_Text_Classifier_fine_tuning.ipynb remove_for_custom_dataset
+
 test: lint unit_test
 
 clean:
diff --git a/README.md b/README.md
index b1bef9ef4..35030f9e9 100644
--- a/README.md
+++ b/README.md
@@ -44,12 +44,10 @@ For best performance on Intel® Data Center GPU Flex and Max Series, please chec
 | ------------------------------------------------------ | ---------- | ----------| ------------------- | ---------------------- |
 | [DenseNet169](https://arxiv.org/pdf/1608.06993.pdf)    | TensorFlow | Inference | [FP32](/benchmarks/image_recognition/tensorflow/densenet169/inference/README.md) | [ImageNet 2012](https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md) |
 | [Inception V3](https://arxiv.org/pdf/1512.00567.pdf)   | TensorFlow | Inference | [Int8 FP32](/benchmarks/image_recognition/tensorflow/inceptionv3/inference/README.md) | [ImageNet 2012](https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md) |
-| [Inception V4](https://arxiv.org/pdf/1602.07261.pdf)   | TensorFlow | Inference | [Int8 FP32](/benchmarks/image_recognition/tensorflow/inceptionv4/inference/README.md) | [ImageNet 2012](https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md) |
 | [MobileNet V1*](https://arxiv.org/pdf/1704.04861.pdf)  | TensorFlow | Inference | [Int8 FP32 BFloat16](/benchmarks/image_recognition/tensorflow/mobilenet_v1/inference/README.md) | [ImageNet 2012](https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md) |
 | [MobileNet V1*](https://arxiv.org/pdf/1704.04861.pdf)  [Sapphire Rapids](https://www.intel.com/content/www/us/en/newsroom/opinion/updates-next-gen-data-center-platform-sapphire-rapids.html#gs.blowcx) | TensorFlow | Inference | [Int8 FP32 BFloat16 BFloat32](/quickstart/image_recognition/tensorflow/mobilenet_v1/inference/cpu/README_SPR_baremetal.md) | [ImageNet 2012](https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md) |
 | [MobileNet V2](https://arxiv.org/pdf/1801.04381.pdf) | Tensorflow | Inference | [FP32 BFloat16 Int8](/benchmarks/image_recognition/tensorflow/mobilenet_v2/inference/README.md) | [ImageNet 2012](https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md)
 | [ResNet 101](https://arxiv.org/pdf/1512.03385.pdf)     | TensorFlow | Inference |  [Int8 FP32](/benchmarks/image_recognition/tensorflow/resnet101/inference/README.md) | [ImageNet 2012](https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md) |
-| [ResNet 50](https://arxiv.org/pdf/1512.03385.pdf)      | TensorFlow | Inference | [Int8 FP32](/benchmarks/image_recognition/tensorflow/resnet50/inference/README.md) | [ImageNet 2012](https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md) |
 | [ResNet 50v1.5](https://github.com/tensorflow/models/tree/v2.11.0/official/legacy/image_classification/resnet) | TensorFlow | Inference | [Int8 FP32 BFloat16 FP16](/benchmarks/image_recognition/tensorflow/resnet50v1_5/inference/README.md) | [ImageNet 2012](https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md) |
 | [ResNet 50v1.5](https://github.com/tensorflow/models/tree/v2.11.0/official/legacy/image_classification/resnet)  [Sapphire Rapids](https://www.intel.com/content/www/us/en/newsroom/opinion/updates-next-gen-data-center-platform-sapphire-rapids.html#gs.blowcx) | TensorFlow | Inference | [Int8 FP32 BFloat16 BFloat32](/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/README_SPR_baremetal.md) | [ImageNet 2012](https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md) |
 | [ResNet 50v1.5](https://github.com/tensorflow/models/tree/v2.11.0/official/legacy/image_classification/resnet) | TensorFlow | Training |  [FP32 BFloat16 FP16](/benchmarks/image_recognition/tensorflow/resnet50v1_5/training/README.md) | [ImageNet 2012](https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md) |
@@ -60,12 +58,12 @@ For best performance on Intel® Data Center GPU Flex and Max Series, please chec
 | [Inception v3](https://arxiv.org/pdf/1512.00567.pdf)   | PyTorch | Inference | [FP32 BFloat16](/quickstart/image_recognition/pytorch/inception_v3/inference/cpu/README.md) | [ImageNet 2012](/quickstart/image_recognition/pytorch/inception_v3/inference/cpu/README.md#datasets) |
 | [MNASNet 0.5](https://arxiv.org/abs/1807.11626)        | PyTorch | Inference | [FP32 BFloat16](/quickstart/image_recognition/pytorch/mnasnet0_5/inference/cpu/README.md) | [ImageNet 2012](/quickstart/image_recognition/pytorch/mnasnet0_5/inference/cpu/README.md#datasets) |
 | [MNASNet 1.0](https://arxiv.org/abs/1807.11626)      | PyTorch | Inference | [FP32 BFloat16](/quickstart/image_recognition/pytorch/mnasnet1_0/inference/cpu/README.md) | [ImageNet 2012](/quickstart/image_recognition/pytorch/mnasnet1_0/inference/cpu/README.md#datasets) |
-| [ResNet 50](https://arxiv.org/pdf/1512.03385.pdf)    | PyTorch | Inference | [FP32 BFloat16 BFloat32](/quickstart/image_recognition/pytorch/resnet50/inference/cpu/README.md) | [ImageNet 2012](/quickstart/image_recognition/pytorch/resnet50/inference/cpu/README.md#datasets) |
+| [ResNet 50](https://arxiv.org/pdf/1512.03385.pdf)    | PyTorch | Inference | [Int8 FP32 BFloat16 BFloat32](/quickstart/image_recognition/pytorch/resnet50/inference/cpu/README.md) | [ImageNet 2012](/quickstart/image_recognition/pytorch/resnet50/inference/cpu/README.md#datasets) |
 | [ResNet 50](https://arxiv.org/pdf/1512.03385.pdf)    | PyTorch | Training  | [FP32 BFloat16 BFloat32](/quickstart/image_recognition/pytorch/resnet50/training/cpu/README.md) | [ImageNet 2012](/quickstart/image_recognition/pytorch/resnet50/training/cpu/README.md#datasets) |
 | [ResNet 101](https://arxiv.org/pdf/1512.03385.pdf)   | PyTorch | Inference | [FP32 BFloat16](/quickstart/image_recognition/pytorch/resnet101/inference/cpu/README.md) | [ImageNet 2012](/quickstart/image_recognition/pytorch/resnet101/inference/cpu/README.md#datasets) |
 | [ResNet 152](https://arxiv.org/pdf/1512.03385.pdf)   | PyTorch | Inference | [FP32 BFloat16](/quickstart/image_recognition/pytorch/resnet152/inference/cpu/README.md) | [ImageNet 2012](/quickstart/image_recognition/pytorch/resnet152/inference/cpu/README.md#datasets) |
 | [ResNext 32x4d](https://arxiv.org/abs/1611.05431)    | PyTorch | Inference | [FP32 BFloat16](/quickstart/image_recognition/pytorch/resnext-32x4d/inference/cpu/README.md) | [ImageNet 2012](/quickstart/image_recognition/pytorch/resnext-32x4d/inference/cpu/README.md#datasets) |
-| [ResNext 32x16d](https://arxiv.org/abs/1611.05431)   | PyTorch | Inference | [FP32 BFloat16 BFloat32](/quickstart/image_recognition/pytorch/resnext-32x16d/inference/cpu/README.md) | [ImageNet 2012](/quickstart/image_recognition/pytorch/resnext-32x16d/inference/cpu/README.md#datasets) |
+| [ResNext 32x16d](https://arxiv.org/abs/1611.05431)   | PyTorch | Inference | [Int8 FP32 BFloat16 BFloat32](/quickstart/image_recognition/pytorch/resnext-32x16d/inference/cpu/README.md) | [ImageNet 2012](/quickstart/image_recognition/pytorch/resnext-32x16d/inference/cpu/README.md#datasets) |
 | [VGG-11](https://arxiv.org/abs/1409.1556)            | PyTorch | Inference | [FP32 BFloat16](/quickstart/image_recognition/pytorch/vgg11/inference/cpu/README.md) | [ImageNet 2012](/quickstart/image_recognition/pytorch/vgg11/inference/cpu/README.md#datasets) |
 | [VGG-11 with batch normalization](https://arxiv.org/abs/1409.1556) | PyTorch | Inference | [FP32 BFloat16](/quickstart/image_recognition/pytorch/vgg11_bn/inference/cpu/README.md) | [ImageNet 2012](/quickstart/image_recognition/pytorch/vgg11_bn/inference/cpu/README.md#datasets) |
 | [Wide ResNet-50-2](https://arxiv.org/pdf/1605.07146.pdf)   | PyTorch | Inference | [FP32 BFloat16](/quickstart/image_recognition/pytorch/wide_resnet50_2/inference/cpu/README.md) | [ImageNet 2012](/quickstart/image_recognition/pytorch/wide_resnet50_2/inference/cpu/README.md#datasets) |
@@ -92,7 +90,7 @@ For best performance on Intel® Data Center GPU Flex and Max Series, please chec
 | [BERT base](https://arxiv.org/pdf/1810.04805.pdf)    | PyTorch | Inference | [FP32 BFloat16](/quickstart/language_modeling/pytorch/bert_base/inference/cpu/README.md) | [BERT Base SQuAD1.1](https://huggingface.co/csarron/bert-base-uncased-squad-v1) |
 | [BERT large](https://arxiv.org/pdf/1810.04805.pdf)   | PyTorch | Inference | [FP32 Int8 BFloat16 BFloat32](/quickstart/language_modeling/pytorch/bert_large/inference/cpu/README.md) | BERT Large SQuAD1.1 |
 | [BERT large](https://arxiv.org/pdf/1810.04805.pdf)   | PyTorch | Training  | [FP32 BFloat16 BFloat32](/quickstart/language_modeling/pytorch/bert_large/training/cpu/README.md) | [preprocessed text dataset](https://drive.google.com/drive/folders/1cywmDnAsrP5-2vsr8GDc6QUc7VWe-M3v) |
-| [DistilBERT base](https://arxiv.org/abs/1910.01108)  | PyTorch | Inference | [FP32 Int8 BFloat16 BFloat32](/quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/README.md) | [ DistilBERT Base SQuAD1.1](https://huggingface.co/distilbert-base-uncased-distilled-squad) |
+| [DistilBERT base](https://arxiv.org/abs/1910.01108)  | PyTorch | Inference | [FP32 Int8-FP32 Int8-BFloat16 BFloat16 BFloat32](/quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/README.md) | [ DistilBERT Base SQuAD1.1](https://huggingface.co/distilbert-base-uncased-distilled-squad) |
 | [RNN-T](https://arxiv.org/abs/2007.15188)            | PyTorch | Inference | [FP32 BFloat16 BFloat32](/quickstart/language_modeling/pytorch/rnnt/inference/cpu/README.md) | [RNN-T dataset](/quickstart/language_modeling/pytorch/rnnt/inference/cpu/download_dataset.sh) |
 | [RNN-T](https://arxiv.org/abs/2007.15188)            | PyTorch | Training  | [FP32 BFloat16 BFloat32](/quickstart/language_modeling/pytorch/rnnt/training/cpu/README.md) | [RNN-T dataset](/quickstart/language_modeling/pytorch/rnnt/training/cpu/download_dataset.sh) |
 | [RoBERTa base](https://arxiv.org/abs/1907.11692)     | PyTorch | Inference | [FP32 BFloat16](/quickstart/language_modeling/pytorch/roberta_base/inference/cpu/README.md) | [RoBERTa Base SQuAD 2.0](https://huggingface.co/deepset/roberta-base-squad2) |
@@ -103,7 +101,6 @@ For best performance on Intel® Data Center GPU Flex and Max Series, please chec
 | Model                                                           | Framework  | Mode      | Model Documentation | Benchmark/Test Dataset |
 | --------------------------------------------------------------- | ---------- | ----------| ------------------- | ---------------------- |
 | [BERT](https://arxiv.org/pdf/1810.04805.pdf)                    | TensorFlow | Inference | [FP32](/benchmarks/language_translation/tensorflow/bert/inference/README.md) | [MRPC](https://github.com/IntelAI/models/tree/master/datasets/bert_data/README.md#classification-training-with-bert) |
-| [GNMT*](https://arxiv.org/pdf/1609.08144.pdf)                   | TensorFlow | Inference | [FP32](/benchmarks/language_translation/tensorflow/mlperf_gnmt/inference/README.md) | [MLPerf GNMT model benchmarking dataset](https://github.com/IntelAI/models/tree/master/benchmarks/language_translation/tensorflow/mlperf_gnmt/inference/fp32#datasets) |
 | [Transformer_LT_mlperf*](https://arxiv.org/pdf/1706.03762.pdf)  | TensorFlow | Inference | [FP32 BFloat16 Int8](/benchmarks/language_translation/tensorflow/transformer_mlperf/inference/README.md) | [WMT English-German data](https://github.com/IntelAI/models/tree/master/datasets/transformer_data#transformer-language-mlperf-dataset) |
 | [Transformer_LT_mlperf*](https://arxiv.org/pdf/1706.03762.pdf) [Sapphire Rapids](https://www.intel.com/content/www/us/en/newsroom/opinion/updates-next-gen-data-center-platform-sapphire-rapids.html#gs.blowcx) | Tensorflow | Inference | [FP32 BFloat16 Int8 BFloat32](/quickstart/language_translation/tensorflow/transformer_mlperf/inference/cpu/README_SPR_Baremetal.md) | [WMT English-German dataset](https://github.com/IntelAI/models/tree/master/datasets/transformer_data#transformer-language-mlperf-dataset) |
 | [Transformer_LT_mlperf*](https://arxiv.org/pdf/1706.03762.pdf)  | TensorFlow | Training | [FP32 BFloat16](/benchmarks/language_translation/tensorflow/transformer_mlperf/training/README.md) | [WMT English-German dataset](https://github.com/IntelAI/models/tree/master/datasets/transformer_data#transformer-language-mlperf-dataset) |
@@ -115,7 +112,6 @@ For best performance on Intel® Data Center GPU Flex and Max Series, please chec
 
 | Model                                                 | Framework  | Mode      | Model Documentation | Benchmark/Test Dataset |
 | ----------------------------------------------------- | ---------- | ----------| ------------------- | ---------------------- |
-| [R-FCN](https://arxiv.org/pdf/1605.06409.pdf)         | TensorFlow | Inference | [Int8 FP32](/benchmarks/object_detection/tensorflow/rfcn/inference/README.md) | [COCO 2017 validation dataset](https://github.com/IntelAI/models/tree/master/datasets/coco#download-and-preprocess-the-coco-validation-images) |
 | [SSD-MobileNet*](https://arxiv.org/pdf/1704.04861.pdf)| TensorFlow | Inference | [Int8 FP32 BFloat16](/benchmarks/object_detection/tensorflow/ssd-mobilenet/inference/README.md) | [COCO 2017 validation dataset](https://github.com/IntelAI/models/tree/master/datasets/coco#download-and-preprocess-the-coco-validation-images) |
 | [SSD-MobileNet*](https://arxiv.org/pdf/1704.04861.pdf)  [Sapphire Rapids](https://www.intel.com/content/www/us/en/newsroom/opinion/updates-next-gen-data-center-platform-sapphire-rapids.html#gs.blowcx) | TensorFlow | Inference | [Int8 FP32 BFloat16 BFloat32](/quickstart/object_detection/tensorflow/ssd-mobilenet/inference/cpu/README_SPR_baremetal.md) | [COCO 2017 validation dataset](https://github.com/IntelAI/models/tree/master/datasets/coco#download-and-preprocess-the-coco-validation-images) |
 | [SSD-ResNet34*](https://arxiv.org/pdf/1512.02325.pdf) | TensorFlow | Inference | [Int8 FP32 BFloat16](/benchmarks/object_detection/tensorflow/ssd-resnet34/inference/README.md) | [COCO 2017 validation dataset](https://github.com/IntelAI/models/tree/master/datasets/coco#download-and-preprocess-the-coco-validation-images) |
@@ -139,21 +135,21 @@ For best performance on Intel® Data Center GPU Flex and Max Series, please chec
 | [DIEN](https://arxiv.org/abs/1809.03672)  [Sapphire Rapids](https://www.intel.com/content/www/us/en/newsroom/opinion/updates-next-gen-data-center-platform-sapphire-rapids.html#gs.blowcx) | TensorFlow | Inference | [FP32 BFloat16 BFloat32](/quickstart/recommendation/tensorflow/dien/inference/cpu/README_SPR_baremetal.md) | [DIEN dataset](https://github.com/IntelAI/models/tree/master/benchmarks/recommendation/tensorflow/dien/inference#datasets) |
 | [DIEN](https://arxiv.org/abs/1809.03672) | TensorFlow | Training | [FP32](/benchmarks/recommendation/tensorflow/dien/training/README.md) | [DIEN dataset](https://github.com/IntelAI/models/tree/master/benchmarks/recommendation/tensorflow/dien#1-prepare-datasets-1) |
 | [DIEN](https://arxiv.org/abs/1809.03672) [Sapphire Rapids](https://www.intel.com/content/www/us/en/newsroom/opinion/updates-next-gen-data-center-platform-sapphire-rapids.html#gs.blowcx) | TensorFlow | Training | [FP32 BFloat16 BFloat32](/quickstart/recommendation/tensorflow/dien/training/cpu/README_SPR_baremetal.md) | [DIEN dataset](https://github.com/IntelAI/models/tree/master/benchmarks/recommendation/tensorflow/dien#1-prepare-datasets-1) |
-| [NCF](https://arxiv.org/pdf/1708.05031.pdf) | TensorFlow | Inference | [FP32](/benchmarks/recommendation/tensorflow/ncf/inference/fp32/README.md) | [MovieLens 1M](https://github.com/IntelAI/models/tree/master/benchmarks/recommendation/tensorflow/ncf/inference/fp32#datasets) |
 | [Wide & Deep](https://arxiv.org/pdf/1606.07792.pdf) | TensorFlow | Inference | [FP32](/benchmarks/recommendation/tensorflow/wide_deep/inference/README.md) | [Census Income dataset](https://github.com/IntelAI/models/tree/master/benchmarks/recommendation/tensorflow/wide_deep/inference/fp32#dataset) |
 | [Wide & Deep Large Dataset](https://arxiv.org/pdf/1606.07792.pdf) | TensorFlow | Inference | [Int8 FP32](/benchmarks/recommendation/tensorflow/wide_deep_large_ds/inference/README.md) | [Large Kaggle Display Advertising Challenge dataset](https://github.com/IntelAI/models/tree/master/datasets/large_kaggle_advertising_challenge/README.md) |
-| [Wide & Deep Large Dataset](https://arxiv.org/pdf/1606.07792.pdf) | TensorFlow | Training | [FP32](/benchmarks/recommendation/tensorflow/wide_deep_large_ds/training/README.md) | [Large Kaggle Display Advertising Challenge dataset](https://github.com/IntelAI/models/tree/master/benchmarks/recommendation/tensorflow/wide_deep_large_ds/training/fp32#dataset) |
 | [DLRM](https://arxiv.org/pdf/1906.00091.pdf)         | PyTorch | Inference | [FP32 Int8 BFloat16 BFloat32](/quickstart/recommendation/pytorch/dlrm/inference/cpu/README.md) | [Criteo Terabyte](/quickstart/recommendation/pytorch/dlrm/inference/cpu/README.md#datasets) |
 | [DLRM](https://arxiv.org/pdf/1906.00091.pdf)         | PyTorch | Training  | [FP32 BFloat16 BFloat32](/quickstart/recommendation/pytorch/dlrm/training/cpu/README.md) | [Criteo Terabyte](/quickstart/recommendation/pytorch/dlrm/training/cpu/README.md#datasets) |
 | [DLRM v2](https://arxiv.org/pdf/1906.00091.pdf)         | PyTorch | Inference | [FP32 FP16 BFloat16 BFloat32 Int8](/quickstart/recommendation/pytorch/torchrec_dlrm/inference/cpu/README.md) | [Criteo 1TB Click Logs dataset](/quickstart/recommendation/pytorch/torchrec_dlrm/inference/cpu#datasets) |
 | [DLRM v2](https://arxiv.org/pdf/1906.00091.pdf)         | PyTorch | Training  | [FP32 FP16 BFloat16 BFloat32](/quickstart/recommendation/pytorch/torchrec_dlrm/training/cpu/README.md) | [Random dataset](/quickstart/recommendation/pytorch/torchrec_dlrm/training/cpu#datasets) |
 | [MEMREC-DLRM](https://arxiv.org/pdf/2305.07205.pdf)         | PyTorch | Inference  | [FP32](/quickstart/recommendation/pytorch/memrec_dlrm/inference/cpu/README.md) | [Criteo Terabyte](/quickstart/recommendation/pytorch/memrec_dlrm/inference/cpu/README.md#datasets) |
 
-### Text-to-Speech
+### Diffusion
 
 | Model                                           | Framework  | Mode      | Model Documentation | Benchmark/Test Dataset |
 | ----------------------------------------------- | ---------- | ----------| ------------------- | ---------------------- |
-| [WaveNet](https://arxiv.org/pdf/1609.03499.pdf) | TensorFlow | Inference | [FP32](/benchmarks/text_to_speech/tensorflow/wavenet/inference/fp32/README.md) |
+| [Stable Diffusion](https://keras.io/guides/keras_cv/generate_images_with_stable_diffusion/) | TensorFlow | Inference | [FP32 BFloat16 FP16](/benchmarks/diffusion/tensorflow/stable_diffusion/inference/README.md) | [COCO 2017 validation dataset](https://github.com/IntelAI/models/tree/master/datasets/coco#download-and-preprocess-the-coco-validation-images)
+| [Stable Diffusion](https://huggingface.co/stabilityai/stable-diffusion-2-1) | PyTorch | Inference | [FP32 BFloat16 FP16 BFloat32 Int8-FP32 Int8-BFloat16](/quickstart/diffusion/pytorch/stable_diffusion/inference/cpu/README.md) | [COCO 2017 validation dataset](https://github.com/IntelAI/models/tree/master/datasets/coco#download-and-preprocess-the-coco-validation-images)
+| [Stable Diffusion](https://huggingface.co/stabilityai/stable-diffusion-2-1) | PyTorch | Training | [FP32 BFloat16 FP16 BFloat32](/quickstart/diffusion/pytorch/stable_diffusion/training/cpu/README.md) | [cat images](https://huggingface.co/datasets/diffusers/cat_toy_example)
 
 ### Shot Boundary Detection
 
@@ -174,26 +170,29 @@ For best performance on Intel® Data Center GPU Flex and Max Series, please chec
 ## Intel® Data Center GPU Workloads
 | Model                             | Framework  | Mode      | GPU Type | Model Documentation |
 | ----------------------------------| ---------- | ----------| -------- | ------------------- |
-| [ResNet 50v1.5](https://github.com/tensorflow/models/tree/v2.11.0/official/legacy/image_classification/resnet) | TensorFlow | Inference | Flex Series | [Int8](/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/gpu/README_Flex_series.md) |
-| [ResNet 50 v1.5](https://arxiv.org/pdf/1512.03385.pdf)    | PyTorch | Inference | Flex Series |[Int8](/quickstart/image_recognition/pytorch/resnet50v1_5/inference/gpu/README_Flex_Series.md) |
+| [ResNet 50v1.5](https://github.com/tensorflow/models/tree/v2.11.0/official/legacy/image_classification/resnet) | TensorFlow | Inference | Flex Series | [Float32 TF32 Float16 BFloat16 Int8](/models_v2/tensorflow/resnet50v1_5/inference/gpu/README.md) |
+| [ResNet 50 v1.5](https://github.com/tensorflow/models/tree/v2.11.0/official/legacy/image_classification/resnet) | TensorFlow | Training | Max Series | [BFloat16 FP32](/models_v2/tensorflow/resnet50v1_5/training/gpu/README.md) |
+| [ResNet 50 v1.5](https://arxiv.org/pdf/1512.03385.pdf)    | PyTorch | Inference | Flex Series, Max Series, ARC Series |[Int8 FP32 FP16 TF32](/models_v2/pytorch/resnet50v1_5/inference/gpu/README.md) |
+| [ResNet 50 v1.5](https://arxiv.org/pdf/1512.03385.pdf)    | PyTorch | Training | Max Series, ARC Series |[BFloat16 TF32 FP32](/models_v2/pytorch/resnet50v1_5/training/gpu/README.md) |
+| [DistilBERT](https://arxiv.org/pdf/1910.01108.pdf) | PyTorch | Inference | Flex Series, Max Series | [FP32 FP16 BF16 TF32](/models_v2/pytorch/distilbert/inference/gpu/README.md) |
+| [DLRM v1](https://arxiv.org/pdf/1906.00091.pdf) | PyTorch | Inference | Flex Series | [FP16 FP32](/models_v2/pytorch/dlrm/inference/gpu/README.md) |
 | [SSD-MobileNet*](https://arxiv.org/pdf/1704.04861.pdf)| TensorFlow | Inference | Flex Series| [Int8](/quickstart/object_detection/tensorflow/ssd-mobilenet/inference/gpu/README.md) |
-| [SSD-MobileNet](https://arxiv.org/pdf/1704.04861.pdf)| PyTorch | Inference | Flex Series | [Int8](/quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/README.md) |
-| [Yolo V4](https://arxiv.org/pdf/1704.04861.pdf)| PyTorch | Inference | Flex Series | [Int8](/quickstart/object_detection/pytorch/yolov4/inference/gpu/README.md) |
-| [EfficientNet](https://arxiv.org/pdf/1905.11946.pdf)  | TensorFlow | Inference | Flex Series | [FP16](/quickstart/image_recognition/tensorflow/efficientnet/inference/gpu/README.md) |
-| [MaskRCNN](https://arxiv.org/pdf/1703.06870.pdf)  | TensorFlow | Inference | Flex Series | [FP16](/quickstart/image_segmentation/tensorflow/maskrcnn/inference/gpu/README.md) | 
-| [Stable Diffusion](https://arxiv.org/pdf/2112.10752.pdf)  | TensorFlow | Inference | Flex Series | [FP16 FP32](/quickstart/generative-ai/tensorflow/stable_diffusion/inference/gpu/README.md) |
-| [Stable Diffusion](https://arxiv.org/pdf/2112.10752.pdf)  | PyTorch | Inference | Flex Series | [FP16 FP32](/quickstart/generative-ai/pytorch/stable_diffusion/inference/gpu/README.md) |
+| [SSD-MobileNet*](https://arxiv.org/pdf/1704.04861.pdf)| PyTorch | Inference | ARC Series| [INT8 FP16 FP32](/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/README.md) |
+| [EfficientNet](https://arxiv.org/pdf/1905.11946.pdf)  | TensorFlow | Inference | Flex Series | [FP16](/models_v2/tensorflow/efficientnet/inference/gpu/README.md) |
+| [Wide Deep Large Dataset](https://arxiv.org/pdf/2112.10752.pdf)  | TensorFlow | Inference | Flex Series | [FP16](/models_v2/tensorflow/wide_deep_large_ds/inference/gpu/README.md) |
 | [Yolo V5](https://arxiv.org/pdf/2108.11539.pdf)  | PyTorch | Inference | Flex Series | [FP16](/quickstart/object_detection/pytorch/yolov5/inference/gpu/README.md) | 
-| [ResNet 50v1.5](https://github.com/tensorflow/models/tree/v2.11.0/official/legacy/image_classification/resnet) | TensorFlow | Inference | Max Series | [Int8 FP32 FP16](/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/gpu/README_Max_Series.md) |
-| [ResNet 50 v1.5](https://github.com/tensorflow/models/tree/v2.11.0/official/legacy/image_classification/resnet) | TensorFlow | Training | Max Series | [BFloat16](/quickstart/image_recognition/tensorflow/resnet50v1_5/training/gpu/README.md) |
-| [ResNet 50 v1.5](https://arxiv.org/pdf/1512.03385.pdf)    | PyTorch | Inference | Max Series |[Int8](/quickstart/image_recognition/pytorch/resnet50v1_5/inference/gpu/README_Max_Series.md) |
-| [ResNet 50 v1.5](https://arxiv.org/pdf/1512.03385.pdf)    | PyTorch | Training | Max Series |[BFloat16](/quickstart/image_recognition/pytorch/resnet50v1_5/training/gpu/README.md) |
-| [BERT large](https://arxiv.org/pdf/1810.04805.pdf)   | PyTorch | Inference | Max Series | [FP16](/quickstart/language_modeling/pytorch/bert_large/inference/gpu/README.md) |
-| [BERT large](https://arxiv.org/pdf/1810.04805.pdf)   | PyTorch | Training  | Max Series | [BFloat16](/quickstart/language_modeling/pytorch/bert_large/training/gpu/README.md) |
+| [BERT large](https://arxiv.org/pdf/1810.04805.pdf)   | PyTorch | Inference | Max Series, ARC Series | [BFloat16 FP32 FP16](/models_v2/pytorch/bert_large/inference/gpu/README.md) |
+| [BERT large](https://arxiv.org/pdf/1810.04805.pdf)   | PyTorch | Training  | Max Series, ARC Series | [BFloat16 FP32 TF32](/models_v2/pytorch/bert_large/training/gpu/README.md) |
 |[BERT large](https://arxiv.org/pdf/1810.04805.pdf) | TensorFlow | Inference | Max Series | [FP32 FP16](/quickstart/language_modeling/tensorflow/bert_large/inference/gpu/README.md) |
-| [BERT large](https://arxiv.org/pdf/1810.04805.pdf) | TensorFlow | Training | Max Series | [BFloat16](/quickstart/language_modeling/tensorflow/bert_large/training/gpu/README.md) |
-| [DLRM](https://arxiv.org/pdf/1906.00091.pdf) | TensorFlow | Inference | Max Series | [FP16](/quickstart/recommendation/pytorch/torchrec_dlrm/inference/gpu/README.md) |
-| [DLRM](https://arxiv.org/pdf/1906.00091.pdf) | TensorFlow | Training | Max Series | [BFloat16](/quickstart/recommendation/pytorch/torchrec_dlrm/training/gpu/README.md) |
+| [BERT large](https://arxiv.org/pdf/1810.04805.pdf) | TensorFlow | Training | Max Series | [BFloat16 TF32 FP32](/models_v2/tensorflow/bert_large/training/gpu/README.md) |
+| [DLRM v2](https://arxiv.org/pdf/1906.00091.pdf) | PyTorch | Inference | Max Series | [FP16 FP32](/models_v2/pytorch/torchrec_dlrm/inference/gpu/README.md) |
+| [DLRM v2](https://arxiv.org/pdf/1906.00091.pdf) | PyTorch | Training | Max Series | [BFloat16 FP32](/models_v2/pytorch/torchrec_dlrm/training/gpu/README.md) |
+| [3D-Unet](https://arxiv.org/pdf/1606.06650.pdf) | PyTorch | Inference | Max Series | [FP16 INT8 FP32](/models_v2/pytorch/3d_unet/inference/gpu/README.md) |
+| [3D-Unet](https://arxiv.org/pdf/1606.06650.pdf) | TensorFlow | Training | Max Series | [BFloat16 FP32](/models_v2/tensorflow/3d_unet/training/gpu/README.md) |
+| [Stable Diffusion](https://arxiv.org/pdf/2112.10752.pdf)  | PyTorch | Inference | Flex Series, Max Series, ARC Series | [FP16 FP32](/models_v2/pytorch/stable_diffusion/inference/gpu/README.md) |
+| [Stable Diffusion](https://arxiv.org/pdf/2112.10752.pdf)  | TensorFlow | Inference | Flex Series | [FP16 FP32](/models_v2/tensorflow/stable_diffusion/inference/gpu/README.md) |
+| [MaskRCNN](https://arxiv.org/pdf/1703.06870.pdf)  | TensorFlow | Inference | Flex Series | [FP32 Float16](/models_v2/tensorflow/maskrcnn/inference/gpu/README.md) | 
+| [MaskRCNN](https://arxiv.org/pdf/1703.06870.pdf)  | TensorFlow | Training | Max Series | [FP32 BFloat16](/models_v2/tensorflow/maskrcnn/training/gpu/README.md) | 
 
 ## How to Contribute
-If you would like to add a new benchmarking script, please use [this guide](/Contribute.md).
+If you would like to add a new benchmarking script, please use [this guide](/CONTRIBUTING.md).
diff --git a/benchmarks/common/base_model_init.py b/benchmarks/common/base_model_init.py
index ca0e3f26d..0795a3d37 100644
--- a/benchmarks/common/base_model_init.py
+++ b/benchmarks/common/base_model_init.py
@@ -219,9 +219,14 @@ def run_numactl_multi_instance(self, cmd, replace_unique_output_dir=None):
             if len(core_list) == 0:
                 continue
 
-            prefix = ("{0}OMP_NUM_THREADS={1} "
-                      "numactl --localalloc --physcpubind={2}").format(
-                ld_preload_prefix, len(core_list), ",".join(core_list))
+            if "OMP_NUM_THREADS" in os.environ:
+                prefix = ("{0}OMP_NUM_THREADS={1} "
+                          "numactl --localalloc --physcpubind={2}").format(
+                              ld_preload_prefix, os.environ["OMP_NUM_THREADS"], ",".join(core_list))
+            else:
+                prefix = ("{0}OMP_NUM_THREADS={1} "
+                          "numactl --localalloc --physcpubind={2}").format(
+                              ld_preload_prefix, len(core_list), ",".join(core_list))
             instance_logfile = log_filename_format.format("instance" + str(instance_num))
 
             unique_command = cmd
diff --git a/benchmarks/common/tensorflow/start.sh b/benchmarks/common/tensorflow/start.sh
index 80d338a6e..7bb50df68 100644
--- a/benchmarks/common/tensorflow/start.sh
+++ b/benchmarks/common/tensorflow/start.sh
@@ -1134,6 +1134,7 @@ function ssd_mobilenet() {
     done
   fi
   CMD="${CMD} $(add_steps_args)"
+  CMD="${CMD} $(add_arg "--input-subgraph" ${INPUT_SUBGRAPH})"
   PYTHONPATH=${PYTHONPATH} CMD=${CMD} run_model
 }
 
@@ -1817,11 +1818,11 @@ function stable_diffusion() {
             "https://github.com/openai/CLIP/blob/main/clip/bpe_simple_vocab_16e6.txt.gz?raw=true",
             file_hash="924691ac288e54409236115652ad4aa250f48203de50a9e4722a6ecd48d6804a",
         )\n_ = keras.utils.get_file(
-          origin="https://huggingface.co/fchollet/stable-diffusion/resolve/main/kcv_encoder.h5",
-          file_hash="4789e63e07c0e54d6a34a29b45ce81ece27060c499a709d556c7755b42bb0dc4",
+          origin="https://huggingface.co/ianstenbit/keras-sd2.1/resolve/main/text_encoder_v2_1.h5",
+          file_hash="985002e68704e1c5c3549de332218e99c5b9b745db7171d5f31fcd9a6089f25b",
         )\n_ = keras.utils.get_file(
-          origin="https://huggingface.co/fchollet/stable-diffusion/resolve/main/kcv_diffusion_model.h5",
-          file_hash="8799ff9763de13d7f30a683d653018e114ed24a6a819667da4f5ee10f9e805fe",
+          origin="https://huggingface.co/ianstenbit/keras-sd2.1/resolve/main/diffusion_model_v2_1.h5",
+          file_hash="c31730e91111f98fe0e2dbde4475d381b5287ebb9672b1821796146a25c5132d",
         )\n_ = keras.utils.get_file(
           origin="https://huggingface.co/fchollet/stable-diffusion/resolve/main/kcv_decoder.h5",
           file_hash="ad350a65cc8bc4a80c8103367e039a3329b4231c2469a1093869a345f55b1962",
@@ -1942,6 +1943,27 @@ function graphsage() {
     fi
 }
 
+function yolov5() {
+    if [ ${MODE} == "inference" ]; then
+      if [ ${PRECISION} == "fp32" ] || [ ${PRECISION} == "bfloat16" ]; then
+        export PYTHONPATH=${PYTHONPATH}:${MOUNT_EXTERNAL_MODELS_SOURCE}
+
+        if [ ${NUM_INTER_THREADS} != "None" ]; then
+          CMD="${CMD} $(add_arg "--num-inter-threads" ${NUM_INTER_THREADS})"
+        fi
+
+        if [ ${NUM_INTRA_THREADS} != "None" ]; then
+          CMD="${CMD} $(add_arg "--num-intra-threads" ${NUM_INTRA_THREADS})"
+        fi
+        CMD="${CMD} $(add_arg "--steps" ${STEPS})"
+        CMD=${CMD} run_model
+      else
+        echo "PRECISION=${PRECISION} not supported for ${MODEL_NAME} in this repo."
+        exit 1
+      fi
+    fi
+}
+
 LOGFILE=${OUTPUT_DIR}/${LOG_FILENAME}
 
 MODEL_NAME=$(echo ${MODEL_NAME} | tr 'A-Z' 'a-z')
@@ -2021,6 +2043,8 @@ elif [ ${MODEL_NAME} == "rgat" ]; then
   rgat
 elif [ ${MODEL_NAME} == "stable_diffusion" ]; then
   stable_diffusion
+elif [ ${MODEL_NAME} == "yolov5" ]; then
+  yolov5
 else
   echo "Unsupported model: ${MODEL_NAME}"
   exit 1
diff --git a/benchmarks/diffusion/__init__.py b/benchmarks/diffusion/__init__.py
new file mode 100644
index 000000000..eb0564d39
--- /dev/null
+++ b/benchmarks/diffusion/__init__.py
@@ -0,0 +1,17 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
diff --git a/benchmarks/diffusion/tensorflow/__init__.py b/benchmarks/diffusion/tensorflow/__init__.py
new file mode 100644
index 000000000..f07dbaf88
--- /dev/null
+++ b/benchmarks/diffusion/tensorflow/__init__.py
@@ -0,0 +1,19 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+#
diff --git a/benchmarks/diffusion/tensorflow/stable_diffusion/__init__.py b/benchmarks/diffusion/tensorflow/stable_diffusion/__init__.py
new file mode 100644
index 000000000..f07dbaf88
--- /dev/null
+++ b/benchmarks/diffusion/tensorflow/stable_diffusion/__init__.py
@@ -0,0 +1,19 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+#
diff --git a/benchmarks/diffusion/tensorflow/stable_diffusion/inference/README.md b/benchmarks/diffusion/tensorflow/stable_diffusion/inference/README.md
new file mode 100644
index 000000000..2e1dc3ffe
--- /dev/null
+++ b/benchmarks/diffusion/tensorflow/stable_diffusion/inference/README.md
@@ -0,0 +1,53 @@
+<!--- 0. Title -->
+# Stable Diffusion Inference
+
+<!-- 10. Description -->
+
+## Description
+This document has instructions for running inference based on a text prompt using the [KerasCV implementation](https://keras.io/guides/keras_cv/generate_images_with_stable_diffusion/) of [stability.ai's](https://stability.ai/) text-to-image model, [Stable Diffusion](https://github.com/CompVis/stable-diffusion). Stable Diffusion is a powerful, open-source text-to-image generation model.
+
+## Run the model
+
+### Run on Linux
+
+Install the Intel-optimized TensorFlow along with model dependencies under [requirements.txt](../../../../../models/diffusion/tensorflow/stable_diffusion/inference/requirements.txt).
+
+```
+# cd to your model zoo directory
+cd models
+export PRECISION=<set the precision to "fp32" or "bfloat16" or "fp16">
+export OUTPUT_DIR=<path to the directory where i) log files will be saved and ii) (only in case of accuracy run) original & generated images will be written>
+# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
+export BATCH_SIZE=<customized batch size value>
+```
+
+### Inference
+1. `inference.sh`
+Runs single-instance realtime inference (`batch_size=1`) with the specified precision (`fp32`, `bfloat16`, or `fp16`).
+```
+./quickstart/diffusion/tensorflow/stable_diffusion/inference/cpu/inference.sh
+```
+
+2. `inference_realtime_multi_instance.sh`
+Runs multi-instance realtime inference (`batch_size=1`) using 4 cores per instance with the specified precision (fp32, bfloat16, or fp16). Waits for all instances to complete, then prints a summarized throughput value.
+```
+./quickstart/diffusion/tensorflow/stable_diffusion/inference/cpu/inference_realtime_multi_instance.sh
+```
+
+3. `inference_throughput_multi_instance.sh`
+Runs multi instance batch inference using 1 socket per instance with the specified precision (fp32, bfloat16, or fp16) with 200 steps. Waits for all instances to complete, then prints a summarized throughput value.
+```
+./quickstart/diffusion/tensorflow/stable_diffusion/inference/cpu/inference_throughput_multi_instance.sh
+```
+
+### Accuracy
+
+1. `accuracy.sh`
+```
+# Additionally, set DATASET_DIR for accuracy runs. The original and the images generated by the model will be saved under OUTPUT_DIR.
+export DATASET_DIR=<path to the dataset directory>
+```
+
+```
+./quickstart/diffusion/tensorflow/stable_diffusion/inference/cpu/accuracy.sh
+```
diff --git a/benchmarks/diffusion/tensorflow/stable_diffusion/inference/__init__.py b/benchmarks/diffusion/tensorflow/stable_diffusion/inference/__init__.py
new file mode 100644
index 000000000..f07dbaf88
--- /dev/null
+++ b/benchmarks/diffusion/tensorflow/stable_diffusion/inference/__init__.py
@@ -0,0 +1,19 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+#
diff --git a/benchmarks/diffusion/tensorflow/stable_diffusion/inference/bfloat16/__init__.py b/benchmarks/diffusion/tensorflow/stable_diffusion/inference/bfloat16/__init__.py
new file mode 100644
index 000000000..f07dbaf88
--- /dev/null
+++ b/benchmarks/diffusion/tensorflow/stable_diffusion/inference/bfloat16/__init__.py
@@ -0,0 +1,19 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+#
diff --git a/benchmarks/object_detection/tensorflow/faster_rcnn/inference/fp32/config.json b/benchmarks/diffusion/tensorflow/stable_diffusion/inference/bfloat16/config.json
similarity index 100%
rename from benchmarks/object_detection/tensorflow/faster_rcnn/inference/fp32/config.json
rename to benchmarks/diffusion/tensorflow/stable_diffusion/inference/bfloat16/config.json
diff --git a/benchmarks/diffusion/tensorflow/stable_diffusion/inference/bfloat16/model_init.py b/benchmarks/diffusion/tensorflow/stable_diffusion/inference/bfloat16/model_init.py
new file mode 100644
index 000000000..58abca295
--- /dev/null
+++ b/benchmarks/diffusion/tensorflow/stable_diffusion/inference/bfloat16/model_init.py
@@ -0,0 +1,93 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+#
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+from common.base_model_init import BaseModelInitializer
+from common.base_model_init import set_env_var
+
+import os
+from argparse import ArgumentParser
+
+
+class ModelInitializer(BaseModelInitializer):
+    """initialize mode and run benchmark"""
+
+    def __init__(self, args, custom_args=[], platform_util=None):
+        super(ModelInitializer, self).__init__(args, custom_args, platform_util)
+
+        self.benchmark_command = ""
+        if not platform_util:
+            raise ValueError("Did not find any platform info.")
+
+        # use default batch size if -1
+        if self.args.batch_size == -1:
+            self.args.batch_size = 1
+
+        arg_parser = ArgumentParser(description='Parse args')
+
+        arg_parser.add_argument(
+            '--kmp-blocktime', dest='kmp_blocktime',
+            help='number of kmp block time',
+            type=int, default=1)
+        arg_parser.add_argument(
+            "--output-dir", dest='output_dir',
+            type=str, default='/tmp/stable_diffusion/',
+            help='Specify the location of the '
+            'output directory for saving original and generated images')
+        arg_parser.add_argument(
+            "--steps", dest='steps',
+            type=int, default=50,
+            help="number of steps for diffusion model")
+
+        self.args = arg_parser.parse_args(self.custom_args, namespace=self.args)
+
+        # Set KMP env vars, if they haven't already been set, but override the default KMP_BLOCKTIME value
+        config_file_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), "config.json")
+        self.set_kmp_vars(config_file_path, kmp_blocktime=str(self.args.kmp_blocktime))
+
+        set_env_var("OMP_NUM_THREADS", self.args.num_intra_threads)
+
+        benchmark_script = os.path.join(
+            self.args.intelai_models, self.args.mode, "eval_inference.py")
+
+        self.benchmark_command = self.get_command_prefix(args.socket_id) + \
+            self.python_exe + " " + benchmark_script
+
+        self.benchmark_command = \
+            self.benchmark_command + \
+            " --precision=" + str(self.args.precision) + \
+            " --batch-size=" + str(self.args.batch_size) + \
+            " --steps=" + str(self.args.steps) + \
+            " --output-dir=" + str(self.args.output_dir)
+
+        # if the data location directory is not empty, then include the arg
+        if self.args.data_location and os.listdir(self.args.data_location):
+            self.benchmark_command += " --data-location=" + self.args.data_location
+
+        if self.args.accuracy_only:
+            self.benchmark_command += " --accuracy-only"
+
+        print("Benchmark command: ", self.benchmark_command)
+
+    def run(self):
+        if self.benchmark_command:
+            self.run_command(self.benchmark_command)
diff --git a/benchmarks/diffusion/tensorflow/stable_diffusion/inference/fp16/__init__.py b/benchmarks/diffusion/tensorflow/stable_diffusion/inference/fp16/__init__.py
new file mode 100644
index 000000000..f07dbaf88
--- /dev/null
+++ b/benchmarks/diffusion/tensorflow/stable_diffusion/inference/fp16/__init__.py
@@ -0,0 +1,19 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+#
diff --git a/benchmarks/recommendation/tensorflow/ncf/inference/fp32/config.json b/benchmarks/diffusion/tensorflow/stable_diffusion/inference/fp16/config.json
similarity index 100%
rename from benchmarks/recommendation/tensorflow/ncf/inference/fp32/config.json
rename to benchmarks/diffusion/tensorflow/stable_diffusion/inference/fp16/config.json
diff --git a/benchmarks/diffusion/tensorflow/stable_diffusion/inference/fp16/model_init.py b/benchmarks/diffusion/tensorflow/stable_diffusion/inference/fp16/model_init.py
new file mode 100644
index 000000000..58abca295
--- /dev/null
+++ b/benchmarks/diffusion/tensorflow/stable_diffusion/inference/fp16/model_init.py
@@ -0,0 +1,93 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+#
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+from common.base_model_init import BaseModelInitializer
+from common.base_model_init import set_env_var
+
+import os
+from argparse import ArgumentParser
+
+
+class ModelInitializer(BaseModelInitializer):
+    """initialize mode and run benchmark"""
+
+    def __init__(self, args, custom_args=[], platform_util=None):
+        super(ModelInitializer, self).__init__(args, custom_args, platform_util)
+
+        self.benchmark_command = ""
+        if not platform_util:
+            raise ValueError("Did not find any platform info.")
+
+        # use default batch size if -1
+        if self.args.batch_size == -1:
+            self.args.batch_size = 1
+
+        arg_parser = ArgumentParser(description='Parse args')
+
+        arg_parser.add_argument(
+            '--kmp-blocktime', dest='kmp_blocktime',
+            help='number of kmp block time',
+            type=int, default=1)
+        arg_parser.add_argument(
+            "--output-dir", dest='output_dir',
+            type=str, default='/tmp/stable_diffusion/',
+            help='Specify the location of the '
+            'output directory for saving original and generated images')
+        arg_parser.add_argument(
+            "--steps", dest='steps',
+            type=int, default=50,
+            help="number of steps for diffusion model")
+
+        self.args = arg_parser.parse_args(self.custom_args, namespace=self.args)
+
+        # Set KMP env vars, if they haven't already been set, but override the default KMP_BLOCKTIME value
+        config_file_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), "config.json")
+        self.set_kmp_vars(config_file_path, kmp_blocktime=str(self.args.kmp_blocktime))
+
+        set_env_var("OMP_NUM_THREADS", self.args.num_intra_threads)
+
+        benchmark_script = os.path.join(
+            self.args.intelai_models, self.args.mode, "eval_inference.py")
+
+        self.benchmark_command = self.get_command_prefix(args.socket_id) + \
+            self.python_exe + " " + benchmark_script
+
+        self.benchmark_command = \
+            self.benchmark_command + \
+            " --precision=" + str(self.args.precision) + \
+            " --batch-size=" + str(self.args.batch_size) + \
+            " --steps=" + str(self.args.steps) + \
+            " --output-dir=" + str(self.args.output_dir)
+
+        # if the data location directory is not empty, then include the arg
+        if self.args.data_location and os.listdir(self.args.data_location):
+            self.benchmark_command += " --data-location=" + self.args.data_location
+
+        if self.args.accuracy_only:
+            self.benchmark_command += " --accuracy-only"
+
+        print("Benchmark command: ", self.benchmark_command)
+
+    def run(self):
+        if self.benchmark_command:
+            self.run_command(self.benchmark_command)
diff --git a/benchmarks/diffusion/tensorflow/stable_diffusion/inference/fp32/__init__.py b/benchmarks/diffusion/tensorflow/stable_diffusion/inference/fp32/__init__.py
new file mode 100644
index 000000000..f07dbaf88
--- /dev/null
+++ b/benchmarks/diffusion/tensorflow/stable_diffusion/inference/fp32/__init__.py
@@ -0,0 +1,19 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+#
diff --git a/benchmarks/recommendation/tensorflow/ncf/training/bfloat16/config.json b/benchmarks/diffusion/tensorflow/stable_diffusion/inference/fp32/config.json
similarity index 100%
rename from benchmarks/recommendation/tensorflow/ncf/training/bfloat16/config.json
rename to benchmarks/diffusion/tensorflow/stable_diffusion/inference/fp32/config.json
diff --git a/benchmarks/diffusion/tensorflow/stable_diffusion/inference/fp32/model_init.py b/benchmarks/diffusion/tensorflow/stable_diffusion/inference/fp32/model_init.py
new file mode 100644
index 000000000..58abca295
--- /dev/null
+++ b/benchmarks/diffusion/tensorflow/stable_diffusion/inference/fp32/model_init.py
@@ -0,0 +1,93 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+#
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+from common.base_model_init import BaseModelInitializer
+from common.base_model_init import set_env_var
+
+import os
+from argparse import ArgumentParser
+
+
+class ModelInitializer(BaseModelInitializer):
+    """initialize mode and run benchmark"""
+
+    def __init__(self, args, custom_args=[], platform_util=None):
+        super(ModelInitializer, self).__init__(args, custom_args, platform_util)
+
+        self.benchmark_command = ""
+        if not platform_util:
+            raise ValueError("Did not find any platform info.")
+
+        # use default batch size if -1
+        if self.args.batch_size == -1:
+            self.args.batch_size = 1
+
+        arg_parser = ArgumentParser(description='Parse args')
+
+        arg_parser.add_argument(
+            '--kmp-blocktime', dest='kmp_blocktime',
+            help='number of kmp block time',
+            type=int, default=1)
+        arg_parser.add_argument(
+            "--output-dir", dest='output_dir',
+            type=str, default='/tmp/stable_diffusion/',
+            help='Specify the location of the '
+            'output directory for saving original and generated images')
+        arg_parser.add_argument(
+            "--steps", dest='steps',
+            type=int, default=50,
+            help="number of steps for diffusion model")
+
+        self.args = arg_parser.parse_args(self.custom_args, namespace=self.args)
+
+        # Set KMP env vars, if they haven't already been set, but override the default KMP_BLOCKTIME value
+        config_file_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), "config.json")
+        self.set_kmp_vars(config_file_path, kmp_blocktime=str(self.args.kmp_blocktime))
+
+        set_env_var("OMP_NUM_THREADS", self.args.num_intra_threads)
+
+        benchmark_script = os.path.join(
+            self.args.intelai_models, self.args.mode, "eval_inference.py")
+
+        self.benchmark_command = self.get_command_prefix(args.socket_id) + \
+            self.python_exe + " " + benchmark_script
+
+        self.benchmark_command = \
+            self.benchmark_command + \
+            " --precision=" + str(self.args.precision) + \
+            " --batch-size=" + str(self.args.batch_size) + \
+            " --steps=" + str(self.args.steps) + \
+            " --output-dir=" + str(self.args.output_dir)
+
+        # if the data location directory is not empty, then include the arg
+        if self.args.data_location and os.listdir(self.args.data_location):
+            self.benchmark_command += " --data-location=" + self.args.data_location
+
+        if self.args.accuracy_only:
+            self.benchmark_command += " --accuracy-only"
+
+        print("Benchmark command: ", self.benchmark_command)
+
+    def run(self):
+        if self.benchmark_command:
+            self.run_command(self.benchmark_command)
diff --git a/benchmarks/diffusion/tensorflow/stable_diffusion/inference/requirements.txt b/benchmarks/diffusion/tensorflow/stable_diffusion/inference/requirements.txt
new file mode 100644
index 000000000..ead0e1463
--- /dev/null
+++ b/benchmarks/diffusion/tensorflow/stable_diffusion/inference/requirements.txt
@@ -0,0 +1,5 @@
+keras_cv==0.5.1
+matplotlib
+scipy
+scikit-image
+pandas
diff --git a/benchmarks/image_recognition/tensorflow/inceptionv4/__init__.py b/benchmarks/image_recognition/tensorflow/inceptionv4/__init__.py
deleted file mode 100644
index 8cb0c8d8d..000000000
--- a/benchmarks/image_recognition/tensorflow/inceptionv4/__init__.py
+++ /dev/null
@@ -1,17 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
diff --git a/benchmarks/image_recognition/tensorflow/inceptionv4/inference/README.md b/benchmarks/image_recognition/tensorflow/inceptionv4/inference/README.md
deleted file mode 100644
index 98033dbb5..000000000
--- a/benchmarks/image_recognition/tensorflow/inceptionv4/inference/README.md
+++ /dev/null
@@ -1,134 +0,0 @@
-<!--- 0. Title -->
-# Inception V4 inference
-
-<!-- 10. Description -->
-## Description
-
-This document has instructions for running Inception V4 inference using
-Intel-optimized TensorFlow.
-
-<!--- 30. Datasets -->
-## Datasets
-
-Download and preprocess the ImageNet dataset using the [instructions here](/datasets/imagenet/README.md).
-After running the conversion script you should have a directory with the
-ImageNet dataset in the TF records format.
-
-Set the `DATASET_DIR` to point to this directory when running Inception V4.
-
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| [`online_inference.sh`](/quickstart/image_recognition/tensorflow/inceptionv4/inference/cpu/online_inference.sh) | Runs online inference (batch_size=1). |
-| [`batch_inference.sh`](/quickstart/image_recognition/tensorflow/inceptionv4/inference/cpu/batch_inference.sh) | Runs batch inference (batch_size=240). |
-| [`accuracy.sh`](/quickstart/image_recognition/tensorflow/inceptionv4/inference/cpu/accuracy.sh) | Measures the model accuracy (batch_size=100). |
-
-<!--- 50. AI Tools -->
-## Run the model
-
-Setup your environment using the instructions below, depending on if you are
-using [AI Tools](/docs/general/tensorflow/AITools.md):
-
-<table>
-  <tr>
-    <th>Setup using AI Tools on Linux</th>
-    <th>Setup without AI Tools on Linux</th>
-    <th>Setup without AI Tools on Windows</th>
-  </tr>
-  <tr>
-    <td>
-      <p>To run using AI Tools on Linux you will need:</p>
-      <ul>
-        <li>numactl
-        <li>wget
-        <li>Activate the tensorflow conda environment
-        <pre>conda activate tensorflow</pre>
-      </ul>
-    </td>
-    <td>
-      <p>To run without AI Tools on Linux you will need:</p>
-      <ul>
-        <li>Python 3
-        <li><a href="https://pypi.org/project/intel-tensorflow/">intel-tensorflow>=2.5.0</a>
-        <li>git
-        <li>numactl
-        <li>wget
-        <li>A clone of the AI Reference Models repo<br />
-        <pre>git clone https://github.com/IntelAI/models.git</pre>
-      </ul>
-    </td>
-    <td>
-      <p>To run without AI Tools on Windows you will need:</p>
-      <ul>
-        <li><a href="/docs/general/Windows.md">Intel AI Reference Models on Windows Systems prerequisites</a>
-        <li>A clone of the AI Reference Models repo<br />
-        <pre>git clone https://github.com/IntelAI/models.git</pre>
-      </ul>
-    </td>
-  </tr>
-</table>
-
-After finishing the setup above, download the pretrained model based on `PRECISION` and set the
-`PRETRAINED_MODEL` environment var to the path to the frozen graph.
-If you run on Windows, please use a browser to download the pretrained model using the link below.
-For Linux, run:
-```
-# Int8 Pretrained model
-wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/inceptionv4_int8_pretrained_model.pb
-export PRETRAINED_MODEL=$(pwd)/inceptionv4_int8_pretrained_model.pb
-
-# FP32 Pretrained model
-wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/inceptionv4_fp32_pretrained_model.pb
-export PRETRAINED_MODEL=$(pwd)/inceptionv4_fp32_pretrained_model.pb
-```
-
-Set the environment variables and run quickstart script on either Linux or Windows systems. See the [list of quickstart scripts](#quick-start-scripts) for details on the different options.
-
-### Run on Linux
-```
-# cd to your AI Reference Models directory
-cd models
-
-export PRETRAINED_MODEL=<path to the frozen graph downloaded above>
-export DATASET_DIR=<path to the ImageNet TF records>
-export PRECISION=<set the precision to "int8" or "fp32">
-export OUTPUT_DIR=<path to the directory where log files will be written>
-# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
-export BATCH_SIZE=<customized batch size value>
-
-./quickstart/image_recognition/tensorflow/inceptionv4/inference/cpu/<script name>.sh
-```
-
-### Run on Windows
-Using `cmd.exe` run:
-```
-# cd to your AI Reference Models directory
-cd models
-
-set PRETRAINED_MODEL=<path to the frozen graph downloaded above>
-set DATASET_DIR=<path to the ImageNet TF records>
-set PRECISION=<set the precision to "int8" or "fp32">
-set OUTPUT_DIR=<directory where log files will be written>
-# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
-set BATCH_SIZE=<customized batch size value>
-
-bash quickstart\image_recognition\tensorflow\inceptionv4\inference\cpu\<script name>.sh
-```
-> Note: You may use `cygpath` to convert the Windows paths to Unix paths before setting the environment variables. 
-As an example, if the dataset location on Windows is `D:\user\ImageNet`, convert the Windows path to Unix as shown:
-> ```
-> cygpath D:\user\ImageNet
-> /d/user/ImageNet
->```
->Then, set the `DATASET_DIR` environment variable `set DATASET_DIR=/d/user/ImageNet`.
-
-<!--- 90. Resource Links-->
-## Additional Resources
-
-* To run more advanced use cases, see the instructions for the available precisions [FP32](fp32/Advanced.md) [Int8](int8/Advanced.md) [<bfloat16 precision>](<bfloat16 advanced readme link>) for calling the `launch_benchmark.py` script directly.
-* To run the model using docker, please see the [Intel® Developer Catalog](https://www.intel.com/content/www/us/en/developer/tools/software-catalog/containers.html)
-  workload container:<br />
-  [https://www.intel.com/content/www/us/en/developer/articles/containers/inceptionv4-fp32-inference-tensorflow-container.html](https://www.intel.com/content/www/us/en/developer/articles/containers/inceptionv4-fp32-inference-tensorflow-container.html).
-
diff --git a/benchmarks/image_recognition/tensorflow/inceptionv4/inference/__init__.py b/benchmarks/image_recognition/tensorflow/inceptionv4/inference/__init__.py
deleted file mode 100644
index c4fdb7d61..000000000
--- a/benchmarks/image_recognition/tensorflow/inceptionv4/inference/__init__.py
+++ /dev/null
@@ -1,19 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
diff --git a/benchmarks/image_recognition/tensorflow/inceptionv4/inference/config.json b/benchmarks/image_recognition/tensorflow/inceptionv4/inference/config.json
deleted file mode 100644
index 6f1228ba7..000000000
--- a/benchmarks/image_recognition/tensorflow/inceptionv4/inference/config.json
+++ /dev/null
@@ -1,7 +0,0 @@
-{
-    "optimization_parameters": {
-        "KMP_AFFINITY": "granularity=fine,verbose,compact,1,0",
-        "KMP_BLOCKTIME": 0,
-        "KMP_SETTINGS": 1
-    }
-}
diff --git a/benchmarks/image_recognition/tensorflow/inceptionv4/inference/fp32/Advanced.md b/benchmarks/image_recognition/tensorflow/inceptionv4/inference/fp32/Advanced.md
deleted file mode 100644
index fb51c8639..000000000
--- a/benchmarks/image_recognition/tensorflow/inceptionv4/inference/fp32/Advanced.md
+++ /dev/null
@@ -1,149 +0,0 @@
-<!--- 0. Title -->
-<!-- This document is auto-generated using markdown fragments and the model-builder -->
-<!-- To make changes to this doc, please change the fragments instead of modifying this doc directly -->
-# Inception V4 FP32 inference - Advanced Instructions
-
-<!-- 10. Description -->
-This document has advanced instructions for running Inception V4 FP32
-inference, which provides more control over the individual parameters that
-are used. For more information on using [`/benchmarks/launch_benchmark.py`](/benchmarks/launch_benchmark.py),
-see the [launch benchmark documentation](/docs/general/tensorflow/LaunchBenchmark.md).
-
-Prior to using these instructions, please follow the setup instructions from
-the model's [README](README.md) and/or the
-[AI Kit documentation](/docs/general/tensorflow/AIKit.md) to get your environment
-setup (if running on bare metal) and download the dataset, pretrained model, etc.
-If you are using AI Kit, please exclude the `--docker-image` flag from the
-commands below, since you will be running the the TensorFlow conda environment
-instead of docker.
-
-<!-- 55. Docker arg -->
-Any of the `launch_benchmark.py` commands below can be run on bare metal by
-removing the `--docker-image` arg. Ensure that you have all of the
-[required prerequisites installed](README.md#run-the-model) in your environment
-before running without the docker container.
-
-If you are new to docker and are running into issues with the container,
-see [this document](/docs/general/docker.md) for troubleshooting tips.
-
-<!-- 50. Launch benchmark instructions -->
-Once your environment is setup, navigate to the `benchmarks` directory of
-the model zoo and set environment variables pointing to the directory for the
-dataset, pretrained model frozen graph, and an output directory where log
-files will be written.
-
-```
-# cd to the benchmarks directory in the model zoo
-cd benchmarks
-
-export DATASET_DIR=<path to the dataset (only required for accuracy)>
-export OUTPUT_DIR=<directory where log files will be written>
-export PRETRAINED_MODEL=<path to the frozen graph that you downloaded>
-```
-
-Inception V4 can be run to test accuracy, batch inference, or online inference.
-Use one of the following examples below, depending on your use case.
-
-* For accuracy run the following command that uses the `DATASET_DIR`, a batch
-  size of 100, and the `--accuracy-only` flag:
-  ```
-  python launch_benchmark.py \
-    --model-name inceptionv4 \
-    --precision fp32 \
-    --mode inference \
-    --framework tensorflow \
-    --accuracy-only \
-    --batch-size 100 \
-    --socket-id 0 \
-    --docker-image intel/intel-optimized-tensorflow:latest \
-    --in-graph ${PRETRAINED_MODEL} \
-    --data-location ${DATASET_DIR} \
-    --output-dir ${OUTPUT_DIR}
-  ```
-
- * For batch inference, use the command below that has a batch size of 240
-   and the `--benchmark-only` flag. Synthetic data will be used, since no
-   dataset is being provided.
-   ```
-   python launch_benchmark.py \
-     --model-name inceptionv4 \
-     --precision fp32 \
-     --mode inference \
-     --framework tensorflow \
-     --benchmark-only \
-     --batch-size 240 \
-     --socket-id 0 \
-     --docker-image intel/intel-optimized-tensorflow:latest \
-     --in-graph ${PRETRAINED_MODEL} \
-     --output-dir ${OUTPUT_DIR}
-   ```
-
-* For online inference use the command below that has a batch size of 1 and
-  the `--benchmark-only` flag. Again, synthetic data is being used in this
-  example.
-  ```
-  python launch_benchmark.py \
-    --model-name inceptionv4 \
-    --precision fp32 \
-    --mode inference \
-    --framework tensorflow \
-    --benchmark-only \
-    --batch-size 1 \
-    --socket-id 0 \
-    --docker-image intel/intel-optimized-tensorflow:latest \
-    --in-graph ${PRETRAINED_MODEL} \
-    --output-dir ${OUTPUT_DIR}
-  ```
-
-Note that the `--verbose` flag can be added to any of the above commands
-to get additional debug output.
-
-The log file is saved to the `OUTPUT_DIR` directory. Below are examples of
-what the tail of your log file should look like for the different configs.
-
-Example log tail when running for accuracy:
-```
-...
-Iteration time: 1337.8728 ms
-Processed 49600 images. (Top1 accuracy, Top5 accuracy) = (0.8015, 0.9517)
-Iteration time: 1331.8253 ms
-Processed 49700 images. (Top1 accuracy, Top5 accuracy) = (0.8017, 0.9518)
-Iteration time: 1339.1553 ms
-Processed 49800 images. (Top1 accuracy, Top5 accuracy) = (0.8017, 0.9518)
-Iteration time: 1334.5991 ms
-Processed 49900 images. (Top1 accuracy, Top5 accuracy) = (0.8018, 0.9519)
-Iteration time: 1336.1905 ms
-Processed 50000 images. (Top1 accuracy, Top5 accuracy) = (0.8018, 0.9519)
-Ran inference with batch size 100
-Log location outside container: ${OUTPUT_DIR}/benchmark_inceptionv4_inference_fp32_20190308_182729.log
-```
-
-Example log tail when running for batch inference:
-```
-[Running warmup steps...]
-steps = 10, 91.4372832625 images/sec
-[Running benchmark steps...]
-steps = 10, 91.0217283977 images/sec
-steps = 20, 90.8331507586 images/sec
-steps = 30, 91.1284943026 images/sec
-steps = 40, 91.1885998597 images/sec
-steps = 50, 91.1905741783 images/sec
-Ran inference with batch size 240
-Log location outside container: ${OUTPUT_DIR}/benchmark_inceptionv4_inference_fp32_20190308_184431.log
-```
-
-Example log tail when running for online inference:
-```
-[Running warmup steps...]
-steps = 10, 15.6993019295 images/sec
-[Running benchmark steps...]
-steps = 10, 16.3553780883 images/sec
-steps = 20, 15.771143231 images/sec
-steps = 30, 15.7133587586 images/sec
-steps = 40, 16.0477494988 images/sec
-steps = 50, 15.483992912 images/sec
-Latency: 63.534 ms
-Ran inference with batch size 1
-Log location outside container: ${OUTPUT_DIR}/benchmark_inceptionv4_inference_fp32_20190307_221954.log
-```
-
diff --git a/benchmarks/image_recognition/tensorflow/inceptionv4/inference/fp32/__init__.py b/benchmarks/image_recognition/tensorflow/inceptionv4/inference/fp32/__init__.py
deleted file mode 100644
index fd96ab7af..000000000
--- a/benchmarks/image_recognition/tensorflow/inceptionv4/inference/fp32/__init__.py
+++ /dev/null
@@ -1,19 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2019 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
diff --git a/benchmarks/image_recognition/tensorflow/inceptionv4/inference/fp32/model_init.py b/benchmarks/image_recognition/tensorflow/inceptionv4/inference/fp32/model_init.py
deleted file mode 100644
index 7de241e2f..000000000
--- a/benchmarks/image_recognition/tensorflow/inceptionv4/inference/fp32/model_init.py
+++ /dev/null
@@ -1,28 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2019 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
-
-from image_recognition.tensorflow.inceptionv4.inference.inceptionv4_model_init import InceptionV4ModelInitializer
-
-
-class ModelInitializer(InceptionV4ModelInitializer):
-    """Model initializer for InceptionV4 FP32 inference"""
-
-    def __init__(self, args, custom_args=[], platform_util=None):
-        super(ModelInitializer, self).__init__(args, custom_args, platform_util)
diff --git a/benchmarks/image_recognition/tensorflow/inceptionv4/inference/inceptionv4_model_init.py b/benchmarks/image_recognition/tensorflow/inceptionv4/inference/inceptionv4_model_init.py
deleted file mode 100644
index 5f29a07f3..000000000
--- a/benchmarks/image_recognition/tensorflow/inceptionv4/inference/inceptionv4_model_init.py
+++ /dev/null
@@ -1,119 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
-
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-
-import argparse
-import os
-
-from common.base_model_init import BaseModelInitializer, set_env_var
-
-
-class InceptionV4ModelInitializer(BaseModelInitializer):
-    """Common model initializer for InceptionV4 inference"""
-
-    def __init__(self, args, custom_args=[], platform_util=None):
-        super(InceptionV4ModelInitializer, self).__init__(args, custom_args, platform_util)
-
-        # Environment variables
-        set_env_var("OMP_NUM_THREADS", platform_util.num_cores_per_socket
-                    if self.args.num_cores == -1 else self.args.num_cores)
-
-        # Set KMP env vars, if they haven't already been set
-        config_file_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), "config.json")
-        self.set_kmp_vars(config_file_path)
-
-        self.set_num_inter_intra_threads(num_inter_threads=platform_util.num_threads_per_core,
-                                         num_intra_threads=platform_util.num_cores_per_socket)
-
-    def parse_args(self):
-        if self.custom_args:
-            parser = argparse.ArgumentParser()
-            parser.add_argument(
-                "--input-height", default=None, dest="input_height",
-                type=int, help="input height")
-            parser.add_argument(
-                "--input-width", default=None,
-                dest="input_width", type=int, help="input width")
-            parser.add_argument(
-                "--warmup-steps", dest="warmup_steps",
-                help="number of warmup steps", type=int, default=10)
-            parser.add_argument(
-                "--steps", dest="steps", help="number of steps", type=int,
-                default=50)
-            parser.add_argument(
-                "--input-layer", dest="input_layer",
-                help="name of input layer", type=str, default=None)
-            parser.add_argument(
-                "--output-layer", dest="output_layer",
-                help="name of output layer", type=str, default=None)
-
-            parser.add_argument(
-                '--kmp-blocktime', dest='kmp_blocktime',
-                help='number of kmp block time',
-                type=int, default=1)
-
-            self.args = parser.parse_args(self.custom_args, namespace=self.args)
-
-            # Set KMP env vars, if they haven't already been set, but override the default KMP_BLOCKTIME value
-            config_file_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), "config.json")
-            self.set_kmp_vars(config_file_path, kmp_blocktime=str(self.args.kmp_blocktime))
-
-    def add_command_prefix(self, script_path):
-        """ Uses the specified script path and adds on the command prefix """
-        return self.get_command_prefix(self.args.socket_id) + self.python_exe + " " + \
-            script_path
-
-    def run_benchmark(self):
-        """ Setup the command string and run the benchmarking script """
-        benchmark_script = os.path.join(
-            self.args.intelai_models, self.args.mode, "benchmark.py")
-        script_args_list = [
-            "input_graph", "input_height", "input_width", "batch_size",
-            "input_layer", "output_layer", "num_inter_threads",
-            "num_intra_threads",
-            "warmup_steps", "steps"]
-        cmd_prefix = self.add_command_prefix(benchmark_script)
-        cmd = self.add_args_to_command(cmd_prefix, script_args_list)
-
-        self.run_command(cmd)
-
-    def run_accuracy(self):
-        """ Setup the command string and run the accuracy test """
-        accuracy_script = os.path.join(
-            self.args.intelai_models, self.args.mode, "accuracy.py")
-        script_args_list = [
-            "input_graph", "data_location", "input_height", "input_width",
-            "batch_size", "input_layer", "output_layer",
-            "num_inter_threads", "num_intra_threads"]
-        cmd_prefix = self.add_command_prefix(accuracy_script)
-        cmd = self.add_args_to_command(cmd_prefix, script_args_list)
-
-        self.run_command(cmd)
-
-    def run(self):
-        self.parse_args()
-        if self.args.benchmark_only:
-            self.run_benchmark()
-        if self.args.accuracy_only:
-            self.run_accuracy()
diff --git a/benchmarks/image_recognition/tensorflow/inceptionv4/inference/int8/Advanced.md b/benchmarks/image_recognition/tensorflow/inceptionv4/inference/int8/Advanced.md
deleted file mode 100644
index 54abe7ec2..000000000
--- a/benchmarks/image_recognition/tensorflow/inceptionv4/inference/int8/Advanced.md
+++ /dev/null
@@ -1,146 +0,0 @@
-<!--- 0. Title -->
-<!-- This document is auto-generated using markdown fragments and the model-builder -->
-<!-- To make changes to this doc, please change the fragments instead of modifying this doc directly -->
-# Inception V4 Int8 inference - Advanced Instructions
-
-<!-- 10. Description -->
-This document has advanced instructions for running Inception V4 Int8
-inference, which provides more control over the individual parameters that
-are used. For more information on using [`/benchmarks/launch_benchmark.py`](/benchmarks/launch_benchmark.py),
-see the [launch benchmark documentation](/docs/general/tensorflow/LaunchBenchmark.md).
-
-Prior to using these instructions, please follow the setup instructions from
-the model's [README](README.md) and/or the
-[AI Kit documentation](/docs/general/tensorflow/AIKit.md) to get your environment
-setup (if running on bare metal) and download the dataset, pretrained model, etc.
-If you are using AI Kit, please exclude the `--docker-image` flag from the
-commands below, since you will be running the the TensorFlow conda environment
-instead of docker.
-
-<!-- 55. Docker arg -->
-Any of the `launch_benchmark.py` commands below can be run on bare metal by
-removing the `--docker-image` arg. Ensure that you have all of the
-[required prerequisites installed](README.md#run-the-model) in your environment
-before running without the docker container.
-
-If you are new to docker and are running into issues with the container,
-see [this document](/docs/general/docker.md) for troubleshooting tips.
-
-<!-- 50. Launch benchmark instructions -->
-Once your environment is setup, navigate to the `benchmarks` directory of
-the model zoo and set environment variables pointing to the directory for the
-dataset, pretrained model frozen graph, and an output directory where log
-files will be written.
-
-```
-# cd to the benchmarks directory in the model zoo
-cd benchmarks
-
-export DATASET_DIR=<path to the dataset>
-export OUTPUT_DIR=<directory where log files will be written>
-export PRETRAINED_MODEL=<path to the frozen graph that you downloaded>
-```
-
-Inception V4 can be run to test accuracy, batch inference, or online inference.
-Use one of the following examples below, depending on your use case.
-
-* Use the command below to test accuracy. This command uses the `DATASET_DIR`,
-  a batch size of 100 and the `--accuracy-only` flag:
-  ```
-  python launch_benchmark.py \
-    --model-name inceptionv4 \
-    --precision int8 \
-    --mode inference \
-    --framework tensorflow \
-    --accuracy-only \
-    --batch-size 100 \
-    --socket-id 0 \
-    --docker-image intel/intel-optimized-tensorflow:latest \
-    --in-graph ${PRETRAINED_MODEL} \
-    --data-location ${DATASET_DIR} \
-    --output-dir ${OUTPUT_DIR}
-  ```
-
-* For batch inference, use the command below that uses a batch size of 240 and
-  the `--benchmark-only` flag. Since no dataset is being provided, synthetic data
-  will be used.
-  ```
-  python launch_benchmark.py \
-    --model-name inceptionv4 \
-    --precision int8 \
-    --mode inference \
-    --framework tensorflow \
-    --benchmark-only \
-    --batch-size 240 \
-    --socket-id 0 \
-    --docker-image intel/intel-optimized-tensorflow:latest \
-    --in-graph ${PRETRAINED_MODEL} \
-    --output-dir ${OUTPUT_DIR}
-  ```
-
-* For online inference, use the command below that has a batch size of 1 and the
-  `--benchmark-only` flag. Again, synthetic data is being used in this example.
-  ```
-  python launch_benchmark.py \
-    --model-name inceptionv4 \
-    --precision int8 \
-    --mode inference \
-    --framework tensorflow \
-    --benchmark-only \
-    --batch-size 1 \
-    --socket-id 0 \
-    --docker-image intel/intel-optimized-tensorflow:latest \
-    --in-graph ${PRETRAINED_MODEL} \
-    --output-dir ${OUTPUT_DIR}
-  ```
-
-Note that the `--verbose` flag can be added to any of the above commands
-to get additional debug output.
-
-The log file is saved to the `OUTPUT_DIR` directory. Below are examples of
-what the tail of your log file should look like for the different configs.
-
-Example log tail when running for accuracy:
-```
-...
-Iteration time: 685.1976 ms
-Processed 49700 images. (Top1 accuracy, Top5 accuracy) = (0.7985, 0.9504)
-Iteration time: 686.3845 ms
-Processed 49800 images. (Top1 accuracy, Top5 accuracy) = (0.7983, 0.9504)
-Iteration time: 686.7021 ms
-Processed 49900 images. (Top1 accuracy, Top5 accuracy) = (0.7984, 0.9504)
-Iteration time: 685.8914 ms
-Processed 50000 images. (Top1 accuracy, Top5 accuracy) = (0.7984, 0.9504)
-Ran inference with batch size 100
-Log location outside container: ${OUTPUT_DIR}/benchmark_inceptionv4_inference_int8_20190306_221608.log
-```
-
-Example log tail when running for batch inference:
-```
-[Running warmup steps...]
-steps = 10, 184.497605972 images/sec
-[Running benchmark steps...]
-steps = 10, 184.664702184 images/sec
-steps = 20, 184.938455688 images/sec
-steps = 30, 184.454197634 images/sec
-steps = 40, 184.491891402 images/sec
-steps = 50, 184.390001575 images/sec
-Ran inference with batch size 240
-Log location outside container: ${OUTPUT_DIR}/benchmark_inceptionv4_inference_int8_20190415_233517.log
-```
-
-Example log tail when running for online inference:
-```
-[Running warmup steps...]
-steps = 10, 32.6095380262 images/sec
-[Running benchmark steps...]
-steps = 10, 32.9024373024 images/sec
-steps = 20, 32.5328989723 images/sec
-steps = 30, 32.5988932413 images/sec
-steps = 40, 31.3991914957 images/sec
-steps = 50, 32.7053998207 images/sec
-Latency: 30.598 ms
-Ran inference with batch size 1
-Log location outside container: ${OUTPUT_DIR}/benchmark_inceptionv4_inference_int8_20190415_232441.log
-```
-
diff --git a/benchmarks/image_recognition/tensorflow/inceptionv4/inference/int8/__init__.py b/benchmarks/image_recognition/tensorflow/inceptionv4/inference/int8/__init__.py
deleted file mode 100644
index c4fdb7d61..000000000
--- a/benchmarks/image_recognition/tensorflow/inceptionv4/inference/int8/__init__.py
+++ /dev/null
@@ -1,19 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
diff --git a/benchmarks/image_recognition/tensorflow/inceptionv4/inference/int8/model_init.py b/benchmarks/image_recognition/tensorflow/inceptionv4/inference/int8/model_init.py
deleted file mode 100644
index 7de241e2f..000000000
--- a/benchmarks/image_recognition/tensorflow/inceptionv4/inference/int8/model_init.py
+++ /dev/null
@@ -1,28 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2019 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
-
-from image_recognition.tensorflow.inceptionv4.inference.inceptionv4_model_init import InceptionV4ModelInitializer
-
-
-class ModelInitializer(InceptionV4ModelInitializer):
-    """Model initializer for InceptionV4 FP32 inference"""
-
-    def __init__(self, args, custom_args=[], platform_util=None):
-        super(ModelInitializer, self).__init__(args, custom_args, platform_util)
diff --git a/benchmarks/language_modeling/tensorflow/bert_large/training/bfloat16/model_init.py b/benchmarks/language_modeling/tensorflow/bert_large/training/bfloat16/model_init.py
index 716288f4b..361383721 100644
--- a/benchmarks/language_modeling/tensorflow/bert_large/training/bfloat16/model_init.py
+++ b/benchmarks/language_modeling/tensorflow/bert_large/training/bfloat16/model_init.py
@@ -200,8 +200,12 @@ def __init__(self, args, custom_args=[], platform_util=None):
             self.benchmark_command = self.benchmark_command + self.python_exe + " " + benchmark_script + "\n"
         elif not self.args.gpu:
             numa_cmd = " -np 1 numactl -N {} -m {} "
-            self.benchmark_command = self.benchmark_command + numa_cmd.format(0, 0) + os.environ["PYTHON_EXE"] + " " \
-                + benchmark_script
+            if int(os.environ["MPI_NUM_PROCESSES"]) > 1 and int(os.environ["MPI_NUM_PROCESSES_PER_SOCKET"]) > 1:
+                self.benchmark_command = "mpirun " + self.benchmark_command + numa_cmd.format(0, 0) \
+                    + os.environ["PYTHON_EXE"] + " " + benchmark_script
+            else:
+                self.benchmark_command = self.benchmark_command + numa_cmd.format(0, 0) \
+                    + os.environ["PYTHON_EXE"] + " " + benchmark_script
             for i in range(1, int(os.environ["MPI_NUM_PROCESSES"])):
                 self.benchmark_command = self.benchmark_command + eoo + " : " + numa_cmd.format(i, i) \
                     + os.environ["PYTHON_EXE"] + " " + benchmark_script
diff --git a/benchmarks/language_modeling/tensorflow/bert_large/training/fp16/model_init.py b/benchmarks/language_modeling/tensorflow/bert_large/training/fp16/model_init.py
index 1c2272ad7..72ae1bef8 100644
--- a/benchmarks/language_modeling/tensorflow/bert_large/training/fp16/model_init.py
+++ b/benchmarks/language_modeling/tensorflow/bert_large/training/fp16/model_init.py
@@ -202,8 +202,12 @@ def __init__(self, args, custom_args=[], platform_util=None):
             self.benchmark_command = self.benchmark_command + self.python_exe + " " + benchmark_script + "\n"
         else:
             numa_cmd = " -np 1 numactl -N {} -m {} "
-            self.benchmark_command = self.benchmark_command + numa_cmd.format(0, 0) + os.environ["PYTHON_EXE"] + " " \
-                + benchmark_script
+            if int(os.environ["MPI_NUM_PROCESSES"]) > 1 and int(os.environ["MPI_NUM_PROCESSES_PER_SOCKET"]) > 1:
+                self.benchmark_command = "mpirun " + self.benchmark_command + numa_cmd.format(0, 0) \
+                    + os.environ["PYTHON_EXE"] + " " + benchmark_script
+            else:
+                self.benchmark_command = self.benchmark_command + numa_cmd.format(0, 0) \
+                    + os.environ["PYTHON_EXE"] + " " + benchmark_script
             for i in range(1, int(os.environ["MPI_NUM_PROCESSES"])):
                 self.benchmark_command = self.benchmark_command + eoo + " : " + numa_cmd.format(i, i) \
                     + os.environ["PYTHON_EXE"] + " " + benchmark_script
diff --git a/benchmarks/language_modeling/tensorflow/bert_large/training/fp32/model_init.py b/benchmarks/language_modeling/tensorflow/bert_large/training/fp32/model_init.py
index eaf002b36..7f88c5503 100644
--- a/benchmarks/language_modeling/tensorflow/bert_large/training/fp32/model_init.py
+++ b/benchmarks/language_modeling/tensorflow/bert_large/training/fp32/model_init.py
@@ -207,8 +207,12 @@ def __init__(self, args, custom_args=[], platform_util=None):
             self.benchmark_command = self.benchmark_command + self.python_exe + " " + benchmark_script + "\n"
         elif not self.args.gpu:
             numa_cmd = " -np 1 numactl -N {} -m {} "
-            self.benchmark_command = self.benchmark_command + numa_cmd.format(0, 0) + os.environ["PYTHON_EXE"] + " " \
-                + benchmark_script
+            if int(os.environ["MPI_NUM_PROCESSES"]) > 1 and int(os.environ["MPI_NUM_PROCESSES_PER_SOCKET"]) > 1:
+                self.benchmark_command = "mpirun " + self.benchmark_command + numa_cmd.format(0, 0) \
+                    + os.environ["PYTHON_EXE"] + " " + benchmark_script
+            else:
+                self.benchmark_command = self.benchmark_command + numa_cmd.format(0, 0) \
+                    + os.environ["PYTHON_EXE"] + " " + benchmark_script
             for i in range(1, int(os.environ["MPI_NUM_PROCESSES"])):
                 self.benchmark_command = self.benchmark_command + eoo + " : " + numa_cmd.format(i, i) \
                     + os.environ["PYTHON_EXE"] + " " + benchmark_script
diff --git a/benchmarks/language_translation/tensorflow/mlperf_gnmt/__init__.py b/benchmarks/language_translation/tensorflow/mlperf_gnmt/__init__.py
deleted file mode 100644
index 6f72f91ea..000000000
--- a/benchmarks/language_translation/tensorflow/mlperf_gnmt/__init__.py
+++ /dev/null
@@ -1,19 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2021 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
diff --git a/benchmarks/language_translation/tensorflow/mlperf_gnmt/inference/README.md b/benchmarks/language_translation/tensorflow/mlperf_gnmt/inference/README.md
deleted file mode 100644
index e94a8fe2b..000000000
--- a/benchmarks/language_translation/tensorflow/mlperf_gnmt/inference/README.md
+++ /dev/null
@@ -1,125 +0,0 @@
-<!--- 0. Title -->
-# MLPerf GNMT inference
-
-<!-- 10. Description -->
-## Description
-
-This document has instructions for running MLPerf GNMT inference using
-Intel-optimized TensorFlow.
-
-<!--- 30. Datasets -->
-## Datasets
-
-Download and unzip the MLPerf GNMT model benchmarking data.
-
-```
-wget https://zenodo.org/record/2531868/files/gnmt_inference_data.zip
-unzip gnmt_inference_data.zip
-export DATASET_DIR=$(pwd)/nmt/data
-```
-
-Set the `DATASET_DIR` to point as instructed above  when running MLPerf GNMT.
-
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| [`online_inference.sh`](/quickstart/language_translation/tensorflow/mlperf_gnmt/inference/cpu/online_inference.sh) | Runs online inference (batch_size=1). |
-| [`batch_inference.sh`](/quickstart/language_translation/tensorflow/mlperf_gnmt/inference/cpu/batch_inference.sh) | Runs batch inference (batch_size=32). |
-| [`accuracy.sh`](/quickstart/language_translation/tensorflow/mlperf_gnmt/inference/cpu/accuracy.sh) | Runs accuracy |
-
-<!--- 50. AI Tools -->
-## Run the model
-
-Setup your environment using the instructions below, depending on if you are
-using [AI Tools](/docs/general/tensorflow/AITools.md):
-
-<table>
-  <tr>
-    <th>Setup using AI Tools</th>
-    <th>Setup without AI Tools</th>
-  </tr>
-  <tr>
-    <td>
-      <p>To run using AI Tools you will need:</p>
-      <ul>
-        <li>git
-        <li>numactl
-        <li>pip
-        <li>wget
-        <li><a href="https://bazel.build/">Bazel</a> to build tensorflow addons
-        <li>Activate the `tensorflow` conda environment
-        <pre>conda activate tensorflow</pre>
-      </ul>
-    </td>
-    <td>
-      <p>To run without AI Tools you will need:</p>
-      <ul>
-        <li>Python 3
-        <li><a href="https://pypi.org/project/intel-tensorflow/">intel-tensorflow>=2.5.0</a>
-        <li>git
-        <li>numactl
-        <li>pip
-        <li>wget
-        <li><a href="https://bazel.build/">Bazel</a> to build tensorflow addons
-        <li>A clone of the AI Reference Models repo<br />
-        <pre>git clone https://github.com/IntelAI/models.git</pre>
-      </ul>
-    </td>
-  </tr>
-</table>
-
-After installing the prerequisites, download the pretrained model and set
-the `PRETRAINED_MODEL` environment variable to the .pb file path:
-```
-wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/mlperf_gnmt_fp32_pretrained_model.pb
-export PRETRAINED_MODEL=$(pwd)/mlperf_gnmt_fp32_pretrained_model.pb
-```
-
-MLPerf GNMT requires TensorFlow addons to be built with a patch from the model
-zoo. The snippet below shows how to clone the addons repo, apply the patch, and
-then build and install the TensorFlow addons wheel.
-```
-# TensorFlow addons (r0.5) build and installation instructions:
-#   Clone TensorFlow addons (r0.5) and apply a patch: A patch file
-#   is attached in Intel AI Reference Models MLpref GNMT model scripts,
-#   it fixes TensorFlow addons (r0.5) to work with TensorFlow
-#   version 2.11, and prevents TensorFlow 2.0.0 to be installed
-#   by default as a required dependency.
-git clone --single-branch --branch=r0.5 https://github.com/tensorflow/addons.git
-cd addons
-git apply ../models/language_translation/tensorflow/mlperf_gnmt/gnmt-fix.patch
-
-#   Build TensorFlow addons source code and create TensorFlow addons
-#   pip wheel. Use bazel 6.0.0 version :
-#   Answer yes to questions while running configure.sh
-bash configure.sh
-bazel build --enable_runfiles build_pip_pkg
-bazel-bin/build_pip_pkg artifacts
-pip install artifacts/tensorflow_addons-*.whl --no-deps
-```
-
-Once that has been completed, ensure you have the required environment variables
-set, and then run a quickstart script.
-
-```
-# cd to your AI Reference Models directory
-cd models
-
-# Set env var paths
-export DATASET_DIR=<path to the dataset>
-export PRECISION=fp32
-export OUTPUT_DIR=<path to the directory where log files will be written>
-export PRETRAINED_MODEL=<path to the pretrained model frozen graph>
-# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
-export BATCH_SIZE=<customized batch size value>
-
-# Run a quickstart script
-./quickstart/language_translation/tensorflow/mlperf_gnmt/inference/cpu/<script name>.sh
-```
-
-<!--- 90. Resource Links-->
-## Additional Resources
-
-* To run more advanced use cases, see the instructions for the available precisions [FP32](fp32/Advanced.md) [<int8 precision>](<int8 advanced readme link>) [<bfloat16 precision>](<bfloat16 advanced readme link>) for calling the `launch_benchmark.py` script directly.
diff --git a/benchmarks/language_translation/tensorflow/mlperf_gnmt/inference/__init__.py b/benchmarks/language_translation/tensorflow/mlperf_gnmt/inference/__init__.py
deleted file mode 100644
index 6f72f91ea..000000000
--- a/benchmarks/language_translation/tensorflow/mlperf_gnmt/inference/__init__.py
+++ /dev/null
@@ -1,19 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2021 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
diff --git a/benchmarks/language_translation/tensorflow/mlperf_gnmt/inference/fp32/Advanced.md b/benchmarks/language_translation/tensorflow/mlperf_gnmt/inference/fp32/Advanced.md
deleted file mode 100644
index bc2b36e83..000000000
--- a/benchmarks/language_translation/tensorflow/mlperf_gnmt/inference/fp32/Advanced.md
+++ /dev/null
@@ -1,94 +0,0 @@
-<!--- 0. Title -->
-<!-- This document is auto-generated using markdown fragments and the model-builder -->
-<!-- To make changes to this doc, please change the fragments instead of modifying this doc directly -->
-# MLPerf GNMT FP32 inference - Advanced Instructions
-
-<!-- 10. Description -->
-This document has advanced instructions for running MLPerf GNMT FP32
-inference, which provides more control over the individual parameters that
-are used. For more information on using [`/benchmarks/launch_benchmark.py`](/benchmarks/launch_benchmark.py),
-see the [launch benchmark documentation](/docs/general/tensorflow/LaunchBenchmark.md).
-
-Prior to using these instructions, please follow the setup instructions from
-the model's [README](README.md) and/or the
-[AI Kit documentation](/docs/general/tensorflow/AIKit.md) to get your environment
-setup (if running on bare metal) and download the dataset, pretrained model, etc.
-If you are using AI Kit, please exclude the `--docker-image` flag from the
-commands below, since you will be running the the TensorFlow conda environment
-instead of docker.
-
-<!-- 55. Docker arg -->
-Any of the `launch_benchmark.py` commands below can be run on bare metal by
-removing the `--docker-image` arg. Ensure that you have all of the
-[required prerequisites installed](README.md#run-the-model) in your environment
-before running without the docker container.
-
-If you are new to docker and are running into issues with the container,
-see [this document](/docs/general/docker.md) for troubleshooting tips.
-
-<!-- 50. Launch benchmark instructions -->
-If you are going to run using docker, copy the `tensorflow-addons` wheel that you built
-during the [model setup](README.md#run-the-model) to the model zoo's `mlperf_gnmt` directory:
-```
-cp <tensorflow-addons repo>/artifacts/tensorflow_addons-*.whl <model zoo directory>/models/language_translation/tensorflow/mlperf_gnmt
-```
-
-Once your environment is setup, navigate to the `benchmarks` directory of
-the model zoo and set environment variables for the dataset, checkpoint
-directory, and an output directory where log files will be written.
-```
-cd benchmarks
-
-export DATASET_DIR=<path to the dataset>
-export OUTPUT_DIR=<directory where log files will be written>
-export PRETRAINED_MODEL=<path to the pretrained model frozen graph .pb file>
-```
-
-MLPerf GNMT inference can be run in three different modes:
-
-* For online inference, use the following command (with `--benchmark-only`, `--socket-id 0` and `--batch-size 1`):
-  ```
-  python launch_benchmark.py \
-    --model-name mlperf_gnmt \
-    --framework tensorflow \
-    --precision fp32 \
-    --mode inference \
-    --batch-size 1 \
-    --socket-id 0 \
-    --data-location $DATASET_DIR \
-    --docker-image intel/intel-optimized-tensorflow:latest \
-    --in-graph $PRETRAINED_MODEL \
-    --output-dir $OUTPUT_DIR \
-    --benchmark-only
-  ```
-* For batch inference, use the following command (with `--benchmark-only`, `--socket-id 0` and `--batch-size 32`):
-  ```
-  python launch_benchmark.py \
-    --model-name mlperf_gnmt \
-    --framework tensorflow \
-    --precision fp32 \
-    --mode inference \
-    --batch-size 32 \
-    --socket-id 0 \
-    --data-location $DATASET_DIR \
-    --docker-image intel/intel-optimized-tensorflow:latest \
-    --in-graph $PRETRAINED_MODEL \
-    --output-dir $OUTPUT_DIR \
-    --benchmark-only
-  ```
-* For accuracy testing, use the following command (with `--accuracy_only`, `--socket-id 0` and `--batch-size 32`):
-  ```
-  python launch_benchmark.py \
-    --model-name mlperf_gnmt \
-    --framework tensorflow \
-    --precision fp32 \
-    --mode inference \
-    --batch-size 32 \
-    --socket-id 0 \
-    --data-location $DATASET_DIR \
-    --docker-image intel/intel-optimized-tensorflow:latest \
-    --in-graph $PRETRAINED_MODEL \
-    --output-dir $OUTPUT_DIR \
-    --accuracy-only
-  ```
-
diff --git a/benchmarks/language_translation/tensorflow/mlperf_gnmt/inference/fp32/__init__.py b/benchmarks/language_translation/tensorflow/mlperf_gnmt/inference/fp32/__init__.py
deleted file mode 100644
index 6f72f91ea..000000000
--- a/benchmarks/language_translation/tensorflow/mlperf_gnmt/inference/fp32/__init__.py
+++ /dev/null
@@ -1,19 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2021 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
diff --git a/benchmarks/language_translation/tensorflow/mlperf_gnmt/inference/fp32/config.json b/benchmarks/language_translation/tensorflow/mlperf_gnmt/inference/fp32/config.json
deleted file mode 100644
index c67a8f22a..000000000
--- a/benchmarks/language_translation/tensorflow/mlperf_gnmt/inference/fp32/config.json
+++ /dev/null
@@ -1,7 +0,0 @@
-{
-  "optimization_parameters": {
-    "KMP_AFFINITY": "granularity=fine,compact,1,0",
-    "KMP_BLOCKTIME": 1,
-    "KMP_SETTINGS": 1
-  }
-}
diff --git a/benchmarks/language_translation/tensorflow/mlperf_gnmt/inference/fp32/model_init.py b/benchmarks/language_translation/tensorflow/mlperf_gnmt/inference/fp32/model_init.py
deleted file mode 100644
index bc805c72e..000000000
--- a/benchmarks/language_translation/tensorflow/mlperf_gnmt/inference/fp32/model_init.py
+++ /dev/null
@@ -1,76 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2019 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-#
-
-import os
-from argparse import ArgumentParser
-
-from common.base_model_init import BaseModelInitializer
-from common.base_model_init import set_env_var
-
-
-class ModelInitializer(BaseModelInitializer):
-    """Model initializer for GNMT FP32 inference"""
-
-    def __init__(self, args, custom_args=[], platform_util=None):
-        super(ModelInitializer, self).__init__(args, custom_args, platform_util)
-        self.cmd = self.get_command_prefix(self.args.socket_id)
-
-        if self.args.socket_id != -1 and self.args.num_cores != -1:
-            self.cmd += "--physcpubind=0-" + \
-                        (str(self.args.num_cores - 1)) + " "
-        self.cmd += "{} ".format(self.python_exe)
-
-        # use default batch size if -1
-        if self.args.batch_size == -1:
-            self.args.batch_size = 32
-
-        # set num_inter_threads and num_intra_threads (override inter threads to 2)
-        self.set_num_inter_intra_threads()
-        set_env_var("OMP_NUM_THREADS", self.args.num_intra_threads)
-        arg_parser = ArgumentParser(description="process custom_args")
-        arg_parser.add_argument('--kmp-blocktime', dest='kmp_blocktime',
-                                help='number of kmp block time',
-                                type=int, default=1)
-        self.args = arg_parser.parse_args(self.custom_args, namespace=self.args)
-        # Set KMP env vars, if they haven't already been set, but override the default KMP_BLOCKTIME value
-        config_file_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), "config.json")
-        self.set_kmp_vars(config_file_path, kmp_blocktime=str(self.args.kmp_blocktime))
-
-        src_vocab_file = os.path.join(self.args.data_location, "vocab.bpe.32000.en")
-        tgt_vocab_file = os.path.join(self.args.data_location, "vocab.bpe.32000.de")
-        inference_input_file = os.path.join(self.args.data_location, "newstest2014.tok.bpe.32000.en")
-        inference_ref_file = os.path.join(self.args.data_location, "newstest2014.tok.bpe.32000.de")
-
-        cmd_args = " --in_graph=" + self.args.input_graph + \
-                   " --batch_size=" + str(self.args.batch_size) + \
-                   " --num_inter_threads=" + str(self.args.num_inter_threads) + \
-                   " --num_intra_threads=" + str(self.args.num_intra_threads) + \
-                   " --src_vocab_file=" + src_vocab_file + \
-                   " --tgt_vocab_file=" + tgt_vocab_file + \
-                   " --inference_input_file=" + inference_input_file + \
-                   " --inference_ref_file=" + inference_ref_file
-
-        run_script = os.path.join(self.args.intelai_models,
-                                  self.args.precision, "run_inference.py")
-
-        self.cmd = self.cmd + run_script + cmd_args
-
-    def run(self):
-        if self.cmd:
-            self.run_command(self.cmd)
diff --git a/benchmarks/launch_benchmark.py b/benchmarks/launch_benchmark.py
index 89a1da3a5..8496d3cea 100644
--- a/benchmarks/launch_benchmark.py
+++ b/benchmarks/launch_benchmark.py
@@ -200,6 +200,7 @@ def get_model_dir(self, benchmark_scripts, use_case, os_type):
         # if we find an optimized model, then we will use that path
         if os.path.isdir(optimized_model_dir):
             intelai_models = optimized_model_dir
+
         return intelai_models
 
     def get_env_vars(self, benchmark_scripts, use_case, intelai_models,
diff --git a/benchmarks/object_detection/tensorflow/faster_rcnn/README.md b/benchmarks/object_detection/tensorflow/faster_rcnn/README.md
deleted file mode 100644
index c4e66fdcf..000000000
--- a/benchmarks/object_detection/tensorflow/faster_rcnn/README.md
+++ /dev/null
@@ -1,5 +0,0 @@
-# Faster R-CNN (ResNet50)
-
-The following documents have instructions for running Faster R-CNN:
-* [FP32 Inference](/benchmarks/object_detection/tensorflow/faster_rcnn/inference/fp32/README.md)
-* [Int8 Inference](/benchmarks/object_detection/tensorflow/faster_rcnn/inference/int8/README.md)
diff --git a/benchmarks/object_detection/tensorflow/faster_rcnn/__init__.py b/benchmarks/object_detection/tensorflow/faster_rcnn/__init__.py
deleted file mode 100644
index cf793ec6a..000000000
--- a/benchmarks/object_detection/tensorflow/faster_rcnn/__init__.py
+++ /dev/null
@@ -1,19 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-# SPDX-License-Identifier: EPL-2.0
-#
diff --git a/benchmarks/object_detection/tensorflow/faster_rcnn/inference/__init__.py b/benchmarks/object_detection/tensorflow/faster_rcnn/inference/__init__.py
deleted file mode 100644
index cf793ec6a..000000000
--- a/benchmarks/object_detection/tensorflow/faster_rcnn/inference/__init__.py
+++ /dev/null
@@ -1,19 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-# SPDX-License-Identifier: EPL-2.0
-#
diff --git a/benchmarks/object_detection/tensorflow/faster_rcnn/inference/fp32/Advanced.md b/benchmarks/object_detection/tensorflow/faster_rcnn/inference/fp32/Advanced.md
deleted file mode 100644
index 4c2e936ba..000000000
--- a/benchmarks/object_detection/tensorflow/faster_rcnn/inference/fp32/Advanced.md
+++ /dev/null
@@ -1,142 +0,0 @@
-<!--- 0. Title -->
-<!-- This document is auto-generated using markdown fragments and the model-builder -->
-<!-- To make changes to this doc, please change the fragments instead of modifying this doc directly -->
-# Faster R-CNN FP32 inference - Advanced Instructions
-
-<!-- 10. Description -->
-This document has advanced instructions for running Faster R-CNN FP32
-inference, which provides more control over the individual parameters that
-are used. For more information on using [`/benchmarks/launch_benchmark.py`](/benchmarks/launch_benchmark.py),
-see the [launch benchmark documentation](/docs/general/tensorflow/LaunchBenchmark.md).
-
-Prior to using these instructions, please follow the setup instructions from
-the model's [README](README.md) and/or the
-[AI Kit documentation](/docs/general/tensorflow/AIKit.md) to get your environment
-setup (if running on bare metal) and download the dataset, pretrained model, etc.
-If you are using AI Kit, please exclude the `--docker-image` flag from the
-commands below, since you will be running the the TensorFlow conda environment
-instead of docker.
-
-<!-- 55. Docker arg -->
-Any of the `launch_benchmark.py` commands below can be run on bare metal by
-removing the `--docker-image` arg. Ensure that you have all of the
-[required prerequisites installed](README.md#run-the-model) in your environment
-before running without the docker container.
-
-If you are new to docker and are running into issues with the container,
-see [this document](/docs/general/docker.md) for troubleshooting tips.
-
-<!-- 50. Launch benchmark instructions -->
-Once your environment is setup, navigate to the `benchmarks` directory of
-the model zoo and set environment variables pointing to the directory for the
-dataset, pretrained model checkpoints, the TensorFlow models repo, and an output
-directory where log files will be written.
-
-```
-# cd to the benchmarks directory in the model zoo
-cd benchmarks
-
-export DATASET_DIR=<path to the directory that contains the coco_val.record file>
-export PRETRAINED_MODEL=<path to the pretrained model checkpoints directory>
-export TF_MODELS_DIR=<path to your clone of the TensorFlow models repo>
-export OUTPUT_DIR=<directory where log files will be saved>
-```
-
-> If you are going to run on bare metal (without the `--docker-image` flag in the
-> commands below) you will need to update a couple of paths in the
-> `$PRETRAINED_MODEL/pipeline.config` file:
-> ```
-> # Replace in the pretrained model path on line 128
-> line_128=$(sed -n '128p' ${PRETRAINED_MODEL}/pipeline.config)
-> new_line_128="  label_map_path: \"$PRETRAINED_MODEL/mscoco_label_map.pbtxt\""
-> sed -i.bak "128s+$line_128+$new_line_128+" ${PRETRAINED_MODEL}/pipeline.config
->
-> # Replace in the dataset directory on line 132
-> line_132=$(sed -n '132p' ${PRETRAINED_MODEL}/pipeline.config)
-> new_line_132="    input_path: \"$DATASET_DIR/coco_val.record\""
-> sed -i.bak "132s+$line_132+$new_line_132+" ${PRETRAINED_MODEL}/pipeline.config
-> ```
-> If you run in a container using the `--docker-image` flag do not edit this
-> file, since the dataset and checkpoints will be mounted to the location
-> that the original `pipeline.config` expects.
-
-Run batch and online inference using the following command:
-```
-python launch_benchmark.py \
- --data-location ${DATASET_DIR} \
- --model-source-dir ${TF_MODELS_DIR} \
- --model-name faster_rcnn \
- --framework tensorflow \
- --precision fp32 \
- --mode inference \
- --socket-id 0 \
- --checkpoint ${PRETRAINED_MODEL} \
- --docker-image intel/intel-optimized-tensorflow:1.15.2 \
- --output-dir ${OUTPUT_DIR} \
- -- config_file=pipeline.config
-```
-
-Test accuracy using the following:
-```
-python launch_benchmark.py \
- --model-name faster_rcnn \
- --mode inference \
- --precision fp32 \
- --framework tensorflow \
- --docker-image intel/intel-optimized-tensorflow:1.15.2 \
- --model-source-dir ${TF_MODELS_DIR} \
- --data-location ${DATASET_DIR} \
- --in-graph ${PRETRAINED_MODEL}/frozen_inference_graph.pb \
- --output-dir ${OUTPUT_DIR} \
- --accuracy-only
-```
-
-Output files and logs are saved to the `${OUTPUT_DIR}` directory.
-
-Below is a sample log file tail when running for batch and online inference:
-```
-I0923 18:13:53.033420 140562203281216 eval_util.py:72] Metrics written to tf summary.
-I0923 18:13:53.033456 140562203281216 eval_util.py:463] Finished evaluation!
-Time spent : ... seconds.
-Time spent per BATCH: ... seconds.
-Log file location: {--output-dir value}/benchmark_faster_rcnn_inference_fp32_20200923_181013.log
-```
-
-And here is a sample log file tail when running for accuracy:
-```
-DONE (t=1.35s).
- Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.316
- Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.489
- Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.355
- Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.316
- Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
- Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.271
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.380
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.383
- Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.383
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
- Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
-Ran inference with batch size 1
-Log location outside container: ${OUTPUT_DIR}/benchmark_faster_rcnn_inference_fp32_20190114_205714.log
-```
-
-And here is a sample log file tail when running for accuracy:
-```
-DONE (t=1.35s).
- Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.316
- Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.489
- Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.355
- Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.316
- Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
- Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.271
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.380
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.383
- Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.383
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
- Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
-Ran inference with batch size 1
-Log location outside container: ${OUTPUT_DIR}/benchmark_faster_rcnn_inference_fp32_20190114_205714.log
-```
-
diff --git a/benchmarks/object_detection/tensorflow/faster_rcnn/inference/fp32/README.md b/benchmarks/object_detection/tensorflow/faster_rcnn/inference/fp32/README.md
deleted file mode 100644
index a27e030a7..000000000
--- a/benchmarks/object_detection/tensorflow/faster_rcnn/inference/fp32/README.md
+++ /dev/null
@@ -1,116 +0,0 @@
-<!--- 0. Title -->
-# Faster R-CNN FP32 inference
-
-<!-- 10. Description -->
-## Description
-
-This document has instructions for running Faster R-CNN FP32 inference using
-Intel-optimized TensorFlow.
-
-<!--- 30. Datasets -->
-## Datasets
-
-The [COCO validation dataset](http://cocodataset.org) is used in the
-Faster R-CNN quickstart scripts. The scripts require that the dataset
-has been converted to the TF records format. See the
-[COCO dataset](/datasets/coco/README.md) for instructions on downloading
-and preprocessing the COCO validation dataset.
-
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| [fp32_inference.sh](/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/fp32/fp32_inference.sh) | Runs batch and online inference using the coco dataset |
-| [fp32_accuracy.sh](/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/fp32/fp32_accuracy.sh) | Runs inference and evaluates the model's accuracy |
-
-<!--- 50. AI Tools -->
-## Run the model
-
-Setup your environment using the instructions below, depending on if you are
-using [AI Tools](/docs/general/tensorflow/AITools.md):
-
-<table>
-  <tr>
-    <th>Setup using AI Tools</th>
-    <th>Setup without AI Tools</th>
-  </tr>
-  <tr>
-    <td>
-      <p>AI Tools does not currently support TF 1.15.2 models</p>
-    </td>
-    <td>
-      <p>To run without AI Tools you will need:</p>
-      <ul>
-        <li>Python 3.6 or 3.7
-        <li>git
-        <li>numactl
-        <li>wget
-        <li><a href="https://github.com/tensorflow/models/blob/v1.12.0/research/object_detection/g3doc/installation.md#protobuf-compilation">Protobuf Compilation</a>
-        <li><a href="https://pypi.org/project/intel-tensorflow/1.15.2/">intel-tensorflow==1.15.2</a>
-        <li>Cython
-        <li>contextlib2
-        <li>jupyter
-        <li>lxml
-        <li>matplotlib
-        <li>pillow>=9.3.0
-        <li>pycocotools
-        <li>A clone of the AI Reference Models repo<br />
-        <pre>git clone https://github.com/IntelAI/models.git</pre>
-      </ul>
-    </td>
-  </tr>
-</table>
-
-Running Faster R-CNN FP32 inference also requires cloning the TensorFlow
-models repo using the tag specified below. Set the `TF_MODELS_DIR` environment
-variable to point to the TensorFlow models directory. Run the
-[protobuf-compiler](https://github.com/tensorflow/models/blob/v1.12.0/research/object_detection/g3doc/installation.md#protobuf-compilation)
-on the `research` directory.
-```
-# Clone the TF models repo
-git clone https://github.com/tensorflow/models.git tf_models
-pushd tf_models
-git checkout tags/v1.12.0
-export TF_MODELS_DIR=$(pwd)
-
-# Run the protobuf-compiler from the TF models research directory
-pushd research
-wget -O protobuf.zip https://github.com/google/protobuf/releases/download/v3.0.0/protoc-3.0.0-linux-x86_64.zip
-unzip protobuf.zip
-./bin/protoc object_detection/protos/*.proto --python_out=.
-rm protobuf.zip
-popd
-popd
-```
-
-Download and extract the pretrained model. The path to this directory
-should be set to the `PRETRAINED_MODEL` environment variable before
-running the quickstart scripts.
-```
-wget https://storage.googleapis.com/intel-optimized-tensorflow/models/faster_rcnn_resnet50_fp32_coco_pretrained_model.tar.gz
-tar -xvf faster_rcnn_resnet50_fp32_coco_pretrained_model.tar.gz
-export PRETRAINED_MODEL=$(pwd)/faster_rcnn_resnet50_fp32_coco
-```
-
-In addition to the `TF_MODELS_DIR` and `PRETRAINED_MODEL` variables from
-above, set environment variables for the path to your `DATASET_DIR` (directory
-where the `coco_val.record` TF records file is located)  and an `OUTPUT_DIR`
-where log files will be written, then run a [quickstart script](#quick-start-scripts).
-```
-# cd to your AI Reference Models directory
-cd models
-
-export DATASET_DIR=<path to the directory that contains the coco_val.record file>
-export OUTPUT_DIR=<path to the directory where log files will be written>
-export TF_MODELS_DIR=<path to the TensorFlow models dir>
-export PRETRAINED_MODEL=<path to the extracted pretrained model dir>
-
-./quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/fp32/<script name>.sh
-```
-
-<!--- 90. Resource Links-->
-## Additional Resources
-
-* To run more advanced use cases, see the instructions [here](Advanced.md)
-  for calling the `launch_benchmark.py` script directly.
diff --git a/benchmarks/object_detection/tensorflow/faster_rcnn/inference/fp32/__init__.py b/benchmarks/object_detection/tensorflow/faster_rcnn/inference/fp32/__init__.py
deleted file mode 100644
index cf793ec6a..000000000
--- a/benchmarks/object_detection/tensorflow/faster_rcnn/inference/fp32/__init__.py
+++ /dev/null
@@ -1,19 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-# SPDX-License-Identifier: EPL-2.0
-#
diff --git a/benchmarks/object_detection/tensorflow/faster_rcnn/inference/fp32/model_init.py b/benchmarks/object_detection/tensorflow/faster_rcnn/inference/fp32/model_init.py
deleted file mode 100644
index 4f352b4e7..000000000
--- a/benchmarks/object_detection/tensorflow/faster_rcnn/inference/fp32/model_init.py
+++ /dev/null
@@ -1,98 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-# SPDX-License-Identifier: EPL-2.0
-#
-
-import argparse
-import os
-import sys
-
-from common.base_model_init import BaseModelInitializer
-from common.base_model_init import set_env_var
-
-
-class ModelInitializer (BaseModelInitializer):
-    def run_inference_sanity_checks(self, args, custom_args):
-        if args.batch_size != -1 and args.batch_size != 1:
-            sys.exit("Faster R-CNN inference supports 'batch-size=1' " +
-                     "only, please modify via the '--batch_size' flag.")
-
-    def __init__(self, args, custom_args, platform_util=None):
-        super(ModelInitializer, self).__init__(args, custom_args, platform_util)
-
-        self.research_dir = os.path.join(self.args.model_source_dir,
-                                         "research")
-        self.run_inference_sanity_checks(self.args, self.custom_args)
-
-        # set num_inter_threads and num_intra_threads
-        self.set_num_inter_intra_threads()
-
-        # Set KMP env vars, if they haven't already been set
-        config_file_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), "config.json")
-        self.set_kmp_vars(config_file_path)
-
-        set_env_var("OMP_NUM_THREADS", self.args.num_intra_threads)
-
-        if self.args.accuracy_only:
-            accuracy_script = os.path.join(
-                self.args.intelai_models, self.args.mode, self.args.precision,
-                "coco_accuracy.sh")
-            if not os.path.exists(accuracy_script):
-                raise ValueError("Unable to locate the Faster R-CNN accuracy "
-                                 "script: {}".format(accuracy_script))
-            self.run_cmd = "bash {} {} {}/coco_val.record {}".format(
-                accuracy_script, self.args.input_graph,
-                self.args.data_location, self.args.model_source_dir)
-        else:
-            self.parse_custom_args()
-
-            benchmark_script = os.path.join(
-                self.args.intelai_models, self.args.mode, self.args.precision,
-                "eval.py")
-            self.command_prefix = \
-                self.get_command_prefix(self.args.socket_id) + self.python_exe + " " + \
-                benchmark_script
-
-            config_file_path = os.path.join(self.args.checkpoint,
-                                            self.args.config_file)
-            self.run_cmd = \
-                self.command_prefix + \
-                " --num_inter_threads " + \
-                str(self.args.num_inter_threads) + \
-                " --num_intra_threads " + \
-                str(self.args.num_intra_threads) + \
-                " --pipeline_config_path " + \
-                config_file_path + \
-                " --checkpoint_dir " + \
-                str(args.checkpoint) + \
-                " --eval_dir " + self.research_dir + \
-                "/object_detection/log/eval"
-
-    def parse_custom_args(self):
-        if self.custom_args:
-            parser = argparse.ArgumentParser()
-            parser.add_argument("--config_file", default=None,
-                                dest="config_file", type=str)
-            self.args = parser.parse_args(self.custom_args,
-                                          namespace=self.args)
-
-    def run(self):
-        original_dir = os.getcwd()
-        os.chdir(self.research_dir)
-        self.run_command(self.run_cmd)
-        os.chdir(original_dir)
diff --git a/benchmarks/object_detection/tensorflow/faster_rcnn/inference/int8/Advanced.md b/benchmarks/object_detection/tensorflow/faster_rcnn/inference/int8/Advanced.md
deleted file mode 100644
index e9b4275d2..000000000
--- a/benchmarks/object_detection/tensorflow/faster_rcnn/inference/int8/Advanced.md
+++ /dev/null
@@ -1,108 +0,0 @@
-<!--- 0. Title -->
-<!-- This document is auto-generated using markdown fragments and the model-builder -->
-<!-- To make changes to this doc, please change the fragments instead of modifying this doc directly -->
-# Faster R-CNN Int8 inference - Advanced Instructions
-
-<!-- 10. Description -->
-This document has advanced instructions for running Faster R-CNN Int8
-inference, which provides more control over the individual parameters that
-are used. For more information on using [`/benchmarks/launch_benchmark.py`](/benchmarks/launch_benchmark.py),
-see the [launch benchmark documentation](/docs/general/tensorflow/LaunchBenchmark.md).
-
-Prior to using these instructions, please follow the setup instructions from
-the model's [README](README.md) and/or the
-[AI Kit documentation](/docs/general/tensorflow/AIKit.md) to get your environment
-setup (if running on bare metal) and download the dataset, pretrained model, etc.
-If you are using AI Kit, please exclude the `--docker-image` flag from the
-commands below, since you will be running the the TensorFlow conda environment
-instead of docker.
-
-<!-- 55. Docker arg -->
-Any of the `launch_benchmark.py` commands below can be run on bare metal by
-removing the `--docker-image` arg. Ensure that you have all of the
-[required prerequisites installed](README.md#run-the-model) in your environment
-before running without the docker container.
-
-If you are new to docker and are running into issues with the container,
-see [this document](/docs/general/docker.md) for troubleshooting tips.
-
-<!-- 50. Launch benchmark instructions -->
-Once your environment is setup, navigate to the `benchmarks` directory of
-the model zoo and set environment variables pointing to the directory for the
-dataset, pretrained model frozen graph, the TensorFlow models repo, and an output
-directory where log files will be written.
-
-```
-# cd to the benchmarks directory in the model zoo
-cd benchmarks
-
-export DATASET_DIR=<path to the dataset>
-export PRETRAINED_MODEL=<path to the pretrained model frozen graph file>
-export TF_MODELS_DIR=<path to your clone of the TensorFlow models repo>
-export OUTPUT_DIR=<directory where log files will be saved>
-```
-Run batch and online inference using the following command.
-```
-python launch_benchmark.py \
- --data-location ${DATASET_DIR} \
- --model-source-dir ${TF_MODELS_DIR} \
- --model-name faster_rcnn \
- --framework tensorflow \
- --precision int8 \
- --mode inference \
- --socket-id 0 \
- --in-graph ${PRETRAINED_MODEL} \
- --docker-image intel/intel-optimized-tensorflow:1.15.2 \
- --output-dir ${OUTPUT_DIR} \
- --benchmark-only \
- -- number_of_steps=5000
-```
-
-Test accuracy where the `--data-location` is the path the directory
-where your `coco_val.record` file is located and the `--in-graph` is
-the pre-trained graph model:
-```
-python launch_benchmark.py \
- --model-name faster_rcnn \
- --mode inference \
- --precision int8 \
- --framework tensorflow \
- --socket-id 0 \
- --docker-image intel/intel-optimized-tensorflow:1.15.2 \
- --model-source-dir ${TF_MODELS_DIR} \
- --data-location ${DATASET_DIR}/coco_val.record \
- --in-graph ${PRETRAINED_MODEL} \
- --output-dir ${OUTPUT_DIR} \
- --accuracy-only
-```
-
-Output files and logs are saved to the `${OUTPUT_DIR}` directory.
-
-Below is a sample log file tail when running for batch and online inference:
-```
-Step 4970: ... seconds
-Step 4980: ... seconds
-Step 4990: ... seconds
-Avg. Duration per Step: ...
-Log location outside container: ${OUTPUT_DIR}/benchmark_faster_rcnn_inference_int8_20190117_232539.log
-```
-
-And here is a sample log file tail when running for accuracy:
-```
-Accumulating evaluation results...
-DONE (t=1.34s).
-Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.310
-Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.479
-Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.351
-Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.310
-Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
-Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
-Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.267
-Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.372
-Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.375
-Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.375
-Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
-Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
-Ran inference with batch size -1
-Log location outside container: ${OUTPUT_DIR}/benchmark_faster_rcnn_inference_int8_20190117_231937.log
-```
diff --git a/benchmarks/object_detection/tensorflow/faster_rcnn/inference/int8/README.md b/benchmarks/object_detection/tensorflow/faster_rcnn/inference/int8/README.md
deleted file mode 100644
index 226973f67..000000000
--- a/benchmarks/object_detection/tensorflow/faster_rcnn/inference/int8/README.md
+++ /dev/null
@@ -1,121 +0,0 @@
-<!--- 0. Title -->
-# Faster R-CNN Int8 inference
-
-<!-- 10. Description -->
-## Description
-
-This document has instructions for running Faster R-CNN Int8 inference using
-Intel Optimized TensorFlow.
-
-<!--- 30. Datasets -->
-## Datasets
-
-The [COCO validation dataset](http://cocodataset.org) is used in the
-Faster R-CNN quickstart scripts. The scripts require that the dataset
-has been converted to the TF records format. See the
-[COCO dataset](/datasets/coco/README.md) for instructions on downloading
-and preprocessing the COCO validation dataset.
-
-Set the `DATASET_DIR` to point to this directory when running Faster R-CNN.
-
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| [int8_inference.sh](int8_inference.sh) | Runs batch and online inference using the raw coco dataset |
-| [int8_accuracy.sh](int8_accuracy.sh) | Runs inference and evaluates the model's accuracy with coco dataset in `coco_val.record` format |
-
-These quickstart scripts can be run in different environments:
-* [Bare Metal](#bare-metal)
-* [Docker](#docker)
-* [Kubernetes](#kubernetes)
-
-<!--- 50. AI Tools -->
-## Run the model
-
-Setup your environment using the instructions below, depending on if you are
-using [AI Tools](/docs/general/tensorflow/AITools.md):
-
-<table>
-  <tr>
-    <th>Setup using AI Tools</th>
-    <th>Setup without AI Tools</th>
-  </tr>
-  <tr>
-    <td>
-      <p>AI Tools does not currently support TF 1.15.2 models</p>
-    </td>
-    <td>
-      <p>To run without AI Tools you will need:</p>
-      <ul>
-        <li>Python 3.6 or 3.7
-        <li>git
-        <li>numactl
-        <li>wget
-        <li><a href="https://github.com/tensorflow/models/blob/v1.12.0/research/object_detection/g3doc/installation.md#protobuf-compilation">Protobuf Compilation</a>
-        <li><a href="https://pypi.org/project/intel-tensorflow/1.15.2/">intel-tensorflow==1.15.2</a>
-        <li>Cython
-        <li>contextlib2
-        <li>jupyter
-        <li>lxml
-        <li>matplotlib
-        <li>pillow>=9.3.0
-        <li>pycocotools
-        <li>A clone of the AI Reference Models repo<br />
-        <pre>git clone https://github.com/IntelAI/models.git</pre>
-      </ul>
-    </td>
-  </tr>
-</table>
-
-Running Faster R-CNN Int8 inference also requires cloning the TensorFlow
-models repo using the tag specified below. Set the `TF_MODELS_DIR` environment
-variable to point to the TensorFlow models directory. Run the
-[protobuf-compiler](https://github.com/tensorflow/models/blob/v1.12.0/research/object_detection/g3doc/installation.md#protobuf-compilation)
-on the `research` directory.
-```
-# Clone the TF models repo
-git clone https://github.com/tensorflow/models.git tf_models
-pushd tf_models
-git checkout tags/v1.12.0
-export TF_MODELS_DIR=$(pwd)
-
-# Run the protobuf-compiler from the TF models research directory
-pushd research
-wget -O protobuf.zip https://github.com/google/protobuf/releases/download/v3.0.0/protoc-3.0.0-linux-x86_64.zip
-unzip protobuf.zip
-./bin/protoc object_detection/protos/*.proto --python_out=.
-rm protobuf.zip
-popd
-popd
-```
-
-Download the pretrained model and set the `PRETRAINED_MODEL` environment
-variable to point to the frozen graph. The quickstart scripts will use this var.
-```
-wget https://storage.googleapis.com/intel-optimized-tensorflow/models/faster_rcnn_int8_pretrained_model.pb
-export PRETRAINED_MODEL=$(pwd)/faster_rcnn_int8_pretrained_model.pb
-```
-
-In addition to the `TF_MODELS_DIR` and `PRETRAINED_MODEL` variables from
-above, set environment variables for the path to your `DATASET_DIR` (where
-the coco TF records file is located) and an `OUTPUT_DIR` where log files
-will be written, then run a [quickstart script](#quick-start-scripts).
-```
-# cd to your AI Reference Models directory
-cd models
-
-export DATASET_DIR=<path to the coco dataset>
-export OUTPUT_DIR=<path to the directory where log files will be written>
-export TF_MODELS_DIR=<path to the TensorFlow models dir>
-export PRETRAINED_MODEL=<path to the pretrained model frozen graph file>
-
-./quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/int8/<script name>.sh
-```
-
-<!--- 90. Resource Links-->
-## Additional Resources
-
-* To run more advanced use cases, see the instructions [here](Advanced.md)
-  for calling the `launch_benchmark.py` script directly.
diff --git a/benchmarks/object_detection/tensorflow/faster_rcnn/inference/int8/__init__.py b/benchmarks/object_detection/tensorflow/faster_rcnn/inference/int8/__init__.py
deleted file mode 100644
index d9c4123de..000000000
--- a/benchmarks/object_detection/tensorflow/faster_rcnn/inference/int8/__init__.py
+++ /dev/null
@@ -1,19 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2019 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-# SPDX-License-Identifier: EPL-2.0
-#
diff --git a/benchmarks/object_detection/tensorflow/faster_rcnn/inference/int8/config.json b/benchmarks/object_detection/tensorflow/faster_rcnn/inference/int8/config.json
deleted file mode 100644
index 6f1228ba7..000000000
--- a/benchmarks/object_detection/tensorflow/faster_rcnn/inference/int8/config.json
+++ /dev/null
@@ -1,7 +0,0 @@
-{
-    "optimization_parameters": {
-        "KMP_AFFINITY": "granularity=fine,verbose,compact,1,0",
-        "KMP_BLOCKTIME": 0,
-        "KMP_SETTINGS": 1
-    }
-}
diff --git a/benchmarks/object_detection/tensorflow/faster_rcnn/inference/int8/model_init.py b/benchmarks/object_detection/tensorflow/faster_rcnn/inference/int8/model_init.py
deleted file mode 100644
index 37eaf2722..000000000
--- a/benchmarks/object_detection/tensorflow/faster_rcnn/inference/int8/model_init.py
+++ /dev/null
@@ -1,120 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2019 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-# SPDX-License-Identifier: EPL-2.0
-#
-
-import argparse
-import os
-
-from common.base_model_init import BaseModelInitializer
-from common.base_model_init import set_env_var
-
-
-class ModelInitializer(BaseModelInitializer):
-    RFCN_PERF_SCRIPT = "run_frozen_graph_rcnn.py"
-    RFCN_ACCURACY_SCRIPT = "coco_int8.sh"
-    perf_script_path = ""
-    accuracy_script_path = ""
-
-    def __init__(self, args, custom_args=[], platform_util=None):
-        super(ModelInitializer, self).__init__(args, custom_args, platform_util)
-
-        self.perf_script_path = os.path.join(
-            self.args.intelai_models, self.args.mode, self.args.precision,
-            self.RFCN_PERF_SCRIPT)
-        self.accuracy_script_path = os.path.join(
-            self.args.intelai_models, self.args.mode, self.args.precision,
-            self.RFCN_ACCURACY_SCRIPT)
-
-        # Set KMP env vars, if they haven't already been set
-        config_file_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), "config.json")
-        self.set_kmp_vars(config_file_path)
-
-        self.validate_args()
-
-        # set num_inter_threads and num_intra_threds (override inter threads to 2)
-        self.set_num_inter_intra_threads(num_inter_threads=2)
-
-    def validate_args(self):
-        if not (self.args.batch_size == -1 or self.args.batch_size == 1):
-            raise ValueError(
-                "Batch Size specified: {}. faster RCNN inference only supports "
-                "batch size = 1".format(self.args.batch_size))
-
-        if not os.path.exists(self.perf_script_path)\
-                and self.args.bechmark_only:
-            raise ValueError("Unable to locate the faster RCNN perf script: {}".
-                             format(self.perf_script_path))
-
-        if not os.path.exists(self.accuracy_script_path)\
-                and self.args.accuracy_only:
-            raise ValueError("Unable to locate the faster RCNN  accuracy script: "
-                             "{}".format(self.accuracy_script_path))
-
-        if not self.args.model_source_dir or not os.path.isdir(
-                self.args.model_source_dir):
-            raise ValueError("Unable to locate TensorFlow models at {}".
-                             format(self.args.model_source_dir))
-
-    def parse_args(self):
-        if self.custom_args:
-            parser = argparse.ArgumentParser()
-            parser.add_argument("-n", "--number-of-steps",
-                                help="Run for n number of steps",
-                                type=int, default=None)
-            self.args = parser.parse_args(self.custom_args,
-                                          namespace=self.args)
-
-    def run_perf_command(self):
-        set_env_var("OMP_NUM_THREADS", self.args.num_intra_threads)
-        self.parse_args()
-        command = self.get_command_prefix(self.args.socket_id)
-        command += " {} ".format(self.python_exe) + self.perf_script_path
-        command += " -g " + self.args.input_graph
-        if self.custom_args:
-            command += " -n " + str(self.args.number_of_steps)
-        if self.args.socket_id != -1:
-            command += " -x "
-        command += \
-            " -d " + self.args.data_location + \
-            " --num-inter-threads " + str(self.args.num_inter_threads) + \
-            " --num-intra-threads " + str(self.args.num_intra_threads)
-        self.run_command(command)
-
-    def run_accuracy_command(self):
-        num_cores = str(self.platform_util.num_cores_per_socket)
-        if self.args.num_cores != -1:
-            num_cores = str(self.args.num_cores)
-
-        set_env_var("OMP_NUM_THREADS", num_cores)
-
-        command = "{} {} {} {}".format(self.accuracy_script_path,
-                                       self.args.input_graph,
-                                       self.args.data_location,
-                                       self.args.model_source_dir)
-        self.run_command(command)
-
-    def run(self):
-        # Run script from the tensorflow models research directory
-        original_dir = os.getcwd()
-        os.chdir(os.path.join(self.args.model_source_dir, "research"))
-        if self.args.accuracy_only:
-            self.run_accuracy_command()
-        else:
-            self.run_perf_command()
-        os.chdir(original_dir)
diff --git a/benchmarks/object_detection/tensorflow/faster_rcnn/requirements.txt b/benchmarks/object_detection/tensorflow/faster_rcnn/requirements.txt
deleted file mode 100644
index 84b53f3fe..000000000
--- a/benchmarks/object_detection/tensorflow/faster_rcnn/requirements.txt
+++ /dev/null
@@ -1,7 +0,0 @@
-Cython
-contextlib2
-pillow>=9.3.0
-lxml
-jupyter
-matplotlib
-pycocotools
diff --git a/benchmarks/object_detection/tensorflow/rfcn/__init__.py b/benchmarks/object_detection/tensorflow/rfcn/__init__.py
deleted file mode 100644
index c4fdb7d61..000000000
--- a/benchmarks/object_detection/tensorflow/rfcn/__init__.py
+++ /dev/null
@@ -1,19 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
diff --git a/benchmarks/object_detection/tensorflow/rfcn/inference/README.md b/benchmarks/object_detection/tensorflow/rfcn/inference/README.md
deleted file mode 100644
index daf51df3b..000000000
--- a/benchmarks/object_detection/tensorflow/rfcn/inference/README.md
+++ /dev/null
@@ -1,220 +0,0 @@
-<!--- 0. Title -->
-# RFCN inference
-
-<!-- 10. Description -->
-
-This document has instructions for running RFCN inference using
-Intel-optimized TensorFlow.
-
-
-<!--- 30. Datasets -->
-## Dataset
-
-The [COCO validation dataset](http://cocodataset.org) is used in these
-RFCN quickstart scripts. The inference quickstart scripts use raw images,
-and the accuracy quickstart scripts require the dataset to be converted
-into the TF records format.
-See the [COCO dataset](/datasets/coco/README.md) for instructions on
-downloading and preprocessing the COCO validation dataset.
-
-
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| [`inference.sh`](/quickstart/object_detection/tensorflow/rfcn/inference/cpu/inference.sh) | Runs inference on a directory of raw images for 500 steps and outputs performance metrics. |
-| [`accuracy.sh`](/quickstart/object_detection/tensorflow/rfcn/inference/cpu/accuracy.sh) | Processes the TF records to run inference and check accuracy on the results. |
-
-<!--- 50. AI Tools -->
-## Run the model
-
-Setup your environment using the instructions below, depending on if you are
-using [AI Tools](/docs/general/tensorflow/AITools.md):
-
-<table>
-  <tr>
-    <th>Setup using AI Tools on Linux</th>
-    <th>Setup without AI Tools on Linux</th>
-    <th>Setup without AI Tools on Windows</th>
-  </tr>
-  <tr>
-    <td>
-      <p>To run using AI Tools on Linux you will need:</p>
-      <ul>
-        <li>git
-        <li>numactl
-        <li>wget
-        <li>build-essential
-        <li>Cython
-        <li>contextlib2
-        <li>jupyter
-        <li>lxml
-        <li>matplotlib
-        <li>pillow>=9.3.0
-        <li>protobuf-compiler
-        <li>pycocotools
-        <li>Activate the `tensorflow` conda environment
-        <pre>conda activate tensorflow</pre>
-      </ul>
-    </td>
-    <td>
-      <p>To run without AI Tools on Linux you will need:</p>
-      <ul>
-        <li>Python 3
-        <li>git
-        <li>numactl
-        <li>wget
-        <li><a href="https://pypi.org/project/intel-tensorflow/">intel-tensorflow>=2.5.0</a>
-        <li>build-essential
-        <li>Cython
-        <li>contextlib2
-        <li>jupyter
-        <li>lxml
-        <li>matplotlib
-        <li>pillow>=9.3.0
-        <li>protobuf-compiler
-        <li>pycocotools
-        <li>A clone of the AI Reference Models repo<br />
-        <pre>git clone https://github.com/IntelAI/models.git</pre>
-      </ul>
-    </td>
-    <td>
-      <p>To run without AI Tools on Windows you will need:</p>
-      <ul>
-        <li><a href="/docs/general/Windows.md">Intel AI Reference Models on Windows Systems prerequisites</a>
-        <li>build-essential
-        <li>Cython
-        <li>contextlib2
-        <li>jupyter
-        <li>lxml
-        <li>matplotlib
-        <li>pillow>=9.3.0
-        <li>protobuf-compiler
-        <li>pycocotools
-        <li>A clone of the AI Reference Models repo<br />
-        <pre>git clone https://github.com/IntelAI/models.git</pre>
-      </ul>
-    </td>
-  </tr>
-</table>
-
-For more information on the required dependencies, see the documentation on [prerequisites](https://github.com/tensorflow/models/blob/6c21084503b27a9ab118e1db25f79957d5ef540b/research/object_detection/g3doc/installation.md#installation)
-in the TensorFlow models repo.
-
-Download and extract the pretrained model and set the `PRETRAINED_MODEL`
-environment variable to point to the frozen graph file.
-If you run on Windows, please use a browser to download the pretrained model using the link below.
-For Linux, run:
-```
-# FP32 Pretrained Model
-https://storage.googleapis.com/intel-optimized-tensorflow/models/2_10_0/rfcn_frozen_inference_graph.pb
-export PRETRAINED_MODEL=$(pwd)/rfcn_frozen_inference_graph.pb
-
-# Int8 Pretrained Model
-https://storage.googleapis.com/intel-optimized-tensorflow/models/2_10_0/rfcn_final_fused_pad_and_conv.pb
-export PRETRAINED_MODEL=$(pwd)/rfcn_final_fused_pad_and_conv.pb
-```
-
-RFCN uses the object detection code from the [TensorFlow Model Garden](https://github.com/tensorflow/models).
-Clone this repo with the SHA specified below and apply the patch from the AI Reference Models directory.
-Set the `TF_MODELS_DIR` environment variable to the path of your clone of the TF Model Garden.
-```
-# Clone the TensorFlow Model Garden
-git clone https://github.com/tensorflow/models.git tensorflow-models
-cd tensorflow-models
-git checkout 6c21084503b27a9ab118e1db25f79957d5ef540b
-
-# Apply the TF2 patch from the AI Reference Models repo directory
-git apply --ignore-space-change --ignore-whitespace <AI Reference Models directory>/models/object_detection/tensorflow/rfcn/inference/tf-2.0.patch
-
-# Set the TF_MODELS_DIR env var
-export TF_MODELS_DIR=$(pwd)
-```
-
-### Run on Linux
-Download and install [Google Protobuf version 3.3.0](https://github.com/google/protobuf/releases/download/v3.3.0/protoc-3.3.0-linux-x86_64.zip), and
-run [protobuf compilation](https://github.com/tensorflow/models/blob/6c21084503b27a9ab118e1db25f79957d5ef540b/research/object_detection/g3doc/installation.md#protobuf-compilation).
-```
-cd TF_MODELS_DIR/research
-protoc object_detection/protos/*.proto --python_out=.
-cd ../..
-```
-
-Once your environment is setup, navigate back to your AI Reference Models directory. Ensure that
-you have set environment variables pointing to the TensorFlow Model Garden repo, the dataset,
-and output directories, and then run a quickstart script.
-
-To run inference with performance metrics:
-```
-# cd to your AI Reference Models directory
-cd models
-
-export DATASET_DIR=<path to the coco val2017 raw image directory (ex: /home/user/coco_dataset/val2017)>
-export PRECISION=<set the precision to "int8" or "fp32">
-export OUTPUT_DIR=<path to the directory where log files will be written>
-export TF_MODELS_DIR=<directory where TensorFlow Model Garden is cloned>
-# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
-export BATCH_SIZE=<customized batch size value>
-
-./quickstart/object_detection/tensorflow/rfcn/inference/cpu/inference.sh
-```
-
-To get accuracy metrics:
-```
-# cd to your AI Reference Models directory
-cd models
-
-export DATASET_DIR=<path to TF record file (ex: /home/user/coco_output/coco_val.record)>
-export PRECISION=<set the precision to "int8" or "fp32">
-export OUTPUT_DIR=<path to the directory where log files will be written>
-export TF_MODELS_DIR=<directory where TensorFlow Model Garden is cloned>
-# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
-export BATCH_SIZE=<customized batch size value>
-
-./quickstart/object_detection/tensorflow/rfcn/inference/cpu/accuracy.sh
-```
-
-### Run on Windows
-* Download and install [Google Protobuf version 3.4.0]((https://github.com/protocolbuffers/protobuf/releases/tag/v3.4.0)) for Windows in addition to the above listed dependencies.
-Download and extract [protoc-3.4.0-win32.zip](https://github.com/protocolbuffers/protobuf/releases/download/v3.4.0/protoc-3.4.0-win32.zip)
-Navigate to the `research` directory in `TF_MODELS_DIR` and install Google Protobuf:
-```
-set TF_MODELS_DIR=<directory where TensorFlow Model Garden is cloned>
-cd %TF_MODELS_DIR%\research
-“C:\<user>\protoc-3.4.0-win32\bin\protoc.exe” object_detection/protos/*.proto --python_out=.
-```
-
-After installing the prerequisites and cloning the TensorFlow models repo, and downloading the pretrained model,
-set the environment variables for the paths to your `PRETRAINED_MODEL`, an `OUTPUT_DIR` where log files will be written,
-TF_MODELS_DIR, and `DATASET_DIR` for COCO raw dataset directory or tf_records file based on whether you run inference or accuracy scripts.
-Navigate to your AI Reference Models directory and then run a [quickstart script](#quick-start-scripts).
-```
-# cd to your AI Reference Models directory
-cd models
-
-set PRETRAINED_MODEL=<path to the frozen graph downloaded above>
-set DATASET_DIR=<path to COCO raw dataset directory or tf_records file based on whether you run inference or accuracy scripts>
-set PRECISION=<set the precision to "int8" or "fp32">
-set OUTPUT_DIR=<directory where log files will be written>
-set TF_MODELS_DIR=<directory where TensorFlow Model Garden is cloned>
-# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
-set BATCH_SIZE=<customized batch size value>
-
-bash quickstart\object_detection\tensorflow\rfcn\inference\cpu\<script name>.sh
-```
-> Note: You may use `cygpath` to convert the Windows paths to Unix paths before setting the environment variables. 
-As an example, if the dataset location on Windows is `D:\user\coco_dataset\val2017`, convert the Windows path to Unix as shown:
-> ```
-> cygpath D:\user\coco_dataset\val2017
-> /d/user/coco_dataset/val2017
->```
->Then, set the `DATASET_DIR` environment variable `set DATASET_DIR=/d/user/coco_dataset/val2017`.
-
-<!--- 90. Resource Links-->
-## Additional Resources
-
-* To run more advanced use cases, see the instructions for the available precisions [FP32](fp32/Advanced.md) [Int8](int8/Advanced.md) [<bfloat16 precision>](<bfloat16 advanced readme link>) for calling the `launch_benchmark.py` script directly.
-* To run the model using docker, please see the [Intel® Developer Catalog](https://www.intel.com/content/www/us/en/developer/tools/software-catalog/containers.html)
-  workload container:<br />
-  [https://www.intel.com/content/www/us/en/developer/articles/containers/rfcn-fp32-inference-tensorflow-container.html](https://www.intel.com/content/www/us/en/developer/articles/containers/rfcn-fp32-inference-tensorflow-container.html).
diff --git a/benchmarks/object_detection/tensorflow/rfcn/inference/__init__.py b/benchmarks/object_detection/tensorflow/rfcn/inference/__init__.py
deleted file mode 100644
index c4fdb7d61..000000000
--- a/benchmarks/object_detection/tensorflow/rfcn/inference/__init__.py
+++ /dev/null
@@ -1,19 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
diff --git a/benchmarks/object_detection/tensorflow/rfcn/inference/fp32/Advanced.md b/benchmarks/object_detection/tensorflow/rfcn/inference/fp32/Advanced.md
deleted file mode 100644
index 065c93d97..000000000
--- a/benchmarks/object_detection/tensorflow/rfcn/inference/fp32/Advanced.md
+++ /dev/null
@@ -1,118 +0,0 @@
-<!--- 0. Title -->
-<!-- This document is auto-generated using markdown fragments and the model-builder -->
-<!-- To make changes to this doc, please change the fragments instead of modifying this doc directly -->
-# RFCN FP32 inference - Advanced Instructions
-
-<!-- 10. Description -->
-This document has advanced instructions for running RFCN FP32
-inference, which provides more control over the individual parameters that
-are used. For more information on using [`/benchmarks/launch_benchmark.py`](/benchmarks/launch_benchmark.py),
-see the [launch benchmark documentation](/docs/general/tensorflow/LaunchBenchmark.md).
-
-Prior to using these instructions, please follow the setup instructions from
-the model's [README](README.md) and/or the
-[AI Kit documentation](/docs/general/tensorflow/AIKit.md) to get your environment
-setup (if running on bare metal) and download the dataset, pretrained model, etc.
-If you are using AI Kit, please exclude the `--docker-image` flag from the
-commands below, since you will be running the the TensorFlow conda environment
-instead of docker.
-
-<!-- 55. Docker arg -->
-Any of the `launch_benchmark.py` commands below can be run on bare metal by
-removing the `--docker-image` arg. Ensure that you have all of the
-[required prerequisites installed](README.md#run-the-model) in your environment
-before running without the docker container.
-
-If you are new to docker and are running into issues with the container,
-see [this document](/docs/general/docker.md) for troubleshooting tips.
-
-<!-- 50. Launch benchmark instructions -->
-Once your environment is setup, navigate to the `benchmarks` directory of
-the model zoo and set environment variables pointing to the directory for the
-dataset, pretrained model frozen graph, the TensorFlow models repo, and an output
-directory where log files will be written.
-
-```
-# cd to the benchmarks directory in the model zoo
-cd benchmarks
-
-export DATASET_DIR=<path to the dataset>
-export PRETRAINED_MODEL=<path to the pretrained model frozen graph file>
-export TF_MODELS_DIR=<path to your clone of the TensorFlow models repo>
-export OUTPUT_DIR=<directory where log files will be saved>
-```
-
-The command below runs batch and online inference. Note that the
-`--data-location ${DATASET_DIR}` should point to the raw COCO dataset images
-(for example: `DATASET_DIR=/home/<user>/coco_dataset/val2017`).
-```
-python launch_benchmark.py \
-    --model-name rfcn \
-    --mode inference \
-    --precision fp32 \
-    --framework tensorflow \
-    --docker-image intel/intel-optimized-tensorflow:latest \
-    --model-source-dir ${TF_MODELS_DIR} \
-    --data-location ${DATASET_DIR} \
-    --in-graph ${PRETRAINED_MODEL} \
-    --output-dir ${OUTPUT_DIR} \
-    --benchmark-only \
-    -- number_of_steps=500
-```
-
-Or for accuracy testing, use the command below and set the `--data-location ${DATASET_DIR}`
-to the path to the COCO validation TF records file (for example:
-`DATASET_DIR=/home/<user>/coco_output/coco_val.record`).
-```
-python launch_benchmark.py \
-    --model-name rfcn \
-    --mode inference \
-    --precision fp32 \
-    --framework tensorflow \
-    --docker-image intel/intel-optimized-tensorflow:latest \
-    --model-source-dir ${TF_MODELS_DIR} \
-    --data-location ${DATASET_DIR} \
-    --in-graph ${PRETRAINED_MODEL} \
-    --output-dir ${OUTPUT_DIR} \
-    --accuracy-only \
-    -- split="accuracy_message"
-```
-
-Note that the `--verbose` flag can be added to any of the above commands
-to get additional debug output .
-
-Log files are located at the value of `$OUTPUT_DIR`. Below is a sample log
-file tail when running for batch and online inference:
-```
-Average time per step: ... sec
-Received these standard args: Namespace(accuracy_only=False, batch_size=1, benchmark_only=False, checkpoint='/checkpoints', data_location='/dataset', framework='tensorflow', input_graph=None, intelai_models='/workspace/intelai_models', mode='inference', model_args=[], model_name='rfcn', model_source_dir='/workspace/models', num_cores=-1, num_inter_threads=2, num_intra_threads=56, precision='fp32, socket_id=0, use_case='object_detection', verbose=True)
-Received these custom args: ['--config_file=rfcn_pipeline.config']
-Run model here.
-current directory: /workspace/models/research
-Running: numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/fp32/eval.py --inter_op 1 --intra_op 28 --omp 28 --pipeline_config_path /checkpoints/rfcn_pipeline.config --checkpoint_dir /checkpoints --eval_dir /workspace/models/research/object_detection/models/rfcn/eval  --logtostderr  --blocktime=0  --run_once=True
-PYTHONPATH: :/workspace/intelai_models:/workspace/models/research:/workspace/models/research/slim:/workspace/models
-RUNCMD: python common/tensorflow/run_tf_benchmark.py --framework=tensorflow --use-case=object_detection --model-name=rfcn --precision=fp32 --mode=inference --model-source-dir=/workspace/models --intelai-models=/workspace/intelai_models --num-cores=-1 --batch-size=1 --data-location=/dataset --socket-id 0 --verbose --checkpoint=/checkpoints         --config_file=rfcn_pipeline.config
-Batch Size: 1
-Ran inference with batch size 1
-Log location outside container: ${OUTPUT_DIR}/benchmark_rfcn_inference.log
-```
-
-And here is a sample log file tail when running for accuracy:
-```
-DONE (t=1.19s).
- Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.347
- Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.532
- Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.389
- Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.347
- Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
- Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.282
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.396
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.400
- Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.400
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
- Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
- Ran inference with batch size 1
-Log location outside container: ${OUTPUT_DIR}/benchmark_rfcn_inference_fp32_20181221_211905.log
-```
-
diff --git a/benchmarks/object_detection/tensorflow/rfcn/inference/fp32/__init__.py b/benchmarks/object_detection/tensorflow/rfcn/inference/fp32/__init__.py
deleted file mode 100644
index c4fdb7d61..000000000
--- a/benchmarks/object_detection/tensorflow/rfcn/inference/fp32/__init__.py
+++ /dev/null
@@ -1,19 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
diff --git a/benchmarks/object_detection/tensorflow/rfcn/inference/fp32/config.json b/benchmarks/object_detection/tensorflow/rfcn/inference/fp32/config.json
deleted file mode 100644
index 6f1228ba7..000000000
--- a/benchmarks/object_detection/tensorflow/rfcn/inference/fp32/config.json
+++ /dev/null
@@ -1,7 +0,0 @@
-{
-    "optimization_parameters": {
-        "KMP_AFFINITY": "granularity=fine,verbose,compact,1,0",
-        "KMP_BLOCKTIME": 0,
-        "KMP_SETTINGS": 1
-    }
-}
diff --git a/benchmarks/object_detection/tensorflow/rfcn/inference/fp32/model_init.py b/benchmarks/object_detection/tensorflow/rfcn/inference/fp32/model_init.py
deleted file mode 100644
index 74a7329c7..000000000
--- a/benchmarks/object_detection/tensorflow/rfcn/inference/fp32/model_init.py
+++ /dev/null
@@ -1,146 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
-
-import argparse
-import os
-import sys
-
-from common.base_model_init import BaseModelInitializer
-from common.base_model_init import set_env_var
-
-
-class ModelInitializer(BaseModelInitializer):
-    accuracy_script = "coco_mAP.sh"
-    accuracy_script_path = ""
-
-    def run_inference_sanity_checks(self, args, custom_args):
-        if args.batch_size != -1 and args.batch_size != 1:
-            sys.exit("R-FCN inference supports 'batch-size=1' " +
-                     "only, please modify via the '--batch_size' flag.")
-
-    def __init__(self, args, custom_args, platform_util):
-        super(ModelInitializer, self).__init__(args, custom_args, platform_util)
-
-        self.accuracy_script_path = os.path.join(
-            self.args.intelai_models, self.args.mode, self.args.precision,
-            self.accuracy_script)
-        self.benchmark_script = os.path.join(
-            self.args.intelai_models, self.args.mode,
-            self.args.precision, "run_rfcn_inference.py")
-
-        # Set KMP env vars, if they haven't already been set
-        config_file_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), "config.json")
-        self.set_kmp_vars(config_file_path)
-
-        # Set num_inter_threads and num_intra_threads
-        self.set_num_inter_intra_threads()
-
-        self.run_inference_sanity_checks(self.args, self.custom_args)
-        self.parse_custom_args()
-        self.research_dir = os.path.join(self.args.model_source_dir,
-                                         "research")
-
-    def parse_custom_args(self):
-        if self.custom_args:
-            parser = argparse.ArgumentParser()
-            mutex_group = parser.add_mutually_exclusive_group()
-            mutex_group.add_argument("-x", "--number_of_steps",
-                                     help="Run for n number of steps",
-                                     type=int, default=None)
-            mutex_group.add_argument(
-                "-v", "--visualize",
-                help="Whether to visualize the output image",
-                action="store_true")
-            parser.add_argument("-q", "--split",
-                                help="Location of accuracy data",
-                                type=str, default=None)
-            self.args = parser.parse_args(self.custom_args, namespace=self.args)
-        else:
-            raise ValueError("Custom parameters are missing...")
-
-    def run_perf_command(self):
-        command = []
-        command.append(self.get_command_prefix(self.args.socket_id))
-        set_env_var("OMP_NUM_THREADS", self.args.num_intra_threads)
-
-        command += (self.python_exe, self.benchmark_script)
-        command += ("-m", self.args.model_source_dir)
-        command += ("-g", self.args.input_graph)
-        command += ("--num-intra-threads", str(self.args.num_intra_threads))
-        command += ("--num-inter-threads", str(self.args.num_inter_threads))
-        if self.args.number_of_steps:
-            command += ("-x", "{}".format(self.args.number_of_steps))
-        if self.args.visualize:
-            command += ("-v")
-        if self.args.data_location:
-            command += ("-d", self.args.data_location)
-        self.run_command(" ".join(command))
-
-    def run_accuracy_command(self):
-        if not os.path.exists(self.accuracy_script_path):
-            raise ValueError("Unable to locate the R-FCN accuracy script: "
-                             "{}".format(self.accuracy_script_path))
-        os.environ["FROZEN_GRAPH"] = "{}".format(self.args.input_graph)
-
-        if self.args.data_location and os.path.exists(
-                self.args.data_location):
-            if os.path.isdir(self.args.data_location):
-                # For directories, create a comma separated list of all the TF record files
-                tf_records_files = []
-                for file in os.listdir(self.args.data_location):
-                    if ".record" in file:
-                        tf_records_files.append(os.path.join(self.args.data_location, file))
-                if len(tf_records_files) == 0:
-                    print("ERROR: No TF records files were found in "
-                          "the data location: {}".format(self.args.data_location))
-                    sys.exit(1)
-                os.environ["TF_RECORD_FILES"] = ",".join(tf_records_files)
-            else:
-                # Single TF record file
-                os.environ["TF_RECORD_FILES"] = "{}".format(self.args.data_location)
-        else:
-            raise ValueError(
-                "Unable to locate the coco data record file at {}".format(
-                    self.args.tf_record_file))
-
-        if self.args.split:
-            os.environ["SPLIT"] = "{}".format(self.args.split)
-        else:
-            raise ValueError("Must specify SPLIT parameter")
-
-        os.environ["TF_MODELS_ROOT"] = "{}".format(self.args.model_source_dir)
-
-        command = "bash {}".format(self.accuracy_script_path)
-        self.run_command(command)
-
-    def run(self):
-        # TODO: make it a property in PlatformUtils (platform_util.os_type) to get the host OS.
-        # We already do the OS check there to see if it's one that we support.
-        if os.environ.get('OS', '') == 'Windows_NT':
-            os.environ["PYTHONPATH"] = "{};{};{}".format(
-                self.research_dir, os.path.join(self.research_dir, "slim"),
-                os.environ["PYTHONPATH"])
-        original_dir = os.getcwd()
-        os.chdir(self.research_dir)
-        if self.args.accuracy_only:
-            self.run_accuracy_command()
-        else:
-            self.run_perf_command()
-        os.chdir(original_dir)
diff --git a/benchmarks/object_detection/tensorflow/rfcn/inference/int8/Advanced.md b/benchmarks/object_detection/tensorflow/rfcn/inference/int8/Advanced.md
deleted file mode 100644
index 4d09abb4c..000000000
--- a/benchmarks/object_detection/tensorflow/rfcn/inference/int8/Advanced.md
+++ /dev/null
@@ -1,121 +0,0 @@
-<!--- 0. Title -->
-<!-- This document is auto-generated using markdown fragments and the model-builder -->
-<!-- To make changes to this doc, please change the fragments instead of modifying this doc directly -->
-# RFCN Int8 inference - Advanced Instructions
-
-<!-- 10. Description -->
-This document has advanced instructions for running RFCN Int8
-inference, which provides more control over the individual parameters that
-are used. For more information on using [`/benchmarks/launch_benchmark.py`](/benchmarks/launch_benchmark.py),
-see the [launch benchmark documentation](/docs/general/tensorflow/LaunchBenchmark.md).
-
-Prior to using these instructions, please follow the setup instructions from
-the model's [README](README.md) and/or the
-[AI Kit documentation](/docs/general/tensorflow/AIKit.md) to get your environment
-setup (if running on bare metal) and download the dataset, pretrained model, etc.
-If you are using AI Kit, please exclude the `--docker-image` flag from the
-commands below, since you will be running the the TensorFlow conda environment
-instead of docker.
-
-<!-- 55. Docker arg -->
-Any of the `launch_benchmark.py` commands below can be run on bare metal by
-removing the `--docker-image` arg. Ensure that you have all of the
-[required prerequisites installed](README.md#run-the-model) in your environment
-before running without the docker container.
-
-If you are new to docker and are running into issues with the container,
-see [this document](/docs/general/docker.md) for troubleshooting tips.
-
-<!-- 50. Launch benchmark instructions -->
-Once your environment is setup, navigate to the `benchmarks` directory of
-the model zoo and set environment variables pointing to the directory for the
-dataset, pretrained model frozen graph, the TensorFlow models repo, and an output
-directory where log files will be written.
-
-```
-# cd to the benchmarks directory in the model zoo
-cd benchmarks
-
-export DATASET_DIR=<path to the dataset (raw images for inference or the TF records file for accuracy)>
-export PRETRAINED_MODEL=<path to the pretrained model frozen graph file>
-export TF_MODELS_DIR=<path to your clone of the TensorFlow models repo>
-export OUTPUT_DIR=<directory where log files will be saved>
-```
-
-The command below runs batch and online inference. Note that the
-`--data-location ${DATASET_DIR}` should point to the raw COCO dataset images
-(for example `DATASET_DIR=/home/<user>/coco_dataset/val2017`).
-```
-python launch_benchmark.py \
-    --model-name rfcn \
-    --mode inference \
-    --precision int8 \
-    --framework tensorflow \
-    --docker-image intel/intel-optimized-tensorflow:latest \
-    --model-source-dir ${TF_MODELS_DIR} \
-    --data-location ${DATASET_DIR} \
-    --in-graph ${PRETRAINED_MODEL} \
-    --benchmark-only \
-    --output-dir ${OUTPUT_DIR} \
-    -- number_of_steps=500
-```
-
-Or for accuracy testing, use the command below and set the `--data-location ${DATASET_DIR}`
-to the path the TF record file (for example `DATASET_DIR=/home/<user>/coco_output/coco_val.record`).
-```
-python launch_benchmark.py \
-    --model-name rfcn \
-    --mode inference \
-    --precision int8 \
-    --framework tensorflow \
-    --docker-image intel/intel-optimized-tensorflow:latest \
-    --model-source-dir ${TF_MODELS_DIR} \
-    --data-location ${DATASET_DIR} \
-    --in-graph ${PRETRAINED_MODEL} \
-    --accuracy-only \
-    --output-dir ${OUTPUT_DIR} \
-    -- split="accuracy_message"
-```
-
-Note that the `--verbose` flag can be added to any of the above commands
-to get additional debug output.
-
-Log files are located at the `${OUTPUT_DIR}` path.
-
-Below is a sample log file tail when running for batch
-and online inference:
-```
-Step 0: ... seconds
-Step 10: ... seconds
-...
-Step 460: ... seconds
-Step 470: ... seconds
-Step 480: ... seconds
-Step 490: ... seconds
-Avg. Duration per Step: ...
-...
-Ran inference with batch size -1
-Log location outside container: ${OUTPUT_DIR}/benchmark_rfcn_inference_int8_20190416_182445.log
-```
-
-And here is a sample log file tail when running for accuracy:
-```
-...
-Accumulating evaluation results...
-DONE (t=1.91s).
- Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.327
- Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.506
- Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.365
- Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.327
- Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
- Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.271
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.376
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.380
- Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.380
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
- Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
-Ran inference with batch size -1
-Log location outside container: ${OUTPUT_DIR}/benchmark_rfcn_inference_int8_20190227_194752.log
-```
-
diff --git a/benchmarks/object_detection/tensorflow/rfcn/inference/int8/__init__.py b/benchmarks/object_detection/tensorflow/rfcn/inference/int8/__init__.py
deleted file mode 100644
index c4fdb7d61..000000000
--- a/benchmarks/object_detection/tensorflow/rfcn/inference/int8/__init__.py
+++ /dev/null
@@ -1,19 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
diff --git a/benchmarks/object_detection/tensorflow/rfcn/inference/int8/config.json b/benchmarks/object_detection/tensorflow/rfcn/inference/int8/config.json
deleted file mode 100644
index 6f1228ba7..000000000
--- a/benchmarks/object_detection/tensorflow/rfcn/inference/int8/config.json
+++ /dev/null
@@ -1,7 +0,0 @@
-{
-    "optimization_parameters": {
-        "KMP_AFFINITY": "granularity=fine,verbose,compact,1,0",
-        "KMP_BLOCKTIME": 0,
-        "KMP_SETTINGS": 1
-    }
-}
diff --git a/benchmarks/object_detection/tensorflow/rfcn/inference/int8/model_init.py b/benchmarks/object_detection/tensorflow/rfcn/inference/int8/model_init.py
deleted file mode 100755
index fcaca9e4f..000000000
--- a/benchmarks/object_detection/tensorflow/rfcn/inference/int8/model_init.py
+++ /dev/null
@@ -1,174 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-#
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-
-import os
-import sys
-import argparse
-
-from common.base_model_init import BaseModelInitializer, set_env_var
-
-
-class ModelInitializer(BaseModelInitializer):
-    command = []
-    RFCN_PERF_SCRIPT = "run_rfcn_inference.py"
-    RFCN_ACCURACY_SCRIPT = "coco_mAP.sh"
-    perf_script_path = ""
-    accuracy_script_path = ""
-
-    def __init__(self, args, custom_args=[], platform_util=None):
-        super(ModelInitializer, self).__init__(args, custom_args, platform_util)
-        self.perf_script_path = os.path.join(
-            self.args.intelai_models, self.args.mode, self.args.precision,
-            self.RFCN_PERF_SCRIPT)
-        self.accuracy_script_path = os.path.join(
-            self.args.intelai_models, self.args.mode, self.args.precision,
-            self.RFCN_ACCURACY_SCRIPT)
-
-        # remove intelai models path, so that imports don't conflict
-        if "MOUNT_BENCHMARK" in os.environ and \
-                os.environ["MOUNT_BENCHMARK"] in sys.path:
-            sys.path.remove(os.environ["MOUNT_BENCHMARK"])
-        if self.args.intelai_models in sys.path:
-            sys.path.remove(self.args.intelai_models)
-
-        self.parse_args()
-
-        # Set KMP env vars, if they haven't already been set
-        config_file_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), "config.json")
-        self.set_kmp_vars(config_file_path)
-
-        # Set num_inter_threads and num_intra_threads
-        self.set_num_inter_intra_threads()
-
-    def parse_args(self):
-        if self.custom_args:
-            parser = argparse.ArgumentParser()
-            mutex_group = parser.add_mutually_exclusive_group()
-            mutex_group.add_argument("-x", "--number_of_steps",
-                                     help="Run for n number of steps",
-                                     type=int, default=None)
-            mutex_group.add_argument(
-                "-v", "--visualize",
-                help="Whether to visualize the output image",
-                action="store_true")
-            parser.add_argument(
-                "-t", "--timeline",
-                help="Output file name for TF timeline",
-                type=str, default=None)
-            parser.add_argument("-e", "--evaluate_tensor",
-                                help="Full tensor name to evaluate",
-                                type=str, default=None)
-            parser.add_argument("-p", "--print_accuracy",
-                                help="Print accuracy results",
-                                action="store_true")
-            parser.add_argument("-q", "--split",
-                                help="Location of accuracy data",
-                                type=str, default=None)
-            self.args = parser.parse_args(self.custom_args,
-                                          namespace=self.args)
-            self.validate_args()
-        else:
-            raise ValueError("Custom parameters are missing...")
-
-    def validate_args(self):
-        if not (self.args.batch_size == -1 or self.args.batch_size == 1):
-            raise ValueError(
-                "Batch Size specified: {}. R-FCN inference only supports "
-                "batch size = 1".format(self.args.batch_size))
-
-        if not os.path.exists(self.perf_script_path):
-            raise ValueError("Unable to locate the R-FCN perf script: {}".
-                             format(self.perf_script_path))
-
-        if not os.path.exists(self.accuracy_script_path):
-            raise ValueError("Unable to locate the R-FCN accuracy script: "
-                             "{}".format(self.accuracy_script_path))
-
-        if not self.args.model_source_dir or not os.path.isdir(
-                self.args.model_source_dir):
-            raise ValueError("Unable to locate TensorFlow models at {}".
-                             format(self.args.model_source_dir))
-
-    def run_perf_command(self):
-        self.command.append(self.get_command_prefix(self.args.socket_id))
-
-        set_env_var("OMP_NUM_THREADS", self.args.num_intra_threads)
-
-        self.command += (self.python_exe, self.perf_script_path)
-        self.command += ("-m", self.args.model_source_dir)
-        self.command += ("-g", self.args.input_graph)
-        self.command += ("--num-intra-threads", str(self.args.num_intra_threads))
-        self.command += ("--num-inter-threads", str(self.args.num_inter_threads))
-        if self.args.number_of_steps:
-            self.command += ("-x", "{}".format(self.args.number_of_steps))
-        if self.args.visualize:
-            self.command += ("-v")
-        if self.args.timeline:
-            self.command += ("-t", self.args.timeline)
-        if self.args.data_location:
-            self.command += ("-d", self.args.data_location)
-        if self.args.evaluate_tensor:
-            self.command += ("-e", self.args.evaluate_tensor)
-        if self.args.print_accuracy:
-            self.command += ("-p")
-        self.run_command(" ".join(self.command))
-
-    def run_accuracy_command(self):
-        # already validated by parent
-        self.command = self.get_command_prefix(self.args.socket_id, numactl=False)
-        os.environ["FROZEN_GRAPH"] = "{}".format(self.args.input_graph)
-
-        if self.args.data_location and os.path.exists(
-                self.args.data_location):
-            os.environ["TF_RECORD_FILE"] = "{}".format(self.args.data_location)
-        else:
-            raise ValueError(
-                "Unable to locate the coco data record file at {}".format(
-                    self.args.tf_record_file))
-
-        if self.args.split:
-            os.environ["SPLIT"] = "{}".format(self.args.split)
-        else:
-            raise ValueError("Must specify SPLIT parameter")
-
-        os.environ["TF_MODELS_ROOT"] = "{}".format(self.args.model_source_dir)
-
-        self.command += "bash " + self.accuracy_script_path
-        self.run_command(self.command)
-
-    def run(self):
-        # TODO: make it a property in PlatformUtils (platform_util.os_type) to get the host OS.
-        # We already do the OS check there to see if it's one that we support.
-        if os.environ.get('OS', '') == 'Windows_NT':
-            os.environ["PYTHONPATH"] = "{};{};{}".format(
-                os.path.join(self.args.model_source_dir, "research"),
-                os.path.join(self.args.model_source_dir, "research", "slim"),
-                os.environ["PYTHONPATH"])
-        # Run script from the tensorflow models research directory
-        original_dir = os.getcwd()
-        os.chdir(os.path.join(self.args.model_source_dir, "research"))
-        if self.args.accuracy_only:
-            self.run_accuracy_command()
-        else:
-            self.run_perf_command()
-        os.chdir(original_dir)
diff --git a/benchmarks/object_detection/tensorflow/rfcn/requirements.txt b/benchmarks/object_detection/tensorflow/rfcn/requirements.txt
deleted file mode 100644
index 6968a3036..000000000
--- a/benchmarks/object_detection/tensorflow/rfcn/requirements.txt
+++ /dev/null
@@ -1,9 +0,0 @@
-numpy>=1.17.4,<=1.23
-Cython
-contextlib2
-pillow>=9.3.0
-lxml
-jupyter
-matplotlib
-pycocotools
-protobuf==3.20.3
diff --git a/benchmarks/object_detection/tensorflow/ssd-mobilenet/inference/README.md b/benchmarks/object_detection/tensorflow/ssd-mobilenet/inference/README.md
index 8758aa3e9..4691e7f65 100644
--- a/benchmarks/object_detection/tensorflow/ssd-mobilenet/inference/README.md
+++ b/benchmarks/object_detection/tensorflow/ssd-mobilenet/inference/README.md
@@ -111,6 +111,10 @@ export PRETRAINED_MODEL=$(pwd)/ssdmobilenet_fp32_pretrained_model_combinedNMS.pb
 wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/ssdmobilenet_int8_pretrained_model_combinedNMS_s8.pb
 export PRETRAINED_MODEL=$(pwd)/ssdmobilenet_int8_pretrained_model_combinedNMS_s8.pb
 
+# Int8 Subgraph model (Int8 Accuracy only)
+wget https://storage.googleapis.com/intel-optimized-tensorflow/models/3_1/ssdmobilenet_preprocess.pb
+export INPUT_SUBGRAPH=$(pwd)/ssdmobilenet_preprocess.pb
+
 # Int8 Pretrained model for OneDNN Graph (Only used when the plugin Intel Extension for Tensorflow is installed, as OneDNN Graph optimization is enabled by default at this point)
 wget https://storage.googleapis.com/intel-optimized-tensorflow/models/2_12_0/ssd_mb_itex_int8.pb
 export PRETRAINED_MODEL=$(pwd)/ssd_mb_itex_int8.pb
diff --git a/benchmarks/object_detection/tensorflow/ssd-mobilenet/inference/int8/model_init.py b/benchmarks/object_detection/tensorflow/ssd-mobilenet/inference/int8/model_init.py
index 9bac17353..6f8b635aa 100644
--- a/benchmarks/object_detection/tensorflow/ssd-mobilenet/inference/int8/model_init.py
+++ b/benchmarks/object_detection/tensorflow/ssd-mobilenet/inference/int8/model_init.py
@@ -38,6 +38,8 @@ def parse_args(self):
             "--steps", dest="steps",
             help="number of steps",
             type=int, default=50)
+        parser.add_argument(
+            "--input-subgraph", dest="input_subgraph", type=str, help="int8 graph")
         self.args = parser.parse_args(self.custom_args,
                                       namespace=self.args)
 
@@ -64,6 +66,7 @@ def __init__(self, args, custom_args=[], platform_util=None):
         self.command_prefix += " -g {0}".format(self.args.input_graph)
         self.command_prefix += " -i {0}".format(self.args.steps)
         self.command_prefix += " -w {0}".format(self.args.warmup_steps)
+        self.command_prefix += " -sg {0}".format(self.args.input_subgraph)
         self.command_prefix += " -a {0}".format(self.args.num_intra_threads)
         self.command_prefix += " -e {0}".format(self.args.num_inter_threads)
         if self.args.data_location:
diff --git a/benchmarks/recommendation/tensorflow/ncf/README.md b/benchmarks/recommendation/tensorflow/ncf/README.md
deleted file mode 100644
index a9e026ccf..000000000
--- a/benchmarks/recommendation/tensorflow/ncf/README.md
+++ /dev/null
@@ -1,115 +0,0 @@
-## Neural Collaborative Filtering (NCF) ##
-
-The following links have instructions on how to run NCF for the
-following modes/precisions:
-* [FP32 training](#training-instructions)
-* [BFloat16 training](#training-instructions)
-* [FP32 inference](/benchmarks/recommendation/tensorflow/ncf/inference/fp32/README.md)
-
-## Training Instructions
-
-1. Dataset
-
-Support two datasets: ml-1m, ml-20m. It can be specified with flag `dataset=ml-1m` or `dataset=ml-20m`.
-This model uses official tensorflow models repo, where [ncf](https://github.com/tensorflow/models/tree/master/official/recommendation)
-model automatically downloads movielens ml-1m dataset as default if the `--data-location` flag is not set.
-If you want to download movielens 1M/20M dataset and provide that path to `--data-location`, check this [reference](https://grouplens.org/datasets/movielens/)
-
-2. Clone the official `tensorflow/models` repository.
-
-For training, please checkout with tag `r2.1_model_reference `:
-```
-git clone https://github.com/tensorflow/models.git
-cd models
-git checkout r2.1_model_reference
-```
-
-3. Now clone `IntelAI/models` repository, then navigate to the `benchmarks` folder:
-
-```
-cd $MODEL_WORK_DIR
-git clone https://github.com/IntelAI/models.git
-cd models/benchmarks
-```
-
-4. Download and extract the pre-trained model, be careful it only works for 1M dataset.
-Skip this step if training only.
-```
-wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_5/ncf_fp32_pretrained_model.tar.gz
-tar -xzvf ncf_fp32_pretrained_model.tar.gz
-```
-
-5. Run the `launch_benchmark.py` script with the appropriate parameters.
-* `--model-source-dir` - Path to official tensorflow models from step2.
-* `--checkpoint` - Path to checkpoint directory for the Pre-trained model from step4. Checkpoint will be stored in this directory while training.
-
-For training, suggest options are:
-* `--batch-size 98304`
-* `--precision fp32` or `--precision bfloat16`
-* `dataset=ml-20m` - suggest to use 20M dataset for training
-* `clean=1` - delete any files stored in `--checkpoint`. Disable this flag if want to reuse pre-trained model.
-* `te=12` - set the max epoch. NCF will train 6+ epochs to SOTA, this flag will stop training when reach specific epoch. Set it to 1 if only want to test performance
-
-```
-python launch_benchmark.py \
-    --checkpoint /home/<user>/ncf_fp32_pretrained_model \
-    --model-source-dir /home/<user>/tensorflow/models \
-    --model-name ncf \
-    --framework tensorflow \
-    --mode training \
-    --precision bfloat16 \
-    --batch-size 98304 \
-    --num-inter-threads 2 \
-    --verbose \
-    --docker-image intel/intel-optimized-tensorflow:1.15.2 \
-    -- dataset=ml-20m clean=1 te=12
-```
-
-NCF will train 6+ epochs to SOTA. The tail of training log looks as below if trained to SOTA.
-HR: Hit Ratio (HR), should >= 0.635 if use 20M dataset
-NDCG: Normalized Discounted Cumulative Gain
-```
-I0122 12:00:14.874159 140303790921536 ncf_estimator_main.py:179] Iteration 6: HR = 0.6356, NDCG = 0.3787, Loss = 0.1567
-I0122 12:00:14.874222 140303790921536 model_helpers.py:53] Stop threshold of 0.635 was passed with metric value 0.635591685772.
-I0122 12:00:14.874658 140303790921536 mlperf_log.py:136] NCF_RAW_:::MLPv0.5.0 ncf 1579665614.874648094 (ncf_estimator_main.py:187) run_stop: {"success": true}
-NCF_RAW_:::MLPv0.5.0 ncf 1579665614.881932020 (ncf_estimator_main.py:193) run_final
-I0122 12:00:14.881944 140303790921536 mlperf_log.py:134] NCF_RAW_:::MLPv0.5.0 ncf 1579665614.881932020 (ncf_estimator_main.py:193) run_final
-```
-
-For training, suggest options are:
-* `--batch-size 98304`
-* `--precision fp32` or `--precision bfloat16`
-* `dataset=ml-20m` - suggest to use 20M dataset for training
-* `clean=1` - delete any files stored in `--checkpoint`. Disable this flag if want to reuse pre-trained model.
-* `te=12` - set the max epoch. NCF will train 6+ epochs to SOTA, this flag will stop training when reach specific epoch. Set it to 1 if only want to test performance
-
-```
-python launch_benchmark.py \
-    --checkpoint /home/<user>/ncf_fp32_pretrained_model \
-    --model-source-dir /home/<user>/tensorflow/models \
-    --model-name ncf \
-    --framework tensorflow \
-    --mode training \
-    --precision bfloat16 \
-    --batch-size 98304 \
-    --num-inter-threads 2 \
-    --verbose \
-    --docker-image intel/intel-optimized-tensorflow:1.15.2 \
-    -- dataset=ml-20m clean=1 te=12
-```
-
-NCF will train 6+ epochs to SOTA. The tail of training log looks as below if trained to SOTA.
-HR: Hit Ratio (HR), should >= 0.635 if use 20M dataset
-NDCG: Normalized Discounted Cumulative Gain
-```
-I0122 12:00:14.874159 140303790921536 ncf_estimator_main.py:179] Iteration 6: HR = 0.6356, NDCG = 0.3787, Loss = 0.1567
-I0122 12:00:14.874222 140303790921536 model_helpers.py:53] Stop threshold of 0.635 was passed with metric value 0.635591685772.
-I0122 12:00:14.874658 140303790921536 mlperf_log.py:136] NCF_RAW_:::MLPv0.5.0 ncf 1579665614.874648094 (ncf_estimator_main.py:187) run_stop: {"success": true}
-NCF_RAW_:::MLPv0.5.0 ncf 1579665614.881932020 (ncf_estimator_main.py:193) run_final
-I0122 12:00:14.881944 140303790921536 mlperf_log.py:134] NCF_RAW_:::MLPv0.5.0 ncf 1579665614.881932020 (ncf_estimator_main.py:193) run_final
-```
-
-6. To return to where you started from:
-```
-popd
-```
diff --git a/benchmarks/recommendation/tensorflow/ncf/__init__.py b/benchmarks/recommendation/tensorflow/ncf/__init__.py
deleted file mode 100644
index c4fdb7d61..000000000
--- a/benchmarks/recommendation/tensorflow/ncf/__init__.py
+++ /dev/null
@@ -1,19 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
diff --git a/benchmarks/recommendation/tensorflow/ncf/inference/__init__.py b/benchmarks/recommendation/tensorflow/ncf/inference/__init__.py
deleted file mode 100644
index c4fdb7d61..000000000
--- a/benchmarks/recommendation/tensorflow/ncf/inference/__init__.py
+++ /dev/null
@@ -1,19 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
diff --git a/benchmarks/recommendation/tensorflow/ncf/inference/fp32/Advanced.md b/benchmarks/recommendation/tensorflow/ncf/inference/fp32/Advanced.md
deleted file mode 100644
index a30c056d5..000000000
--- a/benchmarks/recommendation/tensorflow/ncf/inference/fp32/Advanced.md
+++ /dev/null
@@ -1,92 +0,0 @@
-<!--- 0. Title -->
-<!-- This document is auto-generated using markdown fragments and the model-builder -->
-<!-- To make changes to this doc, please change the fragments instead of modifying this doc directly -->
-# NCF FP32 inference - Advanced Instructions
-
-<!-- 10. Description -->
-This document has advanced instructions for running NCF FP32
-inference, which provides more control over the individual parameters that
-are used. For more information on using [`/benchmarks/launch_benchmark.py`](/benchmarks/launch_benchmark.py),
-see the [launch benchmark documentation](/docs/general/tensorflow/LaunchBenchmark.md).
-
-Prior to using these instructions, please follow the setup instructions from
-the model's [README](README.md) and/or the
-[AI Kit documentation](/docs/general/tensorflow/AIKit.md) to get your environment
-setup (if running on bare metal) and download the dataset, pretrained model, etc.
-If you are using AI Kit, please exclude the `--docker-image` flag from the
-commands below, since you will be running the the TensorFlow conda environment
-instead of docker.
-
-<!-- 55. Docker arg -->
-Any of the `launch_benchmark.py` commands below can be run on bare metal by
-removing the `--docker-image` arg. Ensure that you have all of the
-[required prerequisites installed](README.md#run-the-model) in your environment
-before running without the docker container.
-
-If you are new to docker and are running into issues with the container,
-see [this document](/docs/general/docker.md) for troubleshooting tips.
-
-<!-- 50. Launch benchmark instructions -->
-Once your environment is setup, navigate to the `benchmarks` directory of
-the model zoo and set environment variables pointing to the directory for the
-dataset, pretrained model, and an output directory where log
-files will be written.
-
-```
-# cd to the benchmarks directory in the model zoo
-cd benchmarks
-
-export DATASET_DIR=<path to the dataset>
-export OUTPUT_DIR=<directory where log files will be written>
-export PRETRAINED_MODEL=<path to the pretrained model>
-export TF_MODELS_DIR=<path to your clone of the TensorFlow models repo>
-```
-
-NCF FP32 inference can be run a few different modes:
-* For batch inference, `--batch-size 256`, `--socket-id 0`:
-  ```
-  python launch_benchmark.py \
-    --checkpoint $PRETRAINED_MODEL \
-    --model-source-dir $TF_MODELS_DIR \
-    --data-location ${DATASET_DIR} \
-    --model-name ncf \
-    --socket-id 0 \
-    --batch-size 256 \
-    --framework tensorflow \
-    --precision fp32 \
-    --mode inference \
-    --output-dir $OUTPUT_DIR \
-    --docker-image intel/intel-optimized-tensorflow:1.15.2
-  ```
-* For online inference, `--batch-size 1`, `--socket-id 0`:
-  ```
-  python launch_benchmark.py \
-    --checkpoint $PRETRAINED_MODEL \
-    --model-source-dir $TF_MODELS_DIR \
-    --data-location ${DATASET_DIR} \
-    --model-name ncf \
-    --socket-id 0 \
-    --batch-size 1 \
-    --framework tensorflow \
-    --precision fp32 \
-    --mode inference \
-    --output-dir $OUTPUT_DIR \
-    --docker-image intel/intel-optimized-tensorflow:1.15.2
-  ```
-* For Accuracy, `--batch-size 256`, `--socket-id 0`:
-  ```
-  python launch_benchmark.py \
-    --checkpoint $PRETRAINED_MODEL \
-    --model-source-dir $TF_MODELS_DIR \
-    --data-location ${DATASET_DIR} \
-    --model-name ncf \
-    --socket-id 0 \
-    --accuracy-only \
-    --batch-size 256 \
-    --framework tensorflow \
-    --precision fp32 \
-    --mode inference \
-    --output-dir $OUTPUT_DIR \
-    --docker-image intel/intel-optimized-tensorflow:1.15.2
-  ```
-
diff --git a/benchmarks/recommendation/tensorflow/ncf/inference/fp32/README.md b/benchmarks/recommendation/tensorflow/ncf/inference/fp32/README.md
deleted file mode 100644
index 6351b20e1..000000000
--- a/benchmarks/recommendation/tensorflow/ncf/inference/fp32/README.md
+++ /dev/null
@@ -1,116 +0,0 @@
-<!--- 0. Title -->
-# NCF FP32 inference
-
-<!-- 10. Description -->
-## Description
-
-This document has instructions for running Neural Collaborative Filtering (NCF)
-FP32 inference using Intel-optimized TensorFlow.
-
-<!--- 30. Datasets -->
-## Datasets
-
-Download [movielens 1M dataset](https://grouplens.org/datasets/movielens/1m/)
-```
-wget http://files.grouplens.org/datasets/movielens/ml-1m.zip
-unzip ml-1m.zip
-```
-Set the `DATASET_DIR` to point to this directory when running NCF.
-
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| [`fp32_online_inference.sh`](fp32_online_inference.sh) | Runs online inference (batch_size=1). |
-| [`fp32_batch_inference.sh`](fp32_batch_inference.sh) | Runs batch inference (batch_size=256). |
-| [`fp32_accuracy.sh`](fp32_accuracy.sh) | Measures the model accuracy (batch_size=256). |
-
-<!--- 50. AI Tools -->
-## Run the model
-
-Setup your environment using the instructions below, depending on if you are
-using [AI Tools](/docs/general/tensorflow/AITools.md):
-
-<table>
-  <tr>
-    <th>Setup using AI Tools</th>
-    <th>Setup without AI Tools</th>
-  </tr>
-  <tr>
-    <td>
-      <p>AI Tools does not currently support TF 1.15.2 models</p>
-    </td>
-    <td>
-      <p>To run without AI Tools you will need:</p>
-      <ul>
-        <li>Python 3
-        <li><a href="https://pypi.org/project/intel-tensorflow/1.15.2/">intel-tensorflow==1.15.2</a>
-        <li>numactl
-        <li>git
-        <li>google-api-python-client==1.6.7
-        <li>google-cloud-bigquery==0.31.0
-        <li>kaggle==1.3.9
-        <li>numpy==1.16.3
-        <li>oauth2client==4.1.2
-        <li>pandas
-        <li>psutil>=5.6.7
-        <li>py-cpuinfo==3.3.0
-        <li>tar
-        <li>typing
-        <li>wget
-        <li>A clone of the AI Reference Models repo<br />
-        <pre>git clone https://github.com/IntelAI/models.git</pre>
-      </ul>
-    </td>
-  </tr>
-</table>
-
-
-Running NCF also requires a clone of the
-[TensorFlow models repository](https://github.com/tensorflow/models) with
-at the `v1.11` tag. Set the `TF_MODELS_DIR` env var to the path of your clone.
-```
-git clone https://github.com/tensorflow/models.git tf_models
-cd tf_models
-git checkout v1.11
-export TF_MODELS_DIR=$(pwd)
-cd ..
-```
-
-Download and extract the pretrained model and set the path to the
-`PRETRAINED_MODEL` env var.
-```
-wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_5/ncf_fp32_pretrained_model.tar.gz
-tar -xzvf ncf_fp32_pretrained_model.tar.gz
-export PRETRAINED_MODEL=$(pwd)/ncf_trained_movielens_1m
-```
-
-After your environment is setup, set environment variables to the `DATASET_DIR`
-and an `OUTPUT_DIR` where log files will be written. Ensure that you already have
-the `TF_MODELS_DIR` and `PRETRAINED_MODEL` paths set from the previous commands.
-Once the environment variables are all set, you can run a
-[quickstart script](#quick-start-scripts).
-```
-# cd to your AI Reference Models directory
-cd models
-
-export DATASET_DIR=<path to the dataset>
-export OUTPUT_DIR=<path to the directory where log files will be written>
-export TF_MODELS_DIR=<path to the TensorFlow models directory tf_models>
-export PRETRAINED_MODEL=<path to the pretrained model>
-# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
-export BATCH_SIZE=<customized batch size value>
-
-./quickstart/recommendation/tensorflow/ncf/inference/cpu/fp32/<script name>.sh
-```
-
-<!--- 90. Resource Links-->
-## Additional Resources
-
-* To run more advanced use cases, see the instructions [here](Advanced.md)
-  for calling the `launch_benchmark.py` script directly.
-* To run the model using docker, please see the [oneContainer](https://www.intel.com/content/www/us/en/developer/tools/software-catalog/containers.html)
-  workload container:<br />
-  [https://www.intel.com/content/www/us/en/developer/articles/containers/ncf-fp32-inference-tensorflow-container.html](https://www.intel.com/content/www/us/en/developer/articles/containers/ncf-fp32-inference-tensorflow-container.html).
-
diff --git a/benchmarks/recommendation/tensorflow/ncf/inference/fp32/__init__.py b/benchmarks/recommendation/tensorflow/ncf/inference/fp32/__init__.py
deleted file mode 100644
index c4fdb7d61..000000000
--- a/benchmarks/recommendation/tensorflow/ncf/inference/fp32/__init__.py
+++ /dev/null
@@ -1,19 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
diff --git a/benchmarks/recommendation/tensorflow/ncf/inference/fp32/model_init.py b/benchmarks/recommendation/tensorflow/ncf/inference/fp32/model_init.py
deleted file mode 100644
index 7326ec540..000000000
--- a/benchmarks/recommendation/tensorflow/ncf/inference/fp32/model_init.py
+++ /dev/null
@@ -1,78 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-from common.base_model_init import BaseModelInitializer
-from common.base_model_init import set_env_var
-
-import os
-
-
-class ModelInitializer(BaseModelInitializer):
-    """initialize mode and run benchmark"""
-
-    def __init__(self, args, custom_args=[], platform_util=None):
-        super(ModelInitializer, self).__init__(args, custom_args, platform_util)
-
-        self.benchmark_command = ""
-
-        # use default batch size if -1
-        if self.args.batch_size == -1:
-            self.args.batch_size = 256
-
-        # Set KMP env vars, if they haven't already been set
-        config_file_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), "config.json")
-        self.set_kmp_vars(config_file_path)
-
-        # set num_inter_threads and num_intra_threads
-        self.set_num_inter_intra_threads()
-
-        benchmark_script = os.path.join(
-            self.args.intelai_models, self.args.mode, self.args.precision,
-            "ncf_main.py")
-
-        self.benchmark_command = self.get_command_prefix(args.socket_id) + \
-            self.python_exe + " " + benchmark_script
-
-        set_env_var("OMP_NUM_THREADS", self.args.num_intra_threads)
-
-        self.benchmark_command = self.benchmark_command + \
-            " --data_dir=" + str(args.data_location) + \
-            " --model_dir=" + str(args.checkpoint) + \
-            " --intra_op_parallelism_threads=" + str(
-                self.args.num_intra_threads) + \
-            " --inter_op_parallelism_threads=" + str(
-                self.args.num_inter_threads) + \
-            " --batch_size=" + str(self.args.batch_size) + \
-            " --inference_only"
-
-        if self.args.benchmark_only:
-            self.benchmark_command = self.benchmark_command + \
-                " --benchmark_only"
-
-        if self.args.accuracy_only:
-            self.benchmark_command = self.benchmark_command + \
-                " --accuracy_only"
-
-    def run(self):
-        if self.benchmark_command:
-            self.run_command(self.benchmark_command)
diff --git a/benchmarks/recommendation/tensorflow/ncf/inference/requirements.txt b/benchmarks/recommendation/tensorflow/ncf/inference/requirements.txt
deleted file mode 100644
index bdb967deb..000000000
--- a/benchmarks/recommendation/tensorflow/ncf/inference/requirements.txt
+++ /dev/null
@@ -1,11 +0,0 @@
-google-api-python-client==1.6.7
-google-cloud-bigquery==0.31.0
-kaggle==1.3.9
-numpy>=1.16.3
-oauth2client==4.1.2
-pandas
-psutil>=5.6.7
-py-cpuinfo==3.3.0
-typing
-
-rsa>=4.7 # not directly required, pinned by Snyk to avoid a vulnerability
diff --git a/benchmarks/recommendation/tensorflow/ncf/training/__init__.py b/benchmarks/recommendation/tensorflow/ncf/training/__init__.py
deleted file mode 100644
index cf793ec6a..000000000
--- a/benchmarks/recommendation/tensorflow/ncf/training/__init__.py
+++ /dev/null
@@ -1,19 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-# SPDX-License-Identifier: EPL-2.0
-#
diff --git a/benchmarks/recommendation/tensorflow/ncf/training/bfloat16/__init__.py b/benchmarks/recommendation/tensorflow/ncf/training/bfloat16/__init__.py
deleted file mode 100644
index cf793ec6a..000000000
--- a/benchmarks/recommendation/tensorflow/ncf/training/bfloat16/__init__.py
+++ /dev/null
@@ -1,19 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-# SPDX-License-Identifier: EPL-2.0
-#
diff --git a/benchmarks/recommendation/tensorflow/ncf/training/bfloat16/model_init.py b/benchmarks/recommendation/tensorflow/ncf/training/bfloat16/model_init.py
deleted file mode 100644
index 7543ae6a8..000000000
--- a/benchmarks/recommendation/tensorflow/ncf/training/bfloat16/model_init.py
+++ /dev/null
@@ -1,87 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-# SPDX-License-Identifier: EPL-2.0
-#
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-from common.base_model_init import BaseModelInitializer
-from common.base_model_init import set_env_var
-
-import argparse
-import os
-
-
-class ModelInitializer(BaseModelInitializer):
-    """initialize mode and run benchmark"""
-
-    def __init__(self, args, custom_args=[], platform_util=None):
-        super(ModelInitializer, self).__init__(args, custom_args, platform_util)
-
-        arg_parser = argparse.ArgumentParser(description='Parse additional args')
-        arg_parser.add_argument("--dataset", type=str, default="ml-1m",
-                                choices=["ml-1m", "ml-20m"],
-                                help=("Dataset to be trained and evaluated."),
-                                dest="dataset")
-        self.additional_args, unknown_args = arg_parser.parse_known_args(custom_args)
-
-        self.benchmark_command = ""
-
-        # Add custom parameters for specific dataset
-        if self.additional_args.dataset == "ml-20m":
-            self.additional_args.dataset = " --dataset=ml-20m " + \
-                " --layers=256,256,128,64 --num_factors=64" + \
-                " --eval_batch_size 160000" + \
-                " --learning_rate 0.003821" + \
-                " --beta1 0.783529 --beta2 0.909003 --epsilon 1.45439e-07" + \
-                " --hr_threshold 0.635" + \
-                " --ml_perf"
-
-        # use default batch size if -1
-        if self.args.batch_size == -1:
-            self.args.batch_size = 98304
-
-        # Set KMP env vars, if they haven't already been set
-        config_file_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), "config.json")
-        self.set_kmp_vars(config_file_path)
-
-        # set num_inter_threads and num_intra_threads
-        self.set_num_inter_intra_threads()
-
-        benchmark_script = os.path.join(
-            self.args.intelai_models, self.args.mode, "ncf_estimator_main.py")
-
-        self.benchmark_command = self.get_command_prefix(args.socket_id) + \
-            self.python_exe + " " + benchmark_script
-
-        set_env_var("OMP_NUM_THREADS", self.args.num_intra_threads)
-        set_env_var("TF_NUM_INTRAOP_THREADS", self.args.num_intra_threads)
-        set_env_var("TF_NUM_INTEROP_THREADS", self.args.num_inter_threads)
-
-        self.benchmark_command = self.benchmark_command + \
-            " --use_bfloat16" + \
-            " -dd=" + str(args.data_location) + \
-            " -md=" + str(args.checkpoint) + \
-            " -bs=" + str(self.args.batch_size) + \
-            " -hk=examplespersecondhook" + \
-            self.additional_args.dataset + ' ' + ' '.join(unknown_args)
-
-    def run(self):
-        if self.benchmark_command:
-            self.run_command(self.benchmark_command)
diff --git a/benchmarks/recommendation/tensorflow/ncf/training/fp32/__init__.py b/benchmarks/recommendation/tensorflow/ncf/training/fp32/__init__.py
deleted file mode 100644
index cf793ec6a..000000000
--- a/benchmarks/recommendation/tensorflow/ncf/training/fp32/__init__.py
+++ /dev/null
@@ -1,19 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-# SPDX-License-Identifier: EPL-2.0
-#
diff --git a/benchmarks/recommendation/tensorflow/ncf/training/fp32/config.json b/benchmarks/recommendation/tensorflow/ncf/training/fp32/config.json
deleted file mode 100644
index 273b45b40..000000000
--- a/benchmarks/recommendation/tensorflow/ncf/training/fp32/config.json
+++ /dev/null
@@ -1,7 +0,0 @@
-{
-    "optimization_parameters": {
-        "KMP_AFFINITY": "granularity=fine,verbose,compact,1,0",
-        "KMP_BLOCKTIME": 1,
-        "KMP_SETTINGS": 1
-    }
-}
diff --git a/benchmarks/recommendation/tensorflow/ncf/training/fp32/model_init.py b/benchmarks/recommendation/tensorflow/ncf/training/fp32/model_init.py
deleted file mode 100644
index ffe45759c..000000000
--- a/benchmarks/recommendation/tensorflow/ncf/training/fp32/model_init.py
+++ /dev/null
@@ -1,86 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-# SPDX-License-Identifier: EPL-2.0
-#
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-from common.base_model_init import BaseModelInitializer
-from common.base_model_init import set_env_var
-
-import argparse
-import os
-
-
-class ModelInitializer(BaseModelInitializer):
-    """initialize mode and run benchmark"""
-
-    def __init__(self, args, custom_args=[], platform_util=None):
-        super(ModelInitializer, self).__init__(args, custom_args, platform_util)
-
-        arg_parser = argparse.ArgumentParser(description='Parse additional args')
-        arg_parser.add_argument("--dataset", type=str, default="ml-1m",
-                                choices=["ml-1m", "ml-20m"],
-                                help=("Dataset to be trained and evaluated."),
-                                dest="dataset")
-        self.additional_args, unknown_args = arg_parser.parse_known_args(custom_args)
-
-        self.benchmark_command = ""
-
-        # Add custom parameters for specific dataset
-        if self.additional_args.dataset == "ml-20m":
-            self.additional_args.dataset = " --dataset=ml-20m " + \
-                " --layers=256,256,128,64 --num_factors=64" + \
-                " --eval_batch_size 160000" + \
-                " --learning_rate 0.003821" + \
-                " --beta1 0.783529 --beta2 0.909003 --epsilon 1.45439e-07" + \
-                " --hr_threshold 0.635" + \
-                " --ml_perf"
-
-        # use default batch size if -1
-        if self.args.batch_size == -1:
-            self.args.batch_size = 98304
-
-        # Set KMP env vars, if they haven't already been set
-        config_file_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), "config.json")
-        self.set_kmp_vars(config_file_path)
-
-        # set num_inter_threads and num_intra_threads
-        self.set_num_inter_intra_threads()
-
-        benchmark_script = os.path.join(
-            self.args.intelai_models, self.args.mode, "ncf_estimator_main.py")
-
-        self.benchmark_command = self.get_command_prefix(args.socket_id) + \
-            self.python_exe + " " + benchmark_script
-
-        set_env_var("OMP_NUM_THREADS", self.args.num_intra_threads)
-        set_env_var("TF_NUM_INTRAOP_THREADS", self.args.num_intra_threads)
-        set_env_var("TF_NUM_INTEROP_THREADS", self.args.num_inter_threads)
-
-        self.benchmark_command = self.benchmark_command + \
-            " -dd=" + str(args.data_location) + \
-            " -md=" + str(args.checkpoint) + \
-            " -bs=" + str(self.args.batch_size) + \
-            " -hk=examplespersecondhook" + \
-            self.additional_args.dataset + ' ' + ' '.join(unknown_args)
-
-    def run(self):
-        if self.benchmark_command:
-            self.run_command(self.benchmark_command)
diff --git a/benchmarks/recommendation/tensorflow/wide_deep_large_ds/training/README.md b/benchmarks/recommendation/tensorflow/wide_deep_large_ds/training/README.md
deleted file mode 100644
index e5495c37f..000000000
--- a/benchmarks/recommendation/tensorflow/wide_deep_large_ds/training/README.md
+++ /dev/null
@@ -1,100 +0,0 @@
-<!--- 0. Title -->
-# Wide and Deep Large Dataset training
-
-<!-- 10. Description -->
-
-This document has instructions for training [Wide and Deep](https://arxiv.org/pdf/1606.07792.pdf)
-using a large dataset using Intel-optimized TensorFlow.
-
-
-<!--- 30. Datasets -->
-## Dataset
-
-The Large [Kaggle Display Advertising Challenge Dataset](https://www.kaggle.com/c/criteo-display-ad-challenge/data)
-will be used for training Wide and Deep. The [data](https://www.kaggle.com/c/criteo-display-ad-challenge/data) is from
-[Criteo](https://www.criteo.com) and has a field indicating if an ad was
-clicked (1) or not (0), along with integer and categorical features.
-
-Download the Large Kaggle Display Advertising Challenge Dataset from [Criteo Labs](http://labs.criteo.com/2014/02/kaggle-display-advertising-challenge-dataset/) in `$DATASET_DIR`.
-If the evaluation/train dataset were not available in the above link, it can be downloaded as follow:
-   ```
-    export DATASET_DIR=<location where dataset files will be saved>
-    mkdir $DATASET_DIR && cd $DATASET_DIR
-    wget https://storage.googleapis.com/dataset-uploader/criteo-kaggle/large_version/eval.csv
-    wget https://storage.googleapis.com/dataset-uploader/criteo-kaggle/large_version/train.csv
-   ```
-The `DATASET_DIR` environment variable will be used as the dataset directory when running [quickstart scripts](#quick-start-scripts).
-
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| [`training_check_accuracy.sh`](/quickstart/recommendation/tensorflow/wide_deep_large_ds/training/cpu/training_check_accuracy.sh) | Trains the model for a specified number of steps (default is 500) and then compare the accuracy against the accuracy specified in the `TARGET_ACCURACY` env var (ex: `export TARGET_ACCURACY=0.75`). If the accuracy is not met, then script exits with error code 1. The `CHECKPOINT_DIR` environment variable can optionally be defined to start training based on previous set of checkpoints. |
-| [`training.sh`](/quickstart/recommendation/tensorflow/wide_deep_large_ds/training/cpu/training.sh) | Trains the model for 10 epochs. The `CHECKPOINT_DIR` environment variable can optionally be defined to start training based on previous set of checkpoints. |
-| [`training_demo.sh`](/quickstart/recommendation/tensorflow/wide_deep_large_ds/training/cpu/training_demo.sh) | A short demo run that trains the model for 100 steps. |
-
-<!--- 50. AI Tools -->
-## Run the model
-
-Setup your environment using the instructions below, depending on if you are
-using [AI Tools](/docs/general/tensorflow/AITools.md):
-
-<table>
-  <tr>
-    <th>Setup using AI Tools</th>
-    <th>Setup without AI Tools</th>
-  </tr>
-  <tr>
-    <td>
-      <p>To run using AI Tools you will need:</p>
-      <ul>
-        <li>numactl
-        <li>wget
-        <li>Activate the `tensorflow` conda environment
-        <pre>conda activate tensorflow</pre>
-      </ul>
-    </td>
-    <td>
-      <p>To run without AI Tools you will need:</p>
-      <ul>
-        <li>Python 3
-        <li>intel-tensorflow>=2.5.0
-        <li>numactl
-        <li>git
-        <li>wget
-        <li>A clone of the AI Reference Models repo<br />
-        <pre>git clone https://github.com/IntelAI/models.git</pre>
-      </ul>
-    </td>
-  </tr>
-</table>
-
-After the setup is complete, set environment variables for the path to your
-dataset directory and an output directory where logs will be written. You can
-optionally provide a directory where checkpoint files will be read and
-written. Navigate to your AI Reference Models directory, then select a
-[quickstart script](#quick-start-scripts) to run. Note that some quickstart
-scripts might use other environment variables in addition to the ones below,
-like `STEPS` and `TARGET_ACCURACY` for the `fp32_training_check_accuracy.sh` script.
-```
-# cd to your AI Reference Models directory
-cd models
-
-export DATASET_DIR=<path to the dataset directory>
-export PRECISION=fp32
-export OUTPUT_DIR=<path to the directory where the logs and the saved model will be written>
-export CHECKPOINT_DIR=<Optional directory where checkpoint files will be read and written>
-# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
-export BATCH_SIZE=<customized batch size value>
-
-./quickstart/recommendation/tensorflow/wide_deep_large_ds/training/cpu/<script name>.sh
-```
-
-<!--- 90. Resource Links-->
-## Additional Resources
-
-* To run more advanced use cases, see the instructions for the available precisions [FP32](fp32/Advanced.md) [<int8 precision>](<int8 advanced readme link>) [<bfloat16 precision>](<bfloat16 advanced readme link>) for calling the `launch_benchmark.py` script directly.
-* To run the model using docker, please see the [Intel® Developer Catalog](https://www.intel.com/content/www/us/en/developer/tools/software-catalog/containers.html)
-  workload container:<br />
-  [https://www.intel.com/content/www/us/en/developer/articles/containers/wide-deep-large-dataset-fp32-training-tensorflow-container.html](https://www.intel.com/content/www/us/en/developer/articles/containers/wide-deep-large-dataset-fp32-training-tensorflow-container.html).
diff --git a/benchmarks/recommendation/tensorflow/wide_deep_large_ds/training/__init__.py b/benchmarks/recommendation/tensorflow/wide_deep_large_ds/training/__init__.py
deleted file mode 100644
index cf793ec6a..000000000
--- a/benchmarks/recommendation/tensorflow/wide_deep_large_ds/training/__init__.py
+++ /dev/null
@@ -1,19 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-# SPDX-License-Identifier: EPL-2.0
-#
diff --git a/benchmarks/recommendation/tensorflow/wide_deep_large_ds/training/fp32/Advanced.md b/benchmarks/recommendation/tensorflow/wide_deep_large_ds/training/fp32/Advanced.md
deleted file mode 100644
index ce929f241..000000000
--- a/benchmarks/recommendation/tensorflow/wide_deep_large_ds/training/fp32/Advanced.md
+++ /dev/null
@@ -1,70 +0,0 @@
-<!--- 0. Title -->
-<!-- This document is auto-generated using markdown fragments and the model-builder -->
-<!-- To make changes to this doc, please change the fragments instead of modifying this doc directly -->
-# Wide and Deep using a large dataset FP32 training - Advanced Instructions
-
-<!-- 10. Description -->
-This document has advanced instructions for running Wide and Deep using a large dataset FP32
-training, which provides more control over the individual parameters that
-are used. For more information on using [`/benchmarks/launch_benchmark.py`](/benchmarks/launch_benchmark.py),
-see the [launch benchmark documentation](/docs/general/tensorflow/LaunchBenchmark.md).
-
-Prior to using these instructions, please follow the setup instructions from
-the model's [README](README.md) and/or the
-[AI Kit documentation](/docs/general/tensorflow/AIKit.md) to get your environment
-setup (if running on bare metal) and download the dataset, pretrained model, etc.
-If you are using AI Kit, please exclude the `--docker-image` flag from the
-commands below, since you will be running the the TensorFlow conda environment
-instead of docker.
-
-<!-- 55. Docker arg -->
-Any of the `launch_benchmark.py` commands below can be run on bare metal by
-removing the `--docker-image` arg. Ensure that you have all of the
-[required prerequisites installed](README.md#run-the-model) in your environment
-before running without the docker container.
-
-If you are new to docker and are running into issues with the container,
-see [this document](/docs/general/docker.md) for troubleshooting tips.
-
-<!-- 50. Launch benchmark instructions -->
-Once your environment is setup, navigate to the `benchmarks` directory of
-the model zoo and set environment variables pointing to the directory for the
-dataset (where train.csv and eval.csv are located), an output directory where log
-files will be written, and optionally a directory where checkpoint files can
-be read and written from.
-
-```
-# cd to the benchmarks directory in the model zoo
-cd benchmarks
-
-export DATASET_DIR=<path to the dataset directory>
-export OUTPUT_DIR=<directory where the logs and the saved model will be written>
-export CHECKPOINT_DIR=<Optional directory where checkpoint files will be read and written>
-```
-
-Train the model (The model will be trained for 10 epochs if -- steps is not specified)
-```
-python launch_benchmark.py \
-  --model-name wide_deep_large_ds \
-  --precision fp32 \
-  --mode training  \
-  --framework tensorflow \
-  --batch-size 512 \
-  --data-location ${DATASET_DIR} \
-  --checkpoint ${CHECKPOINT_DIR} \
-  --output-dir ${OUTPUT_DIR} \
-  --docker-image intel/intel-optimized-tensorflow:latest
-```
-Once the training completes successfully the path of checkpoint files and saved_model.pb will be printed as shown below
-```
-INFO:tensorflow:SavedModel written to: $OUTPUT_DIR/temp-1602670603/saved_model.pb
-Using TensorFlow version 2.4.0
-Begin training and evaluation
-Saving model checkpoints to $CHECKPOINT_DIR
-****Computing statistics of train dataset*****
-estimator built
-fit done
-evaluate done
-Model exported to $OUTPUT_DIR
-```
-
diff --git a/benchmarks/recommendation/tensorflow/wide_deep_large_ds/training/fp32/__init__.py b/benchmarks/recommendation/tensorflow/wide_deep_large_ds/training/fp32/__init__.py
deleted file mode 100644
index c4fdb7d61..000000000
--- a/benchmarks/recommendation/tensorflow/wide_deep_large_ds/training/fp32/__init__.py
+++ /dev/null
@@ -1,19 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
diff --git a/benchmarks/recommendation/tensorflow/wide_deep_large_ds/training/fp32/model_init.py b/benchmarks/recommendation/tensorflow/wide_deep_large_ds/training/fp32/model_init.py
deleted file mode 100644
index a28630f78..000000000
--- a/benchmarks/recommendation/tensorflow/wide_deep_large_ds/training/fp32/model_init.py
+++ /dev/null
@@ -1,55 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-import os
-import argparse
-from common.base_model_init import BaseModelInitializer
-
-
-class ModelInitializer(BaseModelInitializer):
-    """Model initializer for Wide and deep large dataset FP32 inference"""
-
-    def __init__(self, args, custom_args=[], platform_util=None):
-        super(ModelInitializer, self).__init__(args, custom_args, platform_util)
-
-    def parse_args(self):
-        parser = argparse.ArgumentParser()
-        parser.add_argument("--steps", dest='steps',
-                            type=int, default=0,
-                            help="number of train steps")
-        self.args = parser.parse_args(self.custom_args,
-                                      namespace=self.args)
-
-    def run_benchmark(self):
-        benchmark_script = os.path.join(self.args.intelai_models,
-                                        self.args.mode, "train.py")
-        script_args_list = ["batch_size", "data_location", "steps", "output_dir", "checkpoint"]
-        command_prefix = self.get_command_prefix(-1)
-        cmd_prefix = command_prefix + self.python_exe + " " + benchmark_script
-        cmd = self.add_args_to_command(cmd_prefix, script_args_list)
-        self.run_command(cmd)
-
-    def run(self):
-        # Parse custom arguments and append to self.args
-        self.parse_args()
-        self.run_benchmark()
diff --git a/benchmarks/text_to_speech/__init__.py b/benchmarks/text_to_speech/__init__.py
deleted file mode 100644
index cf793ec6a..000000000
--- a/benchmarks/text_to_speech/__init__.py
+++ /dev/null
@@ -1,19 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-# SPDX-License-Identifier: EPL-2.0
-#
diff --git a/benchmarks/text_to_speech/tensorflow/__init__.py b/benchmarks/text_to_speech/tensorflow/__init__.py
deleted file mode 100644
index cf793ec6a..000000000
--- a/benchmarks/text_to_speech/tensorflow/__init__.py
+++ /dev/null
@@ -1,19 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-# SPDX-License-Identifier: EPL-2.0
-#
diff --git a/benchmarks/text_to_speech/tensorflow/wavenet/README.md b/benchmarks/text_to_speech/tensorflow/wavenet/README.md
deleted file mode 100644
index 83b9a6253..000000000
--- a/benchmarks/text_to_speech/tensorflow/wavenet/README.md
+++ /dev/null
@@ -1,4 +0,0 @@
-# WaveNet
-
-The following document has instructions for running WaveNet:
-* [FP32 inference](/benchmarks/text_to_speech/tensorflow/wavenet/inference/fp32/README.md)
diff --git a/benchmarks/text_to_speech/tensorflow/wavenet/__init__.py b/benchmarks/text_to_speech/tensorflow/wavenet/__init__.py
deleted file mode 100644
index cf793ec6a..000000000
--- a/benchmarks/text_to_speech/tensorflow/wavenet/__init__.py
+++ /dev/null
@@ -1,19 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-# SPDX-License-Identifier: EPL-2.0
-#
diff --git a/benchmarks/text_to_speech/tensorflow/wavenet/inference/__init__.py b/benchmarks/text_to_speech/tensorflow/wavenet/inference/__init__.py
deleted file mode 100644
index cf793ec6a..000000000
--- a/benchmarks/text_to_speech/tensorflow/wavenet/inference/__init__.py
+++ /dev/null
@@ -1,19 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-# SPDX-License-Identifier: EPL-2.0
-#
diff --git a/benchmarks/text_to_speech/tensorflow/wavenet/inference/fp32/Advanced.md b/benchmarks/text_to_speech/tensorflow/wavenet/inference/fp32/Advanced.md
deleted file mode 100644
index a30461e8b..000000000
--- a/benchmarks/text_to_speech/tensorflow/wavenet/inference/fp32/Advanced.md
+++ /dev/null
@@ -1,85 +0,0 @@
-<!--- 0. Title -->
-<!-- This document is auto-generated using markdown fragments and the model-builder -->
-<!-- To make changes to this doc, please change the fragments instead of modifying this doc directly -->
-# WaveNet FP32 inference - Advanced Instructions
-
-<!-- 10. Description -->
-This document has advanced instructions for running WaveNet FP32
-inference, which provides more control over the individual parameters that
-are used. For more information on using [`/benchmarks/launch_benchmark.py`](/benchmarks/launch_benchmark.py),
-see the [launch benchmark documentation](/docs/general/tensorflow/LaunchBenchmark.md).
-
-Prior to using these instructions, please follow the setup instructions from
-the model's [README](README.md) and/or the
-[AI Kit documentation](/docs/general/tensorflow/AIKit.md) to get your environment
-setup (if running on bare metal) and download the dataset, pretrained model, etc.
-If you are using AI Kit, please exclude the `--docker-image` flag from the
-commands below, since you will be running the the TensorFlow conda environment
-instead of docker.
-
-<!-- 55. Docker arg -->
-Any of the `launch_benchmark.py` commands below can be run on bare metal by
-removing the `--docker-image` arg. Ensure that you have all of the
-[required prerequisites installed](README.md#run-the-model) in your environment
-before running without the docker container.
-
-If you are new to docker and are running into issues with the container,
-see [this document](/docs/general/docker.md) for troubleshooting tips.
-
-<!-- 50. Launch benchmark instructions -->
-Once your environment is setup, navigate to the `benchmarks` directory of
-the model zoo and set environment variables pointing to the directory for the
-dataset, pretrained model, and an output directory where log
-files will be written.
-
-```
-# cd to the benchmarks directory in the model zoo
-cd benchmarks
-
-export OUTPUT_DIR=<directory where log files will be written>
-export PRETRAINED_MODEL=<path to the pretrained model checkpoints>
-export TF_WAVENET_DIR=<path to your clone of the TensorFlow WaveNet repo>
-```
-
-Start a model run by executing the launch script and passing args
-specifying that we are running wavenet fp32 inference using TensorFlow,
-along with a docker image that includes Intel Optimizations for TensorFlow
-and the path to your clone of the WaveNet repo and the checkpoint
-files that were downloaded.  We are also passing a couple of extra model args
-for wavenet: the name of the checkpoint to use and the sample number.
-
-```
-python launch_benchmark.py \
-    --precision fp32 \
-    --model-name wavenet \
-    --mode inference \
-    --framework tensorflow \
-    --socket-id 0 \
-    --num-cores 1 \
-    --docker-image intel/intel-optimized-tensorflow:1.15.2 \
-    --model-source-dir ${TF_WAVENET_DIR} \
-    --checkpoint ${PRETRAINED_MODEL} \
-    --output-dir ${OUTPUT_DIR} \
-    -- checkpoint_name=model.ckpt-99 sample=8510
-```
-
-The logs are displayed in the console output as well as saved to a
-file in the value of `${OUTPUT_DIR}`.
-
-The tail of the log should look something like this:
-```
-Time per 500 Samples: ... sec
-Samples / sec: ...
-msec / sample: ...
-Sample: 8000
-Time per 500 Samples: ... sec
-Samples / sec: ...
-msec / sample: ...
-Sample: 8500
-
-Average Throughput of whole run: Samples / sec: ...
-Average Latency of whole run: msec / sample: ...
-Finished generating. The result can be viewed in TensorBoard.
-Log file location: ${OUTPUT_DIR}/benchmark_wavenet_inference_fp32_20210601_143852.log
-```
-
diff --git a/benchmarks/text_to_speech/tensorflow/wavenet/inference/fp32/README.md b/benchmarks/text_to_speech/tensorflow/wavenet/inference/fp32/README.md
deleted file mode 100644
index 0fc54f963..000000000
--- a/benchmarks/text_to_speech/tensorflow/wavenet/inference/fp32/README.md
+++ /dev/null
@@ -1,94 +0,0 @@
-<!--- 0. Title -->
-# WaveNet FP32 inference
-
-<!-- 10. Description -->
-## Description
-
-This document has instructions for running WaveNet FP32 inference using
-Intel-optimized TensorFlow.
-
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| [fp32_inference.sh](/quickstart/text_to_speech/tensorflow/wavenet/inference/cpu/fp32/fp32_inference.sh) | Runs inference with a pretrained model |
-
-<!--- 50. AI Tools -->
-## Run the model
-
-Setup your environment using the instructions below, depending on if you are
-using [AI Tools](/docs/general/tensorflow/AITools.md):
-
-<table>
-  <tr>
-    <th>Setup using AI Tools</th>
-    <th>Setup without AI Tools</th>
-  </tr>
-  <tr>
-    <td>
-      <p>AI Tools does not currently support TF 1.15.2 models</p>
-    </td>
-    <td>
-      <p>To run without AI Tools you will need:</p>
-      <ul>
-        <li>Python 3.6 or 3.7
-        <li>git
-        <li>numactl
-        <li>wget
-        <li><a href="https://pypi.org/project/intel-tensorflow/1.15.2/">intel-tensorflow==1.15.2</a>
-        <li>librosa==0.5
-        <li>A clone of the AI Reference Models repo<br />
-        <pre>git clone https://github.com/IntelAI/models.git</pre>
-      </ul>
-    </td>
-  </tr>
-</table>
-
-In addition to the requirements specified above, you will also need a clone
-of the [tensorflow-wavenet](https://github.com/ibab/tensorflow-wavenet)
-repo with pull request #352 for the CPU optimizations.  The path to
-the cloned repo needs to be set to the `TF_WAVENET_DIR` environment variable
-before running a quickstart script.
-```
-git clone https://github.com/ibab/tensorflow-wavenet.git
-cd tensorflow-wavenet/
-
-git fetch origin pull/352/head:cpu_optimized
-git checkout cpu_optimized
-export TF_WAVENET_DIR=$(pwd)
-
-cd ..
-```
-
-Download and extract the pretrained model checkpoint files.
-```
-wget https://storage.googleapis.com/intel-optimized-tensorflow/models/wavenet_fp32_pretrained_model.tar.gz
-tar -xvf wavenet_fp32_pretrained_model.tar.gz
-export PRETRAINED_MODEL=$(pwd)/wavenet_checkpoints
-```
-
-Navigate to the your AI Reference Models directory, then set an environment variable
-for an `OUTPUT_DIR` directory where logs will be written and ensure that you
-have the `TF_WAVENET_DIR` and `PRETRAINED_MODEL` variables set. Once this
-setup is done, you can run the `fp32_inference.sh` [quickstart script](#quick-start-scripts).
-```
-# cd to your AI Reference Models directory
-cd models
-
-export OUTPUT_DIR=<path to the directory where log files will be written>
-export TF_WAVENET_DIR=<tensorflow-wavenet directory>
-export PRETRAINED_MODEL=<path to the downloaded and extracted checkpoints>
-
-./quickstart/text_to_speech/tensorflow/wavenet/inference/cpu/fp32/fp32_inference.sh
-```
-
-<!--- 90. Resource Links-->
-## Additional Resources
-
-* To run more advanced use cases, see the instructions [here](Advanced.md)
-  for calling the `launch_benchmark.py` script directly.
-* To run the model using docker, please see the [oneContainer](https://www.intel.com/content/www/us/en/developer/tools/software-catalog/containers.html)
-  workload container:<br />
-  [https://www.intel.com/content/www/us/en/developer/articles/containers/wavenet-fp32-inference-tensorflow-container.html](https://www.intel.com/content/www/us/en/developer/articles/containers/wavenet-fp32-inference-tensorflow-container.html).
-
diff --git a/benchmarks/text_to_speech/tensorflow/wavenet/inference/fp32/__init__.py b/benchmarks/text_to_speech/tensorflow/wavenet/inference/fp32/__init__.py
deleted file mode 100644
index cf793ec6a..000000000
--- a/benchmarks/text_to_speech/tensorflow/wavenet/inference/fp32/__init__.py
+++ /dev/null
@@ -1,19 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-# SPDX-License-Identifier: EPL-2.0
-#
diff --git a/benchmarks/text_to_speech/tensorflow/wavenet/inference/fp32/config.json b/benchmarks/text_to_speech/tensorflow/wavenet/inference/fp32/config.json
deleted file mode 100644
index f0b327528..000000000
--- a/benchmarks/text_to_speech/tensorflow/wavenet/inference/fp32/config.json
+++ /dev/null
@@ -1,6 +0,0 @@
-{
-    "optimization_parameters": {
-        "KMP_AFFINITY": "granularity=fine,verbose,compact,1,0",
-        "KMP_BLOCKTIME": 1
-    }
-}
diff --git a/benchmarks/text_to_speech/tensorflow/wavenet/inference/fp32/model_init.py b/benchmarks/text_to_speech/tensorflow/wavenet/inference/fp32/model_init.py
deleted file mode 100644
index 1756e33ae..000000000
--- a/benchmarks/text_to_speech/tensorflow/wavenet/inference/fp32/model_init.py
+++ /dev/null
@@ -1,83 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-# SPDX-License-Identifier: EPL-2.0
-#
-
-import argparse
-import os
-
-from common.base_model_init import BaseModelInitializer
-from common.base_model_init import set_env_var
-
-
-class ModelInitializer(BaseModelInitializer):
-    def __init__(self, args, custom_args, platform_util):
-        super(ModelInitializer, self).__init__(args, custom_args, platform_util)
-
-        self.command = ""
-        command_prefix = "{} generate.py".format(self.python_exe)
-
-        # Set KMP env vars, if they haven't already been set
-        config_file_path = os.path.join(os.path.dirname(os.path.realpath(__file__)), "config.json")
-        self.set_kmp_vars(config_file_path)
-
-        self.parse_custom_args()
-        # Set the num_inter_threads and num_intra_threads (override inter threads to 1)
-        self.set_num_inter_intra_threads(num_inter_threads=1)
-        set_env_var("OMP_NUM_THREADS", self.args.num_cores)
-
-        if self.args.socket_id != -1:
-            command_prefix = "numactl --physcpubind=0-{} --membind={} {}".\
-                format(str(int(self.args.num_cores) - 1), self.args.socket_id,
-                       command_prefix)
-        else:
-            command_prefix = "numactl --physcpubind=0-{} -l {}".format(
-                str(int(self.args.num_cores) - 1), command_prefix)
-
-        checkpoint_path = os.path.join(self.args.checkpoint,
-                                       self.args.checkpoint_name)
-
-        # create command to run benchmarking
-        self.command = ("{} {} --num_inter_threads={} "
-                        "--num_intra_threads={} --sample={}").format(
-            command_prefix, checkpoint_path, str(self.args.num_inter_threads),
-            str(self.args.num_intra_threads), str(self.args.sample))
-
-    def parse_custom_args(self):
-        if self.custom_args:
-            parser = argparse.ArgumentParser()
-            parser.add_argument("--sample", default=None, dest='sample',
-                                type=int)
-            parser.add_argument("--checkpoint_name", default=None,
-                                dest='checkpoint_name', type=str)
-            self.args = parser.parse_args(self.custom_args,
-                                          namespace=self.args)
-
-    def run(self):
-        if self.command:
-            # The generate.py script expects that we run from the model source
-            # directory.  Save off the current working directory so that we can
-            # restore it when the script is done.
-            original_dir = os.getcwd()
-            os.chdir(self.args.model_source_dir)
-
-            # Run benchmarking
-            self.run_command(self.command)
-
-            # Change context back to the original dir
-            os.chdir(original_dir)
diff --git a/datasets/cloud_data_connector/cloud_data_connector/int/int_downloader.py b/datasets/cloud_data_connector/cloud_data_connector/int/int_downloader.py
index cf7fe4e73..c1126cc97 100644
--- a/datasets/cloud_data_connector/cloud_data_connector/int/int_downloader.py
+++ b/datasets/cloud_data_connector/cloud_data_connector/int/int_downloader.py
@@ -17,7 +17,18 @@
 #
 from abc import ABCMeta, abstractmethod 
 
+#
+
+<<<<<<<< HEAD:benchmarks/recommendation/tensorflow/dien/training/bfloat16/model_init.py
+from recommendation.tensorflow.dien.training.dien_model_init import DienModelInitializer
+
+
+class ModelInitializer(DienModelInitializer):
+    """Model initializer for DIEN bf16 Training"""
 
+    def __init__(self, args, custom_args=[], platform_util=None):
+        super(ModelInitializer, self).__init__(args, custom_args, platform_util)
+========
 class IntDownloader(metaclass=ABCMeta):
     
     def __init__(self, connector: object):
@@ -26,3 +37,4 @@ def __init__(self, connector: object):
     @abstractmethod 
     def download(self, container_obj: object, data_file: object, destiny: object) -> object:
         pass
+>>>>>>>> r3.1:datasets/cloud_data_connector/cloud_data_connector/int/int_downloader.py
diff --git a/datasets/cloud_data_connector/docker/packer.Dockerfile b/datasets/cloud_data_connector/docker/packer.Dockerfile
index 85905ffbe..aa66b516d 100644
--- a/datasets/cloud_data_connector/docker/packer.Dockerfile
+++ b/datasets/cloud_data_connector/docker/packer.Dockerfile
@@ -1,7 +1,11 @@
+<<<<<<<< HEAD:models/aidd/__init__.py
+# Copyright (c) 2022 Intel Corporation
+========
 #
 # -*- coding: utf-8 -*-
 #
 # Copyright (c) 2023 Intel Corporation
+>>>>>>>> r3.1:datasets/cloud_data_connector/docker/packer.Dockerfile
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
diff --git a/docker/flex-gpu/docker-compose.yml b/docker/flex-gpu/docker-compose.yml
index a8d9528f0..4693c114f 100644
--- a/docker/flex-gpu/docker-compose.yml
+++ b/docker/flex-gpu/docker-compose.yml
@@ -27,8 +27,9 @@ services:
         http_proxy: ${http_proxy}
         https_proxy: ${https_proxy}
         no_proxy: ""
-        BASE_IMAGE: ${REGISTRY}/aiops/mlops
-        BASE_TAG: ${FLEX_PYT_BASE_TAG:-ipex-gpu-2.0.110-xpu}
+        NO_PROXY: ""
+        PYT_BASE_IMAGE: ${PYT_BASE_IMAGE:-intel/intel-extension-for-pytorch}
+        PYT_BASE_TAG: ${PYT_BASE_TAG:-2.1.10-xpu}
       dockerfile: docker/flex-gpu/pytorch-resnet50v1-5-inference/pytorch-flex-series-resnet50v1-5-inference.Dockerfile
     command: >
       sh -c "python -c 'import torch; import intel_extension_for_pytorch as ipex; print(\"torch:\", torch.__version__, \" ipex:\",ipex.__version__)'"
@@ -49,8 +50,8 @@ services:
   tf-resnet50v1-5-inference:
     build:
       args:
-        BASE_IMAGE: ${FLEX_TF_BASE_IMAGE:-intel/intel-extension-for-tensorflow}
-        BASE_TAG: ${FLEX_TF_BASE_TAG:-xpu}
+        TF_BASE_IMAGE: ${TF_BASE_IMAGE:-intel/intel-extension-for-tensorflow}
+        TF_BASE_TAG: ${TF_BASE_TAG:-xpu}
         MODEL_URL: ${MODEL_URL:-https://storage.googleapis.com/intel-optimized-tensorflow/models/gpu/resnet50_v1_int8.pb}
       dockerfile: docker/flex-gpu/tf-resnet50v1-5-inference/tf-flex-series-resnet50v1-5-inference.Dockerfile
     command: >
@@ -72,9 +73,6 @@ services:
     image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-segmentation-tf-flex-gpu-maskrcnn-inference
   pytorch-stable-diffusion-inference:
     build:
-      args:
-        BASE_IMAGE: ${REGISTRY}/aiops/mlops
-        BASE_TAG: ${FLEX_PYT_BASE_TAG:-ipex-gpu-2.0.110-xpu}
       dockerfile: docker/flex-gpu/pytorch-stable-diffusion-inference/pytorch-flex-series-stable-diffusion-inference.Dockerfile
     extends: pytorch-resnet50v1-5-inference
     image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-generative-ai-pytorch-flex-gpu-stable-diffusion-inference
@@ -83,6 +81,11 @@ services:
       dockerfile: docker/flex-gpu/pytorch-yolov5-inference/pytorch-flex-series-yolov5-inference.Dockerfile
     extends: pytorch-stable-diffusion-inference
     image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-object-detection-pytorch-flex-gpu-yolov5-inference
+  pytorch-dlrm-v1-inference:
+    build:
+      dockerfile: docker/flex-gpu/pytorch-dlrmv1-inference/pytorch-flex-series-dlrm-v1-inference.Dockerfile
+    extends: pytorch-resnet50v1-5-inference
+    image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-recommendation-pytorch-flex-gpu-dlrm-v1-inference
   tf-stable-diffusion-inference:
     build:
       dockerfile: docker/flex-gpu/tf-stable-diffusion-inference/tf-flex-series-stable-diffusion-inference.Dockerfile
@@ -93,3 +96,13 @@ services:
       dockerfile: docker/flex-gpu/tf-efficientnet-inference/tf-flex-series-efficientnet-inference.Dockerfile
     extends: tf-resnet50v1-5-inference
     image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-flex-gpu-efficientnet-inference
+  pytorch-distilbert-inference:
+    build:
+      dockerfile: docker/flex-gpu/pytorch-distilbert-inference/pytorch-flex-series-distilbert-inference.Dockerfile
+    extends: pytorch-resnet50v1-5-inference
+    image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-pytorch-flex-gpu-distilbert-inference
+  tf-wide-deep-large-inference:
+    build:
+      dockerfile: docker/flex-gpu/tf-wide-deep-large-inference/tf-flex-series-wide-deep-large-inference.Dockerfile
+    extends: tf-resnet50v1-5-inference
+    image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-flex-gpu-wide-deep-large-inference
diff --git a/docker/flex-gpu/pytorch-distilbert-inference/pytorch-flex-series-distilbert-inference.Dockerfile b/docker/flex-gpu/pytorch-distilbert-inference/pytorch-flex-series-distilbert-inference.Dockerfile
new file mode 100644
index 000000000..19eecaa9f
--- /dev/null
+++ b/docker/flex-gpu/pytorch-distilbert-inference/pytorch-flex-series-distilbert-inference.Dockerfile
@@ -0,0 +1,50 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+#
+# THIS IS A GENERATED DOCKERFILE.
+#
+# This file was assembled from multiple pieces, whose use is documented
+# throughout. Please refer to the TensorFlow dockerfiles documentation
+# for more information.
+
+ARG PYT_BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG PYT_BASE_TAG="2.1.10-xpu"
+
+FROM ${PYT_BASE_IMAGE}:${PYT_BASE_TAG}
+
+USER root
+
+WORKDIR /workspace/pytorch-flex-series-distilbert-inference/models
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+RUN apt-get update && apt-get install -y python3-dev && \
+    rm -rf /var/lib/apt/lists/*
+
+RUN python -m pip install transformers==4.25.1 \
+        gitpython==3.1.30 \
+        tensorboard>=1.14.0 \
+        tensorboardX==1.8 \
+        psutil==5.6.6 \
+        scipy>=1.4.1 \
+        h5py
+
+COPY models_v2/pytorch/distilbert/inference/gpu .
+COPY models_v2/common/parse_result.py common/parse_result.py 
+
+COPY LICENSE licenses/LICENSE
+COPY third_party licenses/third_party
+
+USER $USER
diff --git a/docker/flex-gpu/pytorch-distilbert-inference/tests.yaml b/docker/flex-gpu/pytorch-distilbert-inference/tests.yaml
new file mode 100644
index 000000000..d21aadd87
--- /dev/null
+++ b/docker/flex-gpu/pytorch-distilbert-inference/tests.yaml
@@ -0,0 +1,36 @@
+fp32-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-pytorch-flex-gpu-distilbert-inference
+  cmd: bash run_model.sh
+  ipc: host
+  device: /dev/dri
+  env:
+    MULTI_TILE: 'False'
+    PLATFORM: Flex
+    BATCH_SIZE: '32'
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /nfs/site/disks/mlp_pytorch_dataset/hdf5_seq_512/
+    PRECISION: FP32
+    NUM_ITERATIONS: '300'
+  volumes:
+    - src: /nfs/site/disks/mlp_pytorch_dataset/hdf5_seq_512/
+      dst: /nfs/site/disks/mlp_pytorch_dataset/hdf5_seq_512/
+    - src: /tmp
+      dst: /tmp
+fp16-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-pytorch-flex-gpu-distilbert-inference
+  cmd: bash run_model.sh
+  ipc: host
+  device: /dev/dri
+  env:
+    MULTI_TILE: 'False'
+    PLATFORM: Flex
+    BATCH_SIZE: '32'
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /nfs/site/disks/mlp_pytorch_dataset/hdf5_seq_512/
+    PRECISION: FP16
+    NUM_ITERATIONS: '300'
+  volumes:
+    - src: /nfs/site/disks/mlp_pytorch_dataset/hdf5_seq_512/
+      dst: /nfs/site/disks/mlp_pytorch_dataset/hdf5_seq_512/
+    - src: /tmp
+      dst: /tmp
diff --git a/docker/flex-gpu/pytorch-dlrmv1-inference/pytorch-flex-series-dlrm-v1-inference.Dockerfile b/docker/flex-gpu/pytorch-dlrmv1-inference/pytorch-flex-series-dlrm-v1-inference.Dockerfile
new file mode 100644
index 000000000..c9d16d454
--- /dev/null
+++ b/docker/flex-gpu/pytorch-dlrmv1-inference/pytorch-flex-series-dlrm-v1-inference.Dockerfile
@@ -0,0 +1,50 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+#
+# THIS IS A GENERATED DOCKERFILE.
+#
+# This file was assembled from multiple pieces, whose use is documented
+# throughout. Please refer to the TensorFlow dockerfiles documentation
+# for more information.
+
+ARG PYT_BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG PYT_BASE_TAG="2.1.10-xpu"
+
+FROM ${PYT_BASE_IMAGE}:${PYT_BASE_TAG}
+
+ENV DEBIAN_FRONTEND=noninteractive
+USER root
+
+WORKDIR /workspace/pytorch-flex-series-dlrm-v1-inference/models
+
+RUN python -m pip install future \
+                    numpy \
+                    onnx \
+                    pydot \
+                    scikit-learn \
+                    tqdm \
+                    torchrec-nightly \
+                    torchx-nightly \
+                    lark-parser \
+                    scikit-learn \
+                    tensorboard 
+
+COPY models_v2/pytorch/dlrm/inference/gpu .
+COPY models_v2/common common
+
+COPY LICENSE license/LICENSE
+COPY third_party license/third_party
+
+USER $USER
diff --git a/docker/flex-gpu/pytorch-dlrmv1-inference/tests.yaml b/docker/flex-gpu/pytorch-dlrmv1-inference/tests.yaml
new file mode 100644
index 000000000..b16890ab2
--- /dev/null
+++ b/docker/flex-gpu/pytorch-dlrmv1-inference/tests.yaml
@@ -0,0 +1,19 @@
+fp16-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-recommendation-pytorch-flex-gpu-dlrm-v1-inference
+  cmd: bash run_model.sh
+  ipc: host
+  device: /dev/dri
+  env:
+    MULTI_TILE: 'False'
+    PLATFORM: 'Flex'
+    PRECISION: fp16
+    DATASET_DIR: /local_dateset/datasets/Cosim_test/dataset/dlrm_kaggle
+    CKPT_DIR: /local_dateset/datasets/Cosim_test/checkpoints/dlrm_kaggle
+    OUTPUT_DIR: /tmp
+  volumes:
+    - src: /local_dateset/datasets/Cosim_test/dataset/dlrm_kaggle
+      dst: /local_dateset/datasets/Cosim_test/dataset/dlrm_kaggle
+    - src: /local_dateset/datasets/Cosim_test/checkpoints/dlrm_kaggle
+      dst: /local_dateset/datasets/Cosim_test/checkpoints/dlrm_kaggle
+    - src: /tmp
+      dst: /tmp
diff --git a/docker/flex-gpu/pytorch-stable-diffusion-inference/pytorch-flex-series-stable-diffusion-inference.Dockerfile b/docker/flex-gpu/pytorch-stable-diffusion-inference/pytorch-flex-series-stable-diffusion-inference.Dockerfile
index 1bb9a927b..a6b693391 100644
--- a/docker/flex-gpu/pytorch-stable-diffusion-inference/pytorch-flex-series-stable-diffusion-inference.Dockerfile
+++ b/docker/flex-gpu/pytorch-stable-diffusion-inference/pytorch-flex-series-stable-diffusion-inference.Dockerfile
@@ -19,28 +19,36 @@
 # throughout. Please refer to the TensorFlow dockerfiles documentation
 # for more information.
 
-ARG BASE_IMAGE="intel/intel-extension-for-pytorch"
-ARG BASE_TAG="xpu-flex"
+ARG PYT_BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG PYT_BASE_TAG="2.1.10-xpu"
 
-FROM ${BASE_IMAGE}:${BASE_TAG}
+FROM ${PYT_BASE_IMAGE}:${PYT_BASE_TAG}
 
-WORKDIR /home/user/workspace/pytorch-flex-series-stable-diffusion-inference 
+USER root
 
-RUN apt-get update && \
-    apt-get install -y parallel 
-RUN apt-get install -y pciutils
+WORKDIR /workspace/pytorch-flex-series-stable-diffusion-inference/models
 
+ENV DEBIAN_FRONTEND=noninteractive
 ARG PY_VERSION=3.10
 
 RUN apt-get update && \
-    apt-get install -y --no-install-recommends --fix-missing \
+    apt-get install -y --no-install-recommends \
     build-essential \
-    python${PY_VERSION}-dev
+    python${PY_VERSION}-dev && \
+    rm -rf /var/lib/apt/lists/*
 
-RUN pip install diffusers pytorch-fid transformers
+RUN python -m pip install datasets \
+            torchmetrics \
+            diffusers \
+            transformers \
+            accelerate \
+            pytorch-fid \
+            scipy==1.9.1
 
-COPY models/generative-ai/pytorch/stable_diffusion/inference/gpu models/generative-ai/pytorch/stable_diffusion/inference/gpu 
-COPY quickstart/generative-ai/pytorch/stable_diffusion/inference/gpu/online_inference.sh quickstart/online_inference.sh 
+COPY models_v2/pytorch/stable_diffusion/inference/gpu .
+COPY models_v2/common common
 
 COPY LICENSE license/LICENSE
 COPY third_party license/third_party
+
+USER $USER
diff --git a/docker/flex-gpu/pytorch-stable-diffusion-inference/tests.yaml b/docker/flex-gpu/pytorch-stable-diffusion-inference/tests.yaml
index 7a21b4e28..f934cf368 100644
--- a/docker/flex-gpu/pytorch-stable-diffusion-inference/tests.yaml
+++ b/docker/flex-gpu/pytorch-stable-diffusion-inference/tests.yaml
@@ -1,18 +1,11 @@
-fp32-online-inference:
-  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-generative-ai-pytorch-flex-gpu-stable-diffusion-inference
-  cmd: quickstart/online_inference.sh
-  ipc: host
-  device: /dev/dri
-  env:
-    PRECISION: fp32
-    BATCH_SIZE: '1'
-    OUTPUT_DIR: /tmp
 fp16-online-inference:
   img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-generative-ai-pytorch-flex-gpu-stable-diffusion-inference
-  cmd: quickstart/online_inference.sh
+  cmd: bash run_model.sh
   ipc: host
   device: /dev/dri
   env:
     PRECISION: fp16
     BATCH_SIZE: '1'
+    MULTI_TILE: 'False'
     OUTPUT_DIR: /tmp
+    PLATFORM: 'Flex'
diff --git a/docker/flex-gpu/tf-efficientnet-inference/tests.yaml b/docker/flex-gpu/tf-efficientnet-inference/tests.yaml
index bee287706..1f00b67d5 100644
--- a/docker/flex-gpu/tf-efficientnet-inference/tests.yaml
+++ b/docker/flex-gpu/tf-efficientnet-inference/tests.yaml
@@ -1,124 +1,84 @@
 b0-flex170-bs64-batch-inference:
   img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-flex-gpu-efficientnet-inference
-  cmd: quickstart/batch_inference.sh
+  cmd: bash run_model.sh
   ipc: host
   device: /dev/dri
   env: 
     BATCH_SIZE: '64'
     MODEL_NAME: EfficientNetB0
-    PRECISION: fp16
-    GPU_TYPE: flex_170
-    IMAGE_FILE: /dataset/efficient-net/ILSVRC2012_val_00033350.JPEG
     OUTPUT_DIR: /tmp
-  volumes:
-    - src: /dataset/efficient-net/ILSVRC2012_val_00033350.JPEG
-      dst: /dataset/efficient-net/ILSVRC2012_val_00033350.JPEG
 b0-flex170-bs128-batch-inference:
   img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-flex-gpu-efficientnet-inference
-  cmd: quickstart/batch_inference.sh
+  cmd: bash run_model.sh
   ipc: host
   device: /dev/dri
   env: 
     BATCH_SIZE: '128'
     MODEL_NAME: EfficientNetB0
-    PRECISION: fp16
-    GPU_TYPE: flex_170
-    IMAGE_FILE: /dataset/efficient-net/ILSVRC2012_val_00033350.JPEG
     OUTPUT_DIR: /tmp
-  volumes:
-    - src: /dataset/efficient-net/ILSVRC2012_val_00033350.JPEG
-      dst: /dataset/efficient-net/ILSVRC2012_val_00033350.JPEG
 b3-flex170-bs64-batch-inference:
   img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-flex-gpu-efficientnet-inference
-  cmd: quickstart/batch_inference.sh
+  cmd: bash run_model.sh
   ipc: host
   device: /dev/dri
   env: 
     BATCH_SIZE: '64'
     MODEL_NAME: EfficientNetB3
-    PRECISION: fp16
-    GPU_TYPE: flex_170
-    IMAGE_FILE: /dataset/efficient-net/ILSVRC2012_val_00033350.JPEG
     OUTPUT_DIR: /tmp
-  volumes:
-    - src: /dataset/efficient-net/ILSVRC2012_val_00033350.JPEG
-      dst: /dataset/efficient-net/ILSVRC2012_val_00033350.JPEG
 b3-flex170-bs128-batch-inference:
   img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-flex-gpu-efficientnet-inference
-  cmd: quickstart/batch_inference.sh
+  cmd: bash run_model.sh
   ipc: host
   device: /dev/dri
   env: 
     BATCH_SIZE: '128'
     MODEL_NAME: EfficientNetB3
-    PRECISION: fp16
-    GPU_TYPE: flex_170
-    IMAGE_FILE: /dataset/efficient-net/ILSVRC2012_val_00033350.JPEG
     OUTPUT_DIR: /tmp
-  volumes:
-    - src: /dataset/efficient-net/ILSVRC2012_val_00033350.JPEG
-      dst: /dataset/efficient-net/ILSVRC2012_val_00033350.JPEG
-b0-flex140-bs64-batch-inference:
+b4-flex170-bs64-batch-inference:
   img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-flex-gpu-efficientnet-inference
-  cmd: quickstart/batch_inference.sh
+  cmd: bash run_model.sh
   ipc: host
   device: /dev/dri
-  cap_add: SYS_NICE
   env: 
     BATCH_SIZE: '64'
-    MODEL_NAME: EfficientNetB0
-    PRECISION: fp16
-    GPU_TYPE: flex_140
-    IMAGE_FILE: /dataset/efficient-net/ILSVRC2012_val_00033350.JPEG
+    MODEL_NAME: EfficientNetB4
     OUTPUT_DIR: /tmp
-  volumes:
-    - src: /dataset/efficient-net/ILSVRC2012_val_00033350.JPEG
-      dst: /dataset/efficient-net/ILSVRC2012_val_00033350.JPEG
-b0-flex140-bs128-batch-inference:
+b4-flex170-bs128-batch-inference:
   img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-flex-gpu-efficientnet-inference
-  cmd: quickstart/batch_inference.sh
+  cmd: bash run_model.sh
   ipc: host
   device: /dev/dri
-  cap_add: SYS_NICE
   env: 
     BATCH_SIZE: '128'
+    MODEL_NAME: EfficientNetB4
+    OUTPUT_DIR: /tmp
+b0-flex140-bs64-batch-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-flex-gpu-efficientnet-inference
+  cmd: bash run_model.sh
+  ipc: host
+  device: /dev/dri
+  cap_add: SYS_NICE 
+  env: 
+    BATCH_SIZE: '64'
     MODEL_NAME: EfficientNetB0
-    PRECISION: fp16
-    GPU_TYPE: flex_140
-    IMAGE_FILE: /dataset/efficient-net/ILSVRC2012_val_00033350.JPEG
     OUTPUT_DIR: /tmp
-  volumes:
-    - src: /dataset/efficient-net/ILSVRC2012_val_00033350.JPEG
-      dst: /dataset/efficient-net/ILSVRC2012_val_00033350.JPEG
 b3-flex140-bs64-batch-inference:
   img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-flex-gpu-efficientnet-inference
-  cmd: quickstart/batch_inference.sh
+  cmd: bash run_model.sh
   ipc: host
   device: /dev/dri
-  cap_add: SYS_NICE
+  cap_add: SYS_NICE 
   env: 
     BATCH_SIZE: '64'
     MODEL_NAME: EfficientNetB3
-    PRECISION: fp16
-    GPU_TYPE: flex_140
-    IMAGE_FILE: /dataset/efficient-net/ILSVRC2012_val_00033350.JPEG
     OUTPUT_DIR: /tmp
-  volumes:
-    - src: /dataset/efficient-net/ILSVRC2012_val_00033350.JPEG
-      dst: /dataset/efficient-net/ILSVRC2012_val_00033350.JPEG
-b3-flex140-bs128-batch-inference:
+b4-flex140-bs64-batch-inference:
   img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-flex-gpu-efficientnet-inference
-  cmd: quickstart/batch_inference.sh
+  cmd: bash run_model.sh
   ipc: host
   device: /dev/dri
   cap_add: SYS_NICE
   env: 
-    BATCH_SIZE: '128'
-    MODEL_NAME: EfficientNetB3
-    PRECISION: fp16
-    GPU_TYPE: flex_140
-    IMAGE_FILE: /dataset/efficient-net/ILSVRC2012_val_00033350.JPEG
+    BATCH_SIZE: '64'
+    MODEL_NAME: EfficientNetB4
     OUTPUT_DIR: /tmp
-  volumes:
-    - src: /dataset/efficient-net/ILSVRC2012_val_00033350.JPEG
-      dst: /dataset/efficient-net/ILSVRC2012_val_00033350.JPEG
diff --git a/docker/flex-gpu/tf-efficientnet-inference/tf-flex-series-efficientnet-inference.Dockerfile b/docker/flex-gpu/tf-efficientnet-inference/tf-flex-series-efficientnet-inference.Dockerfile
index ec493d9d9..16d029072 100644
--- a/docker/flex-gpu/tf-efficientnet-inference/tf-flex-series-efficientnet-inference.Dockerfile
+++ b/docker/flex-gpu/tf-efficientnet-inference/tf-flex-series-efficientnet-inference.Dockerfile
@@ -19,20 +19,31 @@
 # throughout. Please refer to the TensorFlow dockerfiles documentation
 # for more information.
 
-ARG BASE_IMAGE="intel/intel-extension-for-tensorflow"
-ARG BASE_TAG="xpu"
+<<<<<<<< HEAD:docker/max-gpu/pytorch-resnet50v1-5-inference/pytorch-max-series-resnet50v1-5-inference.Dockerfile
+ARG BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG BASE_TAG="xpu-max"
 
 FROM ${BASE_IMAGE}:${BASE_TAG}
 
-WORKDIR /workspace/tf-flex-series-efficientnet-inference
+WORKDIR /workspace/pytorch-max-series-resnet50v1-5-inference
+COPY quickstart/image_recognition/pytorch/resnet50v1_5/inference/gpu/README_Max_Series.md README.md
+COPY models/image_recognition/pytorch/resnet50v1_5/inference/gpu models/image_recognition/pytorch/resnet50v1_5/inference/gpu
+COPY quickstart/image_recognition/pytorch/resnet50v1_5/inference/gpu/inference_block_format.sh quickstart/inference_block_format.sh
 
-RUN apt-get update && \
-    apt-get install -y --no-install-recommends --fix-missing parallel pciutils numactl
+COPY LICENSE licenses/LICENSE
+COPY third_party licenses/third_party
+========
+ARG TF_BASE_IMAGE="intel/intel-extension-for-tensorflow"
+ARG TF_BASE_TAG="xpu"
 
-RUN pip install pillow 
+FROM ${TF_BASE_IMAGE}:${TF_BASE_TAG}
 
-COPY models/image_recognition/tensorflow/efficientnet/inference/gpu/predict.py models/image_recognition/tensorflow/efficientnet/inference/gpu/predict.py 
-COPY quickstart/image_recognition/tensorflow/efficientnet/inference/gpu/batch_inference.sh quickstart/batch_inference.sh
+WORKDIR /workspace/tf-flex-series-efficientnet-inference/models
+
+RUN pip install pillow
+
+COPY models_v2/tensorflow/efficientnet/inference/gpu . 
 
 COPY LICENSE license/LICENSE
 COPY third_party license/third_party
+>>>>>>>> r3.1:docker/flex-gpu/tf-efficientnet-inference/tf-flex-series-efficientnet-inference.Dockerfile
diff --git a/docker/flex-gpu/tf-maskrcnn-inference/tests.yaml b/docker/flex-gpu/tf-maskrcnn-inference/tests.yaml
index 270fc0575..67e91f0f6 100644
--- a/docker/flex-gpu/tf-maskrcnn-inference/tests.yaml
+++ b/docker/flex-gpu/tf-maskrcnn-inference/tests.yaml
@@ -1,42 +1,32 @@
-170-online-inference:
+170-fp16-inference:
   img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-segmentation-tf-flex-gpu-maskrcnn-inference
-  cmd: quickstart/inference.sh
-  ipc: host
-  device: /dev/dri
-  env:
-    BATCH_SIZE: '1'
-    PRECISION: fp16
-    GPU_TYPE: flex_170
-    DATASET_DIR: /dataset/maskrcnn/coco
-    OUTPUT_DIR: /tmp
-  volumes:
-    - src: /dataset/maskrcnn/coco
-      dst: /dataset/maskrcnn/coco
-170-batch-inference:
-  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-segmentation-tf-flex-gpu-maskrcnn-inference
-  cmd: quickstart/inference.sh
+  cmd: bash run_model.sh
   ipc: host
   device: /dev/dri
   env:
     BATCH_SIZE: '16'
-    PRECISION: fp16
+    PRECISION: float16
     GPU_TYPE: flex_170
-    DATASET_DIR: /dataset/maskrcnn/coco
+    DATASET_DIR: /tf_dataset/dataset/coco_dataset/COCO2017_training_data/
     OUTPUT_DIR: /tmp
   volumes:
-    - src: /dataset/maskrcnn/coco
-      dst: /dataset/maskrcnn/coco
+    - src: ${DATASET_DIR:-/tf_dataset/dataset/coco_dataset/COCO2017_training_data/}
+      dst: /tf_dataset/dataset/coco_dataset/COCO2017_training_data/
+    - src: /tmp
+      dst: /tmp
 140-online-inference:
   img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-segmentation-tf-flex-gpu-maskrcnn-inference
-  cmd: quickstart/inference.sh
+  cmd: bash run_model.sh
   ipc: host
   device: /dev/dri
   env:
     BATCH_SIZE: '1'
-    PRECISION: fp16
+    PRECISION: float16
     GPU_TYPE: flex_140
-    DATASET_DIR: /dataset/maskrcnn/coco
+    DATASET_DIR: /tf_dataset/dataset/coco_dataset/COCO2017_training_data/
     OUTPUT_DIR: /tmp
   volumes:
-    - src: /dataset/maskrcnn/coco
-      dst: /dataset/maskrcnn/coco
+    - src: ${DATASET_DIR:-/tf_dataset/dataset/coco_dataset/COCO2017_training_data/}
+      dst: /tf_dataset/dataset/coco_dataset/COCO2017_training_data/
+    - src: /tmp
+      dst: /tmp
diff --git a/docker/flex-gpu/tf-maskrcnn-inference/tf-flex-series-maskrcnn-inference.Dockerfile b/docker/flex-gpu/tf-maskrcnn-inference/tf-flex-series-maskrcnn-inference.Dockerfile
index 0978c5d2b..ba9b0e874 100644
--- a/docker/flex-gpu/tf-maskrcnn-inference/tf-flex-series-maskrcnn-inference.Dockerfile
+++ b/docker/flex-gpu/tf-maskrcnn-inference/tf-flex-series-maskrcnn-inference.Dockerfile
@@ -19,32 +19,30 @@
 # throughout. Please refer to the TensorFlow dockerfiles documentation
 # for more information.
 
-ARG BASE_IMAGE="intel/intel-extension-for-tensorflow"
-ARG BASE_TAG="xpu"
+ARG TF_BASE_IMAGE="intel/intel-extension-for-tensorflow"
+ARG TF_BASE_TAG="xpu"
 
-FROM ${BASE_IMAGE}:${BASE_TAG}
+FROM ${TF_BASE_IMAGE}:${TF_BASE_TAG}
 
-WORKDIR /workspace/tf-flex-series-maskrcnn-inference
+WORKDIR /workspace/tf-flex-series-maskrcnn-inference/models
 
 RUN apt-get update && \
-    apt-get install -y --no-install-recommends --fix-missing parallel pciutils numactl
+    apt-get install -y --no-install-recommends \
+      numactl \
+      parallel \
+      pciutils && \
+    rm -rf /var/lib/apt/lists/*
     
-RUN apt-get update && \
-    apt-get install -y --no-install-recommends --fix-missing \
-    git build-essential libssl-dev libffi-dev python3.10-dev
-
 RUN python -m pip install opencv-python-headless pycocotools 
 
-COPY models/image_segmentation/tensorflow/maskrcnn/inference/gpu models/image_segmentation/tensorflow/maskrcnn/inference/gpu
-COPY quickstart/image_segmentation/tensorflow/maskrcnn/inference/gpu/inference.sh quickstart/inference.sh
+RUN python -m pip install git+https://github.com/NVIDIA/dllogger.git
+
+COPY models_v2/tensorflow/maskrcnn/inference/gpu .
 
-RUN pip install git+https://github.com/NVIDIA/dllogger.git
 RUN git clone https://github.com/NVIDIA/DeepLearningExamples.git && \
-    cd /workspace/tf-flex-series-maskrcnn-inference/DeepLearningExamples && \
+    cd DeepLearningExamples && \
     git checkout 5be8a3cae21ee2d80e3935a4746827cb3367bcac && \
-    mv /workspace/tf-flex-series-maskrcnn-inference/models/image_segmentation/tensorflow/maskrcnn/inference/gpu/EnableInference.patch . && \
-    git apply EnableInference.patch && \
-    cd -
+    git apply ../EnableInference.patch
 
 COPY LICENSE license/LICENSE
 COPY third_party license/third_party
diff --git a/docker/flex-gpu/tf-resnet50v1-5-inference/tests.yaml b/docker/flex-gpu/tf-resnet50v1-5-inference/tests.yaml
index 80bab7b72..50a1312c9 100644
--- a/docker/flex-gpu/tf-resnet50v1-5-inference/tests.yaml
+++ b/docker/flex-gpu/tf-resnet50v1-5-inference/tests.yaml
@@ -1,52 +1,64 @@
 batch-170-inference:
   img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-flex-gpu-resnet50v1-5-inference
-  cmd: quickstart/batch_inference.sh
+  cmd: bash run_model.sh
   ipc: host
   device: /dev/dri
   env:
-    PRECISION : int8
-    GPU_TYPE: flex_series
+    PRECISION: int8
+    TEST_MODE: inference
+    FLEX_GPU_TYPE: flex_170
+    PB_FILE_PATH: /workspace/tf-flex-series-resnet50v1-5-inference/models/pretrained_models/resnet50v1_5-frozen_graph-int8-gpu.pb
+    BATCH_SIZE: '1024'
     OUTPUT_DIR: /tmp
 online-170-inference:
   img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-flex-gpu-resnet50v1-5-inference
-  cmd: quickstart/online_inference.sh
+  cmd: bash run_model.sh
   ipc: host
   device: /dev/dri
   env:
-    PRECISION : int8
-    GPU_TYPE: flex_series
+    PRECISION: int8
+    TEST_MODE: inference
+    FLEX_GPU_TYPE: flex_170
+    PB_FILE_PATH: /workspace/tf-flex-series-resnet50v1-5-inference/models/pretrained_models/resnet50v1_5-frozen_graph-int8-gpu.pb
+    BATCH_SIZE: '1'
     OUTPUT_DIR: /tmp
 batch-140-inference:
   img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-flex-gpu-resnet50v1-5-inference
-  cmd: quickstart/flex_multi_card_batch_inference.sh
+  cmd: bash run_model.sh
   ipc: host
   device: /dev/dri
   cap_add: SYS_NICE
   env:
     PRECISION : int8
-    GPU_TYPE: flex_series
+    TEST_MODE: inference
+    FLEX_GPU_TYPE: flex_140
+    PB_FILE_PATH: /workspace/tf-flex-series-resnet50v1-5-inference/models/pretrained_models/resnet50v1_5-frozen_graph-int8-gpu.pb
     BATCH_SIZE: '256'
     OUTPUT_DIR: /tmp
-batch-140-inference:
+online-140-inference:
   img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-flex-gpu-resnet50v1-5-inference
-  cmd: quickstart/flex_multi_card_online_inference.sh
+  cmd: bash run_model.sh
   ipc: host
   device: /dev/dri
   cap_add: SYS_NICE
   env:
     PRECISION : int8
-    GPU_TYPE: flex_series
+    TEST_MODE: inference
+    FLEX_GPU_TYPE: flex_140
+    PB_FILE_PATH: /workspace/tf-flex-series-resnet50v1-5-inference/models/pretrained_models/resnet50v1_5-frozen_graph-int8-gpu.pb
     BATCH_SIZE: '1'
     OUTPUT_DIR: /tmp
-170-accuracy:
+accuracy:
   img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-flex-gpu-resnet50v1-5-inference
-  cmd: quickstart/accuracy.sh
+  cmd: bash run_model.sh
   ipc: host
   device: /dev/dri
   env:
-    PRECISION : int8
-    GPU_TYPE: flex_series
+    PRECISION: int8
+    TEST_MODE: accuracy
+    PB_FILE_PATH: /workspace/tf-flex-series-resnet50v1-5-inference/models/pretrained_models/resnet50v1_5-frozen_graph-int8-gpu.pb
+    DATASET_DIR: /local_dateset/TF_Imagenet_FullData/
     OUTPUT_DIR: /tmp
   volumes:
-    - src: /tf_dataset/dataset/TF_Imagenet_FullData
-      dst: /tf_dataset/dataset/TF_Imagenet_FullData
+    - src: /local_dateset/TF_Imagenet_FullData/
+      dst: /local_dateset/TF_Imagenet_FullData/
diff --git a/docker/flex-gpu/tf-resnet50v1-5-inference/tf-flex-series-resnet50v1-5-inference.Dockerfile b/docker/flex-gpu/tf-resnet50v1-5-inference/tf-flex-series-resnet50v1-5-inference.Dockerfile
index 916369381..03ccbb450 100644
--- a/docker/flex-gpu/tf-resnet50v1-5-inference/tf-flex-series-resnet50v1-5-inference.Dockerfile
+++ b/docker/flex-gpu/tf-resnet50v1-5-inference/tf-flex-series-resnet50v1-5-inference.Dockerfile
@@ -19,12 +19,12 @@
 # throughout. Please refer to the TensorFlow dockerfiles documentation
 # for more information.
 
-ARG BASE_IMAGE="intel/intel-extension-for-tensorflow"
-ARG BASE_TAG="gpu"
+ARG TF_BASE_IMAGE="intel/intel-extension-for-tensorflow"
+ARG TF_BASE_TAG="xpu"
 
-FROM ${BASE_IMAGE}:${BASE_TAG}
+FROM ${TF_BASE_IMAGE}:${TF_BASE_TAG}
 
-WORKDIR /workspace/tf-flex-series-resnet50v1-5-inference
+WORKDIR /workspace/tf-flex-series-resnet50v1-5-inference/models
 
 RUN apt-get update && \
     apt-get install -y --no-install-recommends --fix-missing parallel pciutils numactl
@@ -34,22 +34,7 @@ ARG MODEL_URL
 RUN mkdir -p pretrained_models && \ 
     wget ${MODEL_URL} -O pretrained_models/resnet50v1_5-frozen_graph-int8-gpu.pb
     
-    
-COPY benchmarks/common benchmarks/common
-COPY benchmarks/image_recognition/__init__.py benchmarks/image_recognition/__init__.py
-COPY benchmarks/image_recognition/tensorflow/__init__.py benchmarks/image_recognition/tensorflow/__init__.py
-COPY benchmarks/image_recognition/tensorflow/resnet50v1_5/__init__.py benchmarks/image_recognition/tensorflow/resnet50v1_5/__init__.py
-COPY benchmarks/image_recognition/tensorflow/resnet50v1_5/inference benchmarks/image_recognition/tensorflow/resnet50v1_5/inference
-COPY benchmarks/launch_benchmark.py benchmarks/launch_benchmark.py
-COPY models/common models/common
-COPY models/image_recognition/tensorflow/resnet50v1_5 models/image_recognition/tensorflow/resnet50v1_5
-COPY quickstart/common quickstart/common
-COPY quickstart/image_recognition/tensorflow/resnet50v1_5/inference/gpu/README_Flex_series.md README.md
-COPY quickstart/image_recognition/tensorflow/resnet50v1_5/inference/gpu/batch_inference.sh quickstart/batch_inference.sh
-COPY quickstart/image_recognition/tensorflow/resnet50v1_5/inference/gpu/online_inference.sh  quickstart/online_inference.sh
-COPY quickstart/image_recognition/tensorflow/resnet50v1_5/inference/gpu/accuracy.sh quickstart/accuracy.sh
-COPY quickstart/image_recognition/tensorflow/resnet50v1_5/inference/gpu/flex_multi_card_batch_inference.sh quickstart/flex_multi_card_batch_inference.sh
-COPY quickstart/image_recognition/tensorflow/resnet50v1_5/inference/gpu/flex_multi_card_online_inference.sh quickstart/flex_multi_card_online_inference.sh
+COPY models_v2/tensorflow/resnet50v1_5/inference/gpu .
 
 COPY LICENSE license/LICENSE
 COPY third_party license/third_party
diff --git a/docker/flex-gpu/tf-stable-diffusion-inference/tests.yaml b/docker/flex-gpu/tf-stable-diffusion-inference/tests.yaml
index a837a7a1d..c07012ec5 100644
--- a/docker/flex-gpu/tf-stable-diffusion-inference/tests.yaml
+++ b/docker/flex-gpu/tf-stable-diffusion-inference/tests.yaml
@@ -1,32 +1,17 @@
 ---
 fp32-online-inference:
   img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-generative-ai-tf-flex-gpu-stable-diffusion-inference
-  cmd: quickstart/online_inference.sh
+  cmd: bash run_model.sh
   ipc: host
   device: /dev/dri
   env:
-    BATCH_SIZE: '1'
     PRECISION: fp32
     OUTPUT_DIR: /tmp
 fp16-online-inference:
   img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-generative-ai-tf-flex-gpu-stable-diffusion-inference
-  cmd: quickstart/online_inference.sh
+  cmd: bash run_model.sh
   ipc: host
   device: /dev/dri
   env:
-    BATCH_SIZE: '1'
     PRECISION: fp16
     OUTPUT_DIR: /tmp
-fp16-accuracy:
-  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-generative-ai-tf-flex-gpu-stable-diffusion-inference
-  cmd: quickstart/accuracy.sh
-  ipc: host
-  device: /dev/dri
-  env:
-    BATCH_SIZE: '1'
-    PRECISION: fp16
-    REFERENCE_RESULT_FILE: /dataset/stable-diffusion/img_arrays_for_acc.txt
-    OUTPUT_DIR: /tmp
-  volumes:
-    - src: /dataset/stable-diffusion/img_arrays_for_acc.txt
-      dst: /dataset/stable-diffusion/img_arrays_for_acc.txt
diff --git a/docker/flex-gpu/tf-stable-diffusion-inference/tf-flex-series-stable-diffusion-inference.Dockerfile b/docker/flex-gpu/tf-stable-diffusion-inference/tf-flex-series-stable-diffusion-inference.Dockerfile
index 9e6e2865b..2296f14cf 100644
--- a/docker/flex-gpu/tf-stable-diffusion-inference/tf-flex-series-stable-diffusion-inference.Dockerfile
+++ b/docker/flex-gpu/tf-stable-diffusion-inference/tf-flex-series-stable-diffusion-inference.Dockerfile
@@ -19,30 +19,54 @@
 # throughout. Please refer to the TensorFlow dockerfiles documentation
 # for more information.
 
-ARG BASE_IMAGE="intel/intel-extension-for-tensorflow"
-ARG BASE_TAG="xpu"
+<<<<<<<< HEAD:docker/flex-gpu/pytorch-stable-diffusion-inference/pytorch-flex-series-stable-diffusion-inference.Dockerfile
+ARG BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG BASE_TAG="xpu-flex"
 
 FROM ${BASE_IMAGE}:${BASE_TAG}
 
-WORKDIR /workspace/tf-flex-series-stable-diffusion-inference
+WORKDIR /home/user/workspace/pytorch-flex-series-stable-diffusion-inference 
+
+RUN apt-get update && \
+    apt-get install -y parallel 
+RUN apt-get install -y pciutils
+
+ARG PY_VERSION=3.10
+
+RUN apt-get update && \
+    apt-get install -y --no-install-recommends --fix-missing \
+    build-essential \
+    python${PY_VERSION}-dev
+
+RUN pip install diffusers pytorch-fid transformers
+
+COPY models/generative-ai/pytorch/stable_diffusion/inference/gpu models/generative-ai/pytorch/stable_diffusion/inference/gpu 
+COPY quickstart/generative-ai/pytorch/stable_diffusion/inference/gpu/online_inference.sh quickstart/online_inference.sh 
+
+========
+ARG TF_BASE_IMAGE="intel/intel-extension-for-tensorflow"
+ARG TF_BASE_TAG="xpu"
+
+FROM ${TF_BASE_IMAGE}:${TF_BASE_TAG}
+
+WORKDIR /workspace/tf-flex-series-stable-diffusion-inference/models
 
 RUN apt-get update && \
     apt-get install -y --no-install-recommends --fix-missing git
 
-COPY models/generative-ai/tensorflow/stable_diffusion/inference/gpu/ models/generative-ai/tensorflow/stable_diffusion/inference/gpu/
+COPY models_v2/tensorflow/stable_diffusion/inference/gpu/ .
 
-COPY quickstart/generative-ai/tensorflow/stable_diffusion/inference/gpu/online_inference.sh quickstart/online_inference.sh
-COPY quickstart/generative-ai/tensorflow/stable_diffusion/inference/gpu/accuracy.sh quickstart/accuracy.sh
 
 RUN git clone https://github.com/keras-team/keras-cv.git && \
     cd keras-cv && \
     git reset --hard 66fa74b6a2a0bb1e563ae8bce66496b118b95200 && \
-    mv /workspace/tf-flex-series-stable-diffusion-inference/models/generative-ai/tensorflow/stable_diffusion/inference/gpu/patch . && \
+    mv /workspace/tf-flex-series-stable-diffusion-inference/models/patch . && \
     git apply patch && \
     pip install matplotlib && \
     pip install .
 
-RUN python -m pip install scikit-image
+RUN python -m pip install scikit-image scipy==1.11.1
 
+>>>>>>>> r3.1:docker/flex-gpu/tf-stable-diffusion-inference/tf-flex-series-stable-diffusion-inference.Dockerfile
 COPY LICENSE license/LICENSE
 COPY third_party license/third_party
diff --git a/docker/flex-gpu/tf-wide-deep-large-inference/tests.yaml b/docker/flex-gpu/tf-wide-deep-large-inference/tests.yaml
new file mode 100644
index 000000000..d041245bc
--- /dev/null
+++ b/docker/flex-gpu/tf-wide-deep-large-inference/tests.yaml
@@ -0,0 +1,33 @@
+fp16-flex170-10000-batch-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-flex-gpu-wide-deep-large-inference
+  cmd: bash run_model.sh
+  ipc: host
+  device: /dev/dri
+  env: 
+    BATCH_SIZE: '10000'
+    DATASET_PATH: /local_dateset/datasets/wide_deep_large/data/eval_preprocessed_eval.tfrecords
+    PB_FILE_PATH: /local_dateset/datasets/wide_deep_large/wide_deep_fp16_pretrained_model.pb
+    GPU_TYPE: flex_170
+    OUTPUT_DIR: /tmp
+  volumes:
+    - src: /local_dateset/datasets/wide_deep_large/data/eval_preprocessed_eval.tfrecords
+      dst: /local_dateset/datasets/wide_deep_large/data/eval_preprocessed_eval.tfrecords
+    - src: /local_dateset/datasets/wide_deep_large/wide_deep_fp16_pretrained_model.pb
+      dst: /local_dateset/datasets/wide_deep_large/wide_deep_fp16_pretrained_model.pb
+fp16-flex140-10000-batch-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-flex-gpu-wide-deep-large-inference
+  cmd: bash run_model.sh
+  ipc: host
+  device: /dev/dri
+  cap_add: SYS_NICE
+  env: 
+    BATCH_SIZE: '10000'
+    DATASET_PATH: /local_dateset/datasets/wide_deep_large/data/eval_preprocessed_eval.tfrecords
+    PB_FILE_PATH: /local_dateset/datasets/wide_deep_large/wide_deep_fp16_pretrained_model.pb
+    GPU_TYPE: flex_140
+    OUTPUT_DIR: /tmp
+  volumes:
+    - src: /local_dateset/datasets/wide_deep_large/data/eval_preprocessed_eval.tfrecords
+      dst: /local_dateset/datasets/wide_deep_large/data/eval_preprocessed_eval.tfrecords
+    - src: /local_dateset/datasets/wide_deep_large/wide_deep_fp16_pretrained_model.pb
+      dst: /local_dateset/datasets/wide_deep_large/wide_deep_fp16_pretrained_model.pb
diff --git a/docker/flex-gpu/tf-wide-deep-large-inference/tf-flex-series-wide-deep-large-inference.Dockerfile b/docker/flex-gpu/tf-wide-deep-large-inference/tf-flex-series-wide-deep-large-inference.Dockerfile
new file mode 100755
index 000000000..6ba456dcb
--- /dev/null
+++ b/docker/flex-gpu/tf-wide-deep-large-inference/tf-flex-series-wide-deep-large-inference.Dockerfile
@@ -0,0 +1,43 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+#
+# THIS IS A GENERATED DOCKERFILE.
+#
+# This file was assembled from multiple pieces, whose use is documented
+# throughout. Please refer to the TensorFlow dockerfiles documentation
+# for more information
+
+<<<<<<<< HEAD:quickstart/recommendation/pytorch/torchrec_dlrm/training/gpu/setup.sh
+apt-get update && \
+    apt-get install -y --no-install-recommends --fix-missing numactl
+
+pip install -r models/recommendation/pytorch/torchrec_dlrm/training/gpu/requirements.txt
+pip install -e git+https://github.com/mlperf/logging#egg=mlperf-logging
+========
+ARG TF_BASE_IMAGE="intel/intel-extension-for-tensorflow"
+ARG TF_BASE_TAG="xpu"
+
+FROM ${TF_BASE_IMAGE}:${TF_BASE_TAG}
+
+WORKDIR /workspace/tf-flex-series-wide-deep-large-inference/models
+
+RUN apt-get update && \
+    apt-get install -y --no-install-recommends parallel pciutils numactl
+    
+COPY models_v2/tensorflow/wide_deep_large_ds/inference/gpu . 
+
+COPY LICENSE license/LICENSE
+COPY third_party license/third_party
+>>>>>>>> r3.1:docker/flex-gpu/tf-wide-deep-large-inference/tf-flex-series-wide-deep-large-inference.Dockerfile
diff --git a/docker/max-gpu/docker-compose.yml b/docker/max-gpu/docker-compose.yml
index 2c77cf619..999c4def9 100644
--- a/docker/max-gpu/docker-compose.yml
+++ b/docker/max-gpu/docker-compose.yml
@@ -26,9 +26,10 @@ services:
       args:
         http_proxy: ${http_proxy}
         https_proxy: ${https_proxy}
-        no_proxy: ${no_proxy}
-        BASE_IMAGE: ${REGISTRY}/aiops/mlops
-        BASE_TAG: xpu-max
+        no_proxy: ""
+        NO_PROXY: ""
+        PYT_BASE_IMAGE: ${PYT_BASE_IMAGE:-intel/intel-extension-for-pytorch}
+        PYT_BASE_TAG: ${PYT_BASE_TAG:-2.1.10-xpu}   
       dockerfile: docker/max-gpu/pytorch-resnet50v1-5-inference/pytorch-max-series-resnet50v1-5-inference.Dockerfile
     command: >
       sh -c "python -c 'import torch; import intel_extension_for_pytorch as ipex; print(\"torch:\", torch.__version__, \" ipex:\",ipex.__version__)'"
@@ -36,9 +37,7 @@ services:
     pull_policy: always
   pytorch-resnet50v1-5-training:
     build:
-      args:
-        VOC_LABELS_URL: ${VOC_LABELS_URL:-https://drive.google.com/uc?id=1q1sXhKIxniInw3WASnEDKYMqPMuiDDvc}   
-      dockerfile: docker/max-gpu/pytorch-bert-large-inference/pytorch-max-series-bert-large-inference.Dockerfile
+      dockerfile: docker/max-gpu/pytorch-resnet50v1-5-training/pytorch-max-series-resnet50v1-5-training.Dockerfile
     extends: pytorch-resnet50v1-5-inference
     image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-pytorch-max-gpu-resnet50v1-5-training
   pytorch-bert-large-inference:
@@ -54,8 +53,10 @@ services:
   tf-resnet50v1-5-inference:
     build:
       args:
-        BASE_IMAGE: ${BASE_IMAGE:-intel/intel-extension-for-tensorflow}
-        BASE_TAG: ${BASE_TAG:-xpu}   
+        TF_BASE_IMAGE: ${TF_BASE_IMAGE:-intel/intel-extension-for-tensorflow}
+        TF_BASE_TAG: ${TF_BASE_TAG:-2.14.0.1-xpu}  
+        NO_PROXY: ""
+        no_proxy: ""
         INT8_MODEL_URL: ${INT8_MODEL_URL:-https://storage.googleapis.com/intel-optimized-tensorflow/models/gpu/resnet50_v1_int8.pb}
         FP16_MODEL_URL: ${FP16_MODEL_URL:-https://storage.googleapis.com/intel-optimized-tensorflow/models/gpu/resnet50_v1.pb}
         FP32_MODEL_URL: ${FP32_MODEL_URL:-https://storage.googleapis.com/intel-optimized-tensorflow/models/gpu/resnet50_v1.pb}
@@ -81,6 +82,16 @@ services:
       dockerfile: docker/max-gpu/tf-bert-large-training/tf-max-series-bert-large-training.Dockerfile
     extends: tf-resnet50v1-5-inference
     image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-tf-max-gpu-bert-large-training
+  tf-maskrcnn-training:
+    build:
+      dockerfile: docker/max-gpu/tf-maskrcnn-training/tf-max-series-maskrcnn-training.Dockerfile
+    extends: tf-resnet50v1-5-inference
+    image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-segmentation-tf-max-gpu-maskrcnn-training
+  tf-3d-unet-training:
+    build:
+      dockerfile: docker/max-gpu/tf-3d-unet-training/tf-max-series-3d-unet-training.Dockerfile
+    extends: tf-resnet50v1-5-inference
+    image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-segmentation-tf-max-gpu-3d-unet-training
   pytorch-dlrm-inference:
     build:
       dockerfile: docker/max-gpu/pytorch-dlrm-inference/pytorch-max-series-dlrm-inference.Dockerfile
@@ -91,3 +102,26 @@ services:
       dockerfile: docker/max-gpu/pytorch-dlrm-training/pytorch-max-series-dlrm-training.Dockerfile
     extends: pytorch-resnet50v1-5-inference
     image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-recommendation-pytorch-max-gpu-dlrm-training
+  pytorch-distilbert-inference:
+    build:
+      dockerfile: docker/max-gpu/pytorch-distilbert-inference/pytorch-max-series-distilbert-inference.Dockerfile
+    extends: pytorch-resnet50v1-5-inference
+    image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-pytorch-max-gpu-distilbert-inference
+  pytorch-stable-diffusion-inference:
+    build:
+      dockerfile: docker/max-gpu/pytorch-stable-diffusion-inference/pytorch-max-series-stable-diffusion-inference.Dockerfile
+    extends: pytorch-resnet50v1-5-inference
+    image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-generative-ai-pytorch-max-gpu-stable-diffusion-inference
+  pytorch-gpt-j-6b-mlperf-inference:
+    build:
+      context: ../../
+      args:
+        http_proxy: ${http_proxy}
+        https_proxy: ${https_proxy}
+        no_proxy: ''
+        NO_PROXY: ''
+        BASE_IMAGE: intel/oneapi-basekit
+        BASE_TAG: ${BASE_TAG:-2023.1.0-devel-ubuntu22.04}
+        PYTHON: python3.9
+      dockerfile: docker/max-gpu/pytorch-gpt-j-6b-mlperf-inference/pytorch-max-series-gpt-j-6b-mlperf-inference.Dockerfile
+    image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-generative-ai-pytorch-max-gpu-gptj-6b-inference
diff --git a/docker/max-gpu/pytorch-bert-large-inference/pytorch-max-series-bert-large-inference.Dockerfile b/docker/max-gpu/pytorch-bert-large-inference/pytorch-max-series-bert-large-inference.Dockerfile
index 3057e0417..2b195c316 100644
--- a/docker/max-gpu/pytorch-bert-large-inference/pytorch-max-series-bert-large-inference.Dockerfile
+++ b/docker/max-gpu/pytorch-bert-large-inference/pytorch-max-series-bert-large-inference.Dockerfile
@@ -19,20 +19,21 @@
 # throughout. Please refer to the TensorFlow dockerfiles documentation
 # for more information.
 
-ARG BASE_IMAGE="intel/intel-extension-for-pytorch"
-ARG BASE_TAG="xpu-max"
+ARG PYT_BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG PYT_BASE_TAG="2.1.10-xpu"
 
-FROM ${BASE_IMAGE}:${BASE_TAG}
+FROM ${PYT_BASE_IMAGE}:${PYT_BASE_TAG}
 
-WORKDIR /workspace/pytorch-max-series-bert-large-inference
+USER root
 
-COPY quickstart/language_modeling/pytorch/bert_large/inference/gpu/README.md README.md
-COPY models/language_modeling/pytorch/bert_large/inference/gpu models/language_modeling/pytorch/bert_large/inference/gpu
+WORKDIR /workspace/pytorch-max-series-bert-large-inference/models
 
-RUN cd /workspace/pytorch-max-series-bert-large-inference/models/language_modeling/pytorch/bert_large/inference/gpu && \
-    pip install -r requirements.txt 
+COPY models_v2/pytorch/bert_large/inference/gpu .
+COPY models_v2/common/parse_result.py common/parse_result.py 
 
-COPY quickstart/language_modeling/pytorch/bert_large/inference/gpu/fp16_inference_plain_format.sh quickstart/fp16_inference_plain_format.sh
+RUN python -m pip install -r requirements.txt 
 
 COPY LICENSE licenses/LICENSE
 COPY third_party licenses/third_party
+
+USER $USER
diff --git a/docker/max-gpu/pytorch-bert-large-inference/tests.yaml b/docker/max-gpu/pytorch-bert-large-inference/tests.yaml
index dc3f4bf85..535a8de93 100644
--- a/docker/max-gpu/pytorch-bert-large-inference/tests.yaml
+++ b/docker/max-gpu/pytorch-bert-large-inference/tests.yaml
@@ -1,19 +1,120 @@
-fp16-batch-inference:
+# single-tile-fp16-inference:
+#   img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-pytorch-max-gpu-bert-large-inference
+#   cmd: bash run_model.sh
+#   ipc: host
+#   device: /dev/dri
+#   env:
+#     PRECISION: FP16
+#     BATCH_SIZE: '256'
+#     PLATFORM: Max
+#     OUTPUT_DIR: /tmp
+#     MULTI_TILE: 'False'
+#     BERT_WEIGHT: /dataset/squad_large_finetuned_checkpoint
+#     DATASET_DIR: /dataset/squad_dir
+#   volumes:
+#     - src: /dataset/squad_dir
+#       dst: /dataset/squad_dir
+#     - src: /dataset/squad_large_finetuned_checkpoint
+#       dst: /dataset/squad_large_finetuned_checkpoint
+#     - src: /tmp
+#       dst: /tmp
+# multi-tile-fp16-inference:
+#   img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-pytorch-max-gpu-bert-large-inference
+#   cmd: bash run_model.sh
+#   ipc: host
+#   device: /dev/dri
+#   env:
+#     PRECISION: FP16
+#     BATCH_SIZE: '256'
+#     PLATFORM: Max
+#     OUTPUT_DIR: /tmp
+#     MULTI_TILE: 'True'
+#     BERT_WEIGHT: /dataset/squad_large_finetuned_checkpoint
+#     DATASET_DIR: /dataset/squad_dir
+#   volumes:
+#     - src: /dataset/squad_dir
+#       dst: /dataset/squad_dir
+#     - src: /dataset/squad_large_finetuned_checkpoint
+#       dst: /dataset/squad_large_finetuned_checkpoint
+#     - src: /tmp
+#       dst: /tmp
+# single-tile-bf16-inference:
+#   img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-pytorch-max-gpu-bert-large-inference
+#   cmd: bash run_model.sh
+#   ipc: host
+#   device: /dev/dri
+#   env:
+#     PRECISION: BF16
+#     BATCH_SIZE: '256'
+#     PLATFORM: Max
+#     OUTPUT_DIR: /tmp
+#     MULTI_TILE: 'False'
+#     BERT_WEIGHT: /dataset/squad_large_finetuned_checkpoint
+#     DATASET_DIR: /dataset/squad_dir
+#   volumes:
+#     - src: /dataset/squad_dir
+#       dst: /dataset/squad_dir
+#     - src: /dataset/squad_large_finetuned_checkpoint
+#       dst: /dataset/squad_large_finetuned_checkpoint
+#     - src: /tmp
+#       dst: /tmp
+# multi-tile-bf16-inference:
+#   img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-pytorch-max-gpu-bert-large-inference
+#   cmd: bash run_model.sh
+#   ipc: host
+#   device: /dev/dri
+#   env:
+#     PRECISION: BF16
+#     BATCH_SIZE: '256'
+#     PLATFORM: Max
+#     OUTPUT_DIR: /tmp
+#     MULTI_TILE: 'True'
+#     BERT_WEIGHT: /dataset/squad_large_finetuned_checkpoint
+#     DATASET_DIR: /dataset/squad_dir
+#   volumes:
+#     - src: /dataset/squad_dir
+#       dst: /dataset/squad_dir
+#     - src: /dataset/squad_large_finetuned_checkpoint
+#       dst: /dataset/squad_large_finetuned_checkpoint
+#     - src: /tmp
+#       dst: /tmp
+# single-tile-fp32-inference:
+#   img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-pytorch-max-gpu-bert-large-inference
+#   cmd: bash run_model.sh
+#   ipc: host
+#   device: /dev/dri
+#   env:
+#     PRECISION: FP32
+#     BATCH_SIZE: '256'
+#     PLATFORM: Max
+#     OUTPUT_DIR: /tmp
+#     MULTI_TILE: 'False'
+#     BERT_WEIGHT: /dataset/squad_large_finetuned_checkpoint
+#     DATASET_DIR: /dataset/squad_dir
+#   volumes:
+#     - src: /dataset/squad_dir
+#       dst: /dataset/squad_dir
+#     - src: /dataset/squad_large_finetuned_checkpoint
+#       dst: /dataset/squad_large_finetuned_checkpoint
+#     - src: /tmp
+#       dst: /tmp
+multi-tile-fp32-inference:
   img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-pytorch-max-gpu-bert-large-inference
-  cmd: quickstart/fp16_inference_plain_format.sh
+  cmd: bash run_model.sh
   ipc: host
   device: /dev/dri
   env:
-    PRECISION: FP16
-    BATCH_SIZE: '64'
+    PRECISION: FP32
+    BATCH_SIZE: '256'
+    PLATFORM: Max
     OUTPUT_DIR: /tmp
-    NUM_OAM: '4'
+    MULTI_TILE: 'True'
     BERT_WEIGHT: /dataset/squad_large_finetuned_checkpoint
-    DATASET_DIR: /dataset/bert_squad
+    DATASET_DIR: /dataset/squad_dir
   volumes:
-    - src: /dataset/bert_squad
-      dst: /dataset/bert_squad
+    - src: /dataset/squad_dir
+      dst: /dataset/squad_dir
     - src: /dataset/squad_large_finetuned_checkpoint
       dst: /dataset/squad_large_finetuned_checkpoint
-
-      
+    - src: /tmp
+      dst: /tmp
diff --git a/docker/max-gpu/pytorch-bert-large-training/pytorch-max-series-bert-large-training.Dockerfile b/docker/max-gpu/pytorch-bert-large-training/pytorch-max-series-bert-large-training.Dockerfile
index 78b995386..8cdb7884d 100644
--- a/docker/max-gpu/pytorch-bert-large-training/pytorch-max-series-bert-large-training.Dockerfile
+++ b/docker/max-gpu/pytorch-bert-large-training/pytorch-max-series-bert-large-training.Dockerfile
@@ -19,37 +19,37 @@
 # throughout. Please refer to the TensorFlow dockerfiles documentation
 # for more information.
 
-ARG MAX_PYT_BASE_IMAGE="intel/intel-extension-for-pytorch"
-ARG MAX_PYT_BASE_TAG="xpu-max"
+ARG PYT_BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG PYT_BASE_TAG="2.1.10-xpu"
 
-FROM ${MAX_PYT_BASE_IMAGE}:${MAX_PYT_BASE_TAG}
+FROM ${PYT_BASE_IMAGE}:${PYT_BASE_TAG}
 
-    
-RUN curl -fsSL https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS-2023.PUB | apt-key add -
-RUN echo "deb [trusted=yes] https://apt.repos.intel.com/oneapi all main " > /etc/apt/sources.list.d/oneAPI.list
+USER root
 
 RUN apt-get update && \
-  DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
+    DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
     ca-certificates \
-    intel-oneapi-mpi-devel=2021.9.0-43482 \
-    intel-oneapi-ccl=2021.9.0-43543 \
+    curl \
+    intel-oneapi-mpi-devel=2021.11.0-49493  \
+    intel-oneapi-ccl=2021.11.2-5 \
     && \
-  rm -rf /var/lib/apt/lists/*
+    rm -rf /var/lib/apt/lists/*
 
-WORKDIR /workspace/pytorch-max-series-bert-large-training
+WORKDIR /workspace/pytorch-max-series-bert-large-training/models
 
-COPY quickstart/language_modeling/pytorch/bert_large/training/gpu/README.md README.md
-COPY models/language_modeling/pytorch/bert_large/training/gpu models/language_modeling/pytorch/bert_large/training/gpu
-COPY quickstart/language_modeling/pytorch/bert_large/training/gpu/bf16_training_plain_format.sh quickstart/bf16_training_plain_format.sh
-COPY quickstart/language_modeling/pytorch/bert_large/training/gpu/ddp_bf16_training_plain_format.sh quickstart/ddp_bf16_training_plain_format.sh 
+COPY models_v2/pytorch/bert_large/training/gpu .
+COPY models_v2/common common
 
+RUN pip install -r requirements.txt
 
-RUN cd models/language_modeling/pytorch/bert_large/training/gpu/data/ && \
-    wget https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt && \
-    mv bert-base-uncased-vocab.txt vocab.txt && \
-    cd -
-
-RUN pip install -r models/language_modeling/pytorch/bert_large/training/gpu/requirements.txt
+ENV LD_LIBRARY_PATH=/opt/intel/oneapi/ccl/2021.11/lib/:/opt/intel/oneapi/mpi/2021.11/opt/mpi/libfabric/lib:/opt/intel/oneapi/mpi/2021.11/lib:$LD_LIBRARY_PATH
+ENV LIBRARY_PATH=/opt/intel/oneapi/mpi/2021.11/lib:/opt/intel/oneapi/ccl/2021.11/lib/
+ENV PATH=/opt/intel/oneapi/mpi/2021.11/opt/mpi/libfabric/bin:/opt/intel/oneapi/mpi/2021.11/bin:$PATH
+ENV CCL_ROOT=/opt/intel/oneapi/ccl/2021.11
+ENV I_MPI_ROOT=/opt/intel/oneapi/mpi/2021.11
+ENV FI_PROVIDER_PATH=/opt/intel/oneapi/mpi/2021.11/opt/mpi/libfabric/lib/prov:/usr/lib/x86_64-linux-gnu/libfabric
 
 COPY LICENSE licenses/LICENSE
 COPY third_party licenses/third_party
+
+USER $USER
diff --git a/docker/max-gpu/pytorch-bert-large-training/tests.yaml b/docker/max-gpu/pytorch-bert-large-training/tests.yaml
new file mode 100644
index 000000000..5b22cf8c0
--- /dev/null
+++ b/docker/max-gpu/pytorch-bert-large-training/tests.yaml
@@ -0,0 +1,108 @@
+single-tile-bf16-training:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-pytorch-max-gpu-bert-large-training
+  cmd: bash run_model.sh
+  ipc: host
+  device: /dev/dri
+  env:
+    MULTI_TILE: 'False'
+    PLATFORM: Max
+    BATCH_SIZE: '16'
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /nfs/pdx/disks/mlp_pytorch_dataset/hdf5_seq_512/
+    PRECISION: BF16
+  volumes:
+    - src: /nfs/pdx/disks/mlp_pytorch_dataset/hdf5_seq_512/
+      dst: /nfs/pdx/disks/mlp_pytorch_dataset/hdf5_seq_512/
+    - src: /tmp
+      dst: /tmp
+multi-tile-bf16-training:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-pytorch-max-gpu-bert-large-training
+  cmd: bash run_model.sh
+  ipc: host
+  device: /dev/dri
+  env:
+    MULTI_TILE: 'True'
+    PLATFORM: Max
+    BATCH_SIZE: '16'
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /nfs/pdx/disks/mlp_pytorch_dataset/hdf5_seq_512/
+    PRECISION: BF16
+  volumes:
+    - src: /nfs/pdx/disks/mlp_pytorch_dataset/hdf5_seq_512/
+      dst: /nfs/pdx/disks/mlp_pytorch_dataset/hdf5_seq_512/
+    - src: /dev/dri
+      dst: /dev/dri
+    - src: /tmp
+      dst: /tmp
+single-tile-fp32-training:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-pytorch-max-gpu-bert-large-training
+  cmd: bash run_model.sh
+  ipc: host
+  device: /dev/dri
+  env:
+    MULTI_TILE: 'False'
+    PLATFORM: Max
+    BATCH_SIZE: '16'
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /nfs/pdx/disks/mlp_pytorch_dataset/hdf5_seq_512/
+    PRECISION: FP32
+  volumes:
+    - src: /nfs/pdx/disks/mlp_pytorch_dataset/hdf5_seq_512/
+      dst: /nfs/pdx/disks/mlp_pytorch_dataset/hdf5_seq_512/
+    - src: /tmp
+      dst: /tmp
+multi-tile-fp32-training:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-pytorch-max-gpu-bert-large-training
+  cmd: bash run_model.sh
+  ipc: host
+  device: /dev/dri
+  env:
+    MULTI_TILE: 'True'
+    PLATFORM: Max
+    BATCH_SIZE: '16'
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /nfs/pdx/disks/mlp_pytorch_dataset/hdf5_seq_512/
+    PRECISION: FP32
+  volumes:
+    - src: /nfs/pdx/disks/mlp_pytorch_dataset/hdf5_seq_512/
+      dst: /nfs/pdx/disks/mlp_pytorch_dataset/hdf5_seq_512/
+    - src: /dev/dri
+      dst: /dev/dri
+    - src: /tmp
+      dst: /tmp
+single-tile-tf32-training:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-pytorch-max-gpu-bert-large-training
+  cmd: bash run_model.sh
+  ipc: host
+  device: /dev/dri
+  env:
+    MULTI_TILE: 'False'
+    PLATFORM: Max
+    BATCH_SIZE: '16'
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /nfs/pdx/disks/mlp_pytorch_dataset/hdf5_seq_512/
+    PRECISION: TF32
+  volumes:
+    - src: /nfs/pdx/disks/mlp_pytorch_dataset/hdf5_seq_512/
+      dst: /nfs/pdx/disks/mlp_pytorch_dataset/hdf5_seq_512/
+    - src: /tmp
+      dst: /tmp
+multi-tile-tf32-training:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-pytorch-max-gpu-bert-large-training
+  cmd: bash run_model.sh
+  ipc: host
+  device: /dev/dri
+  env:
+    MULTI_TILE: 'True'
+    PLATFORM: Max
+    BATCH_SIZE: '16'
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /nfs/pdx/disks/mlp_pytorch_dataset/hdf5_seq_512/
+    PRECISION: TF32
+  volumes:
+    - src: /nfs/pdx/disks/mlp_pytorch_dataset/hdf5_seq_512/
+      dst: /nfs/pdx/disks/mlp_pytorch_dataset/hdf5_seq_512/
+    - src: /dev/dri
+      dst: /dev/dri
+    - src: /tmp
+      dst: /tmp
diff --git a/docker/max-gpu/pytorch-distilbert-inference/pytorch-max-series-distilbert-inference.Dockerfile b/docker/max-gpu/pytorch-distilbert-inference/pytorch-max-series-distilbert-inference.Dockerfile
new file mode 100644
index 000000000..b1c1f4618
--- /dev/null
+++ b/docker/max-gpu/pytorch-distilbert-inference/pytorch-max-series-distilbert-inference.Dockerfile
@@ -0,0 +1,50 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+#
+# THIS IS A GENERATED DOCKERFILE.
+#
+# This file was assembled from multiple pieces, whose use is documented
+# throughout. Please refer to the TensorFlow dockerfiles documentation
+# for more information.
+
+ARG PYT_BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG PYT_BASE_TAG="2.1.10-xpu"
+
+FROM ${PYT_BASE_IMAGE}:${PYT_BASE_TAG}
+
+USER root
+
+WORKDIR /workspace/pytorch-max-series-distilbert-inference/models
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+RUN apt-get update && apt-get install -y python3-dev && \
+    rm -rf /var/lib/apt/lists/*
+
+RUN python -m pip install transformers==4.25.1 \
+        gitpython==3.1.30 \
+        tensorboard>=1.14.0 \
+        tensorboardX==1.8 \
+        psutil==5.6.6 \
+        scipy>=1.4.1 \
+        h5py
+
+COPY models_v2/pytorch/distilbert/inference/gpu .
+COPY models_v2/common/parse_result.py common/parse_result.py 
+
+COPY LICENSE licenses/LICENSE
+COPY third_party licenses/third_party
+
+USER $USER
diff --git a/docker/max-gpu/pytorch-distilbert-inference/tests.yaml b/docker/max-gpu/pytorch-distilbert-inference/tests.yaml
new file mode 100644
index 000000000..68957c081
--- /dev/null
+++ b/docker/max-gpu/pytorch-distilbert-inference/tests.yaml
@@ -0,0 +1,54 @@
+fp32-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-pytorch-max-gpu-distilbert-inference
+  cmd: bash run_model.sh
+  ipc: host
+  device: /dev/dri
+  env:
+    MULTI_TILE: 'True'
+    PLATFORM: Max
+    BATCH_SIZE: '32'
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /nfs/site/disks/mlp_pytorch_dataset/hdf5_seq_512/
+    PRECISION: FP32
+    NUM_ITERATIONS: '300'
+  volumes:
+    - src: /nfs/site/disks/mlp_pytorch_dataset/hdf5_seq_512/
+      dst: /nfs/site/disks/mlp_pytorch_dataset/hdf5_seq_512/
+    - src: /tmp
+      dst: /tmp
+fp16-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-pytorch-max-gpu-distilbert-inference
+  cmd: bash run_model.sh
+  ipc: host
+  device: /dev/dri
+  env:
+    MULTI_TILE: 'True'
+    PLATFORM: Max
+    BATCH_SIZE: '32'
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /nfs/site/disks/mlp_pytorch_dataset/hdf5_seq_512/
+    PRECISION: FP16
+    NUM_ITERATIONS: '300'
+  volumes:
+    - src: /nfs/site/disks/mlp_pytorch_dataset/hdf5_seq_512/
+      dst: /nfs/site/disks/mlp_pytorch_dataset/hdf5_seq_512/
+    - src: /tmp
+      dst: /tmp
+bf16-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-pytorch-max-gpu-distilbert-inference
+  cmd: bash run_model.sh
+  ipc: host
+  device: /dev/dri
+  env:
+    MULTI_TILE: 'True'
+    PLATFORM: Max
+    BATCH_SIZE: '32'
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /nfs/site/disks/mlp_pytorch_dataset/hdf5_seq_512/
+    PRECISION: BF16
+    NUM_ITERATIONS: '300'
+  volumes:
+    - src: /nfs/site/disks/mlp_pytorch_dataset/hdf5_seq_512/
+      dst: /nfs/site/disks/mlp_pytorch_dataset/hdf5_seq_512/
+    - src: /tmp
+      dst: /tmp
diff --git a/docker/max-gpu/pytorch-resnet50v1-5-inference/pytorch-max-series-resnet50v1-5-inference.Dockerfile b/docker/max-gpu/pytorch-resnet50v1-5-inference/pytorch-max-series-resnet50v1-5-inference.Dockerfile
index c966852ae..1019a2288 100644
--- a/docker/max-gpu/pytorch-resnet50v1-5-inference/pytorch-max-series-resnet50v1-5-inference.Dockerfile
+++ b/docker/max-gpu/pytorch-resnet50v1-5-inference/pytorch-max-series-resnet50v1-5-inference.Dockerfile
@@ -19,15 +19,39 @@
 # throughout. Please refer to the TensorFlow dockerfiles documentation
 # for more information.
 
-ARG BASE_IMAGE="intel/intel-extension-for-pytorch"
-ARG BASE_TAG="xpu-max"
+<<<<<<<< HEAD:docker/flex-gpu/tf-efficientnet-inference/tf-flex-series-efficientnet-inference.Dockerfile
+ARG BASE_IMAGE="intel/intel-extension-for-tensorflow"
+ARG BASE_TAG="xpu"
 
 FROM ${BASE_IMAGE}:${BASE_TAG}
 
-WORKDIR /workspace/pytorch-max-series-resnet50v1-5-inference
-COPY quickstart/image_recognition/pytorch/resnet50v1_5/inference/gpu/README_Max_Series.md README.md
-COPY models/image_recognition/pytorch/resnet50v1_5/inference/gpu models/image_recognition/pytorch/resnet50v1_5/inference/gpu
-COPY quickstart/image_recognition/pytorch/resnet50v1_5/inference/gpu/inference_block_format.sh quickstart/inference_block_format.sh
+WORKDIR /workspace/tf-flex-series-efficientnet-inference
+
+RUN apt-get update && \
+    apt-get install -y --no-install-recommends --fix-missing parallel pciutils numactl
+
+RUN pip install pillow 
+
+COPY models/image_recognition/tensorflow/efficientnet/inference/gpu/predict.py models/image_recognition/tensorflow/efficientnet/inference/gpu/predict.py 
+COPY quickstart/image_recognition/tensorflow/efficientnet/inference/gpu/batch_inference.sh quickstart/batch_inference.sh
+
+COPY LICENSE license/LICENSE
+COPY third_party license/third_party
+========
+ARG PYT_BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG PYT_BASE_TAG="2.1.10-xpu"
+
+FROM ${PYT_BASE_IMAGE}:${PYT_BASE_TAG}
+
+USER root
+
+WORKDIR /workspace/pytorch-max-series-resnet50v1-5-inference/models
+
+COPY models_v2/pytorch/resnet50v1_5/inference/gpu .
+COPY models_v2/common common 
 
 COPY LICENSE licenses/LICENSE
 COPY third_party licenses/third_party
+
+USER $USER
+>>>>>>>> r3.1:docker/max-gpu/pytorch-resnet50v1-5-inference/pytorch-max-series-resnet50v1-5-inference.Dockerfile
diff --git a/docker/max-gpu/pytorch-resnet50v1-5-inference/tests.yaml b/docker/max-gpu/pytorch-resnet50v1-5-inference/tests.yaml
index 6745f1d71..c3a95e9c7 100644
--- a/docker/max-gpu/pytorch-resnet50v1-5-inference/tests.yaml
+++ b/docker/max-gpu/pytorch-resnet50v1-5-inference/tests.yaml
@@ -1,11 +1,75 @@
-int8-batch-inference:
-  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-pytorch-max-gpu-resnet50v1-5-inference
-  cmd: quickstart/inference_block_format.sh
+bf16-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-pytorch-max-gpu-resnet50v1-5-training
+  cmd: bash run_model.sh
   ipc: host
   device: /dev/dri
   env:
-    PRECISION: int8
+    PRECISION: BF16
     NUM_ITERATIONS: '500'
     BATCH_SIZE: '1024'
     OUTPUT_DIR: /tmp
-    NUM_OAM: '4'
+    MULTI_TILE: 'True'
+    PLATFORM: 'Max'
+  volumes:
+    - src: /tmp
+      dst: /tmp
+fp16-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-pytorch-max-gpu-resnet50v1-5-training
+  cmd: bash run_model.sh
+  ipc: host
+  device: /dev/dri
+  env:
+    PRECISION: FP16
+    NUM_ITERATIONS: '500'
+    BATCH_SIZE: '1024'
+    OUTPUT_DIR: /tmp
+    MULTI_TILE: 'True'
+    PLATFORM: 'Max'
+  volumes:
+    - src: /tmp
+      dst: /tmp
+fp32-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-pytorch-max-gpu-resnet50v1-5-training
+  cmd: bash run_model.sh
+  ipc: host
+  device: /dev/dri
+  env:
+    PRECISION: FP32
+    NUM_ITERATIONS: '500'
+    BATCH_SIZE: '1024'
+    OUTPUT_DIR: /tmp
+    MULTI_TILE: 'True'
+    PLATFORM: 'Max'
+  volumes:
+    - src: /tmp
+      dst: /tmp
+tf32-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-pytorch-max-gpu-resnet50v1-5-training
+  cmd: bash run_model.sh
+  ipc: host
+  device: /dev/dri
+  env:
+    PRECISION: TF32
+    NUM_ITERATIONS: '500'
+    BATCH_SIZE: '1024'
+    OUTPUT_DIR: /tmp
+    MULTI_TILE: 'True'
+    PLATFORM: 'Max'
+  volumes:
+    - src: /tmp
+      dst: /tmp
+int8-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-pytorch-max-gpu-resnet50v1-5-training
+  cmd: bash run_model.sh
+  ipc: host
+  device: /dev/dri
+  env:
+    PRECISION: INT8
+    NUM_ITERATIONS: '500'
+    BATCH_SIZE: '1024'
+    OUTPUT_DIR: /tmp
+    MULTI_TILE: 'True'
+    PLATFORM: 'Max'
+  volumes:
+    - src: /tmp
+      dst: /tmp
diff --git a/docker/max-gpu/pytorch-resnet50v1-5-training/pytorch-max-series-resnet50v1-5-training.Dockerfile b/docker/max-gpu/pytorch-resnet50v1-5-training/pytorch-max-series-resnet50v1-5-training.Dockerfile
index c1f30446f..afb6dc01d 100644
--- a/docker/max-gpu/pytorch-resnet50v1-5-training/pytorch-max-series-resnet50v1-5-training.Dockerfile
+++ b/docker/max-gpu/pytorch-resnet50v1-5-training/pytorch-max-series-resnet50v1-5-training.Dockerfile
@@ -19,28 +19,40 @@
 # throughout. Please refer to the TensorFlow dockerfiles documentation
 # for more information.
 
-ARG BASE_IMAGE="intel/intel-extension-for-pytorch"
-ARG BASE_TAG="xpu-max"
+ARG PYT_BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG PYT_BASE_TAG="2.1.10-xpu"
 
-FROM ${BASE_IMAGE}:${BASE_TAG}
+FROM ${PYT_BASE_IMAGE}:${PYT_BASE_TAG}
 
-RUN curl -fsSL https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS-2023.PUB | apt-key add -
-RUN echo "deb [trusted=yes] https://apt.repos.intel.com/oneapi all main " > /etc/apt/sources.list.d/oneAPI.list
+USER root
 
 RUN apt-get update && \
-  DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
+    DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
     ca-certificates \
-    intel-oneapi-mpi-devel=2021.9.0-43482 \
-    intel-oneapi-ccl=2021.9.0-43543 \
+    curl \
+    intel-oneapi-mpi-devel=2021.11.0-49493  \
+    intel-oneapi-ccl=2021.11.2-5 \
     && \
-  rm -rf /var/lib/apt/lists/*
+    rm -rf /var/lib/apt/lists/*
 
-WORKDIR /workspace/pytorch-max-series-resnet50v1-5-training
+WORKDIR /workspace/pytorch-max-series-resnet50v1-5-training/models
 
-COPY quickstart/image_recognition/pytorch/resnet50v1_5/training/gpu/README.md README.md 
-COPY models/image_recognition/pytorch/resnet50v1_5/training/gpu models/image_recognition/pytorch/resnet50v1_5/training/gpu
-COPY quickstart/image_recognition/pytorch/resnet50v1_5/training/gpu/training_plain_format.sh quickstart/training_plain_format.sh
-COPY quickstart/image_recognition/pytorch/resnet50v1_5/training/gpu/ddp_training_plain_format_nchw.sh quickstart/ddp_training_plain_format_nchw.sh
+RUN python -m pip install pillow
+
+COPY models_v2/pytorch/resnet50v1_5/training/gpu .
+COPY models_v2/common common
+
+COPY LICENSE licenses/LICENSE
+COPY third_party licenses/third_party
+
+USER $USER
+
+ENV LD_LIBRARY_PATH=/opt/intel/oneapi/ccl/2021.11/lib/:/opt/intel/oneapi/mpi/2021.11/opt/mpi/libfabric/lib:/opt/intel/oneapi/mpi/2021.11/lib:$LD_LIBRARY_PATH
+ENV LIBRARY_PATH=/opt/intel/oneapi/mpi/2021.11/lib:/opt/intel/oneapi/ccl/2021.11/lib/
+ENV PATH=/opt/intel/oneapi/mpi/2021.11/opt/mpi/libfabric/bin:/opt/intel/oneapi/mpi/2021.11/bin:$PATH
+ENV CCL_ROOT=/opt/intel/oneapi/ccl/2021.11
+ENV I_MPI_ROOT=/opt/intel/oneapi/mpi/2021.11
+ENV FI_PROVIDER_PATH=/opt/intel/oneapi/mpi/2021.11/opt/mpi/libfabric/lib/prov:/usr/lib/x86_64-linux-gnu/libfabric
 
 COPY LICENSE licenses/LICENSE
 COPY third_party licenses/third_party
diff --git a/docker/max-gpu/pytorch-resnet50v1-5-training/tests.yaml b/docker/max-gpu/pytorch-resnet50v1-5-training/tests.yaml
new file mode 100644
index 000000000..79b6bdb09
--- /dev/null
+++ b/docker/max-gpu/pytorch-resnet50v1-5-training/tests.yaml
@@ -0,0 +1,60 @@
+bf16-training:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-pytorch-max-gpu-resnet50v1-5-training
+  cmd: bash run_model.sh
+  ipc: host
+  device: /dev/dri
+  env:
+    PRECISION: BF16
+    NUM_ITERATIONS: '20'
+    BATCH_SIZE: '256'
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /nfs/pdx/disks/mlp_pytorch_dataset/Imagenet_FullData
+    MULTI_TILE: 'True'
+    PLATFORM: 'Max'
+  volumes:
+    - src: /tmp
+      dst: /tmp
+    - src: /dev/dri
+      dst: /dev/dri
+    - src: /nfs/pdx/disks/mlp_pytorch_dataset/Imagenet_FullData
+      dst: /nfs/pdx/disks/mlp_pytorch_dataset/Imagenet_FullData
+fp32-training:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-pytorch-max-gpu-resnet50v1-5-training
+  cmd: bash run_model.sh
+  ipc: host
+  device: /dev/dri
+  env:
+    PRECISION: FP32
+    NUM_ITERATIONS: '20'
+    BATCH_SIZE: '256'
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /nfs/pdx/disks/mlp_pytorch_dataset/Imagenet_FullData
+    MULTI_TILE: 'True'
+    PLATFORM: 'Max'
+  volumes:
+    - src: /tmp
+      dst: /tmp
+    - src: /dev/dri
+      dst: /dev/dri
+    - src: /nfs/pdx/disks/mlp_pytorch_dataset/Imagenet_FullData
+      dst: /nfs/pdx/disks/mlp_pytorch_dataset/Imagenet_FullData
+tf32-training:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-pytorch-max-gpu-resnet50v1-5-training
+  cmd: bash run_model.sh
+  ipc: host
+  device: /dev/dri
+  env:
+    PRECISION: TF32
+    NUM_ITERATIONS: '20'
+    BATCH_SIZE: '256'
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /nfs/pdx/disks/mlp_pytorch_dataset/Imagenet_FullData
+    MULTI_TILE: 'True'
+    PLATFORM: 'Max'
+  volumes:
+    - src: /tmp
+      dst: /tmp
+    - src: /dev/dri
+      dst: /dev/dri
+    - src: /nfs/pdx/disks/mlp_pytorch_dataset/Imagenet_FullData
+      dst: /nfs/pdx/disks/mlp_pytorch_dataset/Imagenet_FullData
diff --git a/docker/max-gpu/pytorch-stable-diffusion-inference/pytorch-max-series-stable-diffusion-inference.Dockerfile b/docker/max-gpu/pytorch-stable-diffusion-inference/pytorch-max-series-stable-diffusion-inference.Dockerfile
new file mode 100644
index 000000000..3c3b49e81
--- /dev/null
+++ b/docker/max-gpu/pytorch-stable-diffusion-inference/pytorch-max-series-stable-diffusion-inference.Dockerfile
@@ -0,0 +1,54 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+#
+# THIS IS A GENERATED DOCKERFILE.
+#
+# This file was assembled from multiple pieces, whose use is documented
+# throughout. Please refer to the TensorFlow dockerfiles documentation
+# for more information.
+
+ARG PYT_BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG PYT_BASE_TAG="2.1.10-xpu"
+
+FROM ${PYT_BASE_IMAGE}:${PYT_BASE_TAG}
+
+USER root
+
+WORKDIR /workspace/pytorch-max-series-stable-diffusion-inference/models
+
+ENV DEBIAN_FRONTEND=noninteractive
+ARG PY_VERSION=3.10
+
+RUN apt-get update && \
+    apt-get install -y --no-install-recommends \
+    build-essential \
+    python${PY_VERSION}-dev && \
+    rm -rf /var/lib/apt/lists/*
+
+RUN python -m pip install datasets \
+            torchmetrics \
+            diffusers \
+            transformers \
+            accelerate \
+            pytorch-fid \
+            scipy==1.9.1
+
+COPY models_v2/pytorch/stable_diffusion/inference/gpu .
+COPY models_v2/common common
+
+COPY LICENSE license/LICENSE
+COPY third_party license/third_party
+
+USER $USER
diff --git a/docker/max-gpu/pytorch-stable-diffusion-inference/tests.yaml b/docker/max-gpu/pytorch-stable-diffusion-inference/tests.yaml
new file mode 100644
index 000000000..fbed73058
--- /dev/null
+++ b/docker/max-gpu/pytorch-stable-diffusion-inference/tests.yaml
@@ -0,0 +1,11 @@
+fp16-online-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-generative-ai-pytorch-max-gpu-stable-diffusion-inference
+  cmd: bash run_model.sh
+  ipc: host
+  device: /dev/dri
+  env:
+    PRECISION: fp16
+    BATCH_SIZE: '1'
+    MULTI_TILE: 'True'
+    OUTPUT_DIR: /tmp
+    PLATFORM: 'Max'
diff --git a/docker/max-gpu/tf-3d-unet-training/tests.yaml b/docker/max-gpu/tf-3d-unet-training/tests.yaml
new file mode 100644
index 000000000..55955461e
--- /dev/null
+++ b/docker/max-gpu/tf-3d-unet-training/tests.yaml
@@ -0,0 +1,35 @@
+single-tile-training:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-segmentation-tf-max-gpu-3d-unet-training
+  cmd: bash run_model.sh
+  ipc: host
+  device: /dev/dri
+  env:
+    PRECISION: bfloat16
+    BATCH_SIZE: '1'
+    MULTI_TILE: 'False'
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /local_dateset/datasets/3dunet/preprocessed
+  volumes:
+    - src: /local_dateset/datasets/3dunet/preprocessed
+      dst: /local_dateset/datasets/3dunet/preprocessed
+    - src: /dev/dri/by-path
+      dst: /dev/dri/by-path
+    - src: /tmp
+      dst: /tmp
+multi-tile-training:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-segmentation-tf-max-gpu-3d-unet-training
+  cmd: bash run_model.sh
+  device: /dev/dri
+  env:
+    PRECISION: bfloat16
+    BATCH_SIZE: '1'
+    MULTI_TILE: 'True'
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /local_dateset/datasets/3dunet/preprocessed
+  volumes:
+    - src: /local_dateset/datasets/3dunet/preprocessed
+      dst: /local_dateset/datasets/3dunet/preprocessed
+    - src: /dev/dri/by-path
+      dst: /dev/dri/by-path
+    - src: /tmp
+      dst: /tmp
diff --git a/docker/max-gpu/tf-3d-unet-training/tf-max-series-3d-unet-training.Dockerfile b/docker/max-gpu/tf-3d-unet-training/tf-max-series-3d-unet-training.Dockerfile
new file mode 100644
index 000000000..62ce7eb97
--- /dev/null
+++ b/docker/max-gpu/tf-3d-unet-training/tf-max-series-3d-unet-training.Dockerfile
@@ -0,0 +1,68 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+#
+# THIS IS A GENERATED DOCKERFILE.
+#
+# This file was assembled from multiple pieces, whose use is documented
+# throughout. Please refer to the TensorFlow dockerfiles documentation
+# for more information.
+
+ARG TF_BASE_IMAGE="intel/intel-extension-for-tensorflow"
+ARG TF_BASE_TAG="2.14.0.1-xpu"
+
+FROM ${TF_BASE_IMAGE}:${TF_BASE_TAG}
+
+RUN apt-get update && \
+    DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends curl
+
+RUN apt-get update && \
+    DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
+    ca-certificates \
+    intel-oneapi-mpi-devel=2021.11.0-49493  \
+    intel-oneapi-ccl=2021.11.2-5 \
+    && \
+    rm -rf /var/lib/apt/lists/*
+
+WORKDIR /workspace/tf-max-series-3d-unet-training/models
+
+RUN python -m pip install tfa-nightly \
+                   git+https://github.com/NVIDIA/dllogger.git 
+
+COPY models_v2/tensorflow/3d_unet/training/gpu .
+
+RUN mkdir -p 3d_unet && \
+    cd 3d_unet && \
+    git clone https://github.com/NVIDIA/DeepLearningExamples.git && \
+    cd DeepLearningExamples && \
+    git checkout 88eb3cff2f03dad85035621d041e23a14345999e && \
+    cd TensorFlow/Segmentation/UNet_3D_Medical && \
+    git apply /workspace/tf-max-series-3d-unet-training/models/3dunet_itex.patch
+
+RUN mkdir -p 3d_unet_hvd && \
+    cd 3d_unet_hvd && \
+    git clone https://github.com/NVIDIA/DeepLearningExamples.git && \
+    cd DeepLearningExamples && \
+    git checkout 88eb3cff2f03dad85035621d041e23a14345999e && \
+    cd TensorFlow/Segmentation/UNet_3D_Medical && \
+    git apply /workspace/tf-max-series-3d-unet-training/models/3dunet_itex_with_horovod.patch
+
+ENV LD_LIBRARY_PATH=/opt/intel/oneapi/mpi/2021.11/opt/mpi/libfabric/lib:/opt/intel/oneapi/mpi/2021.11/lib:/opt/intel/oneapi/ccl/2021.11/lib/:$LD_LIBRARY_PATH
+ENV PATH=/opt/intel/oneapi/mpi/2021.11/opt/mpi/libfabric/bin:/opt/intel/oneapi/mpi/2021.11/bin:$PATH
+ENV CCL_ROOT=/opt/intel/oneapi/ccl/2021.11
+ENV I_MPI_ROOT=/opt/intel/oneapi/mpi/2021.11
+ENV FI_PROVIDER_PATH=/opt/intel/oneapi/mpi/2021.11/opt/mpi/libfabric/lib/prov:/usr/lib/x86_64-linux-gnu/libfabric
+
+COPY LICENSE licenses/LICENSE
+COPY third_party licenses/third_party 
diff --git a/docker/max-gpu/tf-bert-large-training/tests.yaml b/docker/max-gpu/tf-bert-large-training/tests.yaml
new file mode 100644
index 000000000..f4038b04f
--- /dev/null
+++ b/docker/max-gpu/tf-bert-large-training/tests.yaml
@@ -0,0 +1,30 @@
+single-tile-training:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-tf-max-gpu-bert-large-training
+  cmd: bash run_model.sh
+  ipc: host
+  device: /dev/dri
+  env:
+    DATA_DIR: /local_dateset/bert_small_data/bert-large-training/
+    RESULTS_DIR: /tmp
+    DATATYPE: bf16
+    MULTI_TILE: 'False'
+  volumes:
+    - src: /local_dateset/bert_small_data/bert-large-training/
+      dst: /local_dateset/bert_small_data/bert-large-training/
+    - src: /dev/dri/by-path
+      dst: /dev/dri/by-path
+multi-tile-training:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-tf-max-gpu-bert-large-training
+  cmd: bash run_model.sh
+  ipc: host
+  device: /dev/dri
+  env:
+    DATA_DIR: /local_dateset/bert_small_data/bert-large-training/
+    RESULTS_DIR: /tmp
+    DATATYPE: bf16
+    MULTI_TILE: 'True'
+  volumes:
+    - src: /local_dateset/bert_small_data/bert-large-training/
+      dst: /local_dateset/bert_small_data/bert-large-training/
+    - src: /dev/dri/by-path
+      dst: /dev/dri/by-path
diff --git a/docker/max-gpu/tf-bert-large-training/tf-max-series-bert-large-training.Dockerfile b/docker/max-gpu/tf-bert-large-training/tf-max-series-bert-large-training.Dockerfile
index bab2ae609..dfdf0ea6e 100644
--- a/docker/max-gpu/tf-bert-large-training/tf-max-series-bert-large-training.Dockerfile
+++ b/docker/max-gpu/tf-bert-large-training/tf-max-series-bert-large-training.Dockerfile
@@ -19,33 +19,49 @@
 # throughout. Please refer to the TensorFlow dockerfiles documentation
 # for more information.
 
-ARG MAX_TF_BASE_IMAGE="intel/intel-extension-for-tensorflow"
-ARG MAX_TF_BASE_TAG="gpu-horovod"
+ARG TF_BASE_IMAGE="intel/intel-extension-for-tensorflow"
+ARG TF_BASE_TAG="2.14.0.1-xpu"
 
-FROM ${MAX_TF_BASE_IMAGE}:${MAX_TF_BASE_TAG}
+FROM ${TF_BASE_IMAGE}:${TF_BASE_TAG}
 
-RUN curl -fsSL https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS-2023.PUB | apt-key add -
-RUN echo "deb [trusted=yes] https://apt.repos.intel.com/oneapi all main " > /etc/apt/sources.list.d/oneAPI.list
+RUN apt-get update && \
+    DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends curl
 
 RUN apt-get update && \
   DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
     ca-certificates \
-    intel-oneapi-mpi-devel=2021.9.0-43482 \
-    intel-oneapi-ccl=2021.9.0-43543 \
+    intel-oneapi-mpi-devel=2021.11.0-49493  \
+    intel-oneapi-ccl=2021.11.2-5 \
     && \
   rm -rf /var/lib/apt/lists/*
 
-WORKDIR /workspace/tf-max-series-bert-large-training 
+WORKDIR /workspace/tf-max-series-bert-large-training/models
+
+COPY models_v2/tensorflow/bert_large/training/gpu .
+
+RUN git clone https://github.com/titipata/pubmed_parser && \
+    pip install ./pubmed_parser
+
+RUN python -m pip install git+https://github.com/NVIDIA/dllogger \
+              requests \
+              tqdm \
+              sentencepiece \
+              tensorflow_hub \
+              wget \
+              progressbar \
+              nltk \
+              tensorflow-addons 
 
-COPY benchmarks benchmarks
-COPY models models
-COPY quickstart/common quickstart/common
-COPY quickstart/language_modeling/tensorflow/bert_large/training/gpu quickstart/language_modeling/tensorflow/bert_large/training/gpu 
-COPY quickstart/language_modeling/tensorflow/bert_large/training/gpu/bfloat16_training.sh quickstart/bfloat16_training.sh
-COPY quickstart/language_modeling/tensorflow/bert_large/training/gpu/bfloat16_training_hvd.sh quickstart/bfloat16_training_hvd.sh
-COPY quickstart/language_modeling/tensorflow/bert_large/training/gpu/README.md README.md
+RUN git clone https://github.com/NVIDIA/DeepLearningExamples.git && \
+    cd DeepLearningExamples/TensorFlow2/LanguageModeling/BERT && \
+    cp /workspace/tf-max-series-bert-large-training/models/bert-large-itex-bf16.patch . && \
+    git apply bert-large-itex-bf16.patch 
 
-RUN git apply quickstart/language_modeling/tensorflow/bert_large/training/gpu/hvs_support.patch
+ENV LD_LIBRARY_PATH=/opt/intel/oneapi/mpi/2021.11/opt/mpi/libfabric/lib:/opt/intel/oneapi/mpi/2021.11/lib:/opt/intel/oneapi/ccl/2021.11/lib/:$LD_LIBRARY_PATH
+ENV PATH=/opt/intel/oneapi/mpi/2021.11/opt/mpi/libfabric/bin:/opt/intel/oneapi/mpi/2021.11/bin:$PATH
+ENV CCL_ROOT=/opt/intel/oneapi/ccl/2021.11
+ENV I_MPI_ROOT=/opt/intel/oneapi/mpi/2021.11
+ENV FI_PROVIDER_PATH=/opt/intel/oneapi/mpi/2021.11/opt/mpi/libfabric/lib/prov:/usr/lib/x86_64-linux-gnu/libfabric
 
 COPY LICENSE licenses/LICENSE
 COPY third_party licenses/third_party
diff --git a/docker/max-gpu/tf-maskrcnn-training/tests.yaml b/docker/max-gpu/tf-maskrcnn-training/tests.yaml
new file mode 100644
index 000000000..9311d954e
--- /dev/null
+++ b/docker/max-gpu/tf-maskrcnn-training/tests.yaml
@@ -0,0 +1,39 @@
+single-tile-training:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-segmentation-tf-max-gpu-maskrcnn-training
+  ipc: host
+  cmd: mpirun -np 1 -prepend-rank -ppn 1 bash run_model.sh
+  device: /dev/dri
+  env:
+    PRECISION: bfloat16
+    EPOCHS: '1'
+    STEPS_PER_EPOCH: '20'
+    BATCH_SIZE: '4'
+    MULTI_TILE: 'False'
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /tf_dataset/dataset/coco_dataset/COCO2017_training_data/
+  volumes:
+    - src: /tf_dataset/dataset/coco_dataset/COCO2017_training_data/
+      dst: /tf_dataset/dataset/coco_dataset/COCO2017_training_data/
+    - src: /dev/dri/by-path
+      dst: /dev/dri/by-path
+    - src: /tmp
+      dst: /tmp
+multi-tile-training:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-segmentation-tf-max-gpu-maskrcnn-training
+  cmd: bash run_model.sh
+  device: /dev/dri
+  env:
+    PRECISION: bfloat16
+    EPOCHS: '1'
+    STEPS_PER_EPOCH: '20'
+    BATCH_SIZE: '4'
+    MULTI_TILE: 'True'
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /tf_dataset/dataset/coco_dataset/COCO2017_training_data/
+  volumes:
+    - src: /tf_dataset/dataset/coco_dataset/COCO2017_training_data/
+      dst: /tf_dataset/dataset/coco_dataset/COCO2017_training_data/
+    - src: /dev/dri
+      dst: /dev/dri
+    - src: /tmp
+      dst: /tmp
diff --git a/docker/max-gpu/tf-maskrcnn-training/tf-max-series-maskrcnn-training.Dockerfile b/docker/max-gpu/tf-maskrcnn-training/tf-max-series-maskrcnn-training.Dockerfile
new file mode 100644
index 000000000..043b17143
--- /dev/null
+++ b/docker/max-gpu/tf-maskrcnn-training/tf-max-series-maskrcnn-training.Dockerfile
@@ -0,0 +1,58 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+#
+# THIS IS A GENERATED DOCKERFILE.
+#
+# This file was assembled from multiple pieces, whose use is documented
+# throughout. Please refer to the TensorFlow dockerfiles documentation
+# for more information.
+
+ARG TF_BASE_IMAGE="intel/intel-extension-for-tensorflow"
+ARG TF_BASE_TAG="2.14.0.1-xpu"
+
+FROM ${TF_BASE_IMAGE}:${TF_BASE_TAG}
+
+RUN apt-get update && \
+    DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
+    ca-certificates \
+    curl \
+    intel-oneapi-mpi-devel=2021.11.0-49493  \
+    intel-oneapi-ccl=2021.11.2-5 \
+    && \
+    rm -rf /var/lib/apt/lists/*
+
+WORKDIR /workspace/tf-max-series-maskrcnn-inference-training/models
+
+COPY models_v2/tensorflow/maskrcnn/training/gpu .
+
+RUN python -m pip install opencv-python-headless \
+                          pybind11 \
+                          pycocotools \
+                          -e "git+https://github.com/NVIDIA/dllogger#egg=dllogger" 
+
+RUN git clone https://github.com/NVIDIA/DeepLearningExamples.git &&  \
+    cd DeepLearningExamples/TensorFlow2/Segmentation/MaskRCNN && \
+    git checkout c481324031ecf0f70f8939516c02e16cac60446d && \
+    git apply /workspace/tf-max-series-maskrcnn-inference-training/models/EnableBF16.patch
+
+ENV LD_LIBRARY_PATH=/opt/intel/oneapi/mpi/2021.11/opt/mpi/libfabric/lib:/opt/intel/oneapi/mpi/2021.11/lib:/opt/intel/oneapi/ccl/2021.11/lib/:$LD_LIBRARY_PATH
+ENV LIBRARY_PATH=/opt/intel/oneapi/mpi/2021.11/lib:/opt/intel/oneapi/ccl/2021.11/lib/
+ENV PATH=/opt/intel/oneapi/mpi/2021.11/opt/mpi/libfabric/bin:/opt/intel/oneapi/mpi/2021.11/bin:$PATH
+ENV CCL_ROOT=/opt/intel/oneapi/ccl/2021.11
+ENV I_MPI_ROOT=/opt/intel/oneapi/mpi/2021.11
+ENV FI_PROVIDER_PATH=/opt/intel/oneapi/mpi/2021.11/opt/mpi/libfabric/lib/prov:/usr/lib/x86_64-linux-gnu/libfabric
+
+COPY LICENSE licenses/LICENSE
+COPY third_party licenses/third_party 
diff --git a/docker/max-gpu/tf-resnet50v1-5-training/tests.yaml b/docker/max-gpu/tf-resnet50v1-5-training/tests.yaml
new file mode 100644
index 000000000..bc46c6032
--- /dev/null
+++ b/docker/max-gpu/tf-resnet50v1-5-training/tests.yaml
@@ -0,0 +1,49 @@
+single-tile-real-data-training:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-max-gpu-resnet50v1-5-training
+  cmd: bash run_model.sh
+  ipc: host
+  device: /dev/dri
+  env:
+    CONFIG_FILE: /workspace/tf-max-series-resnet50v1-5-training/models/configure/itex_bf16.yaml
+    MULTI_TILE: 'False'
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /tf_dataset/dataset/TF_Imagenet_FullData
+  volumes:
+    - src: /tf_dataset/dataset/TF_Imagenet_FullData
+      dst: /tf_dataset/dataset/TF_Imagenet_FullData
+single-tile-dummy-data-training:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-max-gpu-resnet50v1-5-training
+  cmd: bash run_model.sh
+  ipc: host
+  device: /dev/dri
+  env:
+    CONFIG_FILE: /workspace/tf-max-series-resnet50v1-5-training/models/configure/itex_dummy_bf16.yaml
+    MULTI_TILE: 'False'
+    OUTPUT_DIR: /tmp
+multi-tile-real-data-training:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-max-gpu-resnet50v1-5-training
+  cmd: bash run_model.sh
+  ipc: host
+  device: /dev/dri
+  env:
+    CONFIG_FILE: /workspace/tf-max-series-resnet50v1-5-training/models/hvd_configure/itex_bf16_lars.yaml
+    MULTI_TILE: 'True'
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /tf_dataset/dataset/TF_Imagenet_FullData
+  volumes:
+    - src: /tf_dataset/dataset/TF_Imagenet_FullData
+      dst: /tf_dataset/dataset/TF_Imagenet_FullData
+    - src: /dev/dri/by-path
+      dst: /dev/dri/by-path
+multi-tile-dummy-data-training:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-max-gpu-resnet50v1-5-training
+  cmd: bash run_model.sh
+  ipc: host
+  device: /dev/dri
+  env:
+    CONFIG_FILE: /workspace/tf-max-series-resnet50v1-5-training/models/hvd_configure/itex_dummy_bf16_lars.yaml
+    MULTI_TILE: 'True'
+    OUTPUT_DIR: /tmp
+  volumes:
+    - src: /dev/dri/by-path
+      dst: /dev/dri/by-path
diff --git a/docker/max-gpu/tf-resnet50v1-5-training/tf-max-series-resnet50v1-5-training.Dockerfile b/docker/max-gpu/tf-resnet50v1-5-training/tf-max-series-resnet50v1-5-training.Dockerfile
index 738879d3c..320b93b00 100644
--- a/docker/max-gpu/tf-resnet50v1-5-training/tf-max-series-resnet50v1-5-training.Dockerfile
+++ b/docker/max-gpu/tf-resnet50v1-5-training/tf-max-series-resnet50v1-5-training.Dockerfile
@@ -19,30 +19,50 @@
 # throughout. Please refer to the TensorFlow dockerfiles documentation
 # for more information.
 
-ARG MAX_PYT_BASE_IMAGE="intel/intel-extension-for-tensorflow"
-ARG MAX_PYT_BASE_TAG="gpu-horovod"
+ARG TF_BASE_IMAGE="intel/intel-extension-for-tensorflow"
+ARG TF_BASE_TAG="2.14.0.1-xpu"
 
-FROM ${MAX_PYT_BASE_IMAGE}:${MAX_PYT_BASE_TAG}
+FROM ${TF_BASE_IMAGE}:${TF_BASE_TAG}
 
-RUN curl -fsSL https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS-2023.PUB | apt-key add -
-RUN echo "deb [trusted=yes] https://apt.repos.intel.com/oneapi all main " > /etc/apt/sources.list.d/oneAPI.list
+RUN apt-get update && \
+    DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends curl bc
 
 RUN apt-get update && \
-  DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
+    DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
     ca-certificates \
-    intel-oneapi-mpi-devel=2021.9.0-43482  \
-    intel-oneapi-ccl=2021.9.0-43543 \
+    intel-oneapi-mpi-devel=2021.11.0-49493  \
+    intel-oneapi-ccl=2021.11.2-5 \
     && \
-  rm -rf /var/lib/apt/lists/*
+    rm -rf /var/lib/apt/lists/*
+
+WORKDIR /workspace/tf-max-series-resnet50v1-5-training/models
+
+COPY models_v2/tensorflow/resnet50v1_5/training/gpu .
+
+RUN python -m pip install gin gin-config \
+    tfa-nightly \
+    tensorflow-model-optimization \
+    tensorflow-datasets \
+    protobuf==3.20.3 \
+    pyyaml 
+
+RUN mkdir -p resnet50 && \
+    cd resnet50 && \
+    git clone -b v2.8.0 https://github.com/tensorflow/models.git tensorflow-models && \
+    cd tensorflow-models && \
+    git apply /workspace/tf-max-series-resnet50v1-5-training/models/resnet50.patch 
 
-WORKDIR /workspace/tf-max-series-resnet50v1-5-training
+RUN mkdir -p resnet50_hvd && \
+    cd resnet50_hvd && \
+    git clone -b v2.8.0 https://github.com/tensorflow/models.git tensorflow-models && \
+    cd tensorflow-models && \
+    git apply /workspace/tf-max-series-resnet50v1-5-training/models/hvd_support.patch
 
-COPY benchmarks benchmarks
-COPY models models
-COPY quickstart/common quickstart/common
-COPY quickstart/image_recognition/tensorflow/resnet50v1_5/training/gpu/bfloat16_training_hvd.sh quickstart/bfloat16_training_hvd.sh
-COPY quickstart/image_recognition/tensorflow/resnet50v1_5/training/gpu/bfloat16_training_full.sh quickstart/bfloat16_training_full.sh
-COPY quickstart/image_recognition/tensorflow/resnet50v1_5/training/gpu/README.md README.md
+ENV LD_LIBRARY_PATH=/opt/intel/oneapi/mpi/2021.11/opt/mpi/libfabric/lib:/opt/intel/oneapi/mpi/2021.11/lib:/opt/intel/oneapi/ccl/2021.11/lib/:$LD_LIBRARY_PATH
+ENV PATH=/opt/intel/oneapi/mpi/2021.11/opt/mpi/libfabric/bin:/opt/intel/oneapi/mpi/2021.11/bin:$PATH
+ENV CCL_ROOT=/opt/intel/oneapi/ccl/2021.11
+ENV I_MPI_ROOT=/opt/intel/oneapi/mpi/2021.11
+ENV FI_PROVIDER_PATH=/opt/intel/oneapi/mpi/2021.11/opt/mpi/libfabric/lib/prov:/usr/lib/x86_64-linux-gnu/libfabric
 
 COPY LICENSE licenses/LICENSE
 COPY third_party licenses/third_party 
diff --git a/docker/pyt-cpu/bert-large-inference/pytorch-bert-large-inference.Dockerfile-centos b/docker/pyt-cpu/bert-large-inference/pytorch-bert-large-inference.Dockerfile-centos
new file mode 100644
index 000000000..123fa9fb1
--- /dev/null
+++ b/docker/pyt-cpu/bert-large-inference/pytorch-bert-large-inference.Dockerfile-centos
@@ -0,0 +1,76 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+ARG BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG BASE_TAG="2.0.0-pip-base"
+
+FROM ${BASE_IMAGE}:${BASE_TAG} AS intel-extension-for-pytorch
+
+WORKDIR /workspace/pytorch-bert-large-inference
+
+RUN yum update -y && \
+    yum install -y \
+    ca-certificates \
+    git \
+    cmake>=3.19.6 \
+    make \
+    autoconf \
+    bzip2 \
+    tar
+
+RUN git clone  https://github.com/jemalloc/jemalloc.git && \
+    cd jemalloc && \
+    git checkout c8209150f9d219a137412b06431c9d52839c7272 && \
+    ./autogen.sh && \
+    ./configure --prefix=/workspace/lib/jemalloc && \
+    make && \
+    make install 
+
+RUN wget https://github.com/gperftools/gperftools/releases/download/gperftools-2.7.90/gperftools-2.7.90.tar.gz && \
+    tar -xzf gperftools-2.7.90.tar.gz && \
+    cd gperftools-2.7.90 && \
+    mkdir -p /workspace/lib/ && \
+    ./configure --prefix=/workspace/lib/tcmalloc/ && \
+    make && \
+    make install
+
+COPY models/language_modeling/pytorch/common/enable_ipex_for_transformers.diff models/language_modeling/pytorch/common/enable_ipex_for_transformers.diff
+COPY quickstart/language_modeling/pytorch/bert_large/inference/cpu/configure.json quickstart/configure.json
+COPY quickstart/language_modeling/pytorch/bert_large/inference/cpu/run_accuracy.sh quickstart/run_accuracy.sh
+COPY quickstart/language_modeling/pytorch/bert_large/inference/cpu/run_calibration.sh quickstart/run_calibration.sh
+COPY quickstart/language_modeling/pytorch/bert_large/inference/cpu/run_multi_instance_realtime.sh quickstart/run_multi_instance_realtime.sh
+COPY quickstart/language_modeling/pytorch/bert_large/inference/cpu/run_multi_instance_throughput.sh quickstart/run_multi_instance_throughput.sh
+
+ RUN cd quickstart && \ 
+     git clone https://github.com/huggingface/transformers.git && \
+     cd transformers && \
+     git checkout v4.28.1 && \
+     git apply /workspace/pytorch-bert-large-inference/models/language_modeling/pytorch/common/enable_ipex_for_transformers.diff && \
+     pip install -e ./ && \
+     cd -
+
+RUN pip install packaging intel-openmp tensorboardX
+
+ENV LD_PRELOAD="/workspace/lib/jemalloc/lib/libjemalloc.so":"/workspace/lib/tcmalloc/lib/libtcmalloc.so":"/usr/local/lib/libiomp5.so":$LD_PRELOAD 
+ENV MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+
+RUN yum update -y && \
+    yum install -y \
+    numactl \
+    mesa-libGL && \
+    yum clean all
+
+COPY LICENSE licenses/LICENSE
+COPY third_party licenses/third_party
diff --git a/docker/pyt-cpu/bert-large-inference/pytorch-bert-large-inference.Dockerfile-ubuntu b/docker/pyt-cpu/bert-large-inference/pytorch-bert-large-inference.Dockerfile-ubuntu
new file mode 100644
index 000000000..ab8f06cb1
--- /dev/null
+++ b/docker/pyt-cpu/bert-large-inference/pytorch-bert-large-inference.Dockerfile-ubuntu
@@ -0,0 +1,80 @@
+# Copyright (c) 2020-2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+ARG BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG BASE_TAG="2.0.0-pip-base"
+
+FROM ${BASE_IMAGE}:${BASE_TAG} AS intel-extension-for-pytorch
+
+WORKDIR /workspace/pytorch-bert-large-inference
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+    build-essential \
+    ca-certificates \
+    git \
+    wget \
+    make \
+    cmake \
+    g++ \
+    gcc \
+    autoconf \
+    bzip2 \
+    tar
+
+RUN git clone  https://github.com/jemalloc/jemalloc.git && \
+    cd jemalloc && \
+    git checkout c8209150f9d219a137412b06431c9d52839c7272 && \
+    ./autogen.sh && \
+    ./configure --prefix=/workspace/lib/jemalloc && \
+    make && \
+    make install 
+
+RUN wget https://github.com/gperftools/gperftools/releases/download/gperftools-2.7.90/gperftools-2.7.90.tar.gz && \
+    tar -xzf gperftools-2.7.90.tar.gz && \
+    cd gperftools-2.7.90 && \
+    mkdir -p /workspace/lib/ && \
+    ./configure --prefix=/workspace/lib/tcmalloc/ && \
+    make && \
+    make install
+
+COPY models/language_modeling/pytorch/common/enable_ipex_for_transformers.diff models/language_modeling/pytorch/common/enable_ipex_for_transformers.diff
+COPY quickstart/language_modeling/pytorch/bert_large/inference/cpu/configure.json quickstart/configure.json
+COPY quickstart/language_modeling/pytorch/bert_large/inference/cpu/run_accuracy.sh quickstart/run_accuracy.sh
+COPY quickstart/language_modeling/pytorch/bert_large/inference/cpu/run_calibration.sh quickstart/run_calibration.sh
+COPY quickstart/language_modeling/pytorch/bert_large/inference/cpu/run_multi_instance_realtime.sh quickstart/run_multi_instance_realtime.sh
+COPY quickstart/language_modeling/pytorch/bert_large/inference/cpu/run_multi_instance_throughput.sh quickstart/run_multi_instance_throughput.sh
+
+RUN cd quickstart && \
+    git clone https://github.com/huggingface/transformers.git && \
+    cd transformers && \
+    git checkout v4.28.1 && \
+    git apply /workspace/pytorch-bert-large-inference/models/language_modeling/pytorch/common/enable_ipex_for_transformers.diff && \
+    pip install -e ./ && \
+    pip install tensorboard && \
+    pip install intel-openmp 
+
+RUN pip install packaging intel-openmp tensorboardX
+
+ENV LD_PRELOAD="/workspace/lib/jemalloc/lib/libjemalloc.so":"/workspace/lib/tcmalloc/lib/libtcmalloc.so":"/usr/local/lib/libiomp5.so":$LD_PRELOAD 
+ENV MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+    numactl \
+    libegl1-mesa
+
+COPY LICENSE licenses/LICENSE
+COPY third_party licenses/third_party
diff --git a/docker/pyt-cpu/bert-large-inference/tests.yaml b/docker/pyt-cpu/bert-large-inference/tests.yaml
new file mode 100644
index 000000000..d44eb5ddf
--- /dev/null
+++ b/docker/pyt-cpu/bert-large-inference/tests.yaml
@@ -0,0 +1,192 @@
+fp32-online-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-language-modeling-bert-large-inference
+  cmd: quickstart/run_multi_instance_realtime.sh fp32
+  shm-size: 8G
+  env:
+    PRECISION: fp32
+    EVAL_DATA_FILE: /tf_dataset/dataset/data-bert-squad/squad-data/mini-dev-v1.1.json
+    INT8_CONFIG: /workspace/pytorch-bert-large-inference/quickstart/configure.json
+    OUTPUT_DIR: /tmp
+    FINETUNED_MODEL: /pytorch/pytorch-bert-squad-model
+    EVAL_SCRIPT: /workspace/pytorch-bert-large-inference/quickstart/transformers/examples/legacy/question-answering/run_squad.py
+  volumes:
+    - src: /tf_dataset/dataset/data-bert-squad/squad-data/mini-dev-v1.1.json
+      dst: /tf_dataset/dataset/data-bert-squad/squad-data/mini-dev-v1.1.json
+    - src: /pytorch/pytorch-bert-squad-model
+      dst: /pytorch/pytorch-bert-squad-model
+fp32-throughput-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-language-modeling-bert-large-inference
+  cmd: quickstart/run_multi_instance_throughput.sh fp32
+  shm-size: 8G
+  env:
+    PRECISION: fp32
+    EVAL_DATA_FILE: /tf_dataset/dataset/data-bert-squad/squad-data/mini-dev-v1.1.json
+    INT8_CONFIG: /workspace/pytorch-bert-large-inference/quickstart/configure.json
+    OUTPUT_DIR: /tmp
+    FINETUNED_MODEL: /pytorch/pytorch-bert-squad-model
+    EVAL_SCRIPT: /workspace/pytorch-bert-large-inference/quickstart/transformers/examples/legacy/question-answering/run_squad.py
+  volumes:
+    - src: /tf_dataset/dataset/data-bert-squad/squad-data/mini-dev-v1.1.json
+      dst: /tf_dataset/dataset/data-bert-squad/squad-data/mini-dev-v1.1.json
+    - src: /pytorch/pytorch-bert-squad-model
+      dst: /pytorch/pytorch-bert-squad-model
+bf32-online-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-language-modeling-bert-large-inference
+  cmd: quickstart/run_multi_instance_realtime.sh bf32
+  shm-size: 8G
+  env:
+    PRECISION: bf32
+    EVAL_DATA_FILE: /tf_dataset/dataset/data-bert-squad/squad-data/mini-dev-v1.1.json
+    INT8_CONFIG: /workspace/pytorch-bert-large-inference/quickstart/configure.json
+    OUTPUT_DIR: /tmp
+    FINETUNED_MODEL: /pytorch/pytorch-bert-squad-model
+    EVAL_SCRIPT: /workspace/pytorch-bert-large-inference/quickstart/transformers/examples/legacy/question-answering/run_squad.py
+  volumes:
+    - src: /tf_dataset/dataset/data-bert-squad/squad-data/mini-dev-v1.1.json
+      dst: /tf_dataset/dataset/data-bert-squad/squad-data/mini-dev-v1.1.json
+    - src: /pytorch/pytorch-bert-squad-model
+      dst: /pytorch/pytorch-bert-squad-model
+bf32-throughput-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-language-modeling-bert-large-inference
+  cmd: quickstart/run_multi_instance_throughput.sh bf32
+  shm-size: 8G
+  env:
+    PRECISION: bf32
+    EVAL_DATA_FILE: /tf_dataset/dataset/data-bert-squad/squad-data/mini-dev-v1.1.json
+    INT8_CONFIG: /workspace/pytorch-bert-large-inference/quickstart/configure.json
+    OUTPUT_DIR: /tmp
+    FINETUNED_MODEL: /pytorch/pytorch-bert-squad-model
+    EVAL_SCRIPT: /workspace/pytorch-bert-large-inference/quickstart/transformers/examples/legacy/question-answering/run_squad.py
+  volumes:
+    - src: /tf_dataset/dataset/data-bert-squad/squad-data/mini-dev-v1.1.json
+      dst: /tf_dataset/dataset/data-bert-squad/squad-data/mini-dev-v1.1.json
+    - src: /pytorch/pytorch-bert-squad-model
+      dst: /pytorch/pytorch-bert-squad-model
+int8-online-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-language-modeling-bert-large-inference
+  cmd: quickstart/run_multi_instance_realtime.sh int8
+  shm-size: 8G
+  env:
+    PRECISION: int8
+    EVAL_DATA_FILE: /tf_dataset/dataset/data-bert-squad/squad-data/mini-dev-v1.1.json
+    INT8_CONFIG: /workspace/pytorch-bert-large-inference/quickstart/configure.json
+    OUTPUT_DIR: /tmp
+    FINETUNED_MODEL: /pytorch/pytorch-bert-squad-model
+    EVAL_SCRIPT: /workspace/pytorch-bert-large-inference/quickstart/transformers/examples/legacy/question-answering/run_squad.py
+  volumes:
+    - src: /tf_dataset/dataset/data-bert-squad/squad-data/mini-dev-v1.1.json
+      dst: /tf_dataset/dataset/data-bert-squad/squad-data/mini-dev-v1.1.json
+    - src: /pytorch/pytorch-bert-squad-model
+      dst: /pytorch/pytorch-bert-squad-model
+int8-throughput-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-language-modeling-bert-large-inference
+  cmd: quickstart/run_multi_instance_throughput.sh int8
+  shm-size: 8G
+  env:
+    PRECISION: int8
+    EVAL_DATA_FILE: /tf_dataset/dataset/data-bert-squad/squad-data/mini-dev-v1.1.json
+    INT8_CONFIG: /workspace/pytorch-bert-large-inference/quickstart/configure.json
+    OUTPUT_DIR: /tmp
+    FINETUNED_MODEL: /pytorch/pytorch-bert-squad-model
+    EVAL_SCRIPT: /workspace/pytorch-bert-large-inference/quickstart/transformers/examples/legacy/question-answering/run_squad.py
+  volumes:
+    - src: /tf_dataset/dataset/data-bert-squad/squad-data/mini-dev-v1.1.json
+      dst: /tf_dataset/dataset/data-bert-squad/squad-data/mini-dev-v1.1.json
+    - src: /pytorch/pytorch-bert-squad-model
+      dst: /pytorch/pytorch-bert-squad-model
+bf16-online-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-language-modeling-bert-large-inference
+  cmd: quickstart/run_multi_instance_realtime.sh bf16
+  shm-size: 8G
+  env:
+    PRECISION: bf16
+    EVAL_DATA_FILE: /tf_dataset/dataset/data-bert-squad/squad-data/mini-dev-v1.1.json
+    INT8_CONFIG: /workspace/pytorch-bert-large-inference/quickstart/configure.json
+    OUTPUT_DIR: /tmp
+    FINETUNED_MODEL: /pytorch/pytorch-bert-squad-model
+    EVAL_SCRIPT: /workspace/pytorch-bert-large-inference/quickstart/transformers/examples/legacy/question-answering/run_squad.py
+  volumes:
+    - src: /tf_dataset/dataset/data-bert-squad/squad-data/mini-dev-v1.1.json
+      dst: /tf_dataset/dataset/data-bert-squad/squad-data/mini-dev-v1.1.json
+    - src: /pytorch/pytorch-bert-squad-model
+      dst: /pytorch/pytorch-bert-squad-model
+bf16-throughput-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-language-modeling-bert-large-inference
+  cmd: quickstart/run_multi_instance_throughput.sh bf16
+  shm-size: 8G
+  env:
+    PRECISION: bf16
+    EVAL_DATA_FILE: /tf_dataset/dataset/data-bert-squad/squad-data/mini-dev-v1.1.json
+    INT8_CONFIG: /workspace/pytorch-bert-large-inference/quickstart/configure.json
+    OUTPUT_DIR: /tmp
+    FINETUNED_MODEL: /pytorch/pytorch-bert-squad-model
+    EVAL_SCRIPT: /workspace/pytorch-bert-large-inference/quickstart/transformers/examples/legacy/question-answering/run_squad.py
+  volumes:
+    - src: /tf_dataset/dataset/data-bert-squad/squad-data/mini-dev-v1.1.json
+      dst: /tf_dataset/dataset/data-bert-squad/squad-data/mini-dev-v1.1.json
+    - src: /pytorch/pytorch-bert-squad-model
+      dst: /pytorch/pytorch-bert-squad-model
+avx-int8-online-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-language-modeling-bert-large-inference
+  cmd: quickstart/run_multi_instance_realtime.sh avx-int8
+  shm-size: 8G
+  env:
+    PRECISION: avx-int8
+    EVAL_DATA_FILE: /tf_dataset/dataset/data-bert-squad/squad-data/mini-dev-v1.1.json
+    INT8_CONFIG: /workspace/pytorch-bert-large-inference/quickstart/configure.json
+    OUTPUT_DIR: /tmp
+    FINETUNED_MODEL: /pytorch/pytorch-bert-squad-model
+    EVAL_SCRIPT: /workspace/pytorch-bert-large-inference/quickstart/transformers/examples/legacy/question-answering/run_squad.py
+  volumes:
+    - src: /tf_dataset/dataset/data-bert-squad/squad-data/mini-dev-v1.1.json
+      dst: /tf_dataset/dataset/data-bert-squad/squad-data/mini-dev-v1.1.json
+    - src: /pytorch/pytorch-bert-squad-model
+      dst: /pytorch/pytorch-bert-squad-model
+avx-int8-throughput-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-language-modeling-bert-large-inference
+  cmd: quickstart/run_multi_instance_throughput.sh avx-int8
+  shm-size: 8G
+  env:
+    PRECISION: avx-int8
+    EVAL_DATA_FILE: /tf_dataset/dataset/data-bert-squad/squad-data/mini-dev-v1.1.json
+    INT8_CONFIG: /workspace/pytorch-bert-large-inference/quickstart/configure.json
+    OUTPUT_DIR: /tmp
+    FINETUNED_MODEL: /pytorch/pytorch-bert-squad-model
+    EVAL_SCRIPT: /workspace/pytorch-bert-large-inference/quickstart/transformers/examples/legacy/question-answering/run_squad.py
+  volumes:
+    - src: /tf_dataset/dataset/data-bert-squad/squad-data/mini-dev-v1.1.json
+      dst: /tf_dataset/dataset/data-bert-squad/squad-data/mini-dev-v1.1.json
+    - src: /pytorch/pytorch-bert-squad-model
+      dst: /pytorch/pytorch-bert-squad-model
+avx-fp32-online-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-language-modeling-bert-large-inference
+  cmd: quickstart/run_multi_instance_realtime.sh avx-fp32
+  shm-size: 8G
+  env:
+    PRECISION: avx-fp32
+    EVAL_DATA_FILE: /tf_dataset/dataset/data-bert-squad/squad-data/mini-dev-v1.1.json
+    INT8_CONFIG: /workspace/pytorch-bert-large-inference/quickstart/configure.json
+    OUTPUT_DIR: /tmp
+    FINETUNED_MODEL: /pytorch/pytorch-bert-squad-model
+    EVAL_SCRIPT: /workspace/pytorch-bert-large-inference/quickstart/transformers/examples/legacy/question-answering/run_squad.py
+  volumes:
+    - src: /tf_dataset/dataset/data-bert-squad/squad-data/mini-dev-v1.1.json
+      dst: /tf_dataset/dataset/data-bert-squad/squad-data/mini-dev-v1.1.json
+    - src: /pytorch/pytorch-bert-squad-model
+      dst: /pytorch/pytorch-bert-squad-model
+avx-fp32-throughput-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-language-modeling-bert-large-inference
+  cmd: quickstart/run_multi_instance_throughput.sh avx-fp32
+  shm-size: 8G
+  env:
+    PRECISION: avx-fp32
+    EVAL_DATA_FILE: /tf_dataset/dataset/data-bert-squad/squad-data/mini-dev-v1.1.json
+    INT8_CONFIG: /workspace/pytorch-bert-large-inference/quickstart/configure.json
+    OUTPUT_DIR: /tmp
+    FINETUNED_MODEL: /pytorch/pytorch-bert-squad-model
+    EVAL_SCRIPT: /workspace/pytorch-bert-large-inference/quickstart/transformers/examples/legacy/question-answering/run_squad.py
+  volumes:
+    - src: /tf_dataset/dataset/data-bert-squad/squad-data/mini-dev-v1.1.json
+      dst: /tf_dataset/dataset/data-bert-squad/squad-data/mini-dev-v1.1.json
+    - src: /pytorch/pytorch-bert-squad-model
+      dst: /pytorch/pytorch-bert-squad-model
diff --git a/docker/pyt-cpu/bert-large-training/pytorch-bert-large-training.Dockerfile-centos b/docker/pyt-cpu/bert-large-training/pytorch-bert-large-training.Dockerfile-centos
new file mode 100644
index 000000000..a2025d8b8
--- /dev/null
+++ b/docker/pyt-cpu/bert-large-training/pytorch-bert-large-training.Dockerfile-centos
@@ -0,0 +1,79 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+ARG BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG BASE_TAG="2.0.0-pip-base"
+
+FROM ${BASE_IMAGE}:${BASE_TAG} AS intel-extension-for-pytorch
+
+WORKDIR /workspace/pytorch-bert-large-training
+
+RUN yum update -y && \
+    yum install -y \
+    ca-certificates \
+    git \
+    cmake>=3.19.6 \
+    make \
+    autoconf \
+    bzip2 \
+    tar
+
+RUN wget https://github.com/gperftools/gperftools/releases/download/gperftools-2.7.90/gperftools-2.7.90.tar.gz && \
+    tar -xzf gperftools-2.7.90.tar.gz && \
+    cd gperftools-2.7.90 && \
+    mkdir -p /workspace/lib/ && \
+    ./configure --prefix=/workspace/lib/tcmalloc/ && \
+    make && \
+    make install
+
+RUN git clone https://github.com/jemalloc/jemalloc.git && \
+    cd jemalloc && \
+    git checkout c8209150f9d219a137412b06431c9d52839c7272 && \
+    ./autogen.sh && \
+    ./configure --prefix=/workspace/lib/jemalloc && \
+    make && \ 
+    make install
+
+RUN pip install packaging intel-openmp
+
+ENV LD_PRELOAD="/workspace/lib/jemalloc/lib/libjemalloc.so":"/workspace/lib/tcmalloc/lib/libtcmalloc.so":"/usr/local/lib/libiomp5.so":$LD_PRELOAD 
+ENV MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+
+COPY models/language_modeling/pytorch/bert_large/training models/language_modeling/pytorch/bert_large/training
+COPY models/language_modeling/pytorch/common/enable_ipex_for_transformers.diff models/language_modeling/pytorch/common/enable_ipex_for_transformers.diff
+COPY quickstart/language_modeling/pytorch/bert_large/training/cpu/run_bert_pretrain_phase1.sh quickstart/run_bert_pretrain_phase1.sh
+COPY quickstart/language_modeling/pytorch/bert_large/training/cpu/run_bert_pretrain_phase2.sh quickstart/run_bert_pretrain_phase2.sh
+
+ARG TRANSFORMERS_COMMIT="v4.28.1"
+
+RUN pip install datasets==1.11.0 accelerate tfrecord intel-openmp faiss-cpu pylib-openblas && \
+    git clone https://github.com/huggingface/transformers.git && \
+    cd transformers && \
+    git checkout ${TRANSFORMERS_COMMIT} && \
+    git apply /workspace/pytorch-bert-large-training/models/language_modeling/pytorch/common/enable_ipex_for_transformers.diff && \
+    python -m pip install --upgrade pip && \
+    pip uninstall transformers -y && \
+    pip install -e . && \
+    pip install h5py && \
+    mkdir -p /root/.local
+
+RUN yum update -y && \
+    yum install -y \
+    numactl \
+    mesa-libGL && \
+    yum clean all
+
+COPY LICENSE licenses/LICENSE
+COPY third_party licenses/third_party
diff --git a/docker/pyt-cpu/bert-large-training/pytorch-bert-large-training.Dockerfile-ubuntu b/docker/pyt-cpu/bert-large-training/pytorch-bert-large-training.Dockerfile-ubuntu
new file mode 100644
index 000000000..691093f9c
--- /dev/null
+++ b/docker/pyt-cpu/bert-large-training/pytorch-bert-large-training.Dockerfile-ubuntu
@@ -0,0 +1,80 @@
+# Copyright (c) 2020-2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+ARG BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG BASE_TAG="2.0.0-pip-base"
+
+FROM ${BASE_IMAGE}:${BASE_TAG} AS intel-optimized-pytorch
+
+WORKDIR /workspace/pytorch-bert-large-training
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+    build-essential \
+    ca-certificates \
+    git \
+    wget \
+    make \
+    cmake \
+    autoconf \
+    bzip2 \
+    tar
+
+RUN wget https://github.com/gperftools/gperftools/releases/download/gperftools-2.7.90/gperftools-2.7.90.tar.gz && \
+    tar -xzf gperftools-2.7.90.tar.gz && \
+    cd gperftools-2.7.90 && \
+    mkdir -p /workspace/lib/ && \
+    ./configure --prefix=/workspace/lib/tcmalloc/ && \
+    make && \
+    make install
+
+RUN git clone https://github.com/jemalloc/jemalloc.git && \
+    cd jemalloc && \
+    git checkout c8209150f9d219a137412b06431c9d52839c7272 && \
+    ./autogen.sh && \
+    ./configure --prefix=/workspace/lib/jemalloc && \
+    make && \ 
+    make install
+
+RUN pip install packaging intel-openmp
+
+ENV LD_PRELOAD="/workspace/lib/jemalloc/lib/libjemalloc.so":"/workspace/lib/tcmalloc/lib/libtcmalloc.so":"/usr/local/lib/libiomp5.so":$LD_PRELOAD 
+ENV MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+
+COPY models/language_modeling/pytorch/bert_large/training models/language_modeling/pytorch/bert_large/training
+COPY models/language_modeling/pytorch/common/enable_ipex_for_transformers.diff models/language_modeling/pytorch/common/enable_ipex_for_transformers.diff
+COPY quickstart/language_modeling/pytorch/bert_large/training/cpu/run_bert_pretrain_phase1.sh quickstart/run_bert_pretrain_phase1.sh
+COPY quickstart/language_modeling/pytorch/bert_large/training/cpu/run_bert_pretrain_phase2.sh quickstart/run_bert_pretrain_phase2.sh
+
+ARG TRANSFORMERS_COMMIT="v4.28.1"
+
+RUN pip install datasets==1.11.0 accelerate tfrecord intel-openmp faiss-cpu && \
+    apt-get install -y libopenblas-dev && \
+    git clone https://github.com/huggingface/transformers.git && \
+    cd transformers && \
+    git checkout ${TRANSFORMERS_COMMIT} && \
+    git apply /workspace/pytorch-bert-large-training/models/language_modeling/pytorch/common/enable_ipex_for_transformers.diff && \
+    python -m pip install --upgrade pip && \
+    pip uninstall transformers -y && \
+    pip install -e . && \
+    pip install h5py && \
+    mkdir -p /root/.local
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+    numactl \
+    libegl1-mesa 
+
+COPY LICENSE licenses/LICENSE
+COPY third_party licenses/third_party
diff --git a/docker/pyt-cpu/bert-large-training/tests.yaml b/docker/pyt-cpu/bert-large-training/tests.yaml
new file mode 100644
index 000000000..498e37973
--- /dev/null
+++ b/docker/pyt-cpu/bert-large-training/tests.yaml
@@ -0,0 +1,148 @@
+fp32-training-phase1:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-language-modeling-bert-large-training
+  cmd: quickstart/run_bert_pretrain_phase1.sh fp32
+  shm-size: 8G
+  env:
+    PRECISION: fp32
+    BERT_MODEL_CONFIG: /pytorch/enwiki-20200101/bert_config.json
+    OUTPUT_DIR: /tmp
+    TRAIN_SCRIPT: /workspace/pytorch-bert-large-training/models/language_modeling/pytorch/bert_large/training/run_pretrain_mlperf.py
+    EVAL_SCRIPT: /workspace/pytorch-bert-large-training/quickstart/transformers/examples/legacy/question-answering/run_squad.py
+    DATASET_DIR: /pytorch/tfrecord_dir/bert_results4_dataset/
+  volumes:
+    - src: /pytorch/enwiki-20200101/bert_config.json
+      dst: /pytorch/enwiki-20200101/bert_config.json
+    - src: /tmp
+      dst: /tmp
+    - src: /pytorch/tfrecord_dir/bert_results4_dataset/
+      dst: /pytorch/tfrecord_dir/bert_results4_dataset/
+fp32-training-phase2:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-language-modeling-bert-large-training
+  cmd: quickstart/run_bert_pretrain_phase2.sh fp32
+  shm-size: 8G
+  env:
+    PRECISION: fp32
+    BERT_MODEL_CONFIG: /pytorch/enwiki-20200101/bert_config.json
+    OUTPUT_DIR: /tmp
+    PRETRAINED_MODEL: /tmp/model_save
+    TRAIN_SCRIPT: /workspace/pytorch-bert-large-training/models/language_modeling/pytorch/bert_large/training/run_pretrain_mlperf.py
+    EVAL_SCRIPT: /workspace/pytorch-bert-large-training/quickstart/transformers/examples/legacy/question-answering/run_squad.py
+    DATASET_DIR: /pytorch/tfrecord_dir/bert_results4_dataset/
+  volumes:
+    - src: /pytorch/enwiki-20200101/bert_config.json
+      dst: /pytorch/enwiki-20200101/bert_config.json
+    - src: /tmp/model_save
+      dst: /tmp/model_save
+    - src: /pytorch/tfrecord_dir/bert_results4_dataset/
+      dst: /pytorch/tfrecord_dir/bert_results4_dataset/
+bf16-training-phase1:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-language-modeling-bert-large-training
+  cmd: quickstart/run_bert_pretrain_phase1.sh bf16
+  shm-size: 8G
+  env:
+    PRECISION: bf16
+    BERT_MODEL_CONFIG: /pytorch/enwiki-20200101/bert_config.json
+    OUTPUT_DIR: /tmp
+    TRAIN_SCRIPT: /workspace/pytorch-bert-large-training/models/language_modeling/pytorch/bert_large/training/run_pretrain_mlperf.py
+    EVAL_SCRIPT: /workspace/pytorch-bert-large-training/quickstart/transformers/examples/legacy/question-answering/run_squad.py
+    DATASET_DIR: /pytorch/tfrecord_dir/bert_results4_dataset/
+  volumes:
+    - src: /pytorch/enwiki-20200101/bert_config.json
+      dst: /pytorch/enwiki-20200101/bert_config.json
+    - src: /tmp
+      dst: /tmp
+    - src: /pytorch/tfrecord_dir/bert_results4_dataset/
+      dst: /pytorch/tfrecord_dir/bert_results4_dataset/
+bf16-training-phase2:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-language-modeling-bert-large-training
+  cmd: quickstart/run_bert_pretrain_phase2.sh bf16
+  shm-size: 8G
+  env:
+    PRECISION: bf16
+    BERT_MODEL_CONFIG: /pytorch/enwiki-20200101/bert_config.json
+    OUTPUT_DIR: /tmp
+    PRETRAINED_MODEL: /tmp/model_save
+    TRAIN_SCRIPT: /workspace/pytorch-bert-large-training/models/language_modeling/pytorch/bert_large/training/run_pretrain_mlperf.py
+    EVAL_SCRIPT: /workspace/pytorch-bert-large-training/quickstart/transformers/examples/legacy/question-answering/run_squad.py
+    DATASET_DIR: /pytorch/tfrecord_dir/bert_results4_dataset/
+  volumes:
+    - src: /pytorch/enwiki-20200101/bert_config.json
+      dst: /pytorch/enwiki-20200101/bert_config.json
+    - src: /tmp/model_save
+      dst: /tmp/model_save
+    - src: /pytorch/tfrecord_dir/bert_results4_dataset/
+      dst: /pytorch/tfrecord_dir/bert_results4_dataset/
+bf32-training-phase1:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-language-modeling-bert-large-training
+  cmd: quickstart/run_bert_pretrain_phase1.sh bf32
+  shm-size: 8G
+  env:
+    PRECISION: bf32
+    BERT_MODEL_CONFIG: /pytorch/enwiki-20200101/bert_config.json
+    OUTPUT_DIR: /tmp
+    TRAIN_SCRIPT: /workspace/pytorch-bert-large-training/models/language_modeling/pytorch/bert_large/training/run_pretrain_mlperf.py
+    EVAL_SCRIPT: /workspace/pytorch-bert-large-training/quickstart/transformers/examples/legacy/question-answering/run_squad.py
+    DATASET_DIR: /pytorch/tfrecord_dir/bert_results4_dataset/
+  volumes:
+    - src: /pytorch/enwiki-20200101/bert_config.json
+      dst: /pytorch/enwiki-20200101/bert_config.json
+    - src: /tmp
+      dst: /tmp
+    - src: /pytorch/tfrecord_dir/bert_results4_dataset/
+      dst: /pytorch/tfrecord_dir/bert_results4_dataset/
+bf32-training-phase2:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-language-modeling-bert-large-training
+  cmd: quickstart/run_bert_pretrain_phase2.sh bf32
+  shm-size: 8G
+  env:
+    PRECISION: bf32
+    BERT_MODEL_CONFIG: /pytorch/enwiki-20200101/bert_config.json
+    OUTPUT_DIR: /tmp
+    PRETRAINED_MODEL: /tmp/model_save
+    TRAIN_SCRIPT: /workspace/pytorch-bert-large-training/models/language_modeling/pytorch/bert_large/training/run_pretrain_mlperf.py
+    EVAL_SCRIPT: /workspace/pytorch-bert-large-training/quickstart/transformers/examples/legacy/question-answering/run_squad.py
+    DATASET_DIR: /pytorch/tfrecord_dir/bert_results4_dataset/
+  volumes:
+    - src: /pytorch/enwiki-20200101/bert_config.json
+      dst: /pytorch/enwiki-20200101/bert_config.json
+    - src: /tmp/model_save
+      dst: /tmp/model_save
+    - src: /pytorch/tfrecord_dir/bert_results4_dataset/
+      dst: /pytorch/tfrecord_dir/bert_results4_dataset/
+avx-fp32-training-phase1:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-language-modeling-bert-large-training
+  cmd: quickstart/run_bert_pretrain_phase1.sh avx-fp32
+  shm-size: 8G
+  env:
+    PRECISION: avx-fp32
+    BERT_MODEL_CONFIG: /pytorch/enwiki-20200101/bert_config.json
+    OUTPUT_DIR: /tmp
+    TRAIN_SCRIPT: /workspace/pytorch-bert-large-training/models/language_modeling/pytorch/bert_large/training/run_pretrain_mlperf.py
+    EVAL_SCRIPT: /workspace/pytorch-bert-large-training/quickstart/transformers/examples/legacy/question-answering/run_squad.py
+    DATASET_DIR: /pytorch/tfrecord_dir/bert_results4_dataset/
+  volumes:
+    - src: /pytorch/enwiki-20200101/bert_config.json
+      dst: /pytorch/enwiki-20200101/bert_config.json
+    - src: /tmp
+      dst: /tmp
+    - src: /pytorch/tfrecord_dir/bert_results4_dataset/
+      dst: /pytorch/tfrecord_dir/bert_results4_dataset/
+avx-fp32-training-phase2:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-language-modeling-bert-large-training
+  cmd: quickstart/run_bert_pretrain_phase2.sh avx-fp32
+  shm-size: 8G
+  env:
+    PRECISION: avx-fp32
+    BERT_MODEL_CONFIG: /pytorch/enwiki-20200101/bert_config.json
+    OUTPUT_DIR: /tmp
+    PRETRAINED_MODEL: /tmp/model_save
+    TRAIN_SCRIPT: /workspace/pytorch-bert-large-training/models/language_modeling/pytorch/bert_large/training/run_pretrain_mlperf.py
+    EVAL_SCRIPT: /workspace/pytorch-bert-large-training/quickstart/transformers/examples/legacy/question-answering/run_squad.py
+    DATASET_DIR: /pytorch/tfrecord_dir/bert_results4_dataset/
+  volumes:
+    - src: /pytorch/enwiki-20200101/bert_config.json
+      dst: /pytorch/enwiki-20200101/bert_config.json
+    - src: /tmp/model_save
+      dst: /tmp/model_save
+    - src: /pytorch/tfrecord_dir/bert_results4_dataset/
+      dst: /pytorch/tfrecord_dir/bert_results4_dataset/
diff --git a/docker/pyt-cpu/distilbert-inference/pytorch-distilbert-inference.Dockerfile-centos b/docker/pyt-cpu/distilbert-inference/pytorch-distilbert-inference.Dockerfile-centos
new file mode 100644
index 000000000..1a6dffb6d
--- /dev/null
+++ b/docker/pyt-cpu/distilbert-inference/pytorch-distilbert-inference.Dockerfile-centos
@@ -0,0 +1,79 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+ARG BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG BASE_TAG="2.0.0-pip-base"
+
+FROM ${BASE_IMAGE}:${BASE_TAG} AS intel-extension-for-pytorch
+
+WORKDIR /workspace/pytorch-distilbert-inference
+
+RUN yum update -y && \
+    yum install -y \
+    ca-certificates \
+    git \
+    cmake>=3.19.6 \
+    make \
+    autoconf \
+    bzip2 \
+    tar
+
+RUN git clone  https://github.com/jemalloc/jemalloc.git && \
+    cd jemalloc && \
+    git checkout c8209150f9d219a137412b06431c9d52839c7272 && \
+    ./autogen.sh && \
+    ./configure --prefix=/workspace/lib/jemalloc && \
+    make && \
+    make install 
+
+RUN wget https://github.com/gperftools/gperftools/releases/download/gperftools-2.7.90/gperftools-2.7.90.tar.gz && \
+    tar -xzf gperftools-2.7.90.tar.gz && \
+    cd gperftools-2.7.90 && \
+    mkdir -p /workspace/lib/ && \
+    ./configure --prefix=/workspace/lib/tcmalloc/ && \
+    make && \
+    make install
+
+COPY models/language_modeling/pytorch/common/enable_ipex_for_transformers.diff models/language_modeling/pytorch/common/enable_ipex_for_transformers.diff 
+COPY quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/convert.py quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/convert.py
+COPY quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/configure.json configure.json
+COPY quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/run_accuracy.sh  quickstart/run_accuracy.sh
+COPY quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/do_calibration.sh quickstart/do_calibration.sh
+COPY quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/run_multi_instance_realtime.sh quickstart/run_multi_instance_realtime.sh
+COPY quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/run_multi_instance_throughput.sh quickstart/run_multi_instance_throughput.sh
+COPY quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/download_model_dataset.sh quickstart/download_model_dataset.sh
+
+RUN  cd quickstart && \
+     git clone https://github.com/huggingface/transformers.git && \
+     cd transformers && \
+     git checkout v4.28.1 && \
+     git apply /workspace/pytorch-distilbert-inference/models/language_modeling/pytorch/common/enable_ipex_for_transformers.diff && \
+     pip install -e ./ && \
+     cd -
+
+RUN pip install packaging datasets intel-openmp evaluate \ 
+    datasets scikit-learn scipy
+
+ENV LD_PRELOAD="/workspace/lib/jemalloc/lib/libjemalloc.so":"/workspace/lib/tcmalloc/lib/libtcmalloc.so":"/usr/local/lib/libiomp5.so":$LD_PRELOAD 
+ENV MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+
+RUN yum update -y && \
+    yum install -y \
+    numactl \
+    mesa-libGL && \
+    yum clean all
+
+COPY LICENSE licenses/LICENSE
+COPY third_party licenses/third_party
diff --git a/docker/pyt-cpu/distilbert-inference/pytorch-distilbert-inference.Dockerfile-ubuntu b/docker/pyt-cpu/distilbert-inference/pytorch-distilbert-inference.Dockerfile-ubuntu
new file mode 100644
index 000000000..86c50eeff
--- /dev/null
+++ b/docker/pyt-cpu/distilbert-inference/pytorch-distilbert-inference.Dockerfile-ubuntu
@@ -0,0 +1,84 @@
+# Copyright (c) 2020-2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+
+ARG BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG BASE_TAG="2.0.0-pip-base"
+
+FROM ${BASE_IMAGE}:${BASE_TAG} AS intel-optimized-pytorch
+
+WORKDIR /workspace/pytorch-distilbert-inference
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+    build-essential \
+    ca-certificates \
+    git \
+    wget \
+    make \
+    cmake \
+    autoconf \
+    bzip2 \
+    tar
+
+RUN git clone https://github.com/jemalloc/jemalloc.git && \
+    cd jemalloc && \
+    git checkout c8209150f9d219a137412b06431c9d52839c7272 && \
+    ./autogen.sh && \
+    ./configure --prefix=/workspace/lib/jemalloc && \
+    make && \ 
+    make install
+
+RUN wget https://github.com/gperftools/gperftools/releases/download/gperftools-2.7.90/gperftools-2.7.90.tar.gz && \
+    tar -xzf gperftools-2.7.90.tar.gz && \
+    cd gperftools-2.7.90 && \
+    mkdir -p /workspace/lib/ && \
+    ./configure --prefix=/workspace/lib/tcmalloc/ && \
+    make && \
+    make install
+
+COPY models/language_modeling/pytorch/common/enable_ipex_for_transformers.diff models/language_modeling/pytorch/common/enable_ipex_for_transformers.diff 
+COPY quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/convert.py quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/convert.py
+COPY quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/configure.json configure.json
+COPY quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/run_accuracy.sh  quickstart/run_accuracy.sh
+COPY quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/do_calibration.sh quickstart/do_calibration.sh
+COPY quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/run_multi_instance_realtime.sh quickstart/run_multi_instance_realtime.sh
+COPY quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/run_multi_instance_throughput.sh quickstart/run_multi_instance_throughput.sh
+COPY quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/download_model_dataset.sh quickstart/download_model_dataset.sh
+
+RUN  cd quickstart && \
+     git clone https://github.com/huggingface/transformers.git && \
+     cd transformers && \
+     git checkout v4.28.1 && \
+     git apply /workspace/pytorch-distilbert-inference/models/language_modeling/pytorch/common/enable_ipex_for_transformers.diff && \
+     pip install -e ./ && \
+     cd -
+
+RUN pip install packaging datasets intel-openmp evaluate \ 
+    datasets scikit-learn scipy
+
+ENV LD_PRELOAD="/workspace/lib/jemalloc/lib/libjemalloc.so":"/workspace/lib/tcmalloc/lib/libtcmalloc.so":"/usr/local/lib/libiomp5.so":$LD_PRELOAD 
+ENV MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+
+ENV LD_PRELOAD="/workspace/lib/jemalloc/lib/libjemalloc.so":"/workspace/lib/tcmalloc/lib/libtcmalloc.so":"/usr/local/lib/libiomp5.so":$LD_PRELOAD 
+ENV MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+    numactl \
+    libegl1-mesa 
+
+COPY LICENSE licenses/LICENSE
+COPY third_party licenses/third_party
diff --git a/docker/pyt-cpu/distilbert-inference/tests.yaml b/docker/pyt-cpu/distilbert-inference/tests.yaml
new file mode 100644
index 000000000..225d92f19
--- /dev/null
+++ b/docker/pyt-cpu/distilbert-inference/tests.yaml
@@ -0,0 +1,324 @@
+fp32-accuracy:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-language-modeling-distilbert-inference
+  cmd: quickstart/run_accuracy.sh fp32
+  env:
+    PRECISION: fp32
+    SEQUENCE_LENGTH: '128'
+    CORE_PER_INSTANCE: '32'
+    HF_DATASETS_OFFLINE: '0'
+    OUTPUT_DIR: /tmp
+    PRETRAINED_MODEL: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+    TRAIN_SCRIPT: /workspace/pytorch-distilbert-inference/models/language_modeling/pytorch/bert_large/training/run_pretrain_mlperf.py
+    DATASET_DIR: /pytorch/distilbert/SST-2
+    EVAL_SCRIPT: /workspace/pytorch-distilbert-inference/quickstart/transformers/examples/pytorch/text-classification/run_glue.py
+  volumes:
+    - src: /pytorch/distilbert/SST-2
+      dst: /pytorch/distilbert/SST-2
+    - src: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+      dst: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+fp32-realtime:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-language-modeling-distilbert-inference
+  cmd: quickstart/run_multi_instance_realtime.sh fp32
+  env:
+    PRECISION: fp32
+    SEQUENCE_LENGTH: '128'
+    CORE_PER_INSTANCE: '4'
+    HF_DATASETS_OFFLINE: '0'
+    OUTPUT_DIR: /tmp
+    PRETRAINED_MODEL: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+    TRAIN_SCRIPT: /workspace/pytorch-distilbert-inference/models/language_modeling/pytorch/bert_large/training/run_pretrain_mlperf.py
+    DATASET_DIR: /pytorch/distilbert/SST-2
+    EVAL_SCRIPT: /workspace/pytorch-distilbert-inference/quickstart/transformers/examples/pytorch/text-classification/run_glue.py
+  volumes:
+    - src: /pytorch/distilbert/SST-2
+      dst: /pytorch/distilbert/SST-2
+    - src: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+      dst: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+fp32-throughput:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-language-modeling-distilbert-inference
+  cmd: quickstart/run_multi_instance_throughput.sh fp32
+  env:
+    PRECISION: fp32
+    SEQUENCE_LENGTH: '128'
+    CORE_PER_INSTANCE: '32'
+    HF_DATASETS_OFFLINE: '0'
+    OUTPUT_DIR: /tmp
+    PRETRAINED_MODEL: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+    TRAIN_SCRIPT: /workspace/pytorch-distilbert-inference/models/language_modeling/pytorch/bert_large/training/run_pretrain_mlperf.py
+    DATASET_DIR: /pytorch/distilbert/SST-2
+    EVAL_SCRIPT: /workspace/pytorch-distilbert-inference/quickstart/transformers/examples/pytorch/text-classification/run_glue.py
+  volumes:
+    - src: /pytorch/distilbert/SST-2
+      dst: /pytorch/distilbert/SST-2
+    - src: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+      dst: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+bf32-accuracy:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-language-modeling-distilbert-inference
+  cmd: quickstart/run_accuracy.sh bf32
+  env:
+    PRECISION: bf32
+    SEQUENCE_LENGTH: '128'
+    CORE_PER_INSTANCE: '32'
+    HF_DATASETS_OFFLINE: '0'
+    OUTPUT_DIR: /tmp
+    PRETRAINED_MODEL: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+    TRAIN_SCRIPT: /workspace/pytorch-distilbert-inference/models/language_modeling/pytorch/bert_large/training/run_pretrain_mlperf.py
+    DATASET_DIR: /pytorch/distilbert/SST-2
+    EVAL_SCRIPT: /workspace/pytorch-distilbert-inference/quickstart/transformers/examples/pytorch/text-classification/run_glue.py
+  volumes:
+    - src: /pytorch/distilbert/SST-2
+      dst: /pytorch/distilbert/SST-2
+    - src: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+      dst: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+bf32-realtime:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-language-modeling-distilbert-inference
+  cmd: quickstart/run_multi_instance_realtime.sh bf32
+  env:
+    PRECISION: bf32
+    SEQUENCE_LENGTH: '128'
+    CORE_PER_INSTANCE: '4'
+    HF_DATASETS_OFFLINE: '0'
+    OUTPUT_DIR: /tmp
+    PRETRAINED_MODEL: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+    TRAIN_SCRIPT: /workspace/pytorch-distilbert-inference/models/language_modeling/pytorch/bert_large/training/run_pretrain_mlperf.py
+    DATASET_DIR: /pytorch/distilbert/SST-2
+    EVAL_SCRIPT: /workspace/pytorch-distilbert-inference/quickstart/transformers/examples/pytorch/text-classification/run_glue.py
+  volumes:
+    - src: /pytorch/distilbert/SST-2
+      dst: /pytorch/distilbert/SST-2
+    - src: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+      dst: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+bf32-throughout:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-language-modeling-distilbert-inference
+  cmd: quickstart/run_multi_instance_throughput.sh bf32
+  env:
+    PRECISION: bf32
+    SEQUENCE_LENGTH: '128'
+    CORE_PER_INSTANCE: '32'
+    HF_DATASETS_OFFLINE: '0'
+    OUTPUT_DIR: /tmp
+    PRETRAINED_MODEL: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+    TRAIN_SCRIPT: /workspace/pytorch-distilbert-inference/models/language_modeling/pytorch/bert_large/training/run_pretrain_mlperf.py
+    DATASET_DIR: /pytorch/distilbert/SST-2
+    EVAL_SCRIPT: /workspace/pytorch-distilbert-inference/quickstart/transformers/examples/pytorch/text-classification/run_glue.py
+  volumes:
+    - src: /pytorch/distilbert/SST-2
+      dst: /pytorch/distilbert/SST-2
+    - src: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+      dst: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+int8-bf16-accuracy:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-language-modeling-distilbert-inference
+  cmd: quickstart/run_accuracy.sh int8-bf16
+  env:
+    PRECISION: int8-bf16
+    SEQUENCE_LENGTH: '128'
+    CORE_PER_INSTANCE: '32'
+    HF_DATASETS_OFFLINE: '0'
+    OUTPUT_DIR: /tmp
+    PRETRAINED_MODEL: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+    TRAIN_SCRIPT: /workspace/pytorch-distilbert-inference/models/language_modeling/pytorch/bert_large/training/run_pretrain_mlperf.py
+    DATASET_DIR: /pytorch/distilbert/SST-2
+    EVAL_SCRIPT: /workspace/pytorch-distilbert-inference/quickstart/transformers/examples/pytorch/text-classification/run_glue.py
+  volumes:
+    - src: /pytorch/distilbert/SST-2
+      dst: /pytorch/distilbert/SST-2
+    - src: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+      dst: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+int8-bf16-realtime:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-language-modeling-distilbert-inference
+  cmd: quickstart/run_multi_instance_realtime.sh int8-bf16
+  env:
+    PRECISION: int8-bf16
+    SEQUENCE_LENGTH: '128'
+    CORE_PER_INSTANCE: '4'
+    HF_DATASETS_OFFLINE: '0'
+    OUTPUT_DIR: /tmp
+    PRETRAINED_MODEL: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+    TRAIN_SCRIPT: /workspace/pytorch-distilbert-inference/models/language_modeling/pytorch/bert_large/training/run_pretrain_mlperf.py
+    DATASET_DIR: /pytorch/distilbert/SST-2
+    EVAL_SCRIPT: /workspace/pytorch-distilbert-inference/quickstart/transformers/examples/pytorch/text-classification/run_glue.py
+  volumes:
+    - src: /pytorch/distilbert/SST-2
+      dst: /pytorch/distilbert/SST-2
+    - src: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+      dst: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+int8-bf16-throughput:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-language-modeling-distilbert-inference
+  cmd: quickstart/run_multi_instance_throughput.sh int8-bf16
+  env:
+    PRECISION: int8-bf16
+    SEQUENCE_LENGTH: '128'
+    CORE_PER_INSTANCE: '32'
+    HF_DATASETS_OFFLINE: '0'
+    OUTPUT_DIR: /tmp
+    PRETRAINED_MODEL: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+    TRAIN_SCRIPT: /workspace/pytorch-distilbert-inference/models/language_modeling/pytorch/bert_large/training/run_pretrain_mlperf.py
+    DATASET_DIR: /pytorch/distilbert/SST-2
+    EVAL_SCRIPT: /workspace/pytorch-distilbert-inference/quickstart/transformers/examples/pytorch/text-classification/run_glue.py
+  volumes:
+    - src: /pytorch/distilbert/SST-2
+      dst: /pytorch/distilbert/SST-2
+    - src: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+      dst: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+int8-fp32-accuracy:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-language-modeling-distilbert-inference
+  cmd: quickstart/run_accuracy.sh int8-fp32
+  env:
+    PRECISION: int8-fp32
+    SEQUENCE_LENGTH: '128'
+    CORE_PER_INSTANCE: '32'
+    HF_DATASETS_OFFLINE: '0'
+    OUTPUT_DIR: /tmp
+    PRETRAINED_MODEL: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+    TRAIN_SCRIPT: /workspace/pytorch-distilbert-inference/models/language_modeling/pytorch/bert_large/training/run_pretrain_mlperf.py
+    DATASET_DIR: /pytorch/distilbert/SST-2
+    EVAL_SCRIPT: /workspace/pytorch-distilbert-inference/quickstart/transformers/examples/pytorch/text-classification/run_glue.py
+  volumes:
+    - src: /pytorch/distilbert/SST-2
+      dst: /pytorch/distilbert/SST-2
+    - src: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+      dst: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+int8-fp32-realtime:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-language-modeling-distilbert-inference
+  cmd: quickstart/run_multi_instance_realtime.sh int8-fp32
+  env:
+    PRECISION: int8-fp32
+    SEQUENCE_LENGTH: '128'
+    CORE_PER_INSTANCE: '4'
+    HF_DATASETS_OFFLINE: '0'
+    OUTPUT_DIR: /tmp
+    PRETRAINED_MODEL: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+    TRAIN_SCRIPT: /workspace/pytorch-distilbert-inference/models/language_modeling/pytorch/bert_large/training/run_pretrain_mlperf.py
+    DATASET_DIR: /pytorch/distilbert/SST-2
+    EVAL_SCRIPT: /workspace/pytorch-distilbert-inference/quickstart/transformers/examples/pytorch/text-classification/run_glue.py
+  volumes:
+    - src: /pytorch/distilbert/SST-2
+      dst: /pytorch/distilbert/SST-2
+    - src: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+      dst: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+int8-fp32-throughput:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-language-modeling-distilbert-inference
+  cmd: quickstart/run_multi_instance_throughput.sh int8-fp32
+  env:
+    PRECISION: int8-fp32
+    SEQUENCE_LENGTH: '128'
+    CORE_PER_INSTANCE: '32'
+    HF_DATASETS_OFFLINE: '0'
+    OUTPUT_DIR: /tmp
+    PRETRAINED_MODEL: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+    TRAIN_SCRIPT: /workspace/pytorch-distilbert-inference/models/language_modeling/pytorch/bert_large/training/run_pretrain_mlperf.py
+    DATASET_DIR: /pytorch/distilbert/SST-2
+    EVAL_SCRIPT: /workspace/pytorch-distilbert-inference/quickstart/transformers/examples/pytorch/text-classification/run_glue.py
+  volumes:
+    - src: /pytorch/distilbert/SST-2
+      dst: /pytorch/distilbert/SST-2
+    - src: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+      dst: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+bf16-accuracy:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-language-modeling-distilbert-inference
+  cmd: quickstart/run_accuracy.sh bf16
+  env:
+    PRECISION: bf16
+    SEQUENCE_LENGTH: '128'
+    CORE_PER_INSTANCE: '32'
+    HF_DATASETS_OFFLINE: '0'
+    OUTPUT_DIR: /tmp
+    PRETRAINED_MODEL: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+    TRAIN_SCRIPT: /workspace/pytorch-distilbert-inference/models/language_modeling/pytorch/bert_large/training/run_pretrain_mlperf.py
+    DATASET_DIR: /pytorch/distilbert/SST-2
+    EVAL_SCRIPT: /workspace/pytorch-distilbert-inference/quickstart/transformers/examples/pytorch/text-classification/run_glue.py
+  volumes:
+    - src: /pytorch/distilbert/SST-2
+      dst: /pytorch/distilbert/SST-2
+    - src: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+      dst: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+bf16-realtime:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-language-modeling-distilbert-inference
+  cmd: quickstart/run_multi_instance_realtime.sh bf16
+  env:
+    PRECISION: bf16
+    SEQUENCE_LENGTH: '128'
+    CORE_PER_INSTANCE: '4'
+    HF_DATASETS_OFFLINE: '0'
+    OUTPUT_DIR: /tmp
+    PRETRAINED_MODEL: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+    TRAIN_SCRIPT: /workspace/pytorch-distilbert-inference/models/language_modeling/pytorch/bert_large/training/run_pretrain_mlperf.py
+    DATASET_DIR: /pytorch/distilbert/SST-2
+    EVAL_SCRIPT: /workspace/pytorch-distilbert-inference/quickstart/transformers/examples/pytorch/text-classification/run_glue.py
+  volumes:
+    - src: /pytorch/distilbert/SST-2
+      dst: /pytorch/distilbert/SST-2
+    - src: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+      dst: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+bf16-throughput:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-language-modeling-distilbert-inference
+  cmd: quickstart/run_multi_instance_throughput.sh bf16
+  env:
+    PRECISION: bf16
+    SEQUENCE_LENGTH: '128'
+    CORE_PER_INSTANCE: '32'
+    HF_DATASETS_OFFLINE: '0'
+    OUTPUT_DIR: /tmp
+    PRETRAINED_MODEL: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+    TRAIN_SCRIPT: /workspace/pytorch-distilbert-inference/models/language_modeling/pytorch/bert_large/training/run_pretrain_mlperf.py
+    DATASET_DIR: /pytorch/distilbert/SST-2
+    EVAL_SCRIPT: /workspace/pytorch-distilbert-inference/quickstart/transformers/examples/pytorch/text-classification/run_glue.py
+  volumes:
+    - src: /pytorch/distilbert/SST-2
+      dst: /pytorch/distilbert/SST-2
+    - src: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+      dst: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+bf32-accuracy:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-language-modeling-distilbert-inference
+  cmd: quickstart/run_accuracy.sh bf32
+  env:
+    PRECISION: bf32
+    SEQUENCE_LENGTH: '128'
+    CORE_PER_INSTANCE: '32'
+    HF_DATASETS_OFFLINE: '0'
+    OUTPUT_DIR: /tmp
+    PRETRAINED_MODEL: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+    TRAIN_SCRIPT: /workspace/pytorch-distilbert-inference/models/language_modeling/pytorch/bert_large/training/run_pretrain_mlperf.py
+    DATASET_DIR: /pytorch/distilbert/SST-2
+    EVAL_SCRIPT: /workspace/pytorch-distilbert-inference/quickstart/transformers/examples/pytorch/text-classification/run_glue.py
+  volumes:
+    - src: /pytorch/distilbert/SST-2
+      dst: /pytorch/distilbert/SST-2
+    - src: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+      dst: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+bf32-realtime:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-language-modeling-distilbert-inference
+  cmd: quickstart/run_multi_instance_realtime.sh bf32
+  env:
+    PRECISION: bf32
+    SEQUENCE_LENGTH: '128'
+    CORE_PER_INSTANCE: '4'
+    HF_DATASETS_OFFLINE: '0'
+    OUTPUT_DIR: /tmp
+    PRETRAINED_MODEL: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+    TRAIN_SCRIPT: /workspace/pytorch-distilbert-inference/models/language_modeling/pytorch/bert_large/training/run_pretrain_mlperf.py
+    DATASET_DIR: /pytorch/distilbert/SST-2
+    EVAL_SCRIPT: /workspace/pytorch-distilbert-inference/quickstart/transformers/examples/pytorch/text-classification/run_glue.py
+  volumes:
+    - src: /pytorch/distilbert/SST-2
+      dst: /pytorch/distilbert/SST-2
+    - src: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+      dst: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+bf32-throughput:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-language-modeling-distilbert-inference
+  cmd: quickstart/run_multi_instance_throughput.sh bf32
+  env:
+    PRECISION: bf32
+    SEQUENCE_LENGTH: '128'
+    CORE_PER_INSTANCE: '32'
+    HF_DATASETS_OFFLINE: '0'
+    OUTPUT_DIR: /tmp
+    PRETRAINED_MODEL: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+    TRAIN_SCRIPT: /workspace/pytorch-distilbert-inference/models/language_modeling/pytorch/bert_large/training/run_pretrain_mlperf.py
+    DATASET_DIR: /pytorch/distilbert/SST-2
+    EVAL_SCRIPT: /workspace/pytorch-distilbert-inference/quickstart/transformers/examples/pytorch/text-classification/run_glue.py
+  volumes:
+    - src: /pytorch/distilbert/SST-2
+      dst: /pytorch/distilbert/SST-2
+    - src: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
+      dst: /pytorch/distilbert/distilbert-base-uncased-finetuned-sst-2-english
diff --git a/docker/pyt-cpu/dlrm-inference/pytorch-dlrm-inference.Dockerfile-centos b/docker/pyt-cpu/dlrm-inference/pytorch-dlrm-inference.Dockerfile-centos
new file mode 100644
index 000000000..1befb2b3d
--- /dev/null
+++ b/docker/pyt-cpu/dlrm-inference/pytorch-dlrm-inference.Dockerfile-centos
@@ -0,0 +1,59 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+ARG BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG BASE_TAG="2.0.0-pip-base"
+
+FROM ${BASE_IMAGE}:${BASE_TAG} AS intel-optimized-pytorch
+
+SHELL ["/bin/bash", "-c"]
+
+ENV LANG C.UTF-8
+
+WORKDIR /workspace/pytorch-dlrm-inference
+
+RUN git clone https://github.com/jemalloc/jemalloc.git && \
+    cd jemalloc && \
+    git checkout c8209150f9d219a137412b06431c9d52839c7272 && \
+    ./autogen.sh && \
+    ./configure --prefix=/workspace/lib/jemalloc && \
+    make && \ 
+    make install
+
+RUN pip install packaging intel-openmp
+
+ENV LD_PRELOAD="/workspace/lib/jemalloc/lib/libjemalloc.so":"/usr/local/lib/libiomp5.so":$LD_PRELOAD
+ENV MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+
+COPY models/recommendation/pytorch/dlrm models/recommendation/pytorch/dlrm
+COPY quickstart/recommendation/pytorch/dlrm/inference/cpu/inference_performance.sh quickstart/inference_performance.sh
+COPY quickstart/recommendation/pytorch/dlrm/inference/cpu/accuracy.sh quickstart/accuracy.sh
+COPY quickstart/recommendation/pytorch/dlrm/requirements.txt quickstart/requirements.txt
+
+RUN pip install -r /workspace/pytorch-dlrm-inference/quickstart/requirements.txt
+
+ENV KMP_AFFINITY="granularity=fine,compact,1,0"
+ENV KMP_BLOCKTIME=1
+ENV DNNL_PRIMITIVE_CACHE_CAPACITY=1024
+ENV KMP_SETTINGS=1
+
+RUN yum update -y && \
+    yum install -y \
+    numactl \
+    mesa-libGL && \
+    yum clean all
+
+COPY LICENSE licenses/LICENSE
+COPY third_party licenses/third_party
diff --git a/docker/pyt-cpu/dlrm-inference/pytorch-dlrm-inference.Dockerfile-ubuntu b/docker/pyt-cpu/dlrm-inference/pytorch-dlrm-inference.Dockerfile-ubuntu
new file mode 100644
index 000000000..ea051e792
--- /dev/null
+++ b/docker/pyt-cpu/dlrm-inference/pytorch-dlrm-inference.Dockerfile-ubuntu
@@ -0,0 +1,66 @@
+# Copyright (c) 2020-2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+ARG BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG BASE_TAG="2.0.0-pip-base"
+
+FROM ${BASE_IMAGE}:${BASE_TAG} AS intel-optimized-pytorch
+
+WORKDIR /workspace/pytorch-dlrm-inference
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+    build-essential \
+    ca-certificates \
+    git \
+    wget \
+    make \
+    cmake \
+    autoconf \
+    bzip2 \
+    tar
+
+RUN git clone https://github.com/jemalloc/jemalloc.git && \
+    cd jemalloc && \
+    git checkout c8209150f9d219a137412b06431c9d52839c7272 && \
+    ./autogen.sh && \
+    ./configure --prefix=/workspace/lib/jemalloc && \
+    make && \ 
+    make install
+
+RUN pip install packaging intel-openmp
+
+ENV LD_PRELOAD="/workspace/lib/jemalloc/lib/libjemalloc.so":"/usr/local/lib/libiomp5.so":$LD_PRELOAD
+ENV MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+
+COPY models/recommendation/pytorch/dlrm models/recommendation/pytorch/dlrm
+COPY quickstart/recommendation/pytorch/dlrm/inference/cpu/inference_performance.sh quickstart/inference_performance.sh
+COPY quickstart/recommendation/pytorch/dlrm/inference/cpu/accuracy.sh quickstart/accuracy.sh
+COPY quickstart/recommendation/pytorch/dlrm/requirements.txt quickstart/requirements.txt
+
+RUN pip install -r /workspace/pytorch-dlrm-inference/quickstart/requirements.txt
+
+ENV KMP_AFFINITY="granularity=fine,compact,1,0"
+ENV KMP_BLOCKTIME=1
+ENV DNNL_PRIMITIVE_CACHE_CAPACITY=1024
+ENV KMP_SETTINGS=1
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+      numactl \
+      libegl1-mesa 
+
+COPY LICENSE licenses/LICENSE
+COPY third_party licenses/third_party
diff --git a/docker/pyt-cpu/dlrm-inference/tests.yaml b/docker/pyt-cpu/dlrm-inference/tests.yaml
new file mode 100644
index 000000000..8b0bb873f
--- /dev/null
+++ b/docker/pyt-cpu/dlrm-inference/tests.yaml
@@ -0,0 +1,120 @@
+fp32-accuracy:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-recommendation-dlrm-inference
+  pull_policy: always
+  cmd: quickstart/accuracy.sh
+  shm-size: 8G
+  env:
+    PRECISION: fp32
+    WEIGHT_PATH: /pytorch/dlrm_weights/dlrm/tb00_40M.pt
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/spr_dlrm_preprocessed/dlrm
+  volumes:
+    - src: /pytorch/dlrm_weights/dlrm/tb00_40M.pt
+      dst: /pytorch/dlrm_weights/dlrm/tb00_40M.pt
+    - src: /pytorch/spr_dlrm_preprocessed/dlrm
+      dst: /pytorch/spr_dlrm_preprocessed/dlrm
+bf32-accuracy:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-recommendation-dlrm-inference
+  pull_policy: always
+  cmd: quickstart/accuracy.sh
+  shm-size: 8G
+  env:
+    PRECISION: bf32
+    WEIGHT_PATH: /pytorch/dlrm_weights/dlrm/tb00_40M.pt
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/spr_dlrm_preprocessed/dlrm
+  volumes:
+    - src: /pytorch/dlrm_weights/dlrm/tb00_40M.pt
+      dst: /pytorch/dlrm_weights/dlrm/tb00_40M.pt
+    - src: /pytorch/spr_dlrm_preprocessed/dlrm
+      dst: /pytorch/spr_dlrm_preprocessed/dlrm
+bf16-accuracy:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-recommendation-dlrm-inference
+  pull_policy: always
+  cmd: quickstart/accuracy.sh
+  shm-size: 8G
+  env:
+    PRECISION: bf16
+    WEIGHT_PATH: /pytorch/dlrm_weights/dlrm/tb00_40M.pt
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/spr_dlrm_preprocessed/dlrm
+  volumes:
+    - src: /pytorch/dlrm_weights/dlrm/tb00_40M.pt
+      dst: /pytorch/dlrm_weights/dlrm/tb00_40M.pt
+    - src: /pytorch/spr_dlrm_preprocessed/dlrm
+      dst: /pytorch/spr_dlrm_preprocessed/dlrm
+int8-accuracy:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-recommendation-dlrm-inference
+  pull_policy: always
+  cmd: quickstart/accuracy.sh
+  shm-size: 8G
+  env:
+    PRECISION: int8
+    WEIGHT_PATH: /pytorch/dlrm_weights/dlrm/tb00_40M.pt
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/spr_dlrm_preprocessed/dlrm
+  volumes:
+    - src: /pytorch/dlrm_weights/dlrm/tb00_40M.pt
+      dst: /pytorch/dlrm_weights/dlrm/tb00_40M.pt
+    - src: /pytorch/spr_dlrm_preprocessed/dlrm
+      dst: /pytorch/spr_dlrm_preprocessed/dlrm
+fp32-performance:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-recommendation-dlrm-inference
+  pull_policy: always
+  cmd: quickstart/inference_performance.sh
+  shm-size: 8G
+  env:
+    PRECISION: fp32
+    WEIGHT_PATH: /pytorch/dlrm_weights/dlrm/tb00_40M.pt
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/spr_dlrm_preprocessed/dlrm
+  volumes:
+    - src: /pytorch/dlrm_weights/dlrm/tb00_40M.pt
+      dst: /pytorch/dlrm_weights/dlrm/tb00_40M.pt
+    - src: /pytorch/spr_dlrm_preprocessed/dlrm
+      dst: /pytorch/spr_dlrm_preprocessed/dlrm
+bf32-performance:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-recommendation-dlrm-inference
+  pull_policy: always
+  cmd: quickstart/inference_performance.sh
+  shm-size: 8G
+  env:
+    PRECISION: bf32
+    WEIGHT_PATH: /pytorch/dlrm_weights/dlrm/tb00_40M.pt
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/spr_dlrm_preprocessed/dlrm
+  volumes:
+    - src: /pytorch/dlrm_weights/dlrm/tb00_40M.pt
+      dst: /pytorch/dlrm_weights/dlrm/tb00_40M.pt
+    - src: /pytorch/spr_dlrm_preprocessed/dlrm
+      dst: /pytorch/spr_dlrm_preprocessed/dlrm
+bf16-performance:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-recommendation-dlrm-inference
+  pull_policy: always
+  cmd: quickstart/inference_performance.sh
+  shm-size: 8G
+  env:
+    PRECISION: bf16
+    WEIGHT_PATH: /pytorch/dlrm_weights/dlrm/tb00_40M.pt
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/spr_dlrm_preprocessed/dlrm
+  volumes:
+    - src: /pytorch/dlrm_weights/dlrm/tb00_40M.pt
+      dst: /pytorch/dlrm_weights/dlrm/tb00_40M.pt
+    - src: /pytorch/spr_dlrm_preprocessed/dlrm
+      dst: /pytorch/spr_dlrm_preprocessed/dlrm
+int8-performance:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-recommendation-dlrm-inference
+  pull_policy: always
+  cmd: quickstart/inference_performance.sh
+  shm-size: 8G
+  env:
+    PRECISION: int8
+    WEIGHT_PATH: /pytorch/dlrm_weights/dlrm/tb00_40M.pt
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/spr_dlrm_preprocessed/dlrm
+  volumes:
+    - src: /pytorch/dlrm_weights/dlrm/tb00_40M.pt
+      dst: /pytorch/dlrm_weights/dlrm/tb00_40M.pt
+    - src: /pytorch/spr_dlrm_preprocessed/dlrm
+      dst: /pytorch/spr_dlrm_preprocessed/dlrm
diff --git a/docker/pyt-cpu/dlrm-training/pytorch-dlrm-training.Dockerfile-centos b/docker/pyt-cpu/dlrm-training/pytorch-dlrm-training.Dockerfile-centos
new file mode 100644
index 000000000..b2038d30a
--- /dev/null
+++ b/docker/pyt-cpu/dlrm-training/pytorch-dlrm-training.Dockerfile-centos
@@ -0,0 +1,66 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+ARG BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG BASE_TAG="2.0.0-pip-base"
+
+FROM ${BASE_IMAGE}:${BASE_TAG} AS intel-optimized-pytorch
+
+SHELL ["/bin/bash", "-c"]
+
+ENV LANG C.UTF-8
+
+WORKDIR /workspace/pytorch-dlrm-training
+
+RUN git clone https://github.com/jemalloc/jemalloc.git && \
+    cd jemalloc && \
+    git checkout c8209150f9d219a137412b06431c9d52839c7272 && \
+    ./autogen.sh && \
+    ./configure --prefix=/workspace/lib/jemalloc && \
+    make && \ 
+    make install
+
+RUN wget https://github.com/gperftools/gperftools/releases/download/gperftools-2.7.90/gperftools-2.7.90.tar.gz && \
+    tar -xzf gperftools-2.7.90.tar.gz && \
+    cd gperftools-2.7.90 && \
+    mkdir -p /workspace/lib/ && \
+    ./configure --prefix=/workspace/lib/tcmalloc/ && \
+    make && \
+    make install
+
+RUN pip install packaging intel-openmp
+
+ENV LD_PRELOAD="/workspace/lib/jemalloc/lib/libjemalloc.so":"/workspace/lib/tcmalloc/lib/libtcmalloc.so":"/usr/local/lib/libiomp5.so":$LD_PRELOAD
+ENV MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+
+COPY models/recommendation/pytorch/dlrm/product models/recommendation/pytorch/dlrm/product
+COPY quickstart/recommendation/pytorch/dlrm/training/cpu/training.sh quickstart/training.sh
+COPY quickstart/recommendation/pytorch/dlrm/requirements.txt quickstart/requirements.txt
+
+RUN pip install -r /workspace/pytorch-dlrm-training/quickstart/requirements.txt
+
+ENV KMP_AFFINITY="granularity=fine,compact,1,0"
+ENV KMP_BLOCKTIME=1
+ENV DNNL_PRIMITIVE_CACHE_CAPACITY=1024
+ENV KMP_SETTINGS=1
+
+RUN yum update -y && \
+    yum install -y \
+    numactl \
+    mesa-libGL && \
+    yum clean all
+
+COPY LICENSE licenses/LICENSE
+COPY third_party licenses/third_party
diff --git a/docker/pyt-cpu/dlrm-training/pytorch-dlrm-training.Dockerfile-ubuntu b/docker/pyt-cpu/dlrm-training/pytorch-dlrm-training.Dockerfile-ubuntu
new file mode 100644
index 000000000..4ef6a530f
--- /dev/null
+++ b/docker/pyt-cpu/dlrm-training/pytorch-dlrm-training.Dockerfile-ubuntu
@@ -0,0 +1,72 @@
+# Copyright (c) 2020-2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+ARG BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG BASE_TAG="2.0.0-pip-base"
+
+FROM ${BASE_IMAGE}:${BASE_TAG} AS intel-optimized-pytorch
+
+WORKDIR /workspace/pytorch-dlrm-training
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+    build-essential \
+    ca-certificates \
+    git \
+    wget \
+    make \
+    cmake \
+    autoconf \
+    bzip2 \
+    tar
+
+RUN git clone https://github.com/jemalloc/jemalloc.git && \
+    cd jemalloc && \
+    git checkout c8209150f9d219a137412b06431c9d52839c7272 && \
+    ./autogen.sh && \
+    ./configure --prefix=/workspace/lib/jemalloc && \
+    make && \ 
+    make install
+
+RUN wget https://github.com/gperftools/gperftools/releases/download/gperftools-2.7.90/gperftools-2.7.90.tar.gz && \
+    tar -xzf gperftools-2.7.90.tar.gz && \
+    cd gperftools-2.7.90 && \
+    mkdir -p /workspace/lib/ && \
+    ./configure --prefix=/workspace/lib/tcmalloc/ && \
+    make && \
+    make install
+    
+RUN pip install packaging intel-openmp
+
+ENV LD_PRELOAD="/workspace/lib/jemalloc/lib/libjemalloc.so":"/workspace/lib/tcmalloc/lib/libtcmalloc.so":"/usr/local/lib/libiomp5.so":$LD_PRELOAD
+ENV MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+
+COPY models/recommendation/pytorch/dlrm/product models/recommendation/pytorch/dlrm/product
+COPY quickstart/recommendation/pytorch/dlrm/training/cpu/training.sh quickstart/training.sh
+COPY quickstart/recommendation/pytorch/dlrm/requirements.txt quickstart/requirements.txt
+
+RUN pip install -r /workspace/pytorch-dlrm-training/quickstart/requirements.txt
+
+ENV KMP_AFFINITY="granularity=fine,compact,1,0"
+ENV KMP_BLOCKTIME=1
+ENV DNNL_PRIMITIVE_CACHE_CAPACITY=1024
+ENV KMP_SETTINGS=1
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+      numactl \
+      libegl1-mesa 
+
+COPY LICENSE licenses/LICENSE
+COPY third_party licenses/third_party
diff --git a/docker/pyt-cpu/dlrm-training/tests.yaml b/docker/pyt-cpu/dlrm-training/tests.yaml
new file mode 100644
index 000000000..f8681e1ce
--- /dev/null
+++ b/docker/pyt-cpu/dlrm-training/tests.yaml
@@ -0,0 +1,36 @@
+fp32-training:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-recommendation-dlrm-training
+  cmd: quickstart/training.sh
+  shm-size: 10G
+  env:
+    PRECISION: fp32
+    NUM_BATCH: '10000' 
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/spr_dlrm_preprocessed/dlrm
+  volumes:
+    - src: /pytorch/spr_dlrm_preprocessed/dlrm
+      dst: /pytorch/spr_dlrm_preprocessed/dlrm
+bf16-training:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-recommendation-dlrm-training
+  cmd: quickstart/training.sh
+  shm-size: 10G
+  env:
+    PRECISION: bf16
+    NUM_BATCH: '10000'
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/spr_dlrm_preprocessed/dlrm
+  volumes:
+    - src: /pytorch/spr_dlrm_preprocessed/dlrm
+      dst: /pytorch/spr_dlrm_preprocessed/dlrm
+bf32-training:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-recommendation-dlrm-training
+  cmd: quickstart/training.sh
+  shm-size: 10G
+  env:
+    PRECISION: bf32
+    NUM_BATCH: '10000'
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/spr_dlrm_preprocessed/dlrm
+  volumes:
+    - src: /pytorch/spr_dlrm_preprocessed/dlrm
+      dst: /pytorch/spr_dlrm_preprocessed/dlrm
diff --git a/docker/pyt-cpu/docker-compose.yml b/docker/pyt-cpu/docker-compose.yml
index d3cb03b3e..f5ba4c333 100644
--- a/docker/pyt-cpu/docker-compose.yml
+++ b/docker/pyt-cpu/docker-compose.yml
@@ -17,8 +17,8 @@
 #
 version: '3'
 services:
-  spr-bert-large-inference:
-    image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-spr-bert-large-inference
+  bert-large-inference:
+    image: ${BASE_IMAGE:-intel/intel-optimized-pytorch}-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-pyt-language-modeling-bert-large-inference
     pull_policy: always
     build:
       context: ../../
@@ -26,13 +26,13 @@ services:
         http_proxy: ${http_proxy}
         https_proxy: ${https_proxy}
         no_proxy: ${no_proxy}
-        BASE_IMAGE: ${BASE_IMAGE:-intel/intel-extension-for-pytorch}
-        BASE_TAG: ${BASE_TAG:-2.0.0-pip-base}
-      dockerfile: docker/pyt-cpu/spr-bert-large-inference/pytorch-spr-bert-large-inference.Dockerfile
+        BASE_IMAGE: ${BASE_IMAGE:-intel/intel-optimized-pytorch}
+        BASE_TAG: ${BASE_TAG:-2.1-pip-base}
+      dockerfile: docker/pyt-cpu/bert-large-inference/pytorch-bert-large-inference.Dockerfile-${OS:-ubuntu}
     command: >
       bash -c "python -c 'import torch; import intel_extension_for_pytorch as ipex; print(\"torch:\", torch.__version__, \" ipex:\",ipex.__version__)'"
-  spr-maskrcnn-inference:
-    image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-object-detection-spr-maskrcnn-inference
+  maskrcnn-inference:
+    image: ${BASE_IMAGE:-intel/intel-optimized-pytorch}-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-pyt-object-detection-maskrcnn-inference
     pull_policy: always
     build:
       context: ../../
@@ -40,13 +40,13 @@ services:
         http_proxy: ${http_proxy}
         https_proxy: ${https_proxy}
         no_proxy: ${no_proxy}
-        BASE_IMAGE: ${BASE_IMAGE:-intel/intel-extension-for-pytorch}
-        BASE_TAG: ${BASE_TAG:-2.0.0-pip-base}
-      dockerfile: docker/pyt-cpu/spr-maskrcnn-inference/pytorch-spr-maskrcnn-inference.Dockerfile
+        BASE_IMAGE: ${BASE_IMAGE:-intel/intel-optimized-pytorch}
+        BASE_TAG: ${BASE_TAG:-2.1-pip-base}
+      dockerfile: docker/pyt-cpu/maskrcnn-inference/pytorch-maskrcnn-inference.Dockerfile-${OS:-ubuntu}
     command: >
       bash -c "python -c 'import torch; import intel_extension_for_pytorch as ipex; print(\"torch:\", torch.__version__, \" ipex:\",ipex.__version__)'"
-  spr-maskrcnn-training:
-    image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-object-detection-spr-maskrcnn-training
+  maskrcnn-training:
+    image: ${BASE_IMAGE:-intel/intel-optimized-pytorch}-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-pyt-object-detection-maskrcnn-training
     pull_policy: always
     build:
       context: ../../
@@ -54,13 +54,13 @@ services:
         http_proxy: ${http_proxy}
         https_proxy: ${https_proxy}
         no_proxy: ${no_proxy}
-        BASE_IMAGE: ${BASE_IMAGE:-intel/intel-extension-for-pytorch}
-        BASE_TAG: ${BASE_TAG:-2.0.0-pip-base}
-      dockerfile: docker/pyt-cpu/spr-maskrcnn-training/pytorch-spr-maskrcnn-training.Dockerfile
+        BASE_IMAGE: ${BASE_IMAGE:-intel/intel-optimized-pytorch}
+        BASE_TAG: ${BASE_TAG:-2.1-pip-base}
+      dockerfile: docker/pyt-cpu/maskrcnn-training/pytorch-maskrcnn-training.Dockerfile-${OS:-ubuntu}
     command: >
       bash -c "python -c 'import torch; import intel_extension_for_pytorch as ipex; print(\"torch:\", torch.__version__, \" ipex:\",ipex.__version__)'"
-  spr-resnet50-training:
-    image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-resnet50-training
+  resnet50-training:
+    image: ${BASE_IMAGE:-intel/intel-optimized-pytorch}-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-pyt-pytorch-image-recognition-resnet50-training
     pull_policy: always
     build:
       context: ../../
@@ -68,13 +68,13 @@ services:
         http_proxy: ${http_proxy}
         https_proxy: ${https_proxy}
         no_proxy: ${no_proxy}
-        BASE_IMAGE: ${BASE_IMAGE:-intel/intel-extension-for-pytorch}
-        BASE_TAG: ${BASE_TAG:-2.0.0-pip-base}
-      dockerfile: docker/pyt-cpu/spr-resnet50-training/pytorch-spr-resnet50-training.Dockerfile
+        BASE_IMAGE: ${BASE_IMAGE:-intel/intel-optimized-pytorch}
+        BASE_TAG: ${BASE_TAG:-2.1-pip-base}
+      dockerfile: docker/pyt-cpu/resnet50-training/pytorch-resnet50-training.Dockerfile-${OS:-ubuntu}
     command: >
       bash -c "python -c 'import torch; import intel_extension_for_pytorch as ipex; print(\"torch:\", torch.__version__, \" ipex:\",ipex.__version__)'"
-  spr-resnet50-inference:
-    image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-resnet50-inference
+  resnet50-inference:
+    image: ${BASE_IMAGE:-intel/intel-optimized-pytorch}-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-pyt-pytorch-image-recognition-resnet50-inference
     pull_policy: always
     build:
       context: ../../
@@ -82,13 +82,13 @@ services:
         http_proxy: ${http_proxy}
         https_proxy: ${https_proxy}
         no_proxy: ${no_proxy}
-        BASE_IMAGE: ${BASE_IMAGE:-intel/intel-extension-for-pytorch}
-        BASE_TAG: ${BASE_TAG:-2.0.0-pip-base}
-      dockerfile: docker/pyt-cpu/spr-resnet50-inference/pytorch-spr-resnet50-inference.Dockerfile
+        BASE_IMAGE: ${BASE_IMAGE:-intel/intel-optimized-pytorch}
+        BASE_TAG: ${BASE_TAG:-2.1-pip-base}
+      dockerfile: docker/pyt-cpu/resnet50-inference/pytorch-resnet50-inference.Dockerfile-${OS:-ubuntu}
     command: >
       bash -c "python -c 'import torch; import intel_extension_for_pytorch as ipex; print(\"torch:\", torch.__version__, \" ipex:\",ipex.__version__)'"
-  spr-ssd-resnet34-inference:
-    image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-object-detection-ssd-resnet34-inference
+  ssd-resnet34-inference:
+    image: ${BASE_IMAGE:-intel/intel-optimized-pytorch}-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-pyt-object-detection-ssd-resnet34-inference
     pull_policy: always
     build:
       context: ../../
@@ -96,13 +96,13 @@ services:
         http_proxy: ${http_proxy}
         https_proxy: ${https_proxy}
         no_proxy: ${no_proxy}
-        BASE_IMAGE: ${BASE_IMAGE:-intel/intel-extension-for-pytorch}
-        BASE_TAG: ${BASE_TAG:-2.0.0-pip-base}
-      dockerfile: docker/pyt-cpu/spr-ssd-resnet34-inference/pytorch-spr-ssd-resnet34-inference.Dockerfile
+        BASE_IMAGE: ${BASE_IMAGE:-intel/intel-optimized-pytorch}
+        BASE_TAG: ${BASE_TAG:-2.1-pip-base}
+      dockerfile: docker/pyt-cpu/ssd-resnet34-inference/pytorch-ssd-resnet34-inference.Dockerfile-${OS:-ubuntu}
     command: >
       bash -c "python -c 'import torch; import intel_extension_for_pytorch as ipex; print(\"torch:\", torch.__version__, \" ipex:\",ipex.__version__)'"
-  spr-ssd-resnet34-training:
-    image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-object-detection-ssd-resnet34-training
+  ssd-resnet34-training:
+    image: ${BASE_IMAGE:-intel/intel-optimized-pytorch}-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-pyt-object-detection-ssd-resnet34-training
     pull_policy: always
     build:
       context: ../../
@@ -110,13 +110,13 @@ services:
         http_proxy: ${http_proxy}
         https_proxy: ${https_proxy}
         no_proxy: ${no_proxy}
-        BASE_IMAGE: ${BASE_IMAGE:-intel/intel-extension-for-pytorch}
-        BASE_TAG: ${BASE_TAG:-2.0.0-pip-base}
-      dockerfile: docker/pyt-cpu/spr-ssd-resnet34-training/pytorch-spr-ssd-resnet34-training.Dockerfile
+        BASE_IMAGE: ${BASE_IMAGE:-intel/intel-optimized-pytorch}
+        BASE_TAG: ${BASE_TAG:-2.1-pip-base}
+      dockerfile: docker/pyt-cpu/ssd-resnet34-training/pytorch-ssd-resnet34-training.Dockerfile-${OS:-ubuntu}
     command: >
       bash -c "python -c 'import torch; import intel_extension_for_pytorch as ipex; print(\"torch:\", torch.__version__, \" ipex:\",ipex.__version__)'"
-  spr-resnext-32x16d-inference:
-    image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-resnext-32x16d-inference
+  resnext-32x16d-inference:
+    image: ${BASE_IMAGE:-intel/intel-optimized-pytorch}-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-pyt-resnext-32x16d-inference
     pull_policy: always
     build:
       context: ../../
@@ -124,13 +124,13 @@ services:
         http_proxy: ${http_proxy}
         https_proxy: ${https_proxy}
         no_proxy: ${no_proxy}
-        BASE_IMAGE: ${BASE_IMAGE:-intel/intel-extension-for-pytorch}
-        BASE_TAG: ${BASE_TAG:-2.0.0-pip-base}
-      dockerfile: docker/pyt-cpu/spr-resnext-32x16d-inference/pytorch-resnext-32x16d-inference.Dockerfile
+        BASE_IMAGE: ${BASE_IMAGE:-intel/intel-optimized-pytorch}
+        BASE_TAG: ${BASE_TAG:-2.1-pip-base}
+      dockerfile: docker/pyt-cpu/resnext-32x16d-inference/pytorch-resnext-32x16d-inference.Dockerfile-${OS:-ubuntu}
     command: >
       bash -c "python -c 'import torch; import intel_extension_for_pytorch as ipex; print(\"torch:\", torch.__version__, \" ipex:\",ipex.__version__)'"
-  spr-dlrm-inference:
-    image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-recommendation-dlrm-inference
+  dlrm-inference:
+    image: ${BASE_IMAGE:-intel/intel-optimized-pytorch}-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-pyt-recommendation-dlrm-inference
     pull_policy: always
     build:
       context: ../../
@@ -138,13 +138,13 @@ services:
         http_proxy: ${http_proxy}
         https_proxy: ${https_proxy}
         no_proxy: ${no_proxy}
-        BASE_IMAGE: ${BASE_IMAGE:-intel/intel-extension-for-pytorch}
-        BASE_TAG: ${BASE_TAG:-2.0.0-pip-base}
-      dockerfile: docker/pyt-cpu/spr-dlrm-inference/pytorch-spr-dlrm-inference.Dockerfile
+        BASE_IMAGE: ${BASE_IMAGE:-intel/intel-optimized-pytorch}
+        BASE_TAG: ${BASE_TAG:-2.1-pip-base}
+      dockerfile: docker/pyt-cpu/dlrm-inference/pytorch-dlrm-inference.Dockerfile-${OS:-ubuntu}
     command: >
       bash -c "python -c 'import torch; import intel_extension_for_pytorch as ipex; print(\"torch:\", torch.__version__, \" ipex:\",ipex.__version__)'"
-  spr-dlrm-training:
-    image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-recommendation-dlrm-training
+  dlrm-training:
+    image: ${BASE_IMAGE:-intel/intel-optimized-pytorch}-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-pyt-recommendation-dlrm-training
     pull_policy: always
     build:
       context: ../../
@@ -152,13 +152,13 @@ services:
         http_proxy: ${http_proxy}
         https_proxy: ${https_proxy}
         no_proxy: ${no_proxy}
-        BASE_IMAGE: ${BASE_IMAGE:-intel/intel-extension-for-pytorch}
-        BASE_TAG: ${BASE_TAG:-2.0.0-pip-base}
-      dockerfile: docker/pyt-cpu/spr-dlrm-training/pytorch-spr-dlrm-training.Dockerfile
+        BASE_IMAGE: ${BASE_IMAGE:-intel/intel-optimized-pytorch}
+        BASE_TAG: ${BASE_TAG:-2.1-pip-base}
+      dockerfile: docker/pyt-cpu/dlrm-training/pytorch-dlrm-training.Dockerfile-${OS:-ubuntu}
     command: >
       bash -c "python -c 'import torch; import intel_extension_for_pytorch as ipex; print(\"torch:\", torch.__version__, \" ipex:\",ipex.__version__)'"
-  spr-rnnt-training:
-    image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-rnnt-training
+  rnnt-training:
+    image: ${BASE_IMAGE:-intel/intel-optimized-pytorch}-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-pyt-language-modeling-rnnt-training
     pull_policy: always
     build:
       context: ../../
@@ -166,13 +166,13 @@ services:
         http_proxy: ${http_proxy}
         https_proxy: ${https_proxy}
         no_proxy: ${no_proxy}
-        BASE_IMAGE: ${BASE_IMAGE:-intel/intel-extension-for-pytorch}
-        BASE_TAG: ${BASE_TAG:-2.0.0-pip-base}
-      dockerfile: docker/pyt-cpu/spr-rnnt-training/pytorch-spr-rnnt-training.Dockerfile
+        BASE_IMAGE: ${BASE_IMAGE:-intel/intel-optimized-pytorch}
+        BASE_TAG: ${BASE_TAG:-2.1-pip-base}
+      dockerfile: docker/pyt-cpu/rnnt-training/pytorch-rnnt-training.Dockerfile-${OS:-ubuntu}
     command: >
       bash -c "python -c 'import torch; import intel_extension_for_pytorch as ipex; print(\"torch:\", torch.__version__, \" ipex:\",ipex.__version__)'"
-  spr-rnnt-inference:
-    image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-rnnt-inference
+  rnnt-inference:
+    image: ${BASE_IMAGE:-intel/intel-optimized-pytorch}-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-pyt-language-modeling-rnnt-inference
     pull_policy: always
     build:
       context: ../../
@@ -180,13 +180,13 @@ services:
         http_proxy: ${http_proxy}
         https_proxy: ${https_proxy}
         no_proxy: ${no_proxy}
-        BASE_IMAGE: ${BASE_IMAGE:-intel/intel-extension-for-pytorch}
-        BASE_TAG: ${BASE_TAG:-2.0.0-pip-base}
-      dockerfile: docker/pyt-cpu/spr-rnnt-inference/pytorch-spr-rnnt-inference.Dockerfile
+        BASE_IMAGE: ${BASE_IMAGE:-intel/intel-optimized-pytorch}
+        BASE_TAG: ${BASE_TAG:-2.1-pip-base}
+      dockerfile: docker/pyt-cpu/rnnt-inference/pytorch-rnnt-inference.Dockerfile-${OS:-ubuntu}
     command: >
       bash -c "python -c 'import torch; import intel_extension_for_pytorch as ipex; print(\"torch:\", torch.__version__, \" ipex:\",ipex.__version__)'"
-  spr-distilbert-inference:
-    image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-distilbert-inference
+  distilbert-inference:
+    image: ${BASE_IMAGE:-intel/intel-optimized-pytorch}-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-pyt-language-modeling-distilbert-inference
     pull_policy: always
     build:
       context: ../../
@@ -194,13 +194,13 @@ services:
         http_proxy: ${http_proxy}
         https_proxy: ${https_proxy}
         no_proxy: ${no_proxy}
-        BASE_IMAGE: ${BASE_IMAGE:-intel/intel-extension-for-pytorch}
-        BASE_TAG: ${BASE_TAG:-2.0.0-pip-base}
-      dockerfile: docker/pyt-cpu/spr-distilbert-inference/pytorch-spr-distilbert-inference.Dockerfile
+        BASE_IMAGE: ${BASE_IMAGE:-intel/intel-optimized-pytorch}
+        BASE_TAG: ${BASE_TAG:-2.1-pip-base}
+      dockerfile: docker/pyt-cpu/distilbert-inference/pytorch-distilbert-inference.Dockerfile-${OS:-ubuntu}
     command: >
       bash -c "python -c 'import torch; import intel_extension_for_pytorch as ipex; print(\"torch:\", torch.__version__, \" ipex:\",ipex.__version__)'"
-  spr-bert-large-training:
-    image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-language-modeling-bert-large-training
+  bert-large-training:
+    image: ${BASE_IMAGE:-intel/intel-optimized-pytorch}-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-pyt-language-modeling-bert-large-training
     pull_policy: always
     build:
       context: ../../
@@ -208,8 +208,8 @@ services:
         http_proxy: ${http_proxy}
         https_proxy: ${https_proxy}
         no_proxy: ${no_proxy}
-        BASE_IMAGE: ${BASE_IMAGE:-intel/intel-extension-for-pytorch}
-        BASE_TAG: ${BASE_TAG:-2.0.0-pip-base}
-      dockerfile: docker/pyt-cpu/spr-bert-large-training/pytorch-spr-bert-large-training.Dockerfile
+        BASE_IMAGE: ${BASE_IMAGE:-intel/intel-optimized-pytorch}
+        BASE_TAG: ${BASE_TAG:-2.1-pip-base}
+      dockerfile: docker/pyt-cpu/bert-large-training/pytorch-bert-large-training.Dockerfile-${OS:-ubuntu}
     command: >
       bash -c "python -c 'import torch; import intel_extension_for_pytorch as ipex; print(\"torch:\", torch.__version__, \" ipex:\",ipex.__version__)'"
diff --git a/docker/pyt-cpu/maskrcnn-inference/pytorch-maskrcnn-inference.Dockerfile-centos b/docker/pyt-cpu/maskrcnn-inference/pytorch-maskrcnn-inference.Dockerfile-centos
new file mode 100644
index 000000000..1061c7810
--- /dev/null
+++ b/docker/pyt-cpu/maskrcnn-inference/pytorch-maskrcnn-inference.Dockerfile-centos
@@ -0,0 +1,68 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+ARG BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG BASE_TAG="2.0.0-pip-base"
+
+FROM ${BASE_IMAGE}:${BASE_TAG} AS intel-optimized-pytorch
+
+SHELL ["/bin/bash", "-c"]
+ENV LANG C.UTF-8
+
+WORKDIR /workspace/pytorch-maskrcnn-inference
+
+RUN yum update -y && \
+    yum install -y \
+    ca-certificates \
+    git \
+    cmake>=3.19.6 \
+    make \
+    autoconf \
+    bzip2 \
+    tar
+    
+RUN git clone https://github.com/jemalloc/jemalloc.git && \
+    cd jemalloc && \
+    git checkout c8209150f9d219a137412b06431c9d52839c7272 && \
+    ./autogen.sh && \
+    ./configure --prefix=/workspace/lib/jemalloc && \
+    make && \ 
+    make install
+
+RUN pip install packaging intel-openmp matplotlib Pillow pycocotools \
+    yacs opencv-python cityscapesscripts defusedxml
+    
+ENV LD_PRELOAD="/workspace/lib/jemalloc/lib/libjemalloc.so":"/usr/local/lib/libiomp5.so":$LD_PRELOAD
+ENV MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+
+COPY models/object_detection/pytorch/maskrcnn models/object_detection/pytorch/maskrcnn
+COPY quickstart/common quickstart/common
+
+COPY quickstart/object_detection/pytorch/maskrcnn/inference/cpu/inference_realtime.sh quickstart/inference_realtime.sh
+COPY quickstart/object_detection/pytorch/maskrcnn/inference/cpu/inference_throughput.sh quickstart/inference_throughput.sh
+COPY quickstart/object_detection/pytorch/maskrcnn/inference/cpu/accuracy.sh quickstart/accuracy.sh 
+
+RUN cd /workspace/pytorch-maskrcnn-inference/models/object_detection/pytorch/maskrcnn/maskrcnn-benchmark && \
+    python setup.py develop && \
+    cd -
+
+RUN yum update -y && \
+    yum install -y \
+    numactl \
+    mesa-libGL && \
+    yum clean all
+
+COPY LICENSE licenses/LICENSE
+COPY third_party licenses/third_party
diff --git a/docker/pyt-cpu/maskrcnn-inference/pytorch-maskrcnn-inference.Dockerfile-ubuntu b/docker/pyt-cpu/maskrcnn-inference/pytorch-maskrcnn-inference.Dockerfile-ubuntu
new file mode 100644
index 000000000..8c74f1046
--- /dev/null
+++ b/docker/pyt-cpu/maskrcnn-inference/pytorch-maskrcnn-inference.Dockerfile-ubuntu
@@ -0,0 +1,69 @@
+# Copyright (c) 2020-2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+ARG BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG BASE_TAG="2.0.0-pip-base"
+
+FROM ${BASE_IMAGE}:${BASE_TAG} AS intel-optimized-pytorch
+
+WORKDIR /workspace/pytorch-maskrcnn-inference
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+    build-essential \
+    ca-certificates \
+    git \
+    wget \
+    make \
+    cmake \
+    autoconf \
+    bzip2 \
+    tar
+
+RUN pip install packaging intel-openmp matplotlib Pillow pycocotools \
+    yacs opencv-python cityscapesscripts defusedxml
+
+RUN git clone https://github.com/jemalloc/jemalloc.git && \
+    cd jemalloc && \
+    git checkout c8209150f9d219a137412b06431c9d52839c7272 && \
+    ./autogen.sh && \
+    ./configure --prefix=/workspace/lib/jemalloc && \
+    make && \ 
+    make install
+
+ENV LD_PRELOAD="/workspace/lib/jemalloc/lib/libjemalloc.so":"/usr/local/lib/libiomp5.so":$LD_PRELOAD
+ENV MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+
+COPY models/object_detection/pytorch/maskrcnn models/object_detection/pytorch/maskrcnn
+COPY quickstart/common quickstart/common
+
+COPY quickstart/object_detection/pytorch/maskrcnn/inference/cpu/inference_realtime.sh quickstart/inference_realtime.sh
+COPY quickstart/object_detection/pytorch/maskrcnn/inference/cpu/inference_throughput.sh quickstart/inference_throughput.sh
+COPY quickstart/object_detection/pytorch/maskrcnn/inference/cpu/accuracy.sh quickstart/accuracy.sh 
+
+RUN apt-get update && apt-get install -y python3.10-dev && \
+    cd /workspace/pytorch-maskrcnn-inference/models/object_detection/pytorch/maskrcnn/maskrcnn-benchmark && \
+    python setup.py develop && \
+    cd -
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+    numactl \
+    libgl1 \
+    libglib2.0-0 \
+    libegl1-mesa 
+
+COPY LICENSE licenses/LICENSE
+COPY third_party licenses/third_party
diff --git a/docker/pyt-cpu/maskrcnn-inference/tests.yaml b/docker/pyt-cpu/maskrcnn-inference/tests.yaml
new file mode 100644
index 000000000..9fcb96ba8
--- /dev/null
+++ b/docker/pyt-cpu/maskrcnn-inference/tests.yaml
@@ -0,0 +1,168 @@
+fp32-realtime-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-object-detection-maskrcnn-inference
+  cmd: quickstart/inference_realtime.sh fp32 jit
+  env:
+    PRECISION: fp32
+    MODE: jit
+    CHECKPOINT_DIR: /pytorch/pretrained_models/maskrcnn
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/maskrcnn
+      dst: /pytorch/pretrained_models/maskrcnn
+fp32-throughput-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-object-detection-maskrcnn-inference
+  cmd: quickstart/inference_throughput.sh fp32 jit
+  env:
+    PRECISION: fp32
+    MODE: jit
+    CHECKPOINT_DIR: /pytorch/pretrained_models/maskrcnn
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/maskrcnn
+      dst: /pytorch/pretrained_models/maskrcnn
+fp32-accuracy-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-object-detection-maskrcnn-inference
+  cmd: quickstart/accuracy.sh fp32 jit
+  env:
+    PRECISION: fp32
+    MODE: jit
+    CHECKPOINT_DIR: /pytorch/pretrained_models/maskrcnn
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/maskrcnn
+      dst: /pytorch/pretrained_models/maskrcnn
+avx-fp32-realtime-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-object-detection-maskrcnn-inference
+  cmd: quickstart/inference_realtime.sh avx-fp32 jit
+  env:
+    PRECISION: avx-fp32
+    MODE: jit
+    CHECKPOINT_DIR: /pytorch/pretrained_models/maskrcnn
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/maskrcnn
+      dst: /pytorch/pretrained_models/maskrcnn
+avx-fp32-throughput-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-object-detection-maskrcnn-inference
+  cmd: quickstart/inference_throughput.sh avx-fp32 jit
+  env:
+    PRECISION: avx-fp32
+    MODE: jit
+    CHECKPOINT_DIR: /pytorch/pretrained_models/maskrcnn
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/maskrcnn
+      dst: /pytorch/pretrained_models/maskrcnn
+avx-fp32-accuracy-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-object-detection-maskrcnn-inference
+  cmd: quickstart/accuracy.sh avx-fp32 jit
+  env:
+    PRECISION: avx-fp32
+    MODE: jit
+    CHECKPOINT_DIR: /pytorch/pretrained_models/maskrcnn
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/maskrcnn
+      dst: /pytorch/pretrained_models/maskrcnn
+bf16-realtime-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-object-detection-maskrcnn-inference
+  cmd: quickstart/inference_realtime.sh bf16 jit
+  env:
+    PRECISION: bf16
+    MODE: jit
+    CHECKPOINT_DIR: /pytorch/pretrained_models/maskrcnn
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/maskrcnn
+      dst: /pytorch/pretrained_models/maskrcnn
+bf16-throughput-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-object-detection-maskrcnn-inference
+  cmd: quickstart/inference_throughput.sh bf16 jit
+  env:
+    PRECISION: bf16
+    MODE: jit
+    CHECKPOINT_DIR: /pytorch/pretrained_models/maskrcnn
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/maskrcnn
+      dst: /pytorch/pretrained_models/maskrcnn
+bf16-accuracy-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-object-detection-maskrcnn-inference
+  cmd: quickstart/accuracy.sh bf16 jit
+  env:
+    PRECISION: bf16
+    MODE: jit
+    CHECKPOINT_DIR: /pytorch/pretrained_models/maskrcnn
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/maskrcnn
+      dst: /pytorch/pretrained_models/maskrcnn
+bf32-realtime-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-object-detection-maskrcnn-inference
+  cmd: quickstart/inference_realtime.sh bf32 jit
+  env:
+    PRECISION: bf16
+    MODE: jit
+    CHECKPOINT_DIR: /pytorch/pretrained_models/maskrcnn
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/maskrcnn
+      dst: /pytorch/pretrained_models/maskrcnn
+bf32-throughput-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-object-detection-maskrcnn-inference
+  cmd: quickstart/inference_throughput.sh bf32 jit
+  env:
+    PRECISION: bf32
+    MODE: jit
+    CHECKPOINT_DIR: /pytorch/pretrained_models/maskrcnn
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/maskrcnn
+      dst: /pytorch/pretrained_models/maskrcnn
+bf32-accuracy-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-object-detection-maskrcnn-inference
+  cmd: quickstart/accuracy.sh bf32 jit
+  env:
+    PRECISION: bf32
+    MODE: jit
+    CHECKPOINT_DIR: /pytorch/pretrained_models/maskrcnn
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/maskrcnn
+      dst: /pytorch/pretrained_models/maskrcnn
diff --git a/docker/pyt-cpu/maskrcnn-training/pytorch-maskrcnn-training.Dockerfile-centos b/docker/pyt-cpu/maskrcnn-training/pytorch-maskrcnn-training.Dockerfile-centos
new file mode 100644
index 000000000..cba84bdb6
--- /dev/null
+++ b/docker/pyt-cpu/maskrcnn-training/pytorch-maskrcnn-training.Dockerfile-centos
@@ -0,0 +1,66 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+ARG BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG BASE_TAG="2.0.0-pip-base"
+
+FROM ${BASE_IMAGE}:${BASE_TAG} AS intel-optimized-pytorch
+
+SHELL ["/bin/bash", "-c"]
+ENV LANG C.UTF-8
+
+WORKDIR /workspace/pytorch-maskrcnn-training
+
+RUN yum update -y && \
+    yum install -y \
+    ca-certificates \
+    git \
+    cmake>=3.19.6 \
+    make \
+    autoconf \
+    bzip2 \
+    tar
+    
+RUN git clone https://github.com/jemalloc/jemalloc.git && \
+    cd jemalloc && \
+    git checkout c8209150f9d219a137412b06431c9d52839c7272 && \
+    ./autogen.sh && \
+    ./configure --prefix=/workspace/lib/jemalloc && \
+    make && \ 
+    make install
+
+RUN pip install packaging intel-openmp matplotlib Pillow pycocotools \
+    yacs opencv-python cityscapesscripts defusedxml
+
+ENV LD_PRELOAD="/workspace/lib/jemalloc/lib/libjemalloc.so":"/usr/local/lib/libiomp5.so":$LD_PRELOAD
+ENV MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+
+COPY models/object_detection/pytorch/maskrcnn models/object_detection/pytorch/maskrcnn
+COPY quickstart/object_detection/pytorch/maskrcnn/training/cpu/training.sh quickstart/training.sh
+
+RUN cd /workspace/pytorch-maskrcnn-training/models/object_detection/pytorch/maskrcnn/maskrcnn-benchmark && \
+    python setup.py develop && \
+    cd -
+
+ENV MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+
+RUN yum update -y && \
+    yum install -y \
+    numactl \
+    mesa-libGL && \
+    yum clean all
+
+COPY LICENSE licenses/LICENSE
+COPY third_party licenses/third_party
diff --git a/docker/pyt-cpu/maskrcnn-training/pytorch-maskrcnn-training.Dockerfile-ubuntu b/docker/pyt-cpu/maskrcnn-training/pytorch-maskrcnn-training.Dockerfile-ubuntu
new file mode 100644
index 000000000..204a058cb
--- /dev/null
+++ b/docker/pyt-cpu/maskrcnn-training/pytorch-maskrcnn-training.Dockerfile-ubuntu
@@ -0,0 +1,66 @@
+# Copyright (c) 2020-2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+
+ARG BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG BASE_TAG="2.0.0-pip-base"
+
+FROM ${BASE_IMAGE}:${BASE_TAG} AS intel-optimized-pytorch
+
+WORKDIR /workspace/pytorch-maskrcnn-training 
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+    build-essential \
+    ca-certificates \
+    git \
+    wget \
+    make \
+    cmake \
+    autoconf \
+    bzip2 \
+    tar
+
+RUN pip install matplotlib Pillow pycocotools && \
+    pip install yacs opencv-python cityscapesscripts transformers defusedxml intel-openmp 
+
+RUN git clone https://github.com/jemalloc/jemalloc.git && \
+    cd jemalloc && \
+    git checkout c8209150f9d219a137412b06431c9d52839c7272 && \
+    ./autogen.sh && \
+    ./configure --prefix=/workspace/lib/jemalloc && \
+    make && \ 
+    make install
+
+ENV LD_PRELOAD="/workspace/lib/jemalloc/lib/libjemalloc.so":"/usr/local/lib/libiomp5.so":$LD_PRELOAD
+
+COPY models/object_detection/pytorch/maskrcnn models/object_detection/pytorch/maskrcnn
+COPY quickstart/object_detection/pytorch/maskrcnn/training/cpu/training.sh quickstart/training.sh
+
+RUN apt-get update && apt-get install -y python3.10-dev && \
+    cd /workspace/pytorch-maskrcnn-training/models/object_detection/pytorch/maskrcnn/maskrcnn-benchmark && \
+    python setup.py develop 
+
+ENV MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+    numactl \
+    libgl1 \
+    libglib2.0-0 \
+    libegl1-mesa
+
+COPY LICENSE licenses/LICENSE
+COPY third_party licenses/third_party
diff --git a/docker/pyt-cpu/maskrcnn-training/tests.yaml b/docker/pyt-cpu/maskrcnn-training/tests.yaml
new file mode 100644
index 000000000..9c892ac46
--- /dev/null
+++ b/docker/pyt-cpu/maskrcnn-training/tests.yaml
@@ -0,0 +1,44 @@
+fp32-training:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-object-detection-maskrcnn-training
+  cmd: quickstart/training.sh fp32
+  env:
+    PRECISION: fp32
+    BATCH_SIZE: '112'
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+avx-fp32-training:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-object-detection-maskrcnn-training
+  cmd: quickstart/training.sh avx-fp32
+  env:
+    PRECISION: avx-fp32
+    BATCH_SIZE: '112'
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+bf16-training:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-object-detection-maskrcnn-training
+  cmd: quickstart/training.sh bf16
+  env:
+    PRECISION: bf16
+    BATCH_SIZE: '112'
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+bf32-training:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-object-detection-maskrcnn-training
+  cmd: quickstart/training.sh bf32
+  env:
+    PRECISION: bf32
+    BATCH_SIZE: '112'
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
diff --git a/docker/pyt-cpu/resnet50-inference/pytorch-resnet50-inference.Dockerfile-centos b/docker/pyt-cpu/resnet50-inference/pytorch-resnet50-inference.Dockerfile-centos
new file mode 100644
index 000000000..9655841a8
--- /dev/null
+++ b/docker/pyt-cpu/resnet50-inference/pytorch-resnet50-inference.Dockerfile-centos
@@ -0,0 +1,57 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+ARG BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG BASE_TAG="2.0.0-pip-base"
+
+FROM ${BASE_IMAGE}:${BASE_TAG} AS intel-optimized-pytorch
+
+WORKDIR /workspace/pytorch-resnet50-inference 
+
+RUN yum update -y && \
+    yum install -y \
+    ca-certificates \
+    git \
+    cmake>=3.19.6 \
+    make \
+    autoconf \
+    bzip2 \
+    tar
+
+RUN pip install packaging intel-openmp
+
+RUN git clone https://github.com/jemalloc/jemalloc.git && \
+    cd jemalloc && \
+    git checkout c8209150f9d219a137412b06431c9d52839c7272 && \
+    ./autogen.sh && \
+    ./configure --prefix=/workspace/lib/jemalloc && \
+    make && \ 
+    make install
+
+COPY quickstart/image_recognition/pytorch/resnet50/inference/cpu/calibration.sh quickstart/calibration.sh
+COPY quickstart/image_recognition/pytorch/resnet50/inference/cpu/accuracy.sh quickstart/accuracy.sh
+COPY quickstart/image_recognition/pytorch/resnet50/inference/cpu/inference_realtime.sh quickstart/inference_realtime.sh
+COPY quickstart/image_recognition/pytorch/resnet50/inference/cpu/inference_throughput.sh quickstart/inference_throughput.sh
+COPY models/image_recognition/pytorch/common models/image_recognition/pytorch/common 
+
+ENV LD_PRELOAD="/workspace/lib/jemalloc/lib/libjemalloc.so":"/usr/local/lib/libiomp5.so":$LD_PRELOAD
+ENV MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+
+RUN yum update -y && \
+    yum install -y \
+    numactl \
+    mesa-libGL 
+    
+COPY LICENSE licenses/LICENSE
+COPY third_party licenses/third_party
diff --git a/docker/pyt-cpu/resnet50-inference/pytorch-resnet50-inference.Dockerfile-ubuntu b/docker/pyt-cpu/resnet50-inference/pytorch-resnet50-inference.Dockerfile-ubuntu
new file mode 100644
index 000000000..bb2b9b460
--- /dev/null
+++ b/docker/pyt-cpu/resnet50-inference/pytorch-resnet50-inference.Dockerfile-ubuntu
@@ -0,0 +1,57 @@
+# Copyright (c) 2020-2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+ARG BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG BASE_TAG="2.0.0-pip-base"
+
+FROM ${BASE_IMAGE}:${BASE_TAG} AS intel-optimized-pytorch
+
+WORKDIR /workspace/pytorch-resnet50-inference 
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+    build-essential \
+    git \
+    cmake>=3.19.6 \
+    make \
+    autoconf \
+    bzip2 \
+    tar
+    
+RUN pip install packaging intel-openmp
+
+RUN git clone https://github.com/jemalloc/jemalloc.git && \
+    cd jemalloc && \
+    git checkout c8209150f9d219a137412b06431c9d52839c7272 && \
+    ./autogen.sh && \
+    ./configure --prefix=/workspace/lib/jemalloc && \
+    make && \ 
+    make install
+
+ENV LD_PRELOAD="/workspace/lib/jemalloc/lib/libjemalloc.so":"/usr/local/lib/libiomp5.so":$LD_PRELOAD
+ENV MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+
+COPY quickstart/image_recognition/pytorch/resnet50/inference/cpu/calibration.sh quickstart/calibration.sh
+COPY quickstart/image_recognition/pytorch/resnet50/inference/cpu/accuracy.sh quickstart/accuracy.sh
+COPY quickstart/image_recognition/pytorch/resnet50/inference/cpu/inference_realtime.sh quickstart/inference_realtime.sh
+COPY quickstart/image_recognition/pytorch/resnet50/inference/cpu/inference_throughput.sh quickstart/inference_throughput.sh
+COPY models/image_recognition/pytorch/common models/image_recognition/pytorch/common 
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+      numactl \
+      libegl1-mesa 
+
+COPY LICENSE licenses/LICENSE
+COPY third_party licenses/third_party
diff --git a/docker/pyt-cpu/resnet50-inference/tests.yaml b/docker/pyt-cpu/resnet50-inference/tests.yaml
new file mode 100644
index 000000000..aba0bc694
--- /dev/null
+++ b/docker/pyt-cpu/resnet50-inference/tests.yaml
@@ -0,0 +1,198 @@
+fp32-online_inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-pytorch-image-recognition-resnet50-inference
+  cmd: quickstart/inference_realtime.sh
+  shm-size: 8G
+  env:
+    PRECISION: fp32
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/Imagenet_FullData
+  volumes:
+    - src: /pytorch/Imagenet_FullData
+      dst: /pytorch/Imagenet_FullData
+avx-fp32-online_inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-pytorch-image-recognition-resnet50-inference
+  cmd: quickstart/inference_realtime.sh
+  shm-size: 8G
+  env:
+    PRECISION: avx-fp32
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/Imagenet_FullData
+  volumes:
+    - src: /pytorch/Imagenet_FullData
+      dst: /pytorch/Imagenet_FullData
+bf16-online_inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-pytorch-image-recognition-resnet50-inference
+  cmd: quickstart/inference_realtime.sh
+  shm-size: 8G
+  env:
+    PRECISION: bf16
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/Imagenet_FullData
+  volumes:
+    - src: /pytorch/Imagenet_FullData
+      dst: /pytorch/Imagenet_FullData
+int8-online_inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-pytorch-image-recognition-resnet50-inference
+  cmd: quickstart/inference_realtime.sh
+  shm-size: 8G
+  env:
+    PRECISION: int8
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/Imagenet_FullData
+  volumes:
+    - src: /pytorch/Imagenet_FullData
+      dst: /pytorch/Imagenet_FullData
+avx-int8-online_inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-pytorch-image-recognition-resnet50-inference
+  cmd: quickstart/inference_realtime.sh
+  shm-size: 8G
+  env:
+    PRECISION: avx-int8
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/Imagenet_FullData
+  volumes:
+    - src: /pytorch/Imagenet_FullData
+      dst: /pytorch/Imagenet_FullData
+bf32-online_inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-pytorch-image-recognition-resnet50-inference
+  cmd: quickstart/inference_realtime.sh
+  shm-size: 8G
+  env:
+    PRECISION: bf32
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/Imagenet_FullData
+  volumes:
+    - src: /pytorch/Imagenet_FullData
+      dst: /pytorch/Imagenet_FullData
+fp32-throughput_inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-pytorch-image-recognition-resnet50-inference
+  cmd: quickstart/inference_throughput.sh
+  shm-size: 8G
+  env:
+    PRECISION: fp32
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/Imagenet_FullData
+  volumes:
+    - src: /pytorch/Imagenet_FullData
+      dst: /pytorch/Imagenet_FullData
+avx-fp32-throughput_inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-pytorch-image-recognition-resnet50-inference
+  cmd: quickstart/inference_throughput.sh
+  shm-size: 8G
+  env:
+    PRECISION: avx-fp32
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/Imagenet_FullData
+  volumes:
+    - src: /pytorch/Imagenet_FullData
+      dst: /pytorch/Imagenet_FullData
+bf16-throughput_inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-pytorch-image-recognition-resnet50-inference
+  cmd: quickstart/inference_throughput.sh
+  shm-size: 8G
+  env:
+    PRECISION: bf16
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/Imagenet_FullData
+  volumes:
+    - src: /pytorch/Imagenet_FullData
+      dst: /pytorch/Imagenet_FullData
+int8-throughput_inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-pytorch-image-recognition-resnet50-inference
+  cmd: quickstart/inference_throughput.sh
+  shm-size: 8G
+  env:
+    PRECISION: int8
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/Imagenet_FullData
+  volumes:
+    - src: /pytorch/Imagenet_FullData
+      dst: /pytorch/Imagenet_FullData
+avx-int8-throughput_inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-pytorch-image-recognition-resnet50-inference
+  cmd: quickstart/inference_throughput.sh
+  shm-size: 8G
+  env:
+    PRECISION: avx-int8
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/Imagenet_FullData
+  volumes:
+    - src: /pytorch/Imagenet_FullData
+      dst: /pytorch/Imagenet_FullData
+bf32-throughput_inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-pytorch-image-recognition-resnet50-inference
+  cmd: quickstart/inference_throughput.sh
+  shm-size: 8G
+  env:
+    PRECISION: bf32
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/Imagenet_FullData
+  volumes:
+    - src: /pytorch/Imagenet_FullData
+      dst: /pytorch/Imagenet_FullData
+fp32-accuracy_inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-pytorch-image-recognition-resnet50-inference
+  cmd: quickstart/accuracy.sh
+  shm-size: 8G
+  env:
+    PRECISION: fp32
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/Imagenet_FullData
+  volumes:
+    - src: /pytorch/Imagenet_FullData
+      dst: /pytorch/Imagenet_FullData
+avx-fp32-accuracy_inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-pytorch-image-recognition-resnet50-inference
+  cmd: quickstart/accuracy.sh
+  shm-size: 8G
+  env:
+    PRECISION: avx-fp32
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/Imagenet_FullData
+  volumes:
+    - src: /pytorch/Imagenet_FullData
+      dst: /pytorch/Imagenet_FullData
+bf16-accuracy_inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-pytorch-image-recognition-resnet50-inference
+  cmd: quickstart/accuracy.sh
+  shm-size: 8G
+  env:
+    PRECISION: bf16
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/Imagenet_FullData
+  volumes:
+    - src: /pytorch/Imagenet_FullData
+      dst: /pytorch/Imagenet_FullData
+int8-accuracy_inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-pytorch-image-recognition-resnet50-inference
+  cmd: quickstart/accuracy.sh
+  shm-size: 8G
+  env:
+    PRECISION: int8
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/Imagenet_FullData
+  volumes:
+    - src: /pytorch/Imagenet_FullData
+      dst: /pytorch/Imagenet_FullData
+avx-int8-accuracy_inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-pytorch-image-recognition-resnet50-inference
+  cmd: quickstart/accuracy.sh
+  shm-size: 8G
+  env:
+    PRECISION: avx-int8
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/Imagenet_FullData
+  volumes:
+    - src: /pytorch/Imagenet_FullData
+      dst: /pytorch/Imagenet_FullData
+bf32-accuracy_inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-pytorch-image-recognition-resnet50-inference
+  cmd: quickstart/accuracy.sh
+  shm-size: 8G
+  env:
+    PRECISION: bf32
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/Imagenet_FullData
+  volumes:
+    - src: /pytorch/Imagenet_FullData
+      dst: /pytorch/Imagenet_FullData
diff --git a/docker/pyt-cpu/resnet50-training/pytorch-resnet50-training.Dockerfile-centos b/docker/pyt-cpu/resnet50-training/pytorch-resnet50-training.Dockerfile-centos
new file mode 100644
index 000000000..af8397a2a
--- /dev/null
+++ b/docker/pyt-cpu/resnet50-training/pytorch-resnet50-training.Dockerfile-centos
@@ -0,0 +1,66 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+ARG BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG BASE_TAG="2.0.0-pip-base"
+
+FROM ${BASE_IMAGE}:${BASE_TAG} AS intel-optimized-pytorch
+
+SHELL ["/bin/bash", "-c"]
+ENV LANG C.UTF-8
+WORKDIR /workspace/pytorch-resnet50-training
+
+RUN yum update -y && \
+    yum install -y \
+    ca-certificates \
+    git \
+    cmake>=3.19.6 \
+    make \
+    autoconf \
+    bzip2 \
+    tar
+
+RUN git clone https://github.com/jemalloc/jemalloc.git && \
+    cd jemalloc && \
+    git checkout c8209150f9d219a137412b06431c9d52839c7272 && \
+    ./autogen.sh && \
+    ./configure --prefix=/workspace/lib/jemalloc && \
+    make && \ 
+    make install
+
+RUN wget https://github.com/gperftools/gperftools/releases/download/gperftools-2.7.90/gperftools-2.7.90.tar.gz && \
+    tar -xzf gperftools-2.7.90.tar.gz && \
+    cd gperftools-2.7.90 && \
+    ./configure --prefix=/workspace/lib/tcmalloc && \
+    make && \
+    make install
+
+RUN pip install packaging intel-openmp 
+
+ENV LD_PRELOAD="/workspace/lib/jemalloc/lib/libjemalloc.so":"/workspace/lib/tcmalloc/lib/libtcmalloc.so":"/usr/local/lib/libiomp5.so":$LD_PRELOAD
+ENV MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+
+COPY models/image_recognition/pytorch/common models/image_recognition/pytorch/common
+COPY quickstart/image_recognition/pytorch/resnet50/training/cpu/training.sh quickstart/training.sh 
+
+
+RUN yum update -y && \
+    yum install -y \
+    numactl \
+    mesa-libGL && \
+    yum clean all
+
+COPY LICENSE licenses/LICENSE
+COPY third_party licenses/third_party
diff --git a/docker/pyt-cpu/resnet50-training/pytorch-resnet50-training.Dockerfile-ubuntu b/docker/pyt-cpu/resnet50-training/pytorch-resnet50-training.Dockerfile-ubuntu
new file mode 100644
index 000000000..7651a058f
--- /dev/null
+++ b/docker/pyt-cpu/resnet50-training/pytorch-resnet50-training.Dockerfile-ubuntu
@@ -0,0 +1,65 @@
+# Copyright (c) 2020-2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+ARG BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG BASE_TAG="2.0.0-pip-base"
+
+FROM ${BASE_IMAGE}:${BASE_TAG} AS intel-optimized-pytorch
+
+
+SHELL ["/bin/bash", "-c"]
+
+WORKDIR /workspace/pytorch-resnet50-training 
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+    build-essential \
+    git \
+    make \
+    cmake \
+    autoconf \
+    bzip2 \
+    tar
+
+RUN git clone https://github.com/jemalloc/jemalloc.git && \
+    cd jemalloc && \
+    git checkout c8209150f9d219a137412b06431c9d52839c7272 && \
+    ./autogen.sh && \
+    ./configure --prefix=/workspace/lib/jemalloc && \
+    make && \ 
+    make install
+
+RUN wget https://github.com/gperftools/gperftools/releases/download/gperftools-2.7.90/gperftools-2.7.90.tar.gz && \
+    tar -xzf gperftools-2.7.90.tar.gz && \
+    cd gperftools-2.7.90 && \
+    ./configure --prefix=/workspace/lib/tcmalloc && \
+    make && \
+    make install
+
+RUN pip install packaging intel-openmp 
+
+ENV LD_PRELOAD="/workspace/lib/jemalloc/lib/libjemalloc.so":"/workspace/lib/tcmalloc/lib/libtcmalloc.so":"/usr/local/lib/libiomp5.so":$LD_PRELOAD
+ENV MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+
+COPY models/image_recognition/pytorch/common models/image_recognition/pytorch/common
+COPY quickstart/image_recognition/pytorch/resnet50/training/cpu/training.sh quickstart/training.sh
+    
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+    numactl \
+    libegl1-mesa 
+
+COPY LICENSE licenses/LICENSE
+COPY third_party licenses/third_party
diff --git a/docker/pyt-cpu/resnet50-training/tests.yaml b/docker/pyt-cpu/resnet50-training/tests.yaml
new file mode 100644
index 000000000..6bfd6613c
--- /dev/null
+++ b/docker/pyt-cpu/resnet50-training/tests.yaml
@@ -0,0 +1,45 @@
+fp32-training:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-pytorch-image-recognition-resnet50-training
+  cmd: quickstart/training.sh
+  env:
+    PRECISION: fp32
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/Imagenet_FullData/
+    TRAINING_EPOCHS: '1'
+  volumes:
+    - src: /pytorch/Imagenet_FullData/
+      dst: /pytorch/Imagenet_FullData/
+avx-fp32-training:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-pytorch-image-recognition-resnet50-training
+  cmd: quickstart/training.sh
+  env:
+    PRECISION: avx-fp32
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/Imagenet_FullData/
+    TRAINING_EPOCHS: '1'
+  volumes:
+    - src: /pytorch/Imagenet_FullData/
+      dst: /pytorch/Imagenet_FullData/
+bf16-training:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-pytorch-image-recognition-resnet50-training
+  cmd: quickstart/training.sh
+  env:
+    PRECISION: bf16
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/Imagenet_FullData/
+    TRAINING_EPOCHS: '1'
+  volumes:
+    - src: /pytorch/Imagenet_FullData/
+      dst: /pytorch/Imagenet_FullData/
+bf32-training:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-pytorch-image-recognition-resnet50-training
+  cmd: quickstart/training.sh
+  env:
+    PRECISION: bf32
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/Imagenet_FullData/
+    TRAINING_EPOCHS: '1'
+  volumes:
+    - src: /pytorch/Imagenet_FullData/
+      dst: /pytorch/Imagenet_FullData/
+
diff --git a/docker/pyt-cpu/resnext-32x16d-inference/pytorch-resnext-32x16d-inference.Dockerfile-centos b/docker/pyt-cpu/resnext-32x16d-inference/pytorch-resnext-32x16d-inference.Dockerfile-centos
new file mode 100644
index 000000000..adc93b5b0
--- /dev/null
+++ b/docker/pyt-cpu/resnext-32x16d-inference/pytorch-resnext-32x16d-inference.Dockerfile-centos
@@ -0,0 +1,59 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+ARG BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG BASE_TAG="2.0.0-pip-base"
+
+FROM ${BASE_IMAGE}:${BASE_TAG} AS intel-optimized-pytorch
+
+WORKDIR /workspace/pytorch-resnext-32x16d-inference
+
+RUN yum update -y && \
+    yum install -y \
+    ca-certificates \
+    git \
+    cmake>=3.19.6 \
+    make \
+    autoconf \
+    bzip2 \
+    tar
+
+RUN git clone https://github.com/jemalloc/jemalloc.git && \
+    cd jemalloc && \
+    git checkout c8209150f9d219a137412b06431c9d52839c7272 && \
+    ./autogen.sh && \
+    ./configure --prefix=/workspace/lib/jemalloc && \
+    make && \ 
+    make install
+
+RUN pip install packaging intel-openmp
+
+ENV LD_PRELOAD="/workspace/lib/jemalloc/lib/libjemalloc.so":"/usr/local/lib/libiomp5.so":$LD_PRELOAD
+ENV MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+
+COPY models/image_recognition/pytorch/common models/image_recognition/pytorch/common
+COPY quickstart/common quickstart/common
+COPY quickstart/image_recognition/pytorch/resnext-32x16d/inference/cpu/inference_realtime.sh quickstart/inference_realtime.sh
+COPY quickstart/image_recognition/pytorch/resnext-32x16d/inference/cpu/inference_throughput.sh quickstart//inference_throughput.sh
+COPY quickstart/image_recognition/pytorch/resnext-32x16d/inference/cpu/accuracy.sh quickstart/accuracy.sh
+
+
+
+RUN yum update -y && \
+    yum install -y \
+    numactl \
+    mesa-libGL 
+    
+COPY LICENSE licenses/LICENSE
+COPY third_party licenses/third_party
diff --git a/docker/pyt-cpu/resnext-32x16d-inference/pytorch-resnext-32x16d-inference.Dockerfile-ubuntu b/docker/pyt-cpu/resnext-32x16d-inference/pytorch-resnext-32x16d-inference.Dockerfile-ubuntu
new file mode 100644
index 000000000..5779c746d
--- /dev/null
+++ b/docker/pyt-cpu/resnext-32x16d-inference/pytorch-resnext-32x16d-inference.Dockerfile-ubuntu
@@ -0,0 +1,61 @@
+# Copyright (c) 2020-2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ===========================================================================
+
+ARG BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG BASE_TAG="2.0.0-pip-base"
+
+FROM ${BASE_IMAGE}:${BASE_TAG} AS intel-optimized-pytorch
+
+WORKDIR /workspace/pytorch-resnext-32x16d-inference 
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+    build-essential \
+    ca-certificates \
+    git \
+    wget \
+    make \
+    cmake \
+    autoconf \
+    bzip2 \
+    tar
+
+RUN git clone https://github.com/jemalloc/jemalloc.git && \
+    cd jemalloc && \
+    git checkout c8209150f9d219a137412b06431c9d52839c7272 && \
+    ./autogen.sh && \
+    ./configure --prefix=/workspace/lib/jemalloc && \
+    make && \ 
+    make install
+
+RUN pip install packaging intel-openmp
+
+ENV LD_PRELOAD="/workspace/lib/jemalloc/lib/libjemalloc.so":"/usr/local/lib/libiomp5.so":$LD_PRELOAD
+ENV MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+
+COPY models/image_recognition/pytorch/common models/image_recognition/pytorch/common
+COPY quickstart/common quickstart/common
+
+COPY quickstart/image_recognition/pytorch/resnext-32x16d/inference/cpu/inference_realtime.sh quickstart/inference_realtime.sh
+COPY quickstart/image_recognition/pytorch/resnext-32x16d/inference/cpu/inference_throughput.sh quickstart//inference_throughput.sh
+COPY quickstart/image_recognition/pytorch/resnext-32x16d/inference/cpu/accuracy.sh quickstart/accuracy.sh
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+    numactl \
+    libegl1-mesa 
+
+COPY LICENSE licenses/LICENSE
+COPY third_party licenses/third_party
diff --git a/docker/pyt-cpu/resnext-32x16d-inference/tests.yaml b/docker/pyt-cpu/resnext-32x16d-inference/tests.yaml
new file mode 100644
index 000000000..161695fdb
--- /dev/null
+++ b/docker/pyt-cpu/resnext-32x16d-inference/tests.yaml
@@ -0,0 +1,198 @@
+fp32-realtime-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-pytorch-image-recognition-resnext-32x16d-inference
+  cmd: quickstart/inference_realtime.sh
+  shm-size: 8G
+  env:
+    PRECISION: fp32
+    DATASET_DIR: /pytorch/Imagenet_FullData
+    OUTPUT_DIR: /tmp
+  volumes:
+    - src: /pytorch/Imagenet_FullData
+      dst: /pytorch/Imagenet_FullData
+avx-fp32-realtime-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-pytorch-image-recognition-resnext-32x16d-inference
+  cmd: quickstart/inference_realtime.sh
+  shm-size: 8G
+  env:
+    PRECISION: avx-fp32
+    DATASET_DIR: /pytorch/Imagenet_FullData
+    OUTPUT_DIR: /tmp
+  volumes:
+    - src: /pytorch/Imagenet_FullData
+      dst: /pytorch/Imagenet_FullData
+bf16-realtime-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-pytorch-image-recognition-resnext-32x16d-inference
+  cmd: quickstart/inference_realtime.sh
+  shm-size: 8G
+  env:
+    PRECISION: bf16
+    DATASET_DIR: /pytorch/Imagenet_FullData
+    OUTPUT_DIR: /tmp
+  volumes:
+    - src: /pytorch/Imagenet_FullData
+      dst: /pytorch/Imagenet_FullData
+int8-realtime-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-pytorch-image-recognition-resnext-32x16d-inference
+  cmd: quickstart/inference_realtime.sh
+  shm-size: 8G
+  env:
+    PRECISION: int8
+    DATASET_DIR: /pytorch/Imagenet_FullData
+    OUTPUT_DIR: /tmp
+  volumes:
+    - src: /pytorch/Imagenet_FullData
+      dst: /pytorch/Imagenet_FullData
+avx-int8-realtime-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-pytorch-image-recognition-resnext-32x16d-inference
+  cmd: quickstart/inference_realtime.sh
+  shm-size: 8G
+  env:
+    PRECISION: avx-int8
+    DATASET_DIR: /pytorch/Imagenet_FullData
+    OUTPUT_DIR: /tmp
+  volumes:
+    - src: /pytorch/Imagenet_FullData
+      dst: /pytorch/Imagenet_FullData
+bf32-realtime-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-pytorch-image-recognition-resnext-32x16d-inference
+  cmd: quickstart/inference_realtime.sh
+  shm-size: 8G
+  env:
+    PRECISION: bf32
+    DATASET_DIR: /pytorch/Imagenet_FullData
+    OUTPUT_DIR: /tmp
+  volumes:
+    - src: /pytorch/Imagenet_FullData
+      dst: /pytorch/Imagenet_FullData
+fp32-throughput-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-pytorch-image-recognition-resnext-32x16d-inference
+  cmd: quickstart/inference_throughput.sh 
+  shm-size: 8G
+  env:
+    PRECISION: fp32
+    DATASET_DIR: /pytorch/Imagenet_FullData
+    OUTPUT_DIR: /tmp
+  volumes:
+    - src: /pytorch/Imagenet_FullData
+      dst: /pytorch/Imagenet_FullData
+avx-fp32-throughput-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-pytorch-image-recognition-resnext-32x16d-inference
+  cmd: quickstart/inference_throughput.sh 
+  shm-size: 8G
+  env:
+    PRECISION: avx-fp32
+    DATASET_DIR: /pytorch/Imagenet_FullData
+    OUTPUT_DIR: /tmp
+  volumes:
+    - src: /pytorch/Imagenet_FullData
+      dst: /pytorch/Imagenet_FullData
+bf16-throughput-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-pytorch-image-recognition-resnext-32x16d-inference
+  cmd: quickstart/inference_throughput.sh 
+  shm-size: 8G
+  env:
+    PRECISION: bf16
+    DATASET_DIR: /pytorch/Imagenet_FullData
+    OUTPUT_DIR: /tmp
+  volumes:
+    - src: /pytorch/Imagenet_FullData
+      dst: /pytorch/Imagenet_FullData
+int8-throughput-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-pytorch-image-recognition-resnext-32x16d-inference
+  cmd: quickstart/inference_throughput.sh
+  shm-size: 8G
+  env:
+    PRECISION: int8
+    DATASET_DIR: /pytorch/Imagenet_FullData
+    OUTPUT_DIR: /tmp
+  volumes:
+    - src: /pytorch/Imagenet_FullData
+      dst: /pytorch/Imagenet_FullData
+avx-int8-throughput-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-pytorch-image-recognition-resnext-32x16d-inference
+  cmd: quickstart/inference_throughput.sh
+  shm-size: 8G
+  env:
+    PRECISION: avx-int8
+    DATASET_DIR: /pytorch/Imagenet_FullData
+    OUTPUT_DIR: /tmp
+  volumes:
+    - src: /pytorch/Imagenet_FullData
+      dst: /pytorch/Imagenet_FullData
+bf32-throughput-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-pytorch-image-recognition-resnext-32x16d-inference
+  cmd: quickstart/inference_throughput.sh
+  shm-size: 8G
+  env:
+    PRECISION: bf32
+    DATASET_DIR: /pytorch/Imagenet_FullData
+    OUTPUT_DIR: /tmp
+  volumes:
+    - src: /pytorch/Imagenet_FullData
+      dst: /pytorch/Imagenet_FullData
+fp32-accuracy-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-pytorch-image-recognition-resnext-32x16d-inference
+  cmd: quickstart/accuracy.sh 
+  shm-size: 8G
+  env:
+    PRECISION: fp32
+    DATASET_DIR: /pytorch/Imagenet_FullData
+    OUTPUT_DIR: /tmp
+  volumes:
+    - src: /pytorch/Imagenet_FullData
+      dst: /pytorch/Imagenet_FullData
+avx-fp32-accuracy-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-pytorch-image-recognition-resnext-32x16d-inference
+  cmd: quickstart/accuracy.sh 
+  shm-size: 8G
+  env:
+    PRECISION: avx-fp32
+    DATASET_DIR: /pytorch/Imagenet_FullData
+    OUTPUT_DIR: /tmp
+  volumes:
+    - src: /pytorch/Imagenet_FullData
+      dst: /pytorch/Imagenet_FullData
+bf16-accuracy-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-pytorch-image-recognition-resnext-32x16d-inference
+  cmd: quickstart/accuracy.sh 
+  shm-size: 8G
+  env:
+    PRECISION: bf16
+    DATASET_DIR: /pytorch/Imagenet_FullData
+    OUTPUT_DIR: /tmp
+  volumes:
+    - src: /pytorch/Imagenet_FullData
+      dst: /pytorch/Imagenet_FullData
+int8-accuracy-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-pytorch-image-recognition-resnext-32x16d-inference
+  cmd: quickstart/accuracy.sh 
+  shm-size: 8G
+  env:
+    PRECISION: int8
+    DATASET_DIR: /pytorch/Imagenet_FullData
+    OUTPUT_DIR: /tmp
+  volumes:
+    - src: /pytorch/Imagenet_FullData
+      dst: /pytorch/Imagenet_FullData
+avx-int8-accuracy-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-pytorch-image-recognition-resnext-32x16d-inference
+  cmd: quickstart/accuracy.sh 
+  shm-size: 8G
+  env:
+    PRECISION: avx-int8
+    DATASET_DIR: /pytorch/Imagenet_FullData
+    OUTPUT_DIR: /tmp
+  volumes:
+    - src: /pytorch/Imagenet_FullData
+      dst: /pytorch/Imagenet_FullData
+bf32-accuracy-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-pytorch-image-recognition-resnext-32x16d-inference
+  cmd: quickstart/accuracy.sh
+  shm-size: 8G
+  env:
+    PRECISION: bf32
+    DATASET_DIR: /pytorch/Imagenet_FullData
+    OUTPUT_DIR: /tmp
+  volumes:
+    - src: /pytorch/Imagenet_FullData
+      dst: /pytorch/Imagenet_FullData
diff --git a/docker/pyt-cpu/rnnt-inference/pytorch-rnnt-inference.Dockerfile-centos b/docker/pyt-cpu/rnnt-inference/pytorch-rnnt-inference.Dockerfile-centos
new file mode 100644
index 000000000..96c85938f
--- /dev/null
+++ b/docker/pyt-cpu/rnnt-inference/pytorch-rnnt-inference.Dockerfile-centos
@@ -0,0 +1,76 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+ARG BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG BASE_TAG="2.0.0-pip-base"
+
+FROM ${BASE_IMAGE}:${BASE_TAG} AS intel-extension-for-pytorch
+
+ WORKDIR /workspace/pytorch-rnnt-inference
+
+RUN yum update -y && \
+    yum install -y \
+    ca-certificates \
+    git \
+    cmake>=3.19.6 \
+    make \
+    autoconf \
+    bzip2 \
+    tar
+
+RUN git clone  https://github.com/jemalloc/jemalloc.git && \
+    cd jemalloc && \
+    git checkout c8209150f9d219a137412b06431c9d52839c7272 && \
+    ./autogen.sh && \
+    ./configure --prefix=/workspace/lib/jemalloc && \
+    make && \
+    make install 
+
+COPY models/language_modeling/pytorch/rnnt/inference/cpu models/language_modeling/pytorch/rnnt/inference/cpu
+COPY quickstart/language_modeling/pytorch/rnnt/inference/cpu/enable_warprnnt_c++17.diff quickstart/enable_warprnnt_c++17.diff
+COPY quickstart/language_modeling/pytorch/rnnt/inference/cpu/inference_realtime.sh quickstart/inference_realtime.sh 
+COPY quickstart/language_modeling/pytorch/rnnt/inference/cpu/inference_throughput.sh quickstart/inference_throughput.sh
+COPY quickstart/language_modeling/pytorch/rnnt/inference/cpu/accuracy.sh quickstart/accuracy.sh
+COPY  quickstart/language_modeling/pytorch/rnnt/inference/cpu/download_dataset.sh quickstart/download_dataset.sh
+
+RUN pip install packaging intel-openmp librosa==0.9.1 
+
+ENV LD_PRELOAD="/workspace/lib/jemalloc/lib/libjemalloc.so":"/usr/local/lib/libiomp5.so":$LD_PRELOAD
+ENV MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+
+RUN cd /workspace/pytorch-rnnt-inference/models/language_modeling/pytorch/rnnt/inference/cpu && \
+    pip install -r requirements.txt && \
+    pip install unidecode inflect && \
+    cd -
+
+RUN git clone https://github.com/HawkAaron/warp-transducer && \
+    cd warp-transducer && \
+    git checkout master && \
+    git apply /workspace/pytorch-rnnt-inference/quickstart/enable_warprnnt_c++17.diff && \
+    mkdir build && \
+    cd build && \
+    cmake .. && \
+    make && \
+    cd ../pytorch_binding && \
+    python setup.py install
+
+RUN yum update -y && \
+    yum install -y \
+    numactl \
+    mesa-libGL && \
+    yum clean all
+
+COPY LICENSE licenses/LICENSE
+COPY third_party licenses/third_party
diff --git a/docker/pyt-cpu/rnnt-inference/pytorch-rnnt-inference.Dockerfile-ubuntu b/docker/pyt-cpu/rnnt-inference/pytorch-rnnt-inference.Dockerfile-ubuntu
new file mode 100644
index 000000000..05c82a115
--- /dev/null
+++ b/docker/pyt-cpu/rnnt-inference/pytorch-rnnt-inference.Dockerfile-ubuntu
@@ -0,0 +1,76 @@
+# Copyright (c) 2020-2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+ARG BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG BASE_TAG="2.0.0-pip-base"
+
+FROM ${BASE_IMAGE}:${BASE_TAG} AS intel-optimized-pytorch
+
+WORKDIR /workspace/pytorch-rnnt-inference
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+    build-essential \
+    ca-certificates \
+    git \
+    wget \
+    make \
+    cmake>=3.19.6 \
+    autoconf \
+    bzip2 \
+    tar
+
+RUN git clone  https://github.com/jemalloc/jemalloc.git && \
+    cd jemalloc && \
+    git checkout c8209150f9d219a137412b06431c9d52839c7272 && \
+    ./autogen.sh && \
+    ./configure --prefix=/workspace/lib/jemalloc && \
+    make && \
+    make install 
+
+COPY models/language_modeling/pytorch/rnnt/inference/cpu models/language_modeling/pytorch/rnnt/inference/cpu
+COPY quickstart/language_modeling/pytorch/rnnt/inference/cpu/enable_warprnnt_c++17.diff quickstart/enable_warprnnt_c++17.diff
+COPY quickstart/language_modeling/pytorch/rnnt/inference/cpu/inference_realtime.sh quickstart/inference_realtime.sh 
+COPY quickstart/language_modeling/pytorch/rnnt/inference/cpu/inference_throughput.sh quickstart/inference_throughput.sh
+COPY quickstart/language_modeling/pytorch/rnnt/inference/cpu/accuracy.sh quickstart/accuracy.sh
+COPY  quickstart/language_modeling/pytorch/rnnt/inference/cpu/download_dataset.sh quickstart/download_dataset.sh
+
+RUN cd /workspace/pytorch-rnnt-inference/models/language_modeling/pytorch/rnnt/inference/cpu && \
+    pip install -r requirements.txt && \
+    pip install unidecode inflect && \
+    apt-get update && apt-get install -y gcc python3.10-dev && \
+    git clone https://github.com/HawkAaron/warp-transducer && \
+    cd warp-transducer && \
+    git checkout master && \
+    git apply /workspace/pytorch-rnnt-inference/quickstart/enable_warprnnt_c++17.diff && \
+    mkdir build && \
+    cd build && \
+    cmake .. && \
+    make && \
+    cd ../pytorch_binding && \
+    python setup.py install
+
+RUN pip install packaging intel-openmp librosa==0.9.1
+
+ENV LD_PRELOAD="/workspace/lib/jemalloc/lib/libjemalloc.so":"/usr/local/lib/libiomp5.so":$LD_PRELOAD 
+ENV MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+      numactl \
+      libegl1-mesa 
+
+COPY LICENSE licenses/LICENSE
+COPY third_party licenses/third_party
diff --git a/docker/pyt-cpu/rnnt-inference/tests.yaml b/docker/pyt-cpu/rnnt-inference/tests.yaml
new file mode 100644
index 000000000..88009dc8e
--- /dev/null
+++ b/docker/pyt-cpu/rnnt-inference/tests.yaml
@@ -0,0 +1,168 @@
+fp32-accuracy:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-language-modeling-rnnt-inference
+  cmd: quickstart/accuracy.sh fp32
+  shm-size: 8G
+  env:
+    PRECISION: fp32
+    CHECKPOINT_DIR: /pytorch/rnnt_model
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/rnnt_dataset
+  volumes:
+    - src: /pytorch/rnnt_model
+      dst: /pytorch/rnnt_model
+    - src: /pytorch/rnnt_dataset
+      dst: /pytorch/rnnt_dataset
+fp32-realtime:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-language-modeling-rnnt-inference
+  cmd: quickstart/inference_realtime.sh fp32
+  shm-size: 8G
+  env:
+    PRECISION: fp32
+    CHECKPOINT_DIR: /pytorch/rnnt_model
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/rnnt_dataset
+  volumes:
+    - src: /pytorch/rnnt_model
+      dst: /pytorch/rnnt_model
+    - src: /pytorch/rnnt_dataset
+      dst: /pytorch/rnnt_dataset
+fp32-throughput:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-language-modeling-rnnt-inference
+  cmd: quickstart/inference_throughput.sh fp32
+  shm-size: 8G
+  env:
+    PRECISION: fp32
+    CHECKPOINT_DIR: /pytorch/rnnt_model
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/rnnt_dataset
+  volumes:
+    - src: /pytorch/rnnt_model
+      dst: /pytorch/rnnt_model
+    - src: /pytorch/rnnt_dataset
+      dst: /pytorch/rnnt_dataset
+bf32-accuracy:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-language-modeling-rnnt-inference
+  cmd: quickstart/accuracy.sh bf32
+  shm-size: 8G
+  env:
+    PRECISION: bf32
+    CHECKPOINT_DIR: /pytorch/rnnt_model
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/rnnt_dataset
+  volumes:
+    - src: /pytorch/rnnt_model
+      dst: /pytorch/rnnt_model
+    - src: /pytorch/rnnt_dataset
+      dst: /pytorch/rnnt_dataset
+bf32-realtime:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-language-modeling-rnnt-inference
+  cmd: quickstart/inference_realtime.sh bf32
+  shm-size: 8G
+  env:
+    PRECISION: bf32
+    CHECKPOINT_DIR: /pytorch/rnnt_model
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/rnnt_dataset
+  volumes:
+    - src: /pytorch/rnnt_model
+      dst: /pytorch/rnnt_model
+    - src: /pytorch/rnnt_dataset
+      dst: /pytorch/rnnt_dataset
+bf32-throughput:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-language-modeling-rnnt-inference
+  cmd: quickstart/inference_throughput.sh bf32
+  shm-size: 8G
+  env:
+    PRECISION: bf32
+    CHECKPOINT_DIR: /pytorch/rnnt_model
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/rnnt_dataset
+  volumes:
+    - src: /pytorch/rnnt_model
+      dst: /pytorch/rnnt_model
+    - src: /pytorch/rnnt_dataset
+      dst: /pytorch/rnnt_dataset
+bf16-accuracy:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-language-modeling-rnnt-inference
+  cmd: quickstart/accuracy.sh bf16
+  shm-size: 8G
+  env:
+    PRECISION: bf16
+    CHECKPOINT_DIR: /pytorch/rnnt_model
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/rnnt_dataset
+  volumes:
+    - src: /pytorch/rnnt_model
+      dst: /pytorch/rnnt_model
+    - src: /pytorch/rnnt_dataset
+      dst: /pytorch/rnnt_dataset
+bf16-realtime:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-language-modeling-rnnt-inference
+  cmd: quickstart/inference_realtime.sh bf16
+  shm-size: 8G
+  env:
+    PRECISION: bf16
+    CHECKPOINT_DIR: /pytorch/rnnt_model
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/rnnt_dataset
+  volumes:
+    - src: /pytorch/rnnt_model
+      dst: /pytorch/rnnt_model
+    - src: /pytorch/rnnt_dataset
+      dst: /pytorch/rnnt_dataset
+bf16-throughput:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-language-modeling-rnnt-inference
+  cmd: quickstart/inference_throughput.sh bf16
+  shm-size: 8G
+  env:
+    PRECISION: bf16
+    CHECKPOINT_DIR: /pytorch/rnnt_model
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/rnnt_dataset
+  volumes:
+    - src: /pytorch/rnnt_model
+      dst: /pytorch/rnnt_model
+    - src: /pytorch/rnnt_dataset
+      dst: /pytorch/rnnt_dataset
+avx-fp32-accuracy:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-language-modeling-rnnt-inference
+  cmd: quickstart/accuracy.sh avx-fp32
+  shm-size: 8G
+  env:
+    PRECISION: avx-fp32
+    CHECKPOINT_DIR: /pytorch/rnnt_model
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/rnnt_dataset
+  volumes:
+    - src: /pytorch/rnnt_model
+      dst: /pytorch/rnnt_model
+    - src: /pytorch/rnnt_dataset
+      dst: /pytorch/rnnt_dataset
+avx-fp32-realtime:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-language-modeling-rnnt-inference
+  cmd: quickstart/inference_realtime.sh avx-fp32
+  shm-size: 8G
+  env:
+    PRECISION: avx-fp32
+    CHECKPOINT_DIR: /pytorch/rnnt_model
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/rnnt_dataset
+  volumes:
+    - src: /pytorch/rnnt_model
+      dst: /pytorch/rnnt_model
+    - src: /pytorch/rnnt_dataset
+      dst: /pytorch/rnnt_dataset
+avx-fp32-throughput:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-language-modeling-rnnt-inference
+  cmd: quickstart/inference_throughput.sh avx-fp32
+  shm-size: 8G
+  env:
+    PRECISION: avx-fp32
+    CHECKPOINT_DIR: /pytorch/rnnt_model
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/rnnt_dataset
+  volumes:
+    - src: /pytorch/rnnt_model
+      dst: /pytorch/rnnt_model
+    - src: /pytorch/rnnt_dataset
+      dst: /pytorch/rnnt_dataset
diff --git a/docker/pyt-cpu/rnnt-training/pytorch-rnnt-training.Dockerfile-centos b/docker/pyt-cpu/rnnt-training/pytorch-rnnt-training.Dockerfile-centos
new file mode 100644
index 000000000..d8bb537fb
--- /dev/null
+++ b/docker/pyt-cpu/rnnt-training/pytorch-rnnt-training.Dockerfile-centos
@@ -0,0 +1,78 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+ARG BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG BASE_TAG="2.0.0-pip-base"
+
+FROM ${BASE_IMAGE}:${BASE_TAG} AS intel-extension-for-pytorch
+
+ WORKDIR /workspace/pytorch-rnnt-training
+
+RUN yum update -y && \
+    yum install -y \
+    ca-certificates \
+    git \
+    cmake>=3.19.6 \
+    make \
+    autoconf \
+    bzip2 \
+    tar
+
+RUN git clone  https://github.com/jemalloc/jemalloc.git && \
+    cd jemalloc && \
+    git checkout c8209150f9d219a137412b06431c9d52839c7272 && \
+    ./autogen.sh && \
+    ./configure --prefix=/workspace/lib/jemalloc && \
+    make && \
+    make install 
+
+COPY quickstart/language_modeling/pytorch/rnnt/training/cpu/training.sh quickstart/training.sh
+COPY quickstart/language_modeling/pytorch/rnnt/training/cpu/enable_warprnnt_c++17.diff quickstart/enable_warprnnt_c++17.diff
+COPY models/language_modeling/pytorch/rnnt/training/cpu models/language_modeling/pytorch/rnnt/training/cpu 
+COPY quickstart/language_modeling/pytorch/rnnt/training/cpu/download_dataset.sh quickstart/download_dataset.sh
+
+RUN pip install packaging intel-openmp \
+    librosa==0.9.1 protobuf==3.20.3 \
+    numpy==1.23.4
+
+ENV LD_PRELOAD="/workspace/lib/jemalloc/lib/libjemalloc.so":"/usr/local/lib/libiomp5.so":$LD_PRELOAD
+ENV MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+
+RUN cd /workspace/pytorch-rnnt-training/models/language_modeling/pytorch/rnnt/training/cpu && \
+    pip install -r requirements.txt && \
+    pip install unidecode inflect && \
+    pip install --upgrade pip && pip install sox && \
+    yum install -y libsndfile && \
+    cd -
+
+RUN git clone https://github.com/HawkAaron/warp-transducer && \
+    cd warp-transducer && \
+    git checkout master && \
+    git apply /workspace/pytorch-rnnt-training/quickstart/enable_warprnnt_c++17.diff && \
+    mkdir build && \
+    cd build && \
+    cmake .. && \
+    make && \
+    cd ../pytorch_binding && \
+    python setup.py install
+
+RUN yum update -y && \
+    yum install -y \
+    numactl \
+    mesa-libGL && \
+    yum clean all
+
+COPY LICENSE licenses/LICENSE
+COPY third_party licenses/third_party
diff --git a/docker/pyt-cpu/rnnt-training/pytorch-rnnt-training.Dockerfile-ubuntu b/docker/pyt-cpu/rnnt-training/pytorch-rnnt-training.Dockerfile-ubuntu
new file mode 100644
index 000000000..74f3afa84
--- /dev/null
+++ b/docker/pyt-cpu/rnnt-training/pytorch-rnnt-training.Dockerfile-ubuntu
@@ -0,0 +1,81 @@
+# Copyright (c) 2020-2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+
+ARG BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG BASE_TAG="2.0.0-pip-base"
+
+FROM ${BASE_IMAGE}:${BASE_TAG} AS intel-optimized-pytorch
+
+WORKDIR /workspace/pytorch-rnnt-training 
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+    build-essential \
+    ca-certificates \
+    git \
+    wget \
+    make \
+    cmake \
+    autoconf \
+    bzip2 \
+    tar
+
+RUN git clone https://github.com/jemalloc/jemalloc.git && \
+    cd jemalloc && \
+    git checkout c8209150f9d219a137412b06431c9d52839c7272 && \
+    ./autogen.sh && \
+    ./configure --prefix=/workspace/lib/jemalloc && \
+    make && \ 
+    make install
+
+RUN pip install packaging intel-openmp \
+    librosa==0.9.1 protobuf==3.20.3 \
+    numpy==1.23.4
+
+ENV LD_PRELOAD="/workspace/lib/jemalloc/lib/libjemalloc.so":"/usr/local/lib/libiomp5.so":$LD_PRELOAD
+ENV MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+
+COPY models/language_modeling/pytorch/rnnt/training/cpu models/language_modeling/pytorch/rnnt/training/cpu
+COPY quickstart/language_modeling/pytorch/rnnt/training/cpu/enable_warprnnt_c++17.diff quickstart/enable_warprnnt_c++17.diff
+COPY quickstart/language_modeling/pytorch/rnnt/training/cpu/training.sh quickstart/training.sh
+COPY quickstart/language_modeling/pytorch/rnnt/training/cpu/download_dataset.sh quickstart/download_dataset.sh
+
+RUN cd /workspace/pytorch-rnnt-training/models/language_modeling/pytorch/rnnt/training/cpu && \
+    pip install -r requirements.txt && \
+    pip install unidecode inflect && \
+    pip install --upgrade pip && pip install sox && \
+    apt-get install --no-install-recommends --fix-missing -y \
+    libsndfile1 python3-dev
+
+RUN git clone https://github.com/HawkAaron/warp-transducer && \
+    cd warp-transducer && \
+    git checkout master && \
+    git apply /workspace/pytorch-rnnt-training/quickstart/enable_warprnnt_c++17.diff && \
+    mkdir build && \
+    cd build && \
+    cmake .. && \
+    make && \
+    cd ../pytorch_binding && \
+    python setup.py install
+
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+    numactl \
+    libegl1-mesa 
+
+COPY LICENSE licenses/LICENSE
+COPY third_party licenses/third_party
diff --git a/docker/pyt-cpu/rnnt-training/tests.yaml b/docker/pyt-cpu/rnnt-training/tests.yaml
new file mode 100644
index 000000000..e5a574701
--- /dev/null
+++ b/docker/pyt-cpu/rnnt-training/tests.yaml
@@ -0,0 +1,48 @@
+fp32-training:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-language-modeling-rnnt-training
+  cmd: quickstart/training.sh fp32
+  ipc: host
+  env: 
+    PRECISION: fp32
+    CHECKPOINT_DIR: /pytorch/rnnt_model
+    OUTPUT_DIR: /tmp
+    RESULT_DIR: /tmp
+    NUM_STEPS: '100'
+    DATASET_DIR: /pytorch/rnnt_dataset_training
+  volumes:
+    - src: /pytorch/rnnt_dataset_training
+      dst: /pytorch/rnnt_dataset_training
+    - src: /pytorch/rnnt_model
+      dst: /pytorch/rnnt_model
+bf16-training:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-language-modeling-rnnt-training
+  cmd: quickstart/training.sh bf16
+  ipc: host
+  env: 
+    PRECISION: bf16
+    CHECKPOINT_DIR: /pytorch/rnnt_model
+    OUTPUT_DIR: /tmp
+    RESULT_DIR: /tmp
+    NUM_STEPS: '100'
+    DATASET_DIR: /pytorch/rnnt_dataset_training
+  volumes:
+    - src: /pytorch/rnnt_dataset_training
+      dst: /pytorch/rnnt_dataset_training
+    - src: /pytorch/rnnt_model
+      dst: /pytorch/rnnt_model
+bf32-training:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-language-modeling-rnnt-training
+  cmd: quickstart/training.sh bf32
+  ipc: host
+  env: 
+    PRECISION: bf32
+    CHECKPOINT_DIR: /pytorch/rnnt_model
+    OUTPUT_DIR: /tmp
+    RESULT_DIR: /tmp
+    NUM_STEPS: '100'
+    DATASET_DIR: /pytorch/rnnt_dataset_training
+  volumes:
+    - src: /pytorch/rnnt_dataset_training
+      dst: /pytorch/rnnt_dataset_training
+    - src: /pytorch/rnnt_model
+      dst: /pytorch/rnnt_model
diff --git a/docker/pyt-cpu/ssd-resnet34-inference/pytorch-ssd-resnet34-inference.Dockerfile-centos b/docker/pyt-cpu/ssd-resnet34-inference/pytorch-ssd-resnet34-inference.Dockerfile-centos
new file mode 100644
index 000000000..39dbf8abf
--- /dev/null
+++ b/docker/pyt-cpu/ssd-resnet34-inference/pytorch-ssd-resnet34-inference.Dockerfile-centos
@@ -0,0 +1,52 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+ARG BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG BASE_TAG="2.0.0-pip-base"
+
+FROM ${BASE_IMAGE}:${BASE_TAG} AS intel-optimized-pytorch
+
+SHELL ["/bin/bash", "-c"]
+
+ENV LANG C.UTF-8
+
+WORKDIR /workspace/pytorch-ssd-resnet34-inference
+
+RUN git clone https://github.com/jemalloc/jemalloc.git && \
+    cd jemalloc && \
+    git checkout c8209150f9d219a137412b06431c9d52839c7272 && \
+    ./autogen.sh && \
+    ./configure --prefix=/workspace/lib/jemalloc && \
+    make && \ 
+    make install
+
+RUN pip install packaging intel-openmp matplotlib Pillow pycocotools defusedxml
+
+ENV LD_PRELOAD="/workspace/lib/jemalloc/lib/libjemalloc.so":"/usr/local/lib/libiomp5.so":$LD_PRELOAD
+ENV MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+
+COPY models/object_detection/pytorch/ssd-resnet34/inference/cpu models/object_detection/pytorch/ssd-resnet34/inference/cpu
+COPY quickstart/object_detection/pytorch/ssd-resnet34/inference/cpu/inference_realtime.sh quickstart/inference_realtime.sh
+COPY quickstart/object_detection/pytorch/ssd-resnet34/inference/cpu/inference_throughput.sh quickstart/inference_throughput.sh
+COPY quickstart/object_detection/pytorch/ssd-resnet34/inference/cpu/accuracy.sh quickstart/accuracy.sh
+
+RUN yum update -y && \
+    yum install -y \
+    numactl \
+    mesa-libGL && \
+    yum clean all
+
+COPY LICENSE licenses/LICENSE
+COPY third_party licenses/third_party
diff --git a/docker/pyt-cpu/ssd-resnet34-inference/pytorch-ssd-resnet34-inference.Dockerfile-ubuntu b/docker/pyt-cpu/ssd-resnet34-inference/pytorch-ssd-resnet34-inference.Dockerfile-ubuntu
new file mode 100644
index 000000000..9602cc681
--- /dev/null
+++ b/docker/pyt-cpu/ssd-resnet34-inference/pytorch-ssd-resnet34-inference.Dockerfile-ubuntu
@@ -0,0 +1,59 @@
+# Copyright (c) 2020-2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+ARG BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG BASE_TAG="2.0.0-pip-base"
+
+FROM ${BASE_IMAGE}:${BASE_TAG} AS intel-optimized-pytorch
+
+WORKDIR /workspace/pytorch-ssd-resnet34-inference
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+    build-essential \
+    ca-certificates \
+    git \
+    wget \
+    make \
+    cmake \
+    autoconf \
+    bzip2 \
+    tar
+
+RUN pip install packaging intel-openmp matplotlib Pillow pycocotools defusedxml
+
+RUN git clone https://github.com/jemalloc/jemalloc.git && \
+    cd jemalloc && \
+    git checkout c8209150f9d219a137412b06431c9d52839c7272 && \
+    ./autogen.sh && \
+    ./configure --prefix=/workspace/lib/jemalloc && \
+    make && \ 
+    make install
+
+ENV LD_PRELOAD="/workspace/lib/jemalloc/lib/libjemalloc.so":"/usr/local/lib/libiomp5.so":$LD_PRELOAD
+
+COPY models/object_detection/pytorch/ssd-resnet34/inference/cpu models/object_detection/pytorch/ssd-resnet34/inference/cpu
+COPY quickstart/object_detection/pytorch/ssd-resnet34/inference/cpu/inference_realtime.sh quickstart/inference_realtime.sh
+COPY quickstart/object_detection/pytorch/ssd-resnet34/inference/cpu/inference_throughput.sh quickstart/inference_throughput.sh
+COPY quickstart/object_detection/pytorch/ssd-resnet34/inference/cpu/accuracy.sh quickstart/accuracy.sh
+
+ENV MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+    numactl \
+    libegl1-mesa 
+
+COPY LICENSE licenses/LICENSE
+COPY third_party licenses/third_party
diff --git a/docker/pyt-cpu/ssd-resnet34-inference/tests.yaml b/docker/pyt-cpu/ssd-resnet34-inference/tests.yaml
new file mode 100644
index 000000000..76103ff76
--- /dev/null
+++ b/docker/pyt-cpu/ssd-resnet34-inference/tests.yaml
@@ -0,0 +1,234 @@
+fp32-realtime-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-object-detection-ssd-resnet34-inference
+  cmd: quickstart/inference_realtime.sh fp32
+  env:
+    PRECISION: fp32
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+    CHECKPOINT_DIR: /pytorch/pretrained_models/ssd-resnet34-inference/
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/ssd-resnet34-inference/
+      dst: /pytorch/pretrained_models/ssd-resnet34-inference/
+avx-fp32-realtime-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-object-detection-ssd-resnet34-inference
+  cmd: quickstart/inference_realtime.sh avx-fp32
+  env:
+    PRECISION: avx-fp32
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+    CHECKPOINT_DIR: /pytorch/pretrained_models/ssd-resnet34-inference/
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/ssd-resnet34-inference/
+      dst: /pytorch/pretrained_models/ssd-resnet34-inference/
+bf16-realtime-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-object-detection-ssd-resnet34-inference
+  cmd: quickstart/inference_realtime.sh bf16
+  env:
+    PRECISION: bf16
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+    CHECKPOINT_DIR: /pytorch/pretrained_models/ssd-resnet34-inference/
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/ssd-resnet34-inference/
+      dst: /pytorch/pretrained_models/ssd-resnet34-inference/
+int8-realtime-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-object-detection-ssd-resnet34-inference
+  cmd: quickstart/inference_realtime.sh int8
+  env:
+    PRECISION: int8
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+    CHECKPOINT_DIR: /pytorch/pretrained_models/ssd-resnet34-inference/
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/ssd-resnet34-inference/
+      dst: /pytorch/pretrained_models/ssd-resnet34-inference/
+avx-int8-realtime-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-object-detection-ssd-resnet34-inference
+  cmd: quickstart/inference_realtime.sh avx-int8
+  env:
+    PRECISION: avx-int8
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+    CHECKPOINT_DIR: /pytorch/pretrained_models/ssd-resnet34-inference/
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/ssd-resnet34-inference/
+      dst: /pytorch/pretrained_models/ssd-resnet34-inference/
+bf32-realtime-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-object-detection-ssd-resnet34-inference
+  cmd: quickstart/inference_realtime.sh bf32
+  env:
+    PRECISION: bf32
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+    CHECKPOINT_DIR: /pytorch/pretrained_models/ssd-resnet34-inference/
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/ssd-resnet34-inference/
+      dst: /pytorch/pretrained_models/ssd-resnet34-inference/
+fp32-throughput-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-object-detection-ssd-resnet34-inference
+  cmd: quickstart/inference_throughput.sh fp32
+  env:
+    PRECISION: fp32
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+    CHECKPOINT_DIR: /pytorch/pretrained_models/ssd-resnet34-inference/
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/ssd-resnet34-inference/
+      dst: /pytorch/pretrained_models/ssd-resnet34-inference/
+avx-fp32-throughput-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-object-detection-ssd-resnet34-inference
+  cmd: quickstart/inference_throughput.sh avx-fp32
+  env:
+    PRECISION: avx-fp32
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+    CHECKPOINT_DIR: /pytorch/pretrained_models/ssd-resnet34-inference/
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/ssd-resnet34-inference/
+      dst: /pytorch/pretrained_models/ssd-resnet34-inference/
+bf16-throughput-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-object-detection-ssd-resnet34-inference
+  cmd: quickstart/inference_throughput.sh bf16
+  env:
+    PRECISION: bf16
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+    CHECKPOINT_DIR: /pytorch/pretrained_models/ssd-resnet34-inference/
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/ssd-resnet34-inference/
+      dst: /pytorch/pretrained_models/ssd-resnet34-inference/
+int8-throughput-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-object-detection-ssd-resnet34-inference
+  cmd: quickstart/inference_throughput.sh int8
+  env:
+    PRECISION: int8
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+    CHECKPOINT_DIR: /pytorch/pretrained_models/ssd-resnet34-inference/
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/ssd-resnet34-inference/
+      dst: /pytorch/pretrained_models/ssd-resnet34-inference/
+avx-int8-throughput-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-object-detection-ssd-resnet34-inference
+  cmd: quickstart/inference_throughput.sh avx-int8
+  env:
+    PRECISION: avx-int8
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+    CHECKPOINT_DIR: /pytorch/pretrained_models/ssd-resnet34-inference/
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/ssd-resnet34-inference/
+      dst: /pytorch/pretrained_models/ssd-resnet34-inference/
+bf32-throughput-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-object-detection-ssd-resnet34-inference
+  cmd: quickstart/inference_throughput.sh bf32
+  env:
+    PRECISION: bf32
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+    CHECKPOINT_DIR: /pytorch/pretrained_models/ssd-resnet34-inference/
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/ssd-resnet34-inference/
+      dst: /pytorch/pretrained_models/ssd-resnet34-inference/
+fp32-accuracy-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-object-detection-ssd-resnet34-inference
+  cmd: quickstart/accuracy.sh fp32
+  env:
+    PRECISION: fp32
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+    CHECKPOINT_DIR: /pytorch/pretrained_models/ssd-resnet34-inference/
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/ssd-resnet34-inference/
+      dst: /pytorch/pretrained_models/ssd-resnet34-inference/
+avx-fp32-accuracy-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-object-detection-ssd-resnet34-inference
+  cmd: quickstart/accuracy.sh avx-fp32
+  env:
+    PRECISION: avx-fp32
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+    CHECKPOINT_DIR: /pytorch/pretrained_models/ssd-resnet34-inference/
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/ssd-resnet34-inference/
+      dst: /pytorch/pretrained_models/ssd-resnet34-inference/
+bf16-accuracy-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-object-detection-ssd-resnet34-inference
+  cmd: quickstart/accuracy.sh bf16
+  env:
+    PRECISION: bf16
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+    CHECKPOINT_DIR: /pytorch/pretrained_models/ssd-resnet34-inference/
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/ssd-resnet34-inference/
+      dst: /pytorch/pretrained_models/ssd-resnet34-inference/
+int8-accuracy-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-object-detection-ssd-resnet34-inference
+  cmd: quickstart/accuracy.sh int8
+  env:
+    PRECISION: int8
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+    CHECKPOINT_DIR: /pytorch/pretrained_models/ssd-resnet34-inference/
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/ssd-resnet34-inference/
+      dst: /pytorch/pretrained_models/ssd-resnet34-inference/
+avx-int8-accuracy-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-object-detection-ssd-resnet34-inference
+  cmd: quickstart/accuracy.sh avx-int8
+  env:
+    PRECISION: avx-int8
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+    CHECKPOINT_DIR: /pytorch/pretrained_models/ssd-resnet34-inference/
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/ssd-resnet34-inference/
+      dst: /pytorch/pretrained_models/ssd-resnet34-inference/
+bf32-accuracy-inference:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-object-detection-ssd-resnet34-inference
+  cmd: quickstart/accuracy.sh bf32
+  env:
+    PRECISION: bf32
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+    CHECKPOINT_DIR: /pytorch/pretrained_models/ssd-resnet34-inference/
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/ssd-resnet34-inference/
+      dst: /pytorch/pretrained_models/ssd-resnet34-inference/
diff --git a/docker/pyt-cpu/ssd-resnet34-training/pytorch-ssd-resnet34-training.Dockerfile-centos b/docker/pyt-cpu/ssd-resnet34-training/pytorch-ssd-resnet34-training.Dockerfile-centos
new file mode 100644
index 000000000..10a317df3
--- /dev/null
+++ b/docker/pyt-cpu/ssd-resnet34-training/pytorch-ssd-resnet34-training.Dockerfile-centos
@@ -0,0 +1,70 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+ARG BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG BASE_TAG="2.0.0-pip-base"
+
+FROM ${BASE_IMAGE}:${BASE_TAG} AS intel-optimized-pytorch
+
+WORKDIR /workspace/pytorch-ssd-resnet34-training
+
+SHELL ["/bin/bash", "-c"]
+
+ENV LANG C.UTF-8
+
+RUN yum update -y && \
+    yum install -y \
+    ca-certificates \
+    git \
+    cmake>=3.19.6 \
+    make \
+    autoconf \
+    bzip2 \
+    tar
+    
+RUN git clone https://github.com/jemalloc/jemalloc.git && \
+    cd jemalloc && \
+    git checkout c8209150f9d219a137412b06431c9d52839c7272 && \
+    ./autogen.sh && \
+    ./configure --prefix=/workspace/lib/jemalloc && \
+    make && \ 
+    make install
+
+RUN pip install packaging intel-openmp 
+
+ENV LD_PRELOAD="/workspace/lib/jemalloc/lib/libjemalloc.so":"/usr/local/lib/libiomp5.so":$LD_PRELOAD
+ENV MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+
+RUN pip install matplotlib Pillow pycocotools pytz==2018.5 cython && \
+    pip install yacs opencv-python cityscapesscripts transformers defusedxml
+
+COPY models/object_detection/pytorch/ssd-resnet34/training/cpu models/object_detection/pytorch/ssd-resnet34/training/cpu
+COPY quickstart/object_detection/pytorch/ssd-resnet34/training/cpu/throughput.sh quickstart/throughput.sh
+COPY quickstart/object_detection/pytorch/ssd-resnet34/training/cpu/accuracy.sh quickstart/accuracy.sh
+
+RUN yum update && \
+    yum install -y libpng-devel \
+    freetype-devel && \
+    pip install --upgrade pip && \
+    pip install --no-cache-dir https://github.com/mlperf/logging/archive/9ea0afa.zip 
+
+RUN yum update -y && \
+    yum install -y \
+    numactl \
+    mesa-libGL && \
+    yum clean all
+
+COPY LICENSE licenses/LICENSE
+COPY third_party licenses/third_party
diff --git a/docker/pyt-cpu/ssd-resnet34-training/pytorch-ssd-resnet34-training.Dockerfile-ubuntu b/docker/pyt-cpu/ssd-resnet34-training/pytorch-ssd-resnet34-training.Dockerfile-ubuntu
new file mode 100644
index 000000000..af312c38f
--- /dev/null
+++ b/docker/pyt-cpu/ssd-resnet34-training/pytorch-ssd-resnet34-training.Dockerfile-ubuntu
@@ -0,0 +1,69 @@
+# Copyright (c) 2020-2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+
+ARG BASE_IMAGE="intel/intel-extension-for-pytorch"
+ARG BASE_TAG="2.0.0-pip-base"
+
+FROM ${BASE_IMAGE}:${BASE_TAG} AS intel-optimized-pytorch
+
+WORKDIR /workspace/pytorch-ssd-resnet34-training
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+    build-essential \
+    ca-certificates \
+    git \
+    wget \
+    make \
+    cmake \
+    autoconf \
+    bzip2 \
+    tar
+
+RUN git clone https://github.com/jemalloc/jemalloc.git && \
+    cd jemalloc && \
+    git checkout c8209150f9d219a137412b06431c9d52839c7272 && \
+    ./autogen.sh && \
+    ./configure --prefix=/workspace/lib/jemalloc && \
+    make && \ 
+    make install
+
+RUN pip install packaging intel-openmp 
+
+ENV LD_PRELOAD="/workspace/lib/jemalloc/lib/libjemalloc.so":"/usr/local/lib/libiomp5.so":$LD_PRELOAD
+ENV MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+
+RUN pip install matplotlib Pillow pycocotools pytz==2018.5 cython && \
+    pip install yacs opencv-python cityscapesscripts transformers defusedxml
+
+COPY models/object_detection/pytorch/ssd-resnet34/training/cpu models/object_detection/pytorch/ssd-resnet34/training/cpu
+COPY quickstart/object_detection/pytorch/ssd-resnet34/training/cpu/throughput.sh quickstart/throughput.sh
+COPY quickstart/object_detection/pytorch/ssd-resnet34/training/cpu/accuracy.sh quickstart/accuracy.sh
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+    libpng-dev \
+    libfreetype-dev && \
+    pip install --upgrade pip && \
+    pip install --no-cache-dir https://github.com/mlperf/logging/archive/9ea0afa.zip 
+
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+    numactl \
+    libegl1-mesa 
+
+COPY LICENSE licenses/LICENSE
+COPY third_party licenses/third_party
diff --git a/docker/pyt-cpu/ssd-resnet34-training/tests.yaml b/docker/pyt-cpu/ssd-resnet34-training/tests.yaml
new file mode 100644
index 000000000..06b1400a3
--- /dev/null
+++ b/docker/pyt-cpu/ssd-resnet34-training/tests.yaml
@@ -0,0 +1,112 @@
+fp32-training-perf:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-object-detection-ssd-resnet34-training
+  cmd: quickstart/throughput.sh fp32
+  shm-size: 8G
+  env:
+    PRECISION: fp32
+    CHECKPOINT_DIR: /pytorch/pretrained_models/ssd-resnet34-training
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/ssd-resnet34-training
+      dst: /pytorch/pretrained_models/ssd-resnet34-training
+fp32-training-accuracy:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-object-detection-ssd-resnet34-training
+  cmd: quickstart/accuracy.sh fp32
+  shm-size: 8G
+  env:
+    PRECISION: fp32
+    CHECKPOINT_DIR: /pytorch/pretrained_models/ssd-resnet34-training
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/ssd-resnet34-training
+      dst: /pytorch/pretrained_models/ssd-resnet34-training
+bf16-training-perf:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-object-detection-ssd-resnet34-training
+  cmd: quickstart/throughput.sh bf16
+  shm-size: 8G
+  env:
+    PRECISION: bf16
+    CHECKPOINT_DIR: /pytorch/pretrained_models/ssd-resnet34-training
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/ssd-resnet34-training
+      dst: /pytorch/pretrained_models/ssd-resnet34-training
+bf16-training-accuracy:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-object-detection-ssd-resnet34-training
+  cmd: quickstart/accuracy.sh bf16
+  shm-size: 8G
+  env:
+    PRECISION: bf16
+    CHECKPOINT_DIR: /pytorch/pretrained_models/ssd-resnet34-training
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/ssd-resnet34-training
+      dst: /pytorch/pretrained_models/ssd-resnet34-training
+bf32-training-perf:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-object-detection-ssd-resnet34-training
+  cmd: quickstart/throughput.sh bf32
+  shm-size: 8G
+  env:
+    PRECISION: bf32
+    CHECKPOINT_DIR: /pytorch/pretrained_models/ssd-resnet34-training
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/ssd-resnet34-training
+      dst: /pytorch/pretrained_models/ssd-resnet34-training
+bf32-training-accuracy:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-object-detection-ssd-resnet34-training
+  cmd: quickstart/accuracy.sh bf32
+  shm-size: 8G
+  env:
+    PRECISION: bf32
+    CHECKPOINT_DIR: /pytorch/pretrained_models/ssd-resnet34-training
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/ssd-resnet34-training
+      dst: /pytorch/pretrained_models/ssd-resnet34-training
+avx-fp32-training-perf:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-object-detection-ssd-resnet34-training
+  cmd: quickstart/throughput.sh avx-fp32
+  shm-size: 8G
+  env:
+    PRECISION: avx-fp32
+    CHECKPOINT_DIR: /pytorch/pretrained_models/ssd-resnet34-training
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/ssd-resnet34-training
+      dst: /pytorch/pretrained_models/ssd-resnet34-training
+avx-fp32-training-accuracy:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${BASE_IMAGE_NAME:-ubuntu}-${BASE_IMAGE_TAG:-22.04}-object-detection-ssd-resnet34-training
+  cmd: quickstart/accuracy.sh avx-fp32
+  shm-size: 8G
+  env:
+    PRECISION: avx-fp32
+    CHECKPOINT_DIR: /pytorch/pretrained_models/ssd-resnet34-training
+    OUTPUT_DIR: /tmp
+    DATASET_DIR: /pytorch/coco
+  volumes:
+    - src: /pytorch/coco
+      dst: /pytorch/coco
+    - src: /pytorch/pretrained_models/ssd-resnet34-training
+      dst: /pytorch/pretrained_models/ssd-resnet34-training
diff --git a/docker/tf-cpu/docker-compose.yml b/docker/tf-cpu/docker-compose.yml
index be6d9e302..8c29c509c 100644
--- a/docker/tf-cpu/docker-compose.yml
+++ b/docker/tf-cpu/docker-compose.yml
@@ -21,7 +21,7 @@
 version: '3'
 services: 
   tf-resnet50v1-5-inference:
-    image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-resnet50v1-5-inference
+    image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-image-recognition-tf-resnet50v1-5-inference
     pull_policy: always
     build:
       context: ../../
@@ -29,8 +29,180 @@ services:
         http_proxy: ${http_proxy}
         https_proxy: ${https_proxy}
         no_proxy: ${no_proxy}
+        OS: ${OS:-ubuntu}
+        TF_BASE_IMAGE: ${TF_BASE_IMAGE:-intel/intel-optimized-tensorflow}
+        TF_BASE_TAG: ${TF_BASE_TAG:-centos-tf2.14-py310}
+      dockerfile: docker/tf-cpu/tf-resnet50v1-5-inference/tf-resnet50v1-5-inference.Dockerfile-${OS:-ubuntu}
+    command: >
+      sh -c "python -c 'import tensorflow as tf; print(tf.__version__)'"
+  tf-3d-unet-mlperf-inference:
+    image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-image-segmentation-tf-3d-unet-mlperf-inference
+    pull_policy: always
+    build:
+      context: ../../
+      args:
+        http_proxy: ${http_proxy}
+        https_proxy: ${https_proxy}
+        no_proxy: ${no_proxy}
+        TF_BASE_IMAGE: ${TF_BASE_IMAGE:-intel/intel-optimized-tensorflow}
+        TF_BASE_TAG: ${TF_BASE_TAG:-2.13-pip-base}
+      dockerfile: docker/tf-cpu/tf-3d-unet-mlperf-inference/tf-3d-unet-mlperf-inference.Dockerfile-${OS:-ubuntu}
+    command: >
+      sh -c "python -c 'import tensorflow as tf; print(tf.__version__)'"
+  tf-distilbert-inference:
+    image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-language-modeling-tf-distilbert-inference
+    pull_policy: always
+    build:
+      context: ../../
+      args:
+        http_proxy: ${http_proxy}
+        https_proxy: ${https_proxy}
+        no_proxy: ${no_proxy}
+        TF_BASE_IMAGE: ${TF_BASE_IMAGE:-intel/intel-optimized-tensorflow}
+        TF_BASE_TAG: ${TF_BASE_TAG:-2.13-pip-base}
+      dockerfile: docker/tf-cpu/tf-distilbert-inference/tf-distilbert-inference.Dockerfile-${OS:-ubuntu}
+    command: >
+      sh -c "python -c 'import tensorflow as tf; print(tf.__version__)'"
+  tf-bert-large-inference:
+    image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-language-modeling-tf-bert-large-inference
+    pull_policy: always
+    build:
+      context: ../../
+      args:
+        http_proxy: ${http_proxy}
+        https_proxy: ${https_proxy}
+        no_proxy: ${no_proxy}
+        TF_BASE_IMAGE: ${TF_BASE_IMAGE:-intel/intel-optimized-tensorflow}
+        TF_BASE_TAG: ${TF_BASE_TAG:-2.13-pip-base}
+      dockerfile: docker/tf-cpu/tf-bert-large-inference/tf-bert-large-inference.Dockerfile-${OS:-ubuntu}
+    command: >
+      sh -c "python -c 'import tensorflow as tf; print(tf.__version__)'"
+  tf-bert-large-pretraining:
+    image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-language-modeling-tf-bert-large-pretraining
+    pull_policy: always
+    build:
+      context: ../../
+      args:
+        http_proxy: ${http_proxy}
+        https_proxy: ${https_proxy}
+        no_proxy: ${no_proxy}
+        TF_BASE_IMAGE: ${TF_BASE_IMAGE:-intel/intel-optimized-tensorflow}
+        TF_BASE_TAG: ${TF_BASE_TAG:-2.13-pip-base}
+      dockerfile: docker/tf-cpu/tf-bert-large-pretraining/tf-bert-large-pretraining.Dockerfile-${OS:-ubuntu}
+    command: >
+      sh -c "python -c 'import tensorflow as tf; print(tf.__version__)'"
+  tf-dien-inference:
+    image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-recommendation-tf-dien-inference
+    pull_policy: always
+    build:
+      context: ../../
+      args:
+        http_proxy: ${http_proxy}
+        https_proxy: ${https_proxy}
+        no_proxy: ${no_proxy}
+        TF_BASE_IMAGE: ${TF_BASE_IMAGE:-intel/intel-optimized-tensorflow}
+        TF_BASE_TAG: ${TF_BASE_TAG:-2.13-pip-base}
+      dockerfile: docker/tf-cpu/tf-dien-inference/tf-dien-inference.Dockerfile-${OS:-ubuntu}
+    command: >
+      sh -c "python -c 'import tensorflow as tf; print(tf.__version__)'"
+  tf-mobilenet-v1-inference:
+    image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-image-recognition-tf-mobilenet-v1-inference
+    pull_policy: always
+    build:
+      context: ../../
+      args:
+        http_proxy: ${http_proxy}
+        https_proxy: ${https_proxy}
+        no_proxy: ${no_proxy}
+        TF_BASE_IMAGE: ${TF_BASE_IMAGE:-intel/intel-optimized-tensorflow}
+        TF_BASE_TAG: ${TF_BASE_TAG:-2.13-pip-base}
+      dockerfile: docker/tf-cpu/tf-mobilenet-v1-inference/tf-mobilenet-v1-inference.Dockerfile-${OS:-ubuntu}
+    command: >
+      sh -c "python -c 'import tensorflow as tf; print(tf.__version__)'"
+  tf-resnet50v1-5-training:
+    image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-image-recognition-tf-resnet50v1-5-training
+    pull_policy: always
+    build:
+      context: ../../
+      args:
+        http_proxy: ${http_proxy}
+        https_proxy: ${https_proxy}
+        no_proxy: ${no_proxy}
+        TF_BASE_IMAGE: ${TF_BASE_IMAGE:-intel/intel-optimized-tensorflow}
+        TF_BASE_TAG: ${TF_BASE_TAG:-2.13-pip-base}
+      dockerfile: docker/tf-cpu/tf-resnet50v1-5-training/tf-resnet50v1-5-training.Dockerfile-${OS:-ubuntu}
+    command: >
+      sh -c "python -c 'import tensorflow as tf; print(tf.__version__)'"
+  tf-ssd-resnet34-inference:
+    image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-object-detection-tf-ssd-resnet34-inference
+    pull_policy: always
+    build:
+      context: ../../
+      args:
+        http_proxy: ${http_proxy}
+        https_proxy: ${https_proxy}
+        no_proxy: ${no_proxy}
+        TF_BASE_IMAGE: ${TF_BASE_IMAGE:-intel/intel-optimized-tensorflow}
+        TF_BASE_TAG: ${TF_BASE_TAG:-2.13-pip-base}
+        TF_MODELS_BRANCH: ${TF_MODELS_BRANCH:-f505cec}
+        TF_BENCHMARKS_BRANCH: ${TF_BENCHMARKS_BRANCH:-509b9d2}
+      dockerfile: docker/tf-cpu/tf-ssd-resnet34-inference/tf-ssd-resnet34-inference.Dockerfile-${OS:-ubuntu}
+    command: >
+      sh -c "python -c 'import tensorflow as tf; print(tf.__version__)'"
+  tf-ssd-resnet34-training:
+    image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-object-detection-tf-ssd-resnet34-training
+    pull_policy: always
+    build:
+      context: ../../
+      args:
+        http_proxy: ${http_proxy}
+        https_proxy: ${https_proxy}
+        no_proxy: ${no_proxy}
+        TF_BASE_IMAGE: ${TF_BASE_IMAGE:-intel/intel-optimized-tensorflow}
+        TF_BASE_TAG: ${TF_BASE_TAG:-2.13-pip-base}
+      dockerfile: docker/tf-cpu/tf-ssd-resnet34-training/tf-ssd-resnet34-training.Dockerfile-${OS:-ubuntu}
+    command: >
+      sh -c "python -c 'import tensorflow as tf; print(tf.__version__)'"
+  tf-transformer-mlperf-inference:
+    image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-language-translation-tf-transformer-mlperf-inference
+    pull_policy: always
+    build:
+      context: ../../
+      args:
+        http_proxy: ${http_proxy}
+        https_proxy: ${https_proxy}
+        no_proxy: ${no_proxy}
+        TF_BASE_IMAGE: ${TF_BASE_IMAGE:-intel/intel-optimized-tensorflow}
+        TF_BASE_TAG: ${TF_BASE_TAG:-2.13-pip-base}
+      dockerfile: docker/tf-cpu/tf-transformer-mlperf-inference/tf-transformer-mlperf-inference.Dockerfile-${OS:-ubuntu}
+    command: >
+      sh -c "python -c 'import tensorflow as tf; print(tf.__version__)'"
+  tf-transformer-mlperf-training:
+    image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-language-translation-tf-transformer-mlperf-training
+    pull_policy: always
+    build:
+      context: ../../
+      args:
+        http_proxy: ${http_proxy}
+        https_proxy: ${https_proxy}
+        no_proxy: ${no_proxy}
+        TF_BASE_IMAGE: ${TF_BASE_IMAGE:-intel/intel-optimized-tensorflow}
+        TF_BASE_TAG: ${TF_BASE_TAG:-2.13-pip-base}
+      dockerfile: docker/tf-cpu/tf-transformer-mlperf-training/tf-transformer-mlperf-training.Dockerfile-${OS:-ubuntu}
+    command: >
+      sh -c "python -c 'import tensorflow as tf; print(tf.__version__)'"
+  tf-ssd-mobilenet-inference:
+    image: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-centos}-object-detection-tf-ssd-mobilenet-inference
+    pull_policy: always
+    build:
+      context: ../../
+      args:
+        http_proxy: ${http_proxy}
+        https_proxy: ${https_proxy}
+        no_proxy: ""
         TF_BASE_IMAGE: ${TF_BASE_IMAGE:-intel/intel-optimized-tensorflow}
         TF_BASE_TAG: ${TF_BASE_TAG:-2.13-pip-base}
-      dockerfile: docker/tf-cpu/tf-resnet50v1-5-inference/tf-resnet50v1-5-inference.Dockerfile
+        TF_MODELS_BRANCH: ${TF_MODELS_BRANCH:-7a9934d}
+      dockerfile: docker/tf-cpu/tf-ssd-mobilenet-inference/tf-ssd-mobilenet-inference.Dockerfile-${OS:-centos}
     command: >
       sh -c "python -c 'import tensorflow as tf; print(tf.__version__)'"
diff --git a/docker/tf-cpu/tf-3d-unet-mlperf-inference/tests.yaml b/docker/tf-cpu/tf-3d-unet-mlperf-inference/tests.yaml
new file mode 100644
index 000000000..17f0a10f7
--- /dev/null
+++ b/docker/tf-cpu/tf-3d-unet-mlperf-inference/tests.yaml
@@ -0,0 +1,160 @@
+3d-UNet-MLPerf Accuracy fp32:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-image-segmentation-tf-3d-unet-mlperf-inference
+  cmd: /bin/bash quickstart/accuracy.sh
+  env:
+    DATASET_DIR: /tf_dataset/dataset/3DUNet/MICCAI_BraTS_2019_Data_Training
+    OUTPUT_DIR: /output/tf-3d-unet-mlperf-inference/accuracy_fp32
+    BATCH_SIZE: '1'
+    PRECISION: fp32
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/3DUNet/MICCAI_BraTS_2019_Data_Training
+    dst: /tf_dataset/dataset/3DUNet/MICCAI_BraTS_2019_Data_Training
+  - src: $PWD/output/tf-3d-unet-mlperf-inference/accuracy_fp32
+    dst: /output/tf-3d-unet-mlperf-inference/accuracy_fp32
+3d-UNet-MLPerf Accuracy bf16:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-image-segmentation-tf-3d-unet-mlperf-inference
+  cmd: /bin/bash quickstart/accuracy.sh
+  env:
+    DATASET_DIR: /tf_dataset/dataset/3DUNet/MICCAI_BraTS_2019_Data_Training
+    OUTPUT_DIR: /output/tf-3d-unet-mlperf-inference/accuracy_bf16
+    BATCH_SIZE: '1'
+    PRECISION: bfloat16
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/3DUNet/MICCAI_BraTS_2019_Data_Training
+    dst: /tf_dataset/dataset/3DUNet/MICCAI_BraTS_2019_Data_Training
+  - src: $PWD/output/tf-3d-unet-mlperf-inference/accuracy_bf16
+    dst: /output/tf-3d-unet-mlperf-inference/accuracy_bf16
+3d-UNet-MLPerf Accuracy int8:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-image-segmentation-tf-3d-unet-mlperf-inference
+  cmd: /bin/bash quickstart/accuracy.sh
+  env:
+    DATASET_DIR: /tf_dataset/dataset/3DUNet/MICCAI_BraTS_2019_Data_Training
+    OUTPUT_DIR: /output/tf-3d-unet-mlperf-inference/accuracy_int8
+    BATCH_SIZE: '1'
+    PRECISION: int8
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/3DUNet/MICCAI_BraTS_2019_Data_Training
+    dst: /tf_dataset/dataset/3DUNet/MICCAI_BraTS_2019_Data_Training
+  - src: $PWD/output/tf-3d-unet-mlperf-inference/accuracy_int8
+    dst: /output/tf-3d-unet-mlperf-inference/accuracy_int8
+3d-UNet-MLPerf Realtime Inference int8:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-image-segmentation-tf-3d-unet-mlperf-inference
+  cmd: /bin/bash quickstart/inference_realtime.sh --benchmark-only --num-intra-threads 4 --num-inter-threads 1 
+  env:
+    DATASET_DIR: /tf_dataset/dataset/3DUNet/MICCAI_BraTS_2019_Data_Training
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+    OUTPUT_DIR: /output/tf-3d-unet-mlperf-inference/realtime_int8
+    BATCH_SIZE: '1'
+    PRECISION: int8
+    OMP_NUM_THREADS: '4'
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/3DUNet/MICCAI_BraTS_2019_Data_Training
+    dst: /tf_dataset/dataset/3DUNet/MICCAI_BraTS_2019_Data_Training
+  - src: $PWD/output/tf-3d-unet-mlperf-inference/realtime_int8
+    dst: /output/tf-3d-unet-mlperf-inference/realtime_int8
+3d-UNet-MLPerf Realtime Inference bf16:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-image-segmentation-tf-3d-unet-mlperf-inference
+  cmd: /bin/bash quickstart/inference_realtime.sh
+  env:
+    DATASET_DIR: /tf_dataset/dataset/3DUNet/MICCAI_BraTS_2019_Data_Training
+    OUTPUT_DIR: /output/tf-3d-unet-mlperf-inference/realtime_bf16
+    BATCH_SIZE: '1'
+    PRECISION: bfloat16
+    TF_AUTO_MIXED_PRECISION_GRAPH_REWRITE_DENYLIST_REMOVE: Mean
+    TF_AUTO_MIXED_PRECISION_GRAPH_REWRITE_INFERLIST_ADD: Mean,Rsqrt,SquaredDifference
+    OMP_NUM_THREADS: '4'
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/3DUNet/MICCAI_BraTS_2019_Data_Training
+    dst: /tf_dataset/dataset/3DUNet/MICCAI_BraTS_2019_Data_Training
+  - src: $PWD/output/tf-3d-unet-mlperf-inference/realtime_bf16
+    dst: /output/tf-3d-unet-mlperf-inference/realtime_bf16
+3d-UNet-MLPerf Realtime Inference fp32:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-image-segmentation-tf-3d-unet-mlperf-inference
+  cmd: /bin/bash quickstart/inference_realtime.sh --benchmark-only --num-intra-threads 4 --num-inter-threads 1
+  env:
+    DATASET_DIR: /tf_dataset/dataset/3DUNet/MICCAI_BraTS_2019_Data_Training
+    OUTPUT_DIR: /output/tf-3d-unet-mlperf-inference/realtime_fp32
+    BATCH_SIZE: '1'
+    PRECISION: fp32
+    OMP_NUM_THREADS: '4'
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/3DUNet/MICCAI_BraTS_2019_Data_Training
+    dst: /tf_dataset/dataset/3DUNet/MICCAI_BraTS_2019_Data_Training
+  - src: $PWD/output/tf-3d-unet-mlperf-inference/realtime_fp32
+    dst: /output/tf-3d-unet-mlperf-inference/realtime_fp32
+3d-UNet-MLPerf Throughput int8:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-image-segmentation-tf-3d-unet-mlperf-inference
+  cmd: /bin/bash quickstart/inference_throughput.sh --benchmark-only --num-intra-threads 64 --num-inter-threads 1
+  env:
+    DATASET_DIR: /tf_dataset/dataset/3DUNet/MICCAI_BraTS_2019_Data_Training
+    OUTPUT_DIR: /output/tf-3d-unet-mlperf-inference/throughput_int8
+    BATCH_SIZE: '6'
+    PRECISION: int8
+    OMP_NUM_THREADS: '64'
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/3DUNet/MICCAI_BraTS_2019_Data_Training
+    dst: /tf_dataset/dataset/3DUNet/MICCAI_BraTS_2019_Data_Training
+  - src: $PWD/output/tf-3d-unet-mlperf-inference/throughput_int8
+    dst: /output/tf-3d-unet-mlperf-inference/throughput_int8
+3d-UNet-MLPerf Throughput bf16:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-image-segmentation-tf-3d-unet-mlperf-inference
+  cmd: /bin/bash quickstart/inference_throughput.sh --benchmark-only --num-intra-threads 64 --num-inter-threads 1
+  env:
+    DATASET_DIR: /tf_dataset/dataset/3DUNet/MICCAI_BraTS_2019_Data_Training
+    OUTPUT_DIR: /output/tf-3d-unet-mlperf-inference/throughput_bf16
+    BATCH_SIZE: '6'
+    PRECISION: bfloat16
+    TF_AUTO_MIXED_PRECISION_GRAPH_REWRITE_DENYLIST_REMOVE: Mean
+    TF_AUTO_MIXED_PRECISION_GRAPH_REWRITE_INFERLIST_ADD: Mean,Rsqrt,SquaredDifference
+    OMP_NUM_THREADS: '64'
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/3DUNet/MICCAI_BraTS_2019_Data_Training
+    dst: /tf_dataset/dataset/3DUNet/MICCAI_BraTS_2019_Data_Training
+  - src: $PWD/output/tf-3d-unet-mlperf-inference/throughput_bf16
+    dst: /output/tf-3d-unet-mlperf-inference/throughput_bf16
+3d-UNet-MLPerf Throughput fp32:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-image-segmentation-tf-3d-unet-mlperf-inference
+  cmd: /bin/bash quickstart/inference_throughput.sh --benchmark-only --num-intra-threads 64 --num-inter-threads 1
+  env:
+    DATASET_DIR: /tf_dataset/dataset/3DUNet/MICCAI_BraTS_2019_Data_Training
+    OUTPUT_DIR: /output/tf-3d-unet-mlperf-inference/throughput_fp32
+    BATCH_SIZE: '6'
+    PRECISION: fp32
+    OMP_NUM_THREADS: '64'
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/3DUNet/MICCAI_BraTS_2019_Data_Training
+    dst: /tf_dataset/dataset/3DUNet/MICCAI_BraTS_2019_Data_Training
+  - src: $PWD/output/tf-3d-unet-mlperf-inference/throughput_fp32
+    dst: /output/tf-3d-unet-mlperf-inference/throughput_fp32
diff --git a/docker/tf-cpu/tf-3d-unet-mlperf-inference/tf-3d-unet-mlperf-inference.Dockerfile-centos b/docker/tf-cpu/tf-3d-unet-mlperf-inference/tf-3d-unet-mlperf-inference.Dockerfile-centos
new file mode 100644
index 000000000..d7cd581a5
--- /dev/null
+++ b/docker/tf-cpu/tf-3d-unet-mlperf-inference/tf-3d-unet-mlperf-inference.Dockerfile-centos
@@ -0,0 +1,65 @@
+# Copyright (c) 2020-2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+#
+# THIS IS A GENERATED DOCKERFILE.
+#
+# This file was assembled from multiple pieces, whose use is documented
+# throughout. Please refer to the TensorFlow dockerfiles documentation
+# for more information.
+
+ARG TF_BASE_IMAGE=intel/intel-optimized-tensorflow-avx512
+
+ARG TF_BASE_TAG=latest
+
+FROM ${TF_BASE_IMAGE}:${TF_BASE_TAG}
+
+RUN yum update -y && \
+    yum install -y numactl curl wget
+
+WORKDIR /workspace/tf-3d-unet-mlperf-inference 
+
+COPY benchmarks benchmarks 
+COPY models models
+COPY quickstart/common quickstart/common
+COPY quickstart/image_segmentation/tensorflow/3d_unet_mlperf/inference/cpu/accuracy.sh quickstart/accuracy.sh
+COPY quickstart/image_segmentation/tensorflow/3d_unet_mlperf/inference/cpu/inference_throughput_multi_instance.sh quickstart/inference_throughput.sh
+COPY quickstart/image_segmentation/tensorflow/3d_unet_mlperf/inference/cpu/inference_realtime_multi_instance.sh quickstart/inference_realtime.sh
+
+RUN python -m pip install \
+        torch==1.13.1+cpu \
+        --extra-index-url https://download.pytorch.org/whl/cpu && \
+    python -m pip install \
+        'numpy>=1.20,<=1.23' \
+        'nilearn==0.7.1' \
+        'tables==3.7.0' \
+        'nibabel==3.2.1' \
+        'SimpleITK==2.2.0' \
+        nnunet \
+        tqdm \
+        dicom2nifti \
+        scikit-image \
+        medpy \
+        scipy \
+        batchgenerators==0.24 \
+        scikit-learn \
+        pandas \
+        matplotlib
+
+COPY LICENSE license/LICENSE
+COPY third_party license/third_party
+
+RUN mkdir -p /workspace/tf-3d-unet-mlperf-inference/pretrained_model && \ 
+    wget -qO /workspace/tf-3d-unet-mlperf-inference/pretrained_model/3dunet_dynamic_ndhwc.pb https://storage.googleapis.com/intel-optimized-tensorflow/models/2_10_0/3dunet_dynamic_ndhwc.pb && \
+    wget -qO /workspace/tf-3d-unet-mlperf-inference/pretrained_model/3dunet_new_int8_bf16.pb https://storage.googleapis.com/intel-optimized-tensorflow/models/2_10_0/3dunet_new_int8_bf16.pb
diff --git a/docker/tf-cpu/tf-3d-unet-mlperf-inference/tf-3d-unet-mlperf-inference.Dockerfile-ubuntu b/docker/tf-cpu/tf-3d-unet-mlperf-inference/tf-3d-unet-mlperf-inference.Dockerfile-ubuntu
new file mode 100644
index 000000000..f332d917a
--- /dev/null
+++ b/docker/tf-cpu/tf-3d-unet-mlperf-inference/tf-3d-unet-mlperf-inference.Dockerfile-ubuntu
@@ -0,0 +1,66 @@
+# Copyright (c) 2020-2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+#
+# THIS IS A GENERATED DOCKERFILE.
+#
+# This file was assembled from multiple pieces, whose use is documented
+# throughout. Please refer to the TensorFlow dockerfiles documentation
+# for more information.
+
+ARG TF_BASE_IMAGE=intel/intel-optimized-tensorflow-avx512
+
+ARG TF_BASE_TAG=latest
+
+FROM ${TF_BASE_IMAGE}:${TF_BASE_TAG}
+
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y numactl curl wget 
+
+WORKDIR /workspace/tf-3d-unet-mlperf-inference 
+
+COPY benchmarks benchmarks 
+COPY models models
+COPY quickstart/common quickstart/common
+COPY quickstart/image_segmentation/tensorflow/3d_unet_mlperf/inference/cpu/accuracy.sh quickstart/accuracy.sh
+COPY quickstart/image_segmentation/tensorflow/3d_unet_mlperf/inference/cpu/inference_throughput_multi_instance.sh quickstart/inference_throughput.sh
+COPY quickstart/image_segmentation/tensorflow/3d_unet_mlperf/inference/cpu/inference_realtime_multi_instance.sh quickstart/inference_realtime.sh
+
+RUN python -m pip install \
+        torch==1.13.1+cpu \
+        --extra-index-url https://download.pytorch.org/whl/cpu && \
+    python -m pip install \
+        'numpy>=1.20,<=1.23' \
+        'nilearn==0.7.1' \
+        'tables==3.7.0' \
+        'nibabel==3.2.1' \
+        'SimpleITK==2.2.0' \
+        nnunet \
+        tqdm \
+        dicom2nifti \
+        scikit-image \
+        medpy \
+        scipy \
+        batchgenerators==0.24 \
+        scikit-learn \
+        pandas \
+        matplotlib
+
+COPY LICENSE license/LICENSE
+COPY third_party license/third_party
+
+RUN mkdir -p /workspace/tf-3d-unet-mlperf-inference/pretrained_model && \ 
+    wget -qO /workspace/tf-3d-unet-mlperf-inference/pretrained_model/3dunet_dynamic_ndhwc.pb https://storage.googleapis.com/intel-optimized-tensorflow/models/2_10_0/3dunet_dynamic_ndhwc.pb && \
+    wget -qO /workspace/tf-3d-unet-mlperf-inference/pretrained_model/3dunet_new_int8_bf16.pb https://storage.googleapis.com/intel-optimized-tensorflow/models/2_10_0/3dunet_new_int8_bf16.pb
diff --git a/docker/tf-cpu/tf-bert-large-inference/tests.yaml b/docker/tf-cpu/tf-bert-large-inference/tests.yaml
new file mode 100644
index 000000000..d3bf9aacd
--- /dev/null
+++ b/docker/tf-cpu/tf-bert-large-inference/tests.yaml
@@ -0,0 +1,226 @@
+BERT-Large Realtime Inference fp32:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-language-modeling-tf-bert-large-inference
+  cmd: /bin/bash quickstart/inference_realtime.sh
+  env:
+    DATASET_DIR: /tf_dataset/dataset/bert_large_wwm/wwm_uncased_L-24_H-1024_A-16
+    OUTPUT_DIR: /output/tf-bert-large-inference/realtime_fp32
+    BATCH_SIZE: '1'
+    PRECISION: fp32
+    OMP_NUM_THREADS: '64'
+    CORES_PER_INSTANCE: '64'
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/bert_large_wwm/wwm_uncased_L-24_H-1024_A-16
+    dst: /tf_dataset/dataset/bert_large_wwm/wwm_uncased_L-24_H-1024_A-16
+  - src: $PWD/output/tf-bert-large-inference/realtime_fp32
+    dst: /output/tf-bert-large-inference/realtime_fp32
+BERT-Large Weightsharing Inference fp32:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-language-modeling-tf-bert-large-inference
+  cmd: /bin/bash quickstart/inference_realtime_weight_sharing.sh
+  env:
+    TF_ONEDNN_USE_SYSTEM_ALLOCATOR: '1'
+    OMP_NUM_THREADS: '4'
+    CORES_PER_INSTANCE: '4'
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0 
+    DATASET_DIR: /tf_dataset/dataset/bert_large_wwm/wwm_uncased_L-24_H-1024_A-16
+    OUTPUT_DIR: /output/tf-bert-large-inference/weightsharing_fp32
+    BATCH_SIZE: '1'
+    PRECISION: fp32
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/bert_large_wwm/wwm_uncased_L-24_H-1024_A-16
+    dst: /tf_dataset/dataset/bert_large_wwm/wwm_uncased_L-24_H-1024_A-16
+  - src: $PWD/output/tf-bert-large-inference/weightsharing_fp32
+    dst: /output/tf-bert-large-inference/weightsharing_fp32
+BERT-Large Throughput fp32:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-language-modeling-tf-bert-large-inference
+  cmd: /bin/bash quickstart/inference_throughput.sh
+  env:
+    DATASET_DIR: /tf_dataset/dataset/bert_large_wwm/wwm_uncased_L-24_H-1024_A-16
+    OUTPUT_DIR: /output/tf-bert-large-inference/throughput_fp32
+    BATCH_SIZE: '128'
+    PRECISION: fp32
+    OMP_NUM_THREADS: '64'
+    CORES_PER_INSTANCE: '64'
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/bert_large_wwm/wwm_uncased_L-24_H-1024_A-16
+    dst: /tf_dataset/dataset/bert_large_wwm/wwm_uncased_L-24_H-1024_A-16
+  - src: $PWD/output/tf-bert-large-inference/throughput_fp32
+    dst: /output/tf-bert-large-inference/throughput_fp32
+BERT-Large Accuracy fp32:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-language-modeling-tf-bert-large-inference
+  cmd: /bin/bash quickstart/accuracy.sh --verbose
+  env:
+    DATASET_DIR: /tf_dataset/dataset/bert_large_wwm/wwm_uncased_L-24_H-1024_A-16
+    OUTPUT_DIR: /output/tf-bert-large-inference/accuracy_fp32
+    BATCH_SIZE: '56'
+    PRECISION: fp32
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/bert_large_wwm/wwm_uncased_L-24_H-1024_A-16
+    dst: /tf_dataset/dataset/bert_large_wwm/wwm_uncased_L-24_H-1024_A-16
+  - src: $PWD/output/tf-bert-large-inference/accuracy_fp32
+    dst: /output/tf-bert-large-inference/accuracy_fp32
+BERT-Large Accuracy bf16:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-language-modeling-tf-bert-large-inference
+  cmd: /bin/bash quickstart/accuracy.sh --verbose
+  env:
+    DATASET_DIR: /tf_dataset/dataset/bert_large_wwm/wwm_uncased_L-24_H-1024_A-16
+    OUTPUT_DIR: /output/tf-bert-large-inference/accuracy_bf16
+    BATCH_SIZE: '56'
+    PRECISION: bfloat16
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/bert_large_wwm/wwm_uncased_L-24_H-1024_A-16
+    dst: /tf_dataset/dataset/bert_large_wwm/wwm_uncased_L-24_H-1024_A-16
+  - src: $PWD/output/tf-bert-large-inference/accuracy_bf16
+    dst: /output/tf-bert-large-inference/accuracy_bf16
+BERT-Large Realtime Inference bf16:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-language-modeling-tf-bert-large-inference
+  cmd: /bin/bash quickstart/inference_realtime.sh
+  env:
+    DATASET_DIR: /tf_dataset/dataset/bert_large_wwm/wwm_uncased_L-24_H-1024_A-16
+    OUTPUT_DIR: /output/tf-bert-large-inference/realtime_bf16
+    BATCH_SIZE: '1'
+    PRECISION: bfloat16
+    LD_PRELOAD: /workspace/tcmalloc/install/lib/libtcmalloc.so.4
+    OMP_NUM_THREADS: '4'
+    CORES_PER_INSTANCE: '4'
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/bert_large_wwm/wwm_uncased_L-24_H-1024_A-16
+    dst: /tf_dataset/dataset/bert_large_wwm/wwm_uncased_L-24_H-1024_A-16
+  - src: $PWD/output/tf-bert-large-inference/realtime_bf16
+    dst: /output/tf-bert-large-inference/realtime_bf16
+BERT-Large Weightsharing Inference bf16:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-language-modeling-tf-bert-large-inference
+  cmd: /bin/bash quickstart/inference_realtime_weight_sharing.sh
+  env:
+    DATASET_DIR: /tf_dataset/dataset/bert_large_wwm/wwm_uncased_L-24_H-1024_A-16
+    OUTPUT_DIR: /output/tf-bert-large-inference/weightsharing_bf16
+    BATCH_SIZE: '1'
+    PRECISION: bfloat16
+    LD_PRELOAD: /workspace/tcmalloc/install/lib/libtcmalloc.so.4
+    TF_ONEDNN_USE_SYSTEM_ALLOCATOR: '1'
+    OMP_NUM_THREADS: '64'
+    CORES_PER_INSTANCE: '64'
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/bert_large_wwm/wwm_uncased_L-24_H-1024_A-16
+    dst: /tf_dataset/dataset/bert_large_wwm/wwm_uncased_L-24_H-1024_A-16
+  - src: $PWD/output/tf-bert-large-inference/weightsharing_bf16
+    dst: /output/tf-bert-large-inference/weightsharing_bf16
+BERT-Large Throughput bf16:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-language-modeling-tf-bert-large-inference
+  cmd: /bin/bash quickstart/inference_throughput.sh
+  env:
+    DATASET_DIR: /tf_dataset/dataset/bert_large_wwm/wwm_uncased_L-24_H-1024_A-16
+    OUTPUT_DIR: /output/tf-bert-large-inference/throughput_bf16
+    BATCH_SIZE: '128'
+    PRECISION: bfloat16
+    OMP_NUM_THREADS: '64'
+    CORES_PER_INSTANCE: '64'
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/bert_large_wwm/wwm_uncased_L-24_H-1024_A-16
+    dst: /tf_dataset/dataset/bert_large_wwm/wwm_uncased_L-24_H-1024_A-16
+  - src: $PWD/output/tf-bert-large-inference/throughput_bf16
+    dst: /output/tf-bert-large-inference/throughput_bf16
+BERT-Large Accuracy int8:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-language-modeling-tf-bert-large-inference
+  cmd: /bin/bash quickstart/accuracy.sh
+  env:
+    DATASET_DIR: /tf_dataset/dataset/bert_large_wwm/wwm_uncased_L-24_H-1024_A-16
+    OUTPUT_DIR: /output/tf-bert-large-inference/accuracy_int8
+    BATCH_SIZE: '56'
+    PRECISION: int8
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/bert_large_wwm/wwm_uncased_L-24_H-1024_A-16
+    dst: /tf_dataset/dataset/bert_large_wwm/wwm_uncased_L-24_H-1024_A-16
+  - src: $PWD/output/tf-bert-large-inference/accuracy_int8
+    dst: /output/tf-bert-large-inference/accuracy_int8
+BERT-Large Realtime Inference int8:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-language-modeling-tf-bert-large-inference
+  cmd: /bin/bash quickstart/inference_realtime.sh
+  env:
+    DATASET_DIR: /tf_dataset/dataset/bert_large_wwm/wwm_uncased_L-24_H-1024_A-16
+    OUTPUT_DIR: /output/tf-bert-large-inference/realtime_int8
+    BATCH_SIZE: '1'
+    PRECISION: int8
+    LD_PRELOAD: /workspace/tcmalloc/install/lib/libtcmalloc.so.4
+    OMP_NUM_THREADS: '4'
+    CORES_PER_INSTANCE: '4'
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/bert_large_wwm/wwm_uncased_L-24_H-1024_A-16
+    dst: /tf_dataset/dataset/bert_large_wwm/wwm_uncased_L-24_H-1024_A-16
+  - src: $PWD/output/tf-bert-large-inference/realtime_int8
+    dst: /output/tf-bert-large-inference/realtime_int8
+BERT-Large Weightsharing Inference int8:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-language-modeling-tf-bert-large-inference
+  cmd: /bin/bash quickstart/inference_realtime_weight_sharing.sh
+  env:
+    DATASET_DIR: /tf_dataset/dataset/bert_large_wwm/wwm_uncased_L-24_H-1024_A-16
+    OUTPUT_DIR: /output/tf-bert-large-inference/weightsharing_int8
+    BATCH_SIZE: '1'
+    PRECISION: int8
+    LD_PRELOAD: /workspace/tcmalloc/install/lib/libtcmalloc.so.4
+    TF_ONEDNN_USE_SYSTEM_ALLOCATOR: '1'
+    OMP_NUM_THREADS: '64'
+    CORES_PER_INSTANCE: '64'
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/bert_large_wwm/wwm_uncased_L-24_H-1024_A-16
+    dst: /tf_dataset/dataset/bert_large_wwm/wwm_uncased_L-24_H-1024_A-16
+  - src: $PWD/output/tf-bert-large-inference/weightsharing_int8
+    dst: /output/tf-bert-large-inference/weightsharing_int8
+BERT-Large Throughput int8:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-language-modeling-tf-bert-large-inference
+  cmd: /bin/bash quickstart/inference_throughput.sh
+  env:
+    DATASET_DIR: /tf_dataset/dataset/bert_large_wwm/wwm_uncased_L-24_H-1024_A-16
+    OUTPUT_DIR: /output/tf-bert-large-inference/throughput_int8
+    BATCH_SIZE: '16'
+    PRECISION: int8
+    OMP_NUM_THREADS: '4'
+    CORES_PER_INSTANCE: '4'
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/bert_large_wwm/wwm_uncased_L-24_H-1024_A-16
+    dst: /tf_dataset/dataset/bert_large_wwm/wwm_uncased_L-24_H-1024_A-16
+  - src: $PWD/output/tf-bert-large-inference/throughput_int8
+    dst: /output/tf-bert-large-inference/throughput_int8
diff --git a/docker/tf-cpu/tf-bert-large-inference/tf-bert-large-inference.Dockerfile-centos b/docker/tf-cpu/tf-bert-large-inference/tf-bert-large-inference.Dockerfile-centos
new file mode 100644
index 000000000..5a2b83df4
--- /dev/null
+++ b/docker/tf-cpu/tf-bert-large-inference/tf-bert-large-inference.Dockerfile-centos
@@ -0,0 +1,67 @@
+# Copyright (c) 2020-2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+#
+# THIS IS A GENERATED DOCKERFILE.
+#
+# This file was assembled from multiple pieces, whose use is documented
+# throughout. Please refer to the TensorFlow dockerfiles documentation
+# for more information.
+
+ARG TF_BASE_IMAGE=intel/intel-optimized-tensorflow-avx512
+
+ARG TF_BASE_TAG=latest
+
+FROM ${TF_BASE_IMAGE}:${TF_BASE_TAG}
+
+WORKDIR /workspace/tf-bert-large-inference
+
+RUN yum update -y && \
+    yum install -y numactl unzip wget ca-certificates && \
+    yum clean all
+
+RUN mkdir -p /workspace/tcmalloc && \
+    cd /workspace/tcmalloc && \
+    wget https://github.com/gperftools/gperftools/releases/download/gperftools-2.7.90/gperftools-2.7.90.tar.gz && \
+    tar -xzf gperftools-2.7.90.tar.gz && \
+    rm gperftools-2.7.90.tar.gz && \
+    cd gperftools-2.7.90 && \
+    ./configure --prefix=/workspace/tcmalloc/install && \
+    make && \
+    make install && \
+    cd /workspace && \
+    rm -rf gperftools-2.7.90
+
+ENV LD_LIBRARY_PATH="/workspace/tcmalloc/install/lib/":${LD_LIBRARY_PATH}
+ENV LD_PRELOAD=/workspace/tcmalloc/install/lib/libtcmalloc.so.4
+
+COPY benchmarks benchmarks
+COPY models/common models/common
+COPY models/language_modeling/tensorflow/bert_large/inference models/language_modeling/tensorflow/bert_large/inference
+COPY quickstart/common quickstart/common 
+COPY quickstart/language_modeling/tensorflow/bert_large/inference/cpu/inference_realtime_multi_instance.sh quickstart/inference_realtime.sh
+COPY quickstart/language_modeling/tensorflow/bert_large/inference/cpu/inference_realtime_weight_sharing.sh quickstart/inference_realtime_weight_sharing.sh
+COPY quickstart/language_modeling/tensorflow/bert_large/inference/cpu/inference_throughput_multi_instance.sh quickstart/inference_throughput.sh
+COPY quickstart/language_modeling/tensorflow/bert_large/inference/cpu/accuracy.sh quickstart/accuracy.sh
+
+COPY LICENSE license/LICENSE
+COPY third_party license/third_party
+
+RUN mkdir -p /workspace/tf-bert-large-inference/pretrained_model && \ 
+    wget -qO /workspace/tf-bert-large-inference/pretrained_model/bert_large_fp32_pretrained_model.pb https://storage.googleapis.com/intel-optimized-tensorflow/models/2_10_0/fp32_bert_squad.pb && \
+    wget -qO /workspace/tf-bert-large-inference/pretrained_model/bert_large_bfloat16_pretrained_model.pb https://storage.googleapis.com/intel-optimized-tensorflow/models/2_10_0/optimized_bf16_bert.pb && \
+    wget -qO /workspace/tf-bert-large-inference/pretrained_model/bert_large_int8_pretrained_model.pb https://storage.googleapis.com/intel-optimized-tensorflow/models/2_10_0/per_channel_opt_int8_bf16_bert.pb && \
+    wget -qO /workspace/tf-bert-large-inference/bert_large_checkpoints.zip https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/bert_large_checkpoints.zip && \
+    unzip /workspace/tf-bert-large-inference/bert_large_checkpoints.zip -d /workspace/tf-bert-large-inference/pretrained_model/ && \
+    rm /workspace/tf-bert-large-inference/bert_large_checkpoints.zip
diff --git a/docker/tf-cpu/tf-bert-large-inference/tf-bert-large-inference.Dockerfile-ubuntu b/docker/tf-cpu/tf-bert-large-inference/tf-bert-large-inference.Dockerfile-ubuntu
new file mode 100644
index 000000000..ac21edfdb
--- /dev/null
+++ b/docker/tf-cpu/tf-bert-large-inference/tf-bert-large-inference.Dockerfile-ubuntu
@@ -0,0 +1,53 @@
+# Copyright (c) 2020-2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+#
+# THIS IS A GENERATED DOCKERFILE.
+#
+# This file was assembled from multiple pieces, whose use is documented
+# throughout. Please refer to the TensorFlow dockerfiles documentation
+# for more information.
+
+ARG TF_BASE_IMAGE=intel/intel-optimized-tensorflow-avx512
+
+ARG TF_BASE_TAG=latest
+
+FROM ${TF_BASE_IMAGE}:${TF_BASE_TAG}
+
+WORKDIR /workspace/tf-bert-large-inference
+
+RUN apt-get update && \
+    apt-get install -y numactl unzip wget google-perftools
+
+ENV LD_PRELOAD="/usr/lib/x86_64-linux-gnu/libtcmalloc.so.4.5.9"
+
+COPY benchmarks benchmarks
+COPY models/common models/common
+COPY models/language_modeling/tensorflow/bert_large/inference models/language_modeling/tensorflow/bert_large/inference
+COPY quickstart/common quickstart/common 
+COPY quickstart/language_modeling/tensorflow/bert_large/inference/cpu/inference_realtime_multi_instance.sh quickstart/inference_realtime.sh
+COPY quickstart/language_modeling/tensorflow/bert_large/inference/cpu/inference_realtime_weight_sharing.sh quickstart/inference_realtime_weight_sharing.sh
+COPY quickstart/language_modeling/tensorflow/bert_large/inference/cpu/inference_throughput_multi_instance.sh quickstart/inference_throughput.sh
+COPY quickstart/language_modeling/tensorflow/bert_large/inference/cpu/accuracy.sh quickstart/accuracy.sh
+
+COPY LICENSE license/LICENSE
+COPY third_party license/third_party
+
+RUN mkdir -p /workspace/tf-bert-large-inference/pretrained_model && \ 
+    wget -qO /workspace/tf-bert-large-inference/pretrained_model/bert_large_fp32_pretrained_model.pb https://storage.googleapis.com/intel-optimized-tensorflow/models/2_10_0/fp32_bert_squad.pb && \
+    wget -qO /workspace/tf-bert-large-inference/pretrained_model/bert_large_bfloat16_pretrained_model.pb https://storage.googleapis.com/intel-optimized-tensorflow/models/2_10_0/optimized_bf16_bert.pb && \
+    wget -qO /workspace/tf-bert-large-inference/pretrained_model/bert_large_int8_pretrained_model.pb https://storage.googleapis.com/intel-optimized-tensorflow/models/2_10_0/per_channel_opt_int8_bf16_bert.pb && \
+    wget -qO /workspace/tf-bert-large-inference/bert_large_checkpoints.zip https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/bert_large_checkpoints.zip && \
+    unzip /workspace/tf-bert-large-inference/bert_large_checkpoints.zip -d /workspace/tf-bert-large-inference/pretrained_model/ && \
+    rm /workspace/tf-bert-large-inference/bert_large_checkpoints.zip
diff --git a/docker/tf-cpu/tf-bert-large-pretraining/tests.yaml b/docker/tf-cpu/tf-bert-large-pretraining/tests.yaml
new file mode 100644
index 000000000..bd4f41ce6
--- /dev/null
+++ b/docker/tf-cpu/tf-bert-large-pretraining/tests.yaml
@@ -0,0 +1,36 @@
+BERT-Large Pretraining fp32:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-language-modeling-tf-bert-large-pretraining
+  cmd: /bin/bash quickstart/pretraining.sh -a 64 -e 2
+  env:
+    DATASET_DIR: /tf_dataset/dataset/bert_large_wwm
+    OUTPUT_DIR: /output/tf-bert-large-pretraining/fp32
+    PRECISION: fp32
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+    OMP_NUM_THREADS: '64'
+    BATCH_SIZE: '32'
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/bert_large_wwm
+    dst: /tf_dataset/dataset/bert_large_wwm
+  - src: $PWD/output/tf-bert-large-pretraining/fp32
+    dst: /output/tf-bert-large-pretraining/fp32
+BERT-Large Pretraining bf16:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-language-modeling-tf-bert-large-pretraining
+  cmd: /bin/bash quickstart/pretraining.sh -a 64 -e 2
+  env:
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+    OMP_NUM_THREADS: '64'
+    DATASET_DIR: /tf_dataset/dataset/bert_large_wwm
+    OUTPUT_DIR: /output/tf-bert-large-pretraining/bf16
+    PRECISION: bfloat16
+    BATCH_SIZE: '128'
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/bert_large_wwm
+    dst: /tf_dataset/dataset/bert_large_wwm
+  - src: $PWD/output/tf-bert-large-pretraining/bf16
+    dst: /output/tf-bert-large-pretraining/bf16
diff --git a/docker/tf-cpu/tf-bert-large-pretraining/tf-bert-large-pretraining.Dockerfile-centos b/docker/tf-cpu/tf-bert-large-pretraining/tf-bert-large-pretraining.Dockerfile-centos
new file mode 100644
index 000000000..c12647dda
--- /dev/null
+++ b/docker/tf-cpu/tf-bert-large-pretraining/tf-bert-large-pretraining.Dockerfile-centos
@@ -0,0 +1,90 @@
+# Copyright (c) 2020-2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+#
+# THIS IS A GENERATED DOCKERFILE.
+#
+# This file was assembled from multiple pieces, whose use is documented
+# throughout. Please refer to the TensorFlow dockerfiles documentation
+# for more information.
+
+
+ARG TF_BASE_IMAGE=intel/intel-optimized-tensorflow-avx512
+
+ARG TF_BASE_TAG=latest
+
+FROM ${TF_BASE_IMAGE}:${TF_BASE_TAG}
+
+WORKDIR /workspace/tf-bert-large-pretraining
+
+RUN yum update -y && \
+    yum install -y git cmake \
+        ca-certificates \
+        python3-tkinter && \
+    yum clean all
+
+# Install OpenMPI
+ARG OPENMPI_VERSION="openmpi-4.1.0"
+ARG OPENMPI_DOWNLOAD_URL="https://www.open-mpi.org/software/ompi/v4.1/downloads/openmpi-4.1.0.tar.gz"
+
+RUN mkdir /tmp/openmpi && \
+    cd /tmp/openmpi && \
+    curl -fSsL -O ${OPENMPI_DOWNLOAD_URL} && \
+    tar zxf ${OPENMPI_VERSION}.tar.gz && \
+    cd ${OPENMPI_VERSION} && \
+    ./configure --enable-mpirun-prefix-by-default && \
+    make -j $(nproc) all && \
+    make install && \
+    ldconfig && \
+    cd / && \
+    rm -rf /tmp/openmpi
+
+# Create a wrapper for OpenMPI to allow running as root by default
+RUN mv /usr/local/bin/mpirun /usr/local/bin/mpirun.real && \
+    echo '#!/bin/bash' > /usr/local/bin/mpirun && \
+    echo 'mpirun.real --allow-run-as-root "$@"' >> /usr/local/bin/mpirun && \
+    chmod a+x /usr/local/bin/mpirun
+
+# Configure OpenMPI to run good defaults:
+RUN echo "btl_tcp_if_exclude = lo,docker0" >> /usr/local/etc/openmpi-mca-params.conf
+
+# Install OpenSSH for MPI to communicate between containers
+RUN yum update -y && \
+    yum install -y  \
+    openssh-server \
+    openssh-clients && \
+    yum clean all
+
+ARG HOROVOD_VERSION=39c8f7c
+ENV HOROVOD_WITHOUT_MXNET=1 \
+    HOROVOD_WITHOUT_PYTORCH=1 \
+    HOROVOD_WITH_TENSORFLOW=1 \
+    HOROVOD_CPU_OPERATIONS=MPI \
+    HOROVOD_WITH_MPI=1 \
+    HOROVOD_WITHOUT_GLOO=1
+
+
+RUN python3 -m pip install git+https://github.com/horovod/horovod.git@${HOROVOD_VERSION}
+
+RUN yum update -y && \
+    yum install -y numactl && \
+    yum clean all
+
+COPY benchmarks benchmarks
+COPY models models
+COPY quickstart/common quickstart/common
+COPY quickstart/language_modeling/tensorflow/bert_large/training/cpu/pretraining.sh quickstart/pretraining.sh
+
+COPY LICENSE license/LICENSE
+COPY third_party license/third_party
diff --git a/docker/tf-cpu/tf-bert-large-pretraining/tf-bert-large-pretraining.Dockerfile-ubuntu b/docker/tf-cpu/tf-bert-large-pretraining/tf-bert-large-pretraining.Dockerfile-ubuntu
new file mode 100644
index 000000000..ef5163b82
--- /dev/null
+++ b/docker/tf-cpu/tf-bert-large-pretraining/tf-bert-large-pretraining.Dockerfile-ubuntu
@@ -0,0 +1,112 @@
+# Copyright (c) 2020-2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+#
+# THIS IS A GENERATED DOCKERFILE.
+#
+# This file was assembled from multiple pieces, whose use is documented
+# throughout. Please refer to the TensorFlow dockerfiles documentation
+# for more information.
+
+
+ARG TF_BASE_IMAGE=intel/intel-optimized-tensorflow-avx512
+
+ARG TF_BASE_TAG=latest
+
+FROM ${TF_BASE_IMAGE}:${TF_BASE_TAG}
+
+WORKDIR /workspace/tf-bert-large-pretraining
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+ARG PY_VERSION=3.10
+RUN apt-get update && apt-get install --no-install-recommends --fix-missing -y gcc 
+
+RUN apt-get update && \
+    apt-get install -y software-properties-common && \
+    add-apt-repository ppa:deadsnakes/ppa -y  && \
+    apt install -y python${PY_VERSION} \
+    python${PY_VERSION}-dev
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y curl build-essential 
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y git cmake libxext-dev \
+        libsm-dev \
+        ca-certificates \
+        python3-tk && \
+    apt-get clean all
+
+# Install OpenMPI
+ARG OPENMPI_VERSION="openmpi-4.1.0"
+ARG OPENMPI_DOWNLOAD_URL="https://www.open-mpi.org/software/ompi/v4.1/downloads/openmpi-4.1.0.tar.gz"
+
+RUN mkdir /tmp/openmpi && \
+    cd /tmp/openmpi && \
+    curl -fSsL -O ${OPENMPI_DOWNLOAD_URL} && \
+    tar zxf ${OPENMPI_VERSION}.tar.gz && \
+    cd ${OPENMPI_VERSION} && \
+    ./configure --enable-mpirun-prefix-by-default && \
+    make -j $(nproc) all && \
+    make install && \
+    ldconfig && \
+    cd / && \
+    rm -rf /tmp/openmpi
+
+# Create a wrapper for OpenMPI to allow running as root by default
+RUN mv /usr/local/bin/mpirun /usr/local/bin/mpirun.real && \
+    echo '#!/bin/bash' > /usr/local/bin/mpirun && \
+    echo 'mpirun.real --allow-run-as-root "$@"' >> /usr/local/bin/mpirun && \
+    chmod a+x /usr/local/bin/mpirun
+
+# Configure OpenMPI to run good defaults:
+RUN echo "btl_tcp_if_exclude = lo,docker0" >> /usr/local/etc/openmpi-mca-params.conf
+
+# Install OpenSSH for MPI to communicate between containers
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y  \
+    openssh-server \
+    openssh-client && \
+    rm  /etc/ssh/ssh_host_*_key \
+    /etc/ssh/ssh_host_*_key.pub && \
+    apt-get clean all
+
+ARG HOROVOD_VERSION=39c8f7c
+ENV HOROVOD_WITHOUT_MXNET=1 \
+    HOROVOD_WITHOUT_PYTORCH=1 \
+    HOROVOD_WITH_TENSORFLOW=1 \
+    HOROVOD_CPU_OPERATIONS=MPI \
+    HOROVOD_WITH_MPI=1 \
+    HOROVOD_WITHOUT_GLOO=1
+
+
+RUN python3 -m pip install git+https://github.com/horovod/horovod.git@${HOROVOD_VERSION}
+
+RUN apt-get update && apt-get install --no-install-recommends --fix-missing -y numactl
+
+COPY benchmarks benchmarks
+COPY models models
+COPY quickstart/common quickstart/common
+COPY quickstart/language_modeling/tensorflow/bert_large/training/cpu/pretraining.sh quickstart/pretraining.sh
+
+COPY LICENSE license/LICENSE
+COPY third_party license/third_party
+
+COPY docker/tf-cpu/generate_ssh_key.sh /workspace/generate_ssh_key.sh
+
+RUN cat /workspace/generate_ssh_key.sh >> ${HOME}/.bash_profile && \
+    cat /workspace/generate_ssh_key.sh >> ${HOME}/.bashrc
+
+ENV BASH_ENV=/root/.bash_profile
diff --git a/docker/tf-cpu/tf-dien-inference/tests.yaml b/docker/tf-cpu/tf-dien-inference/tests.yaml
new file mode 100644
index 000000000..5d11dab06
--- /dev/null
+++ b/docker/tf-cpu/tf-dien-inference/tests.yaml
@@ -0,0 +1,104 @@
+DIEN Accuracy bf16:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-recommendation-tf-dien-inference
+  cmd: /bin/bash quickstart/accuracy.sh
+  env:
+    DATASET_DIR: /tf_dataset/dataset/DIEN
+    OUTPUT_DIR: /output/tf-dien-inference/accuracy_bf16
+    PRECISION: bfloat16
+    BATCH_SIZE: '128'
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/DIEN
+    dst: /tf_dataset/dataset/DIEN
+  - src: $PWD/output/tf-dien-inference/accuracy_bf16
+    dst: /output/tf-dien-inference/accuracy_bf16
+DIEN Accuracy fp32:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-recommendation-tf-dien-inference
+  cmd: /bin/bash quickstart/accuracy.sh
+  env:
+    DATASET_DIR: /tf_dataset/dataset/DIEN
+    OUTPUT_DIR: /output/tf-dien-inference/accuracy_fp32
+    PRECISION: fp32
+    BATCH_SIZE: '128'
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/DIEN
+    dst: /tf_dataset/dataset/DIEN
+  - src: $PWD/output/tf-dien-inference/accuracy_fp32
+    dst: /output/tf-dien-inference/accuracy_fp32
+DIEN Realtime Inference bf16:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-recommendation-tf-dien-inference
+  cmd: /bin/bash quickstart/inference_realtime.sh --num-inter-threads 1  --num-intra-threads 4 --benchmark-only --verbose
+  env:
+    DATASET_DIR: /tf_dataset/dataset/DIEN
+    OUTPUT_DIR: /output/tf-dien-inference/realtime_bf16
+    PRECISION: bfloat16
+    BATCH_SIZE: '16'
+    OMP_NUM_THREADS: '4'
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/DIEN
+    dst: /tf_dataset/dataset/DIEN
+  - src: $PWD/output/tf-dien-inference/realtime_bf16
+    dst: /output/tf-dien-inference/realtime_bf16
+DIEN Realtime Inference fp32:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-recommendation-tf-dien-inference
+  cmd: /bin/bash quickstart/inference_realtime.sh --num-inter-threads 1  --num-intra-threads 4 --benchmark-only --verbose
+  env:
+    DATASET_DIR: /tf_dataset/dataset/DIEN
+    OUTPUT_DIR: /output/tf-dien-inference/realtime_fp32
+    PRECISION: fp32
+    BATCH_SIZE: '16'
+    OMP_NUM_THREADS: '4'
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/DIEN
+    dst: /tf_dataset/dataset/DIEN
+  - src: $PWD/output/tf-dien-inference/realtime_fp32
+    dst: /output/tf-dien-inference/realtime_fp32
+DIEN Throughput bf16:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-recommendation-tf-dien-inference
+  cmd: /bin/bash quickstart/inference_throughput.sh --num-inter-threads 1  --num-intra-threads 64 --benchmark-only --verbose
+  env:
+    DATASET_DIR: /tf_dataset/dataset/DIEN
+    OUTPUT_DIR: /output/tf-dien-inference/throughput_bf16
+    PRECISION: bfloat16
+    BATCH_SIZE: '65536'
+    OMP_NUM_THREADS: '64'
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/DIEN
+    dst: /tf_dataset/dataset/DIEN
+  - src: $PWD/output/tf-dien-inference/throughput_bf16
+    dst: /output/tf-dien-inference/throughput_bf16
+DIEN Throughput fp32:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-recommendation-tf-dien-inference
+  cmd: /bin/bash quickstart/inference_throughput.sh --num-inter-threads 1  --num-intra-threads 64 --benchmark-only --verbose
+  env:
+    DATASET_DIR: /tf_dataset/dataset/DIEN
+    OUTPUT_DIR: /output/tf-dien-inference/throughput_fp32
+    PRECISION: fp32
+    BATCH_SIZE: '65536'
+    OMP_NUM_THREADS: '64'
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/DIEN
+    dst: /tf_dataset/dataset/DIEN
+  - src: $PWD/output/tf-dien-inference/throughput_fp32
+    dst: /output/tf-dien-inference/throughput_fp32
diff --git a/docker/tf-cpu/tf-dien-inference/tf-dien-inference.Dockerfile-centos b/docker/tf-cpu/tf-dien-inference/tf-dien-inference.Dockerfile-centos
new file mode 100644
index 000000000..5c79d676c
--- /dev/null
+++ b/docker/tf-cpu/tf-dien-inference/tf-dien-inference.Dockerfile-centos
@@ -0,0 +1,46 @@
+# Copyright (c) 2020-2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+#
+# THIS IS A GENERATED DOCKERFILE.
+#
+# This file was assembled from multiple pieces, whose use is documented
+# throughout. Please refer to the TensorFlow dockerfiles documentation
+# for more information.
+
+ARG TF_BASE_IMAGE=intel/intel-optimized-tensorflow-avx512
+
+ARG TF_BASE_TAG=latest
+
+FROM ${TF_BASE_IMAGE}:${TF_BASE_TAG}
+
+RUN yum update -y && \
+    yum install -y numactl wget
+
+WORKDIR /workspace/tf-dien-inference
+
+COPY benchmarks benchmarks
+COPY models/common models/common
+COPY models/recommendation/tensorflow/dien/inference models/recommendation/tensorflow/dien/inference
+COPY quickstart/common quickstart/common 
+COPY quickstart/recommendation/tensorflow/dien/inference/cpu/inference_realtime_multi_instance.sh quickstart/inference_realtime.sh
+COPY quickstart/recommendation/tensorflow/dien/inference/cpu/inference_throughput_multi_instance.sh quickstart/inference_throughput.sh
+COPY quickstart/recommendation/tensorflow/dien/inference/cpu/accuracy.sh quickstart/accuracy.sh
+
+COPY LICENSE license/LICENSE
+COPY third_party license/third_party
+
+RUN mkdir -p /workspace/tf-dien-inference/pretrained_model && \
+    wget -qO /workspace/tf-dien-inference/pretrained_model/dien_fp32_static_mklgrus.pb https://storage.googleapis.com/intel-optimized-tensorflow/models/2_10_0/dien_fp32_static_mklgrus.pb && \
+    wget -qO /workspace/tf-dien-inference/pretrained_model/dien_fp32_dynamic_mklgrus.pb https://storage.googleapis.com/intel-optimized-tensorflow/models/2_10_0/dien_fp32_dynamic_mklgrus.pb
diff --git a/docker/tf-cpu/tf-dien-inference/tf-dien-inference.Dockerfile-ubuntu b/docker/tf-cpu/tf-dien-inference/tf-dien-inference.Dockerfile-ubuntu
new file mode 100644
index 000000000..ee6644710
--- /dev/null
+++ b/docker/tf-cpu/tf-dien-inference/tf-dien-inference.Dockerfile-ubuntu
@@ -0,0 +1,45 @@
+# Copyright (c) 2020-2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+#
+# THIS IS A GENERATED DOCKERFILE.
+#
+# This file was assembled from multiple pieces, whose use is documented
+# throughout. Please refer to the TensorFlow dockerfiles documentation
+# for more information.
+
+ARG TF_BASE_IMAGE=intel/intel-optimized-tensorflow-avx512
+
+ARG TF_BASE_TAG=latest
+
+FROM ${TF_BASE_IMAGE}:${TF_BASE_TAG}
+
+RUN apt-get update && apt-get install -y numactl wget 
+
+WORKDIR /workspace/tf-dien-inference
+
+COPY benchmarks benchmarks
+COPY models/common models/common
+COPY models/recommendation/tensorflow/dien/inference models/recommendation/tensorflow/dien/inference
+COPY quickstart/common quickstart/common 
+COPY quickstart/recommendation/tensorflow/dien/inference/cpu/inference_realtime_multi_instance.sh quickstart/inference_realtime.sh
+COPY quickstart/recommendation/tensorflow/dien/inference/cpu/inference_throughput_multi_instance.sh quickstart/inference_throughput.sh
+COPY quickstart/recommendation/tensorflow/dien/inference/cpu/accuracy.sh quickstart/accuracy.sh
+
+COPY LICENSE license/LICENSE
+COPY third_party license/third_party
+
+RUN mkdir -p /workspace/tf-dien-inference/pretrained_model && \
+    wget -qO /workspace/tf-dien-inference/pretrained_model/dien_fp32_static_mklgrus.pb https://storage.googleapis.com/intel-optimized-tensorflow/models/2_10_0/dien_fp32_static_mklgrus.pb && \
+    wget -qO /workspace/tf-dien-inference/pretrained_model/dien_fp32_dynamic_mklgrus.pb https://storage.googleapis.com/intel-optimized-tensorflow/models/2_10_0/dien_fp32_dynamic_mklgrus.pb
diff --git a/docker/tf-cpu/tf-distilbert-inference/tests.yaml b/docker/tf-cpu/tf-distilbert-inference/tests.yaml
new file mode 100644
index 000000000..ec8b6c447
--- /dev/null
+++ b/docker/tf-cpu/tf-distilbert-inference/tests.yaml
@@ -0,0 +1,234 @@
+DistilBERT Realtime Inference fp32:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-language-modeling-tf-distilbert-inference
+  cmd: /bin/bash quickstart/inference_realtime.sh
+  env:
+    DATASET_DIR: /localdisk/datasets/sst-2/sst2_validation_dataset/
+    OUTPUT_DIR: /output/tf-distilbert-inference/realtime_fp32
+    IN_GRAPH: /workspace/tf-distilbert-inference/pretrained_model/distilbert_frozen_graph_fp32_final.pb
+    BATCH_SIZE: '1'
+    PRECISION: fp32
+    OMP_NUM_THREADS: '4'
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+    WARMUP_STEPS: '20'
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /localdisk/datasets/sst-2/sst2_validation_dataset/
+    dst: /localdisk/datasets/sst-2/sst2_validation_dataset/
+  - src: $PWD/output/tf-distilbert-inference/realtime_fp32
+    dst: /output/tf-distilbert-inference/realtime_fp32
+DistilBERT Weightsharing Inference fp32:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-language-modeling-tf-distilbert-inference
+  cmd: /bin/bash quickstart/inference_realtime_weight_sharing.sh
+  env:
+    TF_ONEDNN_USE_SYSTEM_ALLOCATOR: '1'
+    OMP_NUM_THREADS: '4'
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+    IN_GRAPH: /workspace/tf-distilbert-inference/pretrained_model/distilbert_frozen_graph_fp32_final.pb  
+    DATASET_DIR: /localdisk/datasets/sst-2/sst2_validation_dataset/
+    OUTPUT_DIR: /output/tf-distilbert-inference/weightsharing_fp32
+    BATCH_SIZE: '1'
+    PRECISION: fp32
+    WARMUP_STEPS: '20'
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /localdisk/datasets/sst-2/sst2_validation_dataset/
+    dst: /localdisk/datasets/sst-2/sst2_validation_dataset/
+  - src: $PWD/output/tf-distilbert-inference/weightsharing_fp32
+    dst: /output/tf-distilbert-inference/weightsharing_fp32
+DistilBERT Throughput fp32:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-language-modeling-tf-distilbert-inference
+  cmd: /bin/bash quickstart/inference_throughput.sh
+  env:
+    DATASET_DIR: /localdisk/datasets/sst-2/sst2_validation_dataset/
+    OUTPUT_DIR: /output/tf-distilbert-inference/throughput_fp32
+    BATCH_SIZE: '56'
+    PRECISION: fp32
+    OMP_NUM_THREADS: '64'
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+    IN_GRAPH: /workspace/tf-distilbert-inference/pretrained_model/distilbert_frozen_graph_fp32_final.pb
+    WARMUP_STEPS: '20'
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /localdisk/datasets/sst-2/sst2_validation_dataset/
+    dst: /localdisk/datasets/sst-2/sst2_validation_dataset/
+  - src: $PWD/output/tf-distilbert-inference/throughput_fp32
+    dst: /output/tf-distilbert-inference/throughput_fp32
+DistilBERT Accuracy fp32:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-language-modeling-tf-distilbert-inference
+  cmd: /bin/bash quickstart/accuracy.sh --verbose -- DEBIAN_FRONTEND=noninteractive
+  env:
+    DATASET_DIR: /localdisk/datasets/sst-2/sst2_validation_dataset/
+    OUTPUT_DIR: /output/tf-distilbert-inference/accuracy_fp32
+    IN_GRAPH: /workspace/tf-distilbert-inference/pretrained_model/distilbert_frozen_graph_fp32_final.pb
+    BATCH_SIZE: '56'
+    PRECISION: fp32
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /localdisk/datasets/sst-2/sst2_validation_dataset/
+    dst: /localdisk/datasets/sst-2/sst2_validation_dataset/
+  - src: $PWD/output/tf-distilbert-inference/accuracy_fp32
+    dst: /output/tf-distilbert-inference/accuracy_fp32
+DistilBERT Accuracy bf16:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-language-modeling-tf-distilbert-inference
+  cmd: /bin/bash quickstart/accuracy.sh --verbose -- DEBIAN_FRONTEND=noninteractive 
+  env:
+    DATASET_DIR: /localdisk/datasets/sst-2/sst2_validation_dataset/
+    OUTPUT_DIR: /output/tf-distilbert-inference/accuracy_bf16
+    IN_GRAPH: /workspace/tf-distilbert-inference/pretrained_model/distilbert_frozen_graph_fp32_final.pb
+    BATCH_SIZE: '56'
+    PRECISION: bfloat16
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /localdisk/datasets/sst-2/sst2_validation_dataset/
+    dst: /localdisk/datasets/sst-2/sst2_validation_dataset/
+  - src: $PWD/output/tf-distilbert-inference/accuracy_bf16
+    dst: /output/tf-distilbert-inference/accuracy_bf16
+DistilBERT Realtime Inference bf16:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-language-modeling-tf-distilbert-inference
+  cmd: /bin/bash quickstart/inference_realtime.sh
+  env:
+    DATASET_DIR: /localdisk/datasets/sst-2/sst2_validation_dataset/
+    OUTPUT_DIR: /output/tf-distilbert-inference/realtime_bf16
+    BATCH_SIZE: '1'
+    PRECISION: bfloat16
+    OMP_NUM_THREADS: '4'
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+    IN_GRAPH: /workspace/tf-distilbert-inference/pretrained_model/distilbert_frozen_graph_fp32_final.pb
+    WARMUP_STEPS: '20'
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /localdisk/datasets/sst-2/sst2_validation_dataset/
+    dst: /localdisk/datasets/sst-2/sst2_validation_dataset/
+  - src: $PWD/output/tf-distilbert-inference/realtime_bf16
+    dst: /output/tf-distilbert-inference/realtime_bf16
+DistilBERT Weightsharing Inference bf16:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-language-modeling-tf-distilbert-inference
+  cmd: /bin/bash quickstart/inference_realtime_weight_sharing.sh
+  env:
+    DATASET_DIR: /localdisk/datasets/sst-2/sst2_validation_dataset/
+    OUTPUT_DIR: /output/tf-distilbert-inference/weightsharing_bf16
+    BATCH_SIZE: '1'
+    PRECISION: bfloat16
+    TF_ONEDNN_USE_SYSTEM_ALLOCATOR: '1'
+    OMP_NUM_THREADS: '4'
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+    IN_GRAPH: /workspace/tf-distilbert-inference/pretrained_model/distilbert_frozen_graph_fp32_final.pb
+    WARMUP_STEPS: '20'
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /localdisk/datasets/sst-2/sst2_validation_dataset/
+    dst: /localdisk/datasets/sst-2/sst2_validation_dataset/
+  - src: $PWD/output/tf-distilbert-inference/weightsharing_bf16
+    dst: /output/tf-distilbert-inference/weightsharing_bf16
+DistilBERT Throughput bf16:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-language-modeling-tf-distilbert-inference
+  cmd: /bin/bash quickstart/inference_throughput.sh
+  env:
+    DATASET_DIR: /localdisk/datasets/sst-2/sst2_validation_dataset/
+    OUTPUT_DIR: /output/tf-distilbert-inference/throughput_bf16
+    BATCH_SIZE: '198'
+    PRECISION: bfloat16
+    OMP_NUM_THREADS: '64'
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+    IN_GRAPH: /workspace/tf-distilbert-inference/pretrained_model/distilbert_frozen_graph_fp32_final.pb
+    WARMUP_STEPS: '20'
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /localdisk/datasets/sst-2/sst2_validation_dataset/
+    dst: /localdisk/datasets/sst-2/sst2_validation_dataset/
+  - src: $PWD/output/tf-distilbert-inference/throughput_bf16
+    dst: /output/tf-distilbert-inference/throughput_bf16
+DistilBERT Accuracy int8:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-language-modeling-tf-distilbert-inference
+  cmd: /bin/bash quickstart/accuracy.sh -- DEBIAN_FRONTEND=noninteractive
+  env:
+    DATASET_DIR: /localdisk/datasets/sst-2/sst2_validation_dataset/
+    OUTPUT_DIR: /output/tf-distilbert-inference/accuracy_int8
+    IN_GRAPH: /workspace/tf-distilbert-inference/pretrained_model/distilbert_itex_int8.pb
+    BATCH_SIZE: '56'
+    PRECISION: int8
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /localdisk/datasets/sst-2/sst2_validation_dataset/
+    dst: /localdisk/datasets/sst-2/sst2_validation_dataset/
+  - src: $PWD/output/tf-distilbert-inference/accuracy_int8
+    dst: /output/tf-distilbert-inference/accuracy_int8
+DistilBERT Realtime Inference int8:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-language-modeling-tf-distilbert-inference
+  cmd: /bin/bash quickstart/inference_realtime.sh
+  env:
+    DATASET_DIR: /localdisk/datasets/sst-2/sst2_validation_dataset/
+    OUTPUT_DIR: /output/tf-distilbert-inference/realtime_int8
+    BATCH_SIZE: '1'
+    PRECISION: int8
+    OMP_NUM_THREADS: '4'
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+    IN_GRAPH: /workspace/tf-distilbert-inference/pretrained_model/distilbert_itex_int8.pb
+    WARMUP_STEPS: '20'
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /localdisk/datasets/sst-2/sst2_validation_dataset/
+    dst: /localdisk/datasets/sst-2/sst2_validation_dataset/
+  - src: $PWD/output/tf-distilbert-inference/realtime_int8
+    dst: /output/tf-distilbert-inference/realtime_int8
+DistilBERT Weightsharing Inference int8:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-language-modeling-tf-distilbert-inference
+  cmd: /bin/bash quickstart/inference_realtime_weight_sharing.sh
+  env:
+    DATASET_DIR: /localdisk/datasets/sst-2/sst2_validation_dataset/
+    OUTPUT_DIR: /output/tf-distilbert-inference/weightsharing_int8
+    BATCH_SIZE: '1'
+    PRECISION: int8
+    TF_ONEDNN_USE_SYSTEM_ALLOCATOR: '1'
+    OMP_NUM_THREADS: '4'
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+    IN_GRAPH: /workspace/tf-distilbert-inference/pretrained_model/distilbert_itex_int8.pb
+    WARMUP_STEPS: '20'
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /localdisk/datasets/sst-2/sst2_validation_dataset/
+    dst: /localdisk/datasets/sst-2/sst2_validation_dataset/
+  - src: $PWD/output/tf-distilbert-inference/weightsharing_int8
+    dst: /output/tf-distilbert-inference/weightsharing_int8
+DistilBERT Throughput int8:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-language-modeling-tf-distilbert-inference
+  cmd: /bin/bash quickstart/inference_throughput.sh
+  env:
+    DATASET_DIR: /localdisk/datasets/sst-2/sst2_validation_dataset/
+    OUTPUT_DIR: /output/tf-distilbert-inference/throughput_int8
+    BATCH_SIZE: '110'
+    PRECISION: int8
+    OMP_NUM_THREADS: '64'
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+    IN_GRAPH: /workspace/tf-distilbert-inference/pretrained_model/distilbert_itex_int8.pb
+    WARMUP_STEPS: '20'
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /localdisk/datasets/sst-2/sst2_validation_dataset/
+    dst: /localdisk/datasets/sst-2/sst2_validation_dataset/
+  - src: $PWD/output/tf-distilbert-inference/throughput_int8
+    dst: /output/tf-distilbert-inference/throughput_int8
diff --git a/docker/tf-cpu/tf-distilbert-inference/tf-distilbert-inference.Dockerfile-centos b/docker/tf-cpu/tf-distilbert-inference/tf-distilbert-inference.Dockerfile-centos
new file mode 100644
index 000000000..4aaad21e3
--- /dev/null
+++ b/docker/tf-cpu/tf-distilbert-inference/tf-distilbert-inference.Dockerfile-centos
@@ -0,0 +1,57 @@
+# Copyright (c) 2020-2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+#
+# THIS IS A GENERATED DOCKERFILE.
+#
+# This file was assembled from multiple pieces, whose use is documented
+# throughout. Please refer to the TensorFlow dockerfiles documentation
+# for more information.
+
+ARG TF_BASE_IMAGE=intel/intel-optimized-tensorflow-avx512
+
+ARG TF_BASE_TAG=latest
+
+FROM ${TF_BASE_IMAGE}:${TF_BASE_TAG}
+
+WORKDIR /workspace/tf-distilbert-inference
+
+RUN yum update -y && \
+    yum install -y numactl unzip wget && \
+    yum clean all
+
+RUN python -m pip install \
+    datasets>=2.6.1 \
+    evaluate==0.3.0 \
+    scikit-learn>=1.1.2 \
+    scipy>=1.9.3 \
+    sklearn==0.0 \
+    tokenizers==0.13.1 \
+    transformers==4.30.0
+
+COPY benchmarks benchmarks
+COPY models/common models/common
+COPY models/language_modeling/tensorflow/distilbert_base/inference models/language_modeling/tensorflow/distilbert_base/inference
+COPY quickstart/common quickstart/common 
+COPY quickstart/language_modeling/tensorflow/distilbert_base/inference/cpu/inference_realtime_multi_instance.sh quickstart/inference_realtime.sh
+COPY quickstart/language_modeling/tensorflow/distilbert_base/inference/cpu/inference_realtime_weight_sharing.sh quickstart/inference_realtime_weight_sharing.sh
+COPY quickstart/language_modeling/tensorflow/distilbert_base/inference/cpu/inference_throughput_multi_instance.sh quickstart/inference_throughput.sh
+COPY quickstart/language_modeling/tensorflow/distilbert_base/inference/cpu/accuracy.sh quickstart/accuracy.sh
+
+COPY LICENSE license/LICENSE
+COPY third_party license/third_party
+
+RUN mkdir -p /workspace/tf-distilbert-inference/pretrained_model && \
+    wget -qO /workspace/tf-distilbert-inference/pretrained_model/distilbert_frozen_graph_fp32_final.pb https://storage.googleapis.com/intel-optimized-tensorflow/models/2_10_0/distilbert_frozen_graph_fp32_final.pb && \
+    wget -qO /workspace/tf-distilbert-inference/pretrained_model/distilbert_itex_int8.pb https://storage.googleapis.com/intel-optimized-tensorflow/models/2_12_0/distilbert_itex_int8.pb
diff --git a/docker/tf-cpu/tf-distilbert-inference/tf-distilbert-inference.Dockerfile-ubuntu b/docker/tf-cpu/tf-distilbert-inference/tf-distilbert-inference.Dockerfile-ubuntu
new file mode 100644
index 000000000..0cf1c5b8c
--- /dev/null
+++ b/docker/tf-cpu/tf-distilbert-inference/tf-distilbert-inference.Dockerfile-ubuntu
@@ -0,0 +1,56 @@
+# Copyright (c) 2020-2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+#
+# THIS IS A GENERATED DOCKERFILE.
+#
+# This file was assembled from multiple pieces, whose use is documented
+# throughout. Please refer to the TensorFlow dockerfiles documentation
+# for more information.
+
+ARG TF_BASE_IMAGE=intel/intel-optimized-tensorflow-avx512
+
+ARG TF_BASE_TAG=latest
+
+FROM ${TF_BASE_IMAGE}:${TF_BASE_TAG}
+
+WORKDIR /workspace/tf-distilbert-inference
+
+RUN apt-get update && \
+    apt-get install -y numactl unzip wget
+
+RUN python -m pip install \
+    datasets>=2.6.1 \
+    evaluate==0.3.0 \
+    scikit-learn>=1.1.2 \
+    scipy>=1.9.3 \
+    sklearn==0.0 \
+    tokenizers==0.13.1 \
+    transformers==4.30.0
+
+COPY benchmarks benchmarks
+COPY models/common models/common
+COPY models/language_modeling/tensorflow/distilbert_base/inference models/language_modeling/tensorflow/distilbert_base/inference
+COPY quickstart/common quickstart/common 
+COPY quickstart/language_modeling/tensorflow/distilbert_base/inference/cpu/inference_realtime_multi_instance.sh quickstart/inference_realtime.sh
+COPY quickstart/language_modeling/tensorflow/distilbert_base/inference/cpu/inference_realtime_weight_sharing.sh quickstart/inference_realtime_weight_sharing.sh
+COPY quickstart/language_modeling/tensorflow/distilbert_base/inference/cpu/inference_throughput_multi_instance.sh quickstart/inference_throughput.sh
+COPY quickstart/language_modeling/tensorflow/distilbert_base/inference/cpu/accuracy.sh quickstart/accuracy.sh
+
+COPY LICENSE license/LICENSE
+COPY third_party license/third_party
+
+RUN mkdir -p /workspace/tf-distilbert-inference/pretrained_model && \
+    wget -qO /workspace/tf-distilbert-inference/pretrained_model/distilbert_frozen_graph_fp32_final.pb https://storage.googleapis.com/intel-optimized-tensorflow/models/2_10_0/distilbert_frozen_graph_fp32_final.pb && \
+    wget -qO /workspace/tf-distilbert-inference/pretrained_model/distilbert_itex_int8.pb https://storage.googleapis.com/intel-optimized-tensorflow/models/2_12_0/distilbert_itex_int8.pb
diff --git a/docker/tf-cpu/tf-mobilenet-v1-inference/tests.yaml b/docker/tf-cpu/tf-mobilenet-v1-inference/tests.yaml
new file mode 100644
index 000000000..968486cf2
--- /dev/null
+++ b/docker/tf-cpu/tf-mobilenet-v1-inference/tests.yaml
@@ -0,0 +1,156 @@
+MobileNet-V1 Accuracy fp32:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-image-recognition-tf-mobilenet-v1-inference
+  cmd: /bin/bash quickstart/accuracy.sh
+  env:
+    DATASET_DIR: /tf_dataset/dataset/TF_Imagenet_FullData
+    OUTPUT_DIR: /output/tf-mobilenet-v1-inference/accuracy_fp32
+    BATCH_SIZE: '100'
+    PRECISION: fp32
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/TF_Imagenet_FullData
+    dst: /tf_dataset/dataset/TF_Imagenet_FullData
+  - src: $PWD/output/tf-mobilenet-v1-inference/accuracy_fp32
+    dst: /output/tf-mobilenet-v1-inference/accuracy_fp32
+MobileNet-V1 Accuracy bf16:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-image-recognition-tf-mobilenet-v1-inference
+  cmd: /bin/bash quickstart/accuracy.sh
+  env:
+    DATASET_DIR: /tf_dataset/dataset/TF_Imagenet_FullData
+    OUTPUT_DIR: /output/tf-mobilenet-v1-inference/accuracy_bf16
+    BATCH_SIZE: '100'
+    PRECISION: bfloat16
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/TF_Imagenet_FullData
+    dst: /tf_dataset/dataset/TF_Imagenet_FullData
+  - src: $PWD/output/tf-mobilenet-v1-inference/accuracy_bf16
+    dst: /output/tf-mobilenet-v1-inference/accuracy_bf16
+MobileNet-V1 Accuracy int8:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-image-recognition-tf-mobilenet-v1-inference
+  cmd: /bin/bash quickstart/accuracy.sh
+  env:
+    DATASET_DIR: /tf_dataset/dataset/TF_Imagenet_FullData
+    OUTPUT_DIR: /output/tf-mobilenet-v1-inference/accuracy_int8
+    BATCH_SIZE: '100'
+    PRECISION: int8
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/TF_Imagenet_FullData
+    dst: /tf_dataset/dataset/TF_Imagenet_FullData
+  - src: $PWD/output/tf-mobilenet-v1-inference/accuracy_int8
+    dst: /output/tf-mobilenet-v1-inference/accuracy_int8
+MobileNet-V1 Realtime Inference fp32:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-image-recognition-tf-mobilenet-v1-inference
+  cmd: /bin/bash quickstart/inference_realtime_multi_instance.sh --benchmark-only --num-intra-threads 4 --num-inter-threads 1 --verbose
+  env:
+    DATASET_DIR: /tf_dataset/dataset/TF_Imagenet_FullData
+    OUTPUT_DIR: /output/tf-mobilenet-v1-inference/realtime_fp32
+    BATCH_SIZE: '1'
+    PRECISION: fp32
+    OMP_NUM_THREADS: '4'
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/TF_Imagenet_FullData
+    dst: /tf_dataset/dataset/TF_Imagenet_FullData
+  - src: $PWD/output/tf-mobilenet-v1-inference/realtime_fp32
+    dst: /output/tf-mobilenet-v1-inference/realtime_fp32
+MobileNet-V1 Realtime Inference bf16:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-image-recognition-tf-mobilenet-v1-inference
+  cmd: /bin/bash quickstart/inference_realtime_multi_instance.sh --benchmark-only --num-intra-threads 4 --num-inter-threads 1 --verbose
+  env:
+    DATASET_DIR: /tf_dataset/dataset/TF_Imagenet_FullData
+    OUTPUT_DIR: /output/tf-mobilenet-v1-inference/realtime_bf16
+    BATCH_SIZE: '1'
+    PRECISION: bfloat16
+    OMP_NUM_THREADS: '4'
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/TF_Imagenet_FullData
+    dst: /tf_dataset/dataset/TF_Imagenet_FullData
+  - src: $PWD/output/tf-mobilenet-v1-inference/realtime_bf16
+    dst: /output/tf-mobilenet-v1-inference/realtime_bf16
+MobileNet-V1 Realtime Inference int8:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-image-recognition-tf-mobilenet-v1-inference
+  cmd: /bin/bash quickstart/inference_realtime_multi_instance.sh --benchmark-only --num-intra-threads 4 --num-inter-threads 1 --verbose
+  env:
+    DATASET_DIR: /tf_dataset/dataset/TF_Imagenet_FullData
+    OUTPUT_DIR: /output/tf-mobilenet-v1-inference/realtime_int8
+    BATCH_SIZE: '1'
+    PRECISION: int8
+    OMP_NUM_THREADS: '4'
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/TF_Imagenet_FullData
+    dst: /tf_dataset/dataset/TF_Imagenet_FullData
+  - src: $PWD/output/tf-mobilenet-v1-inference/realtime_int8
+    dst: /output/tf-mobilenet-v1-inference/realtime_int8
+MobileNet-V1 Throughput fp32:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-image-recognition-tf-mobilenet-v1-inference
+  cmd: /bin/bash quickstart/inference_throughput_multi_instance.sh --benchmark-only --num-intra-threads 64 --num-inter-threads 1 --verbose
+  env:
+    DATASET_DIR: /tf_dataset/dataset/TF_Imagenet_FullData
+    OUTPUT_DIR: /output/tf-mobilenet-v1-inference/throughput_fp32
+    BATCH_SIZE: '448'
+    PRECISION: fp32
+    OMP_NUM_THREADS: '64'
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/TF_Imagenet_FullData
+    dst: /tf_dataset/dataset/TF_Imagenet_FullData
+  - src: $PWD/output/tf-mobilenet-v1-inference/throughput_fp32
+    dst: /output/tf-mobilenet-v1-inference/throughput_fp32
+MobileNet-V1 Throughput bf16:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-image-recognition-tf-mobilenet-v1-inference
+  cmd: /bin/bash quickstart/inference_throughput_multi_instance.sh --benchmark-only --num-intra-threads 64 --num-inter-threads 1 --verbose
+  env:
+    DATASET_DIR: /tf_dataset/dataset/TF_Imagenet_FullData
+    OUTPUT_DIR: /output/tf-mobilenet-v1-inference/throughput_bf16
+    BATCH_SIZE: '448'
+    PRECISION: bfloat16
+    OMP_NUM_THREADS: '64'
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/TF_Imagenet_FullData
+    dst: /tf_dataset/dataset/TF_Imagenet_FullData
+  - src: $PWD/output/tf-mobilenet-v1-inference/throughput_bf16
+    dst: /output/tf-mobilenet-v1-inference/throughput_bf16
+MobileNet-V1 Throughput int8:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-image-recognition-tf-mobilenet-v1-inference
+  cmd: /bin/bash quickstart/inference_throughput_multi_instance.sh --benchmark-only --num-intra-threads 64 --num-inter-threads 1 --verbose
+  env:
+    DATASET_DIR: /tf_dataset/dataset/TF_Imagenet_FullData
+    OUTPUT_DIR: /output/tf-mobilenet-v1-inference/throughput_int8
+    BATCH_SIZE: '448'
+    PRECISION: int8
+    OMP_NUM_THREADS: '64'
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/TF_Imagenet_FullData
+    dst: /tf_dataset/dataset/TF_Imagenet_FullData
+  - src: $PWD/output/tf-mobilenet-v1-inference/throughput_int8
+    dst: /output/tf-mobilenet-v1-inference/throughput_int8
diff --git a/docker/tf-cpu/tf-mobilenet-v1-inference/tf-mobilenet-v1-inference.Dockerfile-centos b/docker/tf-cpu/tf-mobilenet-v1-inference/tf-mobilenet-v1-inference.Dockerfile-centos
new file mode 100644
index 000000000..3a50d6f1e
--- /dev/null
+++ b/docker/tf-cpu/tf-mobilenet-v1-inference/tf-mobilenet-v1-inference.Dockerfile-centos
@@ -0,0 +1,45 @@
+# Copyright (c) 2020-2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+#
+# THIS IS A GENERATED DOCKERFILE.
+#
+# This file was assembled from multiple pieces, whose use is documented
+# throughout. Please refer to the TensorFlow dockerfiles documentation
+# for more information.
+
+ARG TF_BASE_IMAGE=intel/intel-optimized-tensorflow-avx512
+
+ARG TF_BASE_TAG=latest
+
+FROM ${TF_BASE_IMAGE}:${TF_BASE_TAG}
+
+WORKDIR /workspace/tf-mobilenet-v1-inference
+
+RUN yum update -y && \
+    yum install -y ca-certificates numactl wget && \
+    yum clean all
+
+COPY benchmarks benchmarks
+COPY models/common models/common
+COPY models/image_recognition/tensorflow/mobilenet_v1/inference models/image_recognition/tensorflow/mobilenet_v1/inference
+COPY quickstart/common quickstart/common 
+COPY quickstart/image_recognition/tensorflow/mobilenet_v1/inference/cpu quickstart
+
+COPY LICENSE license/LICENSE
+COPY third_party license/third_party
+
+RUN mkdir -p /workspace/tf-mobilenet-v1-inference/pretrained_model && \
+    wget -qO /workspace/tf-mobilenet-v1-inference/pretrained_model/mobilenetv1_fp32_pretrained_model_new.pb https://storage.googleapis.com/intel-optimized-tensorflow/models/2_10_0/mobilenetv1_fp32_pretrained_model.pb && \
+    wget -qO /workspace/tf-mobilenet-v1-inference/pretrained_model/mobilenetv1_int8_pretrained_model_new.pb https://storage.googleapis.com/intel-optimized-tensorflow/models/2_10_0/mobilenetv1_int8_pretrained_model.pb
diff --git a/docker/tf-cpu/tf-mobilenet-v1-inference/tf-mobilenet-v1-inference.Dockerfile-ubuntu b/docker/tf-cpu/tf-mobilenet-v1-inference/tf-mobilenet-v1-inference.Dockerfile-ubuntu
new file mode 100644
index 000000000..db06c3465
--- /dev/null
+++ b/docker/tf-cpu/tf-mobilenet-v1-inference/tf-mobilenet-v1-inference.Dockerfile-ubuntu
@@ -0,0 +1,44 @@
+# Copyright (c) 2020-2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+#
+# THIS IS A GENERATED DOCKERFILE.
+#
+# This file was assembled from multiple pieces, whose use is documented
+# throughout. Please refer to the TensorFlow dockerfiles documentation
+# for more information.
+
+ARG TF_BASE_IMAGE=intel/intel-optimized-tensorflow-avx512
+
+ARG TF_BASE_TAG=latest
+
+FROM ${TF_BASE_IMAGE}:${TF_BASE_TAG}
+
+WORKDIR /workspace/tf-mobilenet-v1-inference
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y ca-certificates numactl wget 
+
+COPY benchmarks benchmarks
+COPY models/common models/common
+COPY models/image_recognition/tensorflow/mobilenet_v1/inference models/image_recognition/tensorflow/mobilenet_v1/inference
+COPY quickstart/common quickstart/common 
+COPY quickstart/image_recognition/tensorflow/mobilenet_v1/inference/cpu quickstart
+
+COPY LICENSE license/LICENSE
+COPY third_party license/third_party
+
+RUN mkdir -p /workspace/tf-mobilenet-v1-inference/pretrained_model && \
+    wget -qO /workspace/tf-mobilenet-v1-inference/pretrained_model/mobilenetv1_fp32_pretrained_model_new.pb https://storage.googleapis.com/intel-optimized-tensorflow/models/2_10_0/mobilenetv1_fp32_pretrained_model.pb && \
+    wget -qO /workspace/tf-mobilenet-v1-inference/pretrained_model/mobilenetv1_int8_pretrained_model_new.pb https://storage.googleapis.com/intel-optimized-tensorflow/models/2_10_0/mobilenetv1_int8_pretrained_model.pb
diff --git a/docker/tf-cpu/tf-resnet50v1-5-inference/tests.yaml b/docker/tf-cpu/tf-resnet50v1-5-inference/tests.yaml
index 7c2ac92c6..09ad0f976 100644
--- a/docker/tf-cpu/tf-resnet50v1-5-inference/tests.yaml
+++ b/docker/tf-cpu/tf-resnet50v1-5-inference/tests.yaml
@@ -1,9 +1,9 @@
-ResNet50 v1.5 BFloat32 accuracy:
-  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-spr-resnet50v1-5-inference
+ResNet50 v1.5 Accuracy bf32:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-image-recognition-tf-resnet50v1-5-inference
   cmd: /bin/bash quickstart/accuracy.sh
   env:
     DATASET_DIR: /tf_dataset/dataset/TF_Imagenet_FullData
-    OUTPUT_DIR: /output/tf-spr-resnet50v1-5-inference/bf32_accuracy
+    OUTPUT_DIR: /output/tf-resnet50v1-5-inference/accuracy_bf32
     BATCH_SIZE: '100'
     PRECISION: bfloat32
   shm_size: 8G
@@ -12,14 +12,14 @@ ResNet50 v1.5 BFloat32 accuracy:
   volumes:
   - src: /tf_dataset/dataset/TF_Imagenet_FullData
     dst: /tf_dataset/dataset/TF_Imagenet_FullData
-  - src: /output/tf-spr-resnet50v1-5-inference/bf32_accuracy
-    dst: /output/tf-spr-resnet50v1-5-inference/bf32_accuracy
-ResNet50 v1.5 BFloat16 accuracy:
-  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-spr-resnet50v1-5-inference
+  - src: $PWD/output/tf-resnet50v1-5-inference/accuracy_bf32
+    dst: /output/tf-resnet50v1-5-inference/accuracy_bf32
+ResNet50 v1.5 Accuracy bf16:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-image-recognition-tf-resnet50v1-5-inference
   cmd: /bin/bash quickstart/accuracy.sh
   env:
     DATASET_DIR: /tf_dataset/dataset/TF_Imagenet_FullData
-    OUTPUT_DIR: /output/tf-spr-resnet50v1-5-inference/bf16_accuracy
+    OUTPUT_DIR: /output/tf-resnet50v1-5-inference/accuracy_bf16
     BATCH_SIZE: '100'
     PRECISION: bfloat16
   shm_size: 8G
@@ -28,14 +28,14 @@ ResNet50 v1.5 BFloat16 accuracy:
   volumes:
   - src: /tf_dataset/dataset/TF_Imagenet_FullData
     dst: /tf_dataset/dataset/TF_Imagenet_FullData
-  - src: /output/tf-spr-resnet50v1-5-inference/bf16_accuracy
-    dst: /output/tf-spr-resnet50v1-5-inference/bf16_accuracy
-ResNet50 v1.5 fp32 accuracy:
-  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-spr-resnet50v1-5-inference
+  - src: $PWD/output/tf-resnet50v1-5-inference/accuracy_bf16
+    dst: /output/tf-resnet50v1-5-inference/accuracy_bf16
+ResNet50 v1.5 Accuracy fp32:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-image-recognition-tf-resnet50v1-5-inference
   cmd: /bin/bash quickstart/accuracy.sh
   env:
     DATASET_DIR: /tf_dataset/dataset/TF_Imagenet_FullData
-    OUTPUT_DIR: /output/tf-spr-resnet50v1-5-inference/fp32_accuracy
+    OUTPUT_DIR: /output/tf-resnet50v1-5-inference/accuracy_fp32
     BATCH_SIZE: '100'
     PRECISION: fp32
   shm_size: 8G
@@ -44,14 +44,14 @@ ResNet50 v1.5 fp32 accuracy:
   volumes:
   - src: /tf_dataset/dataset/TF_Imagenet_FullData
     dst: /tf_dataset/dataset/TF_Imagenet_FullData
-  - src: /output/tf-spr-resnet50v1-5-inference/fp32_accuracy
-    dst: /output/tf-spr-resnet50v1-5-inference/fp32_accuracy
-ResNet50 v1.5 int8 accuracy:
-  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-spr-resnet50v1-5-inference
+  - src: $PWD/output/tf-resnet50v1-5-inference/accuracy_fp32
+    dst: /output/tf-resnet50v1-5-inference/accuracy_fp32
+ResNet50 v1.5 Accuracy int8:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-image-recognition-tf-resnet50v1-5-inference
   cmd: /bin/bash quickstart/accuracy.sh
   env:
     DATASET_DIR: /tf_dataset/dataset/TF_Imagenet_FullData
-    OUTPUT_DIR: /output/tf-spr-resnet50v1-5-inference/int8_accuracy
+    OUTPUT_DIR: /output/tf-resnet50v1-5-inference/accuracy_int8
     BATCH_SIZE: '100'
     PRECISION: int8
   shm_size: 8G
@@ -60,165 +60,187 @@ ResNet50 v1.5 int8 accuracy:
   volumes:
   - src: /tf_dataset/dataset/TF_Imagenet_FullData
     dst: /tf_dataset/dataset/TF_Imagenet_FullData
-  - src: /output/tf-spr-resnet50v1-5-inference/int8_accuracy
-    dst: /output/tf-spr-resnet50v1-5-inference/int8_accuracy
-ResNet50 v1.5 bfloat32 realtime inference:
-  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-spr-resnet50v1-5-inference
+  - src: $PWD/output/tf-resnet50v1-5-inference/accuracy_int8
+    dst: /output/tf-resnet50v1-5-inference/accuracy_int8
+ResNet50 v1.5 Realtime Inference bf32:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-image-recognition-tf-resnet50v1-5-inference
   cmd: /bin/bash quickstart/inference_realtime.sh
   env:
     DATASET_DIR: /tf_dataset/dataset/TF_Imagenet_FullData
-    OUTPUT_DIR: /output/tf-spr-resnet50v1-5-inference/bf32_realtime
+    OUTPUT_DIR: /output/tf-resnet50v1-5-inference/realtime_bf32
     BATCH_SIZE: '1'
     PRECISION: bfloat32
+    CORES_PER_INSTANCE: '4'
+    OMP_NUM_THREADS: '4'
   shm_size: 8G
   privileged: true
   init: true
   volumes:
   - src: /tf_dataset/dataset/TF_Imagenet_FullData
     dst: /tf_dataset/dataset/TF_Imagenet_FullData
-  - src: /output/tf-spr-resnet50v1-5-inference/bf32_realtime
-    dst: /output/tf-spr-resnet50v1-5-inference/bf32_realtime
-ResNet50 v1.5 bfloat16 realtime inference:
-  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-spr-resnet50v1-5-inference
-  cmd: /bin/bash quickstart/inference_realtime.sh
+  - src: $PWD/output/tf-resnet50v1-5-inference/realtime_bf32
+    dst: /output/tf-resnet50v1-5-inference/realtime_bf32
+ResNet50 v1.5 Realtime Inference bf16:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-image-recognition-tf-resnet50v1-5-inference
+  cmd: /bin/bash quickstart/inference_realtime.sh --benchmark-only --num-intra-threads 4 --num-inter-threads 1
   env:
     DATASET_DIR: /tf_dataset/dataset/TF_Imagenet_FullData
-    OUTPUT_DIR: /output/tf-spr-resnet50v1-5-inference/bf16_realtime
+    OUTPUT_DIR: /output/tf-resnet50v1-5-inference/realtime_bf16
     BATCH_SIZE: '1'
     PRECISION: bfloat16
+    CORES_PER_INSTANCE: '64'
+    OMP_NUM_THREADS: '64'
   shm_size: 8G
   privileged: true
   init: true
   volumes:
   - src: /tf_dataset/dataset/TF_Imagenet_FullData
     dst: /tf_dataset/dataset/TF_Imagenet_FullData
-  - src: /output/tf-spr-resnet50v1-5-inference/bf16_realtime
-    dst: /output/tf-spr-resnet50v1-5-inference/bf16_realtime
-ResNet50 v1.5 fp32 realtime inference:
-  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-spr-resnet50v1-5-inference
-  cmd: /bin/bash quickstart/inference_realtime.sh
+  - src: $PWD/output/tf-resnet50v1-5-inference/realtime_bf16
+    dst: /output/tf-resnet50v1-5-inference/realtime_bf16
+ResNet50 v1.5 Realtime Inference fp32:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-image-recognition-tf-resnet50v1-5-inference
+  cmd: /bin/bash quickstart/inference_realtime.sh --benchmark-only --num-intra-threads 4 --num-inter-threads 1
   env:
     DATASET_DIR: /tf_dataset/dataset/TF_Imagenet_FullData
-    OUTPUT_DIR: /output/tf-spr-resnet50v1-5-inference/fp32_realtime
+    OUTPUT_DIR: /output/tf-resnet50v1-5-inference/realtime_fp32
     BATCH_SIZE: '1'
     PRECISION: fp32
+    OMP_NUM_THREADS: '4'
+    CORES_PER_INSTANCE: '4'
   shm_size: 8G
   privileged: true
   init: true
   volumes:
   - src: /tf_dataset/dataset/TF_Imagenet_FullData
     dst: /tf_dataset/dataset/TF_Imagenet_FullData
-  - src: /output/tf-spr-resnet50v1-5-inference/fp32_realtime
-    dst: /output/tf-spr-resnet50v1-5-inference/fp32_realtime
-ResNet50 v1.5 int8 realtime inference:
-  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-spr-resnet50v1-5-inference
-  cmd: /bin/bash quickstart/inference_realtime.sh
+  - src: $PWD/output/tf-resnet50v1-5-inference/realtime_fp32
+    dst: /output/tf-resnet50v1-5-inference/realtime_fp32
+ResNet50 v1.5 Realtime Inference int8:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-image-recognition-tf-resnet50v1-5-inference
+  cmd: /bin/bash quickstart/inference_realtime.sh --benchmark-only --num-intra-threads 4 --num-inter-threads 1 
   env:
     DATASET_DIR: /tf_dataset/dataset/TF_Imagenet_FullData
-    OUTPUT_DIR: /output/tf-spr-resnet50v1-5-inference/int8_realtime
+    OUTPUT_DIR: /output/tf-resnet50v1-5-inference/realtime_int8
     BATCH_SIZE: '1'
     PRECISION: int8
+    OMP_NUM_THREADS: '4'
+    CORES_PER_INSTANCE: '4'
   shm_size: 8G
   privileged: true
   init: true
   volumes:
   - src: /tf_dataset/dataset/TF_Imagenet_FullData
     dst: /tf_dataset/dataset/TF_Imagenet_FullData
-  - src: /output/tf-spr-resnet50v1-5-inference/int8_realtime
-    dst: /output/tf-spr-resnet50v1-5-inference/int8_realtime
-ResNet50 v1.5 bfloat32 throughput inference:
-  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-spr-resnet50v1-5-inference
+  - src: $PWD/output/tf-resnet50v1-5-inference/realtime_int8
+    dst: /output/tf-resnet50v1-5-inference/realtime_int8
+ResNet50 v1.5 Throughput bf32:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-image-recognition-tf-resnet50v1-5-inference
   cmd: /bin/bash quickstart/inference_throughput.sh
   env:
     DATASET_DIR: /tf_dataset/dataset/TF_Imagenet_FullData
-    OUTPUT_DIR: /output/tf-spr-resnet50v1-5-inference/bf32_throughput
+    OUTPUT_DIR: /output/tf-resnet50v1-5-inference/throughput_bf32
     BATCH_SIZE: '448'
     PRECISION: bfloat32
+    CORES_PER_INSTANCE: '64'
+    OMP_NUM_THREADS: '64'
   shm_size: 8G
   privileged: true
   init: true
   volumes:
   - src: /tf_dataset/dataset/TF_Imagenet_FullData
     dst: /tf_dataset/dataset/TF_Imagenet_FullData
-  - src: /output/tf-spr-resnet50v1-5-inference/bf32_throughput
-    dst: /output/tf-spr-resnet50v1-5-inference/bf32_throughput
-ResNet50 v1.5 bfloat16 throughput inference:
-  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-spr-resnet50v1-5-inference
-  cmd: /bin/bash quickstart/inference_throughput.sh
+  - src: $PWD/output/tf-resnet50v1-5-inference/throughput_bf32
+    dst: /output/tf-resnet50v1-5-inference/throughput_bf32
+ResNet50 v1.5 Throughput bf16:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-image-recognition-tf-resnet50v1-5-inference
+  cmd: /bin/bash quickstart/inference_throughput.sh --benchmark-only --num-intra-threads 64 --num-inter-threads 1
   env:
     DATASET_DIR: /tf_dataset/dataset/TF_Imagenet_FullData
-    OUTPUT_DIR: /output/tf-spr-resnet50v1-5-inference/bf16_throughput
-    BATCH_SIZE: '448'
+    OUTPUT_DIR: /output/tf-resnet50v1-5-inference/throughput_bf16
+    BATCH_SIZE: '80'
     PRECISION: bfloat16
+    CORES_PER_INSTANCE: '64'
+    OMP_NUM_THREADS: '64'
   shm_size: 8G
   privileged: true
   init: true
   volumes:
   - src: /tf_dataset/dataset/TF_Imagenet_FullData
     dst: /tf_dataset/dataset/TF_Imagenet_FullData
-  - src: /output/tf-spr-resnet50v1-5-inference/bf16_throughput
-    dst: /output/tf-spr-resnet50v1-5-inference/bf16_throughput
-ResNet50 v1.5 fp32 throughput inference:
-  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-spr-resnet50v1-5-inference
-  cmd: /bin/bash quickstart/inference_throughput.sh
+  - src: $PWD/output/tf-resnet50v1-5-inference/throughput_bf16
+    dst: /output/tf-resnet50v1-5-inference/throughput_bf16
+ResNet50 v1.5 Throughput fp32:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-image-recognition-tf-resnet50v1-5-inference
+  cmd: /bin/bash quickstart/inference_throughput.sh --benchmark-only --num-intra-threads 64 --num-inter-threads 1
   env:
     DATASET_DIR: /tf_dataset/dataset/TF_Imagenet_FullData
-    OUTPUT_DIR: /output/tf-spr-resnet50v1-5-inference/fp32_throughput
-    BATCH_SIZE: '448'
+    OUTPUT_DIR: /output/tf-resnet50v1-5-inference/throughput_fp32
+    BATCH_SIZE: '116'
     PRECISION: fp32
+    CORES_PER_INSTANCE: '64'
+    OMP_NUM_THREADS: '64'
   shm_size: 8G
   privileged: true
   init: true
   volumes:
   - src: /tf_dataset/dataset/TF_Imagenet_FullData
     dst: /tf_dataset/dataset/TF_Imagenet_FullData
-  - src: /output/tf-spr-resnet50v1-5-inference/fp32_throughput
-    dst: /output/tf-spr-resnet50v1-5-inference/fp32_throughput
-ResNet50 v1.5 int8 throughput inference:
-  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-spr-resnet50v1-5-inference
-  cmd: /bin/bash quickstart/inference_throughput.sh
+  - src: $PWD/output/tf-resnet50v1-5-inference/throughput_fp32
+    dst: /output/tf-resnet50v1-5-inference/throughput_fp32
+ResNet50 v1.5 Throughput int8:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-image-recognition-tf-resnet50v1-5-inference
+  cmd: /bin/bash quickstart/inference_throughput.sh --benchmark-only --num-intra-threads 64 --num-inter-threads 1
   env:
     DATASET_DIR: /tf_dataset/dataset/TF_Imagenet_FullData
-    OUTPUT_DIR: /output/tf-spr-resnet50v1-5-inference/int8_throughput
-    BATCH_SIZE: '448'
+    OUTPUT_DIR: /output/tf-resnet50v1-5-inference/throughput_int8
+    BATCH_SIZE: '116'
     PRECISION: int8
+    OMP_NUM_THREADS: '64'
+    CORES_PER_INSTANCE: '64'
   shm_size: 8G
   privileged: true
   init: true
   volumes:
   - src: /tf_dataset/dataset/TF_Imagenet_FullData
     dst: /tf_dataset/dataset/TF_Imagenet_FullData
-  - src: /output/tf-spr-resnet50v1-5-inference/int8_throughput
-    dst: /output/tf-spr-resnet50v1-5-inference/int8_throughput
-ResNet50 v1.5 bfloat16 weightsharing inference:
-  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-spr-resnet50v1-5-inference
-  cmd: /bin/bash quickstart/inference_realtime_weightsharing.sh
+  - src: $PWD/output/tf-resnet50v1-5-inference/throughput_int8
+    dst: /output/tf-resnet50v1-5-inference/throughput_int8
+ResNet50 v1.5 Weightsharing Inference bf16:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-image-recognition-tf-resnet50v1-5-inference
+  cmd: /bin/bash quickstart/inference_realtime_weightsharing.sh --benchmark-only
   env:
     DATASET_DIR: /tf_dataset/dataset/TF_Imagenet_FullData
-    OUTPUT_DIR: /output/tf-spr-resnet50v1-5-inference/bf16_weightsharing
+    OUTPUT_DIR: /output/tf-resnet50v1-5-inference/weightsharing_bf16
     BATCH_SIZE: '1'
     PRECISION: bfloat16
+    OMP_NUM_THREADS: '4'
+    CORES_PER_INSTANCE: '4'
+    KMP_BLOCKTIME: '1'
   shm_size: 8G
   privileged: true
   init: true
   volumes:
   - src: /tf_dataset/dataset/TF_Imagenet_FullData
     dst: /tf_dataset/dataset/TF_Imagenet_FullData
-  - src: /output/tf-spr-resnet50v1-5-inference/bf16_weightsharing
-    dst: /output/tf-spr-resnet50v1-5-inference/bf16_weightsharing
-ResNet50 v1.5 int8 weightsharing inference:
-  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-image-recognition-tf-spr-resnet50v1-5-inference
-  cmd: /bin/bash quickstart/inference_realtime_weightsharing.sh
+  - src: $PWD/output/tf-resnet50v1-5-inference/weightsharing_bf16
+    dst: /output/tf-resnet50v1-5-inference/weightsharing_bf16
+ResNet50 v1.5 Weightsharing Inference int8:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-image-recognition-tf-resnet50v1-5-inference
+  cmd: /bin/bash quickstart/inference_realtime_weightsharing.sh --benchmark-only
   env:
     DATASET_DIR: /tf_dataset/dataset/TF_Imagenet_FullData
-    OUTPUT_DIR: /output/tf-spr-resnet50v1-5-inference/int8_weightsharing
+    OUTPUT_DIR: /output/tf-resnet50v1-5-inference/weightsharing_int8
     BATCH_SIZE: '1'
     PRECISION: int8
+    OMP_NUM_THREADS: '4'
+    CORES_PER_INSTANCE: '4'
+    KMP_BLOCKTIME: '1'
   shm_size: 8G
   privileged: true
   init: true
   volumes:
   - src: /tf_dataset/dataset/TF_Imagenet_FullData
     dst: /tf_dataset/dataset/TF_Imagenet_FullData
-  - src: /output/tf-spr-resnet50v1-5-inference/int8_weightsharing
-    dst: /output/tf-spr-resnet50v1-5-inference/int8_weightsharing
+  - src: $PWD/output/tf-resnet50v1-5-inference/weightsharing_int8
+    dst: /output/tf-resnet50v1-5-inference/weightsharing_int8
diff --git a/docker/tf-cpu/tf-resnet50v1-5-inference/tf-resnet50v1-5-inference.Dockerfile-centos b/docker/tf-cpu/tf-resnet50v1-5-inference/tf-resnet50v1-5-inference.Dockerfile-centos
new file mode 100644
index 000000000..675aa68f2
--- /dev/null
+++ b/docker/tf-cpu/tf-resnet50v1-5-inference/tf-resnet50v1-5-inference.Dockerfile-centos
@@ -0,0 +1,47 @@
+# Copyright (c) 2020-2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+#
+# THIS IS A GENERATED DOCKERFILE.
+#
+# This file was assembled from multiple pieces, whose use is documented
+# throughout. Please refer to the TensorFlow dockerfiles documentation
+# for more information.
+
+ARG TF_BASE_IMAGE=intel/intel-optimized-tensorflow-avx512
+
+ARG TF_BASE_TAG=latest
+
+FROM ${TF_BASE_IMAGE}:${TF_BASE_TAG}
+
+WORKDIR /workspace/tf-resnet50v1-5-inference 
+
+RUN yum update -y && \
+    yum install -y --skip-broken ca-certificates numactl wget 
+
+RUN mkdir -p pretrained_model && \
+    wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/resnet50v1_5_int8_pretrained_model.pb -O pretrained_model/bias_resnet50.pb && \
+    wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/resnet50_v1_5_bfloat16.pb -O pretrained_model/bf16_resnet50_v1.pb && \
+    wget https://zenodo.org/record/2535873/files/resnet50_v1.pb -O pretrained_model/resnet50_v1.pb
+
+COPY benchmarks benchmarks
+COPY models models
+COPY quickstart/common quickstart/common
+COPY quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/accuracy.sh quickstart/accuracy.sh
+COPY quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/inference_throughput_multi_instance.sh quickstart/inference_throughput.sh
+COPY quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/inference_realtime_multi_instance.sh quickstart/inference_realtime.sh
+COPY quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/inference_realtime_weightsharing.sh quickstart/inference_realtime_weightsharing.sh
+
+COPY LICENSE license/LICENSE
+COPY third_party license/third_party
diff --git a/docker/tf-cpu/tf-resnet50v1-5-inference/tf-resnet50v1-5-inference.Dockerfile-ubuntu b/docker/tf-cpu/tf-resnet50v1-5-inference/tf-resnet50v1-5-inference.Dockerfile-ubuntu
new file mode 100644
index 000000000..84474b54e
--- /dev/null
+++ b/docker/tf-cpu/tf-resnet50v1-5-inference/tf-resnet50v1-5-inference.Dockerfile-ubuntu
@@ -0,0 +1,47 @@
+# Copyright (c) 2020-2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+#
+# THIS IS A GENERATED DOCKERFILE.
+#
+# This file was assembled from multiple pieces, whose use is documented
+# throughout. Please refer to the TensorFlow dockerfiles documentation
+# for more information.
+
+ARG TF_BASE_IMAGE=intel/intel-optimized-tensorflow-avx512
+
+ARG TF_BASE_TAG=latest
+
+FROM ${TF_BASE_IMAGE}:${TF_BASE_TAG}
+
+WORKDIR /workspace/tf-resnet50v1-5-inference 
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y ca-certificates numactl wget 
+
+RUN mkdir -p pretrained_model && \
+    wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/resnet50v1_5_int8_pretrained_model.pb -O pretrained_model/bias_resnet50.pb && \
+    wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/resnet50_v1_5_bfloat16.pb -O pretrained_model/bf16_resnet50_v1.pb && \
+    wget https://zenodo.org/record/2535873/files/resnet50_v1.pb -O pretrained_model/resnet50_v1.pb
+
+COPY benchmarks benchmarks
+COPY models models
+COPY quickstart/common quickstart/common
+COPY quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/accuracy.sh quickstart/accuracy.sh
+COPY quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/inference_throughput_multi_instance.sh quickstart/inference_throughput.sh
+COPY quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/inference_realtime_multi_instance.sh quickstart/inference_realtime.sh
+COPY quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/inference_realtime_weightsharing.sh quickstart/inference_realtime_weightsharing.sh
+
+COPY LICENSE license/LICENSE
+COPY third_party license/third_party
diff --git a/docker/tf-cpu/tf-resnet50v1-5-training/tests.yaml b/docker/tf-cpu/tf-resnet50v1-5-training/tests.yaml
new file mode 100644
index 000000000..4e2a0240c
--- /dev/null
+++ b/docker/tf-cpu/tf-resnet50v1-5-training/tests.yaml
@@ -0,0 +1,34 @@
+ResNet50 v1.5 Training fp32:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-image-recognition-tf-resnet50v1-5-training
+  cmd: /bin/bash quickstart/training.sh
+  env:
+    DATASET_DIR: /tf_dataset/dataset/TF_Imagenet_FullData
+    OUTPUT_DIR: /output/tf-resnet50v1-5-training/fp32
+    BATCH_SIZE: '512'
+    PRECISION: fp32
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/TF_Imagenet_FullData
+    dst: /tf_dataset/dataset/TF_Imagenet_FullData
+  - src: $PWD/output/tf-resnet50v1-5-training/fp32
+    dst: /output/tf-resnet50v1-5-training/fp32
+ResNet50 v1.5 Training bf16:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-image-recognition-tf-resnet50v1-5-training
+  cmd: /bin/bash quickstart/training.sh
+  env:
+    DATASET_DIR: /tf_dataset/dataset/TF_Imagenet_FullData
+    OUTPUT_DIR: /output/tf-resnet50v1-5-training/bf16
+    BATCH_SIZE: '1024'
+    PRECISION: bfloat16
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/TF_Imagenet_FullData
+    dst: /tf_dataset/dataset/TF_Imagenet_FullData
+  - src: $PWD/output/tf-resnet50v1-5-training/bf16
+    dst: /output/tf-resnet50v1-5-training/bf16
diff --git a/docker/tf-cpu/tf-resnet50v1-5-training/tf-resnet50v1-5-training.Dockerfile-centos b/docker/tf-cpu/tf-resnet50v1-5-training/tf-resnet50v1-5-training.Dockerfile-centos
new file mode 100644
index 000000000..24ce41530
--- /dev/null
+++ b/docker/tf-cpu/tf-resnet50v1-5-training/tf-resnet50v1-5-training.Dockerfile-centos
@@ -0,0 +1,97 @@
+# Copyright (c) 2020-2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+#
+# THIS IS A GENERATED DOCKERFILE.
+#
+# This file was assembled from multiple pieces, whose use is documented
+# throughout. Please refer to the TensorFlow dockerfiles documentation
+# for more information.
+
+ARG TF_BASE_IMAGE=intel/intel-optimized-tensorflow-avx512
+
+ARG TF_BASE_TAG=latest
+
+FROM ${TF_BASE_IMAGE}:${TF_BASE_TAG}
+
+WORKDIR /workspace/tf-resnet50v1-5-training
+
+RUN yum update -y && \
+    yum install -y git cmake \
+        ca-certificates \
+        python3-tkinter && \
+    yum clean all
+
+# Install OpenMPI
+ARG OPENMPI_VERSION="openmpi-4.1.0"
+ARG OPENMPI_DOWNLOAD_URL="https://www.open-mpi.org/software/ompi/v4.1/downloads/openmpi-4.1.0.tar.gz"
+
+RUN mkdir /tmp/openmpi && \
+    cd /tmp/openmpi && \
+    curl -fSsL -O ${OPENMPI_DOWNLOAD_URL} && \
+    tar zxf ${OPENMPI_VERSION}.tar.gz && \
+    cd ${OPENMPI_VERSION} && \
+    ./configure --enable-mpirun-prefix-by-default && \
+    make -j $(nproc) all && \
+    make install && \
+    ldconfig && \
+    cd / && \
+    rm -rf /tmp/openmpi
+
+# Create a wrapper for OpenMPI to allow running as root by default
+RUN mv /usr/local/bin/mpirun /usr/local/bin/mpirun.real && \
+    echo '#!/bin/bash' > /usr/local/bin/mpirun && \
+    echo 'mpirun.real --allow-run-as-root "$@"' >> /usr/local/bin/mpirun && \
+    chmod a+x /usr/local/bin/mpirun
+
+# Configure OpenMPI to run good defaults:
+RUN echo "btl_tcp_if_exclude = lo,docker0" >> /usr/local/etc/openmpi-mca-params.conf
+
+# Install OpenSSH for MPI to communicate between containers
+RUN yum update -y && \
+    yum install -y  \
+    openssh-server \
+    openssh-clients && \
+    yum clean all
+
+ARG HOROVOD_VERSION=39c8f7c
+ENV HOROVOD_WITHOUT_MXNET=1 \
+    HOROVOD_WITHOUT_PYTORCH=1 \
+    HOROVOD_WITH_TENSORFLOW=1 \
+    HOROVOD_CPU_OPERATIONS=MPI \
+    HOROVOD_WITH_MPI=1 \
+    HOROVOD_WITHOUT_GLOO=1
+
+
+RUN python3 -m pip install git+https://github.com/horovod/horovod.git@${HOROVOD_VERSION}
+
+RUN yum update -y && \
+    yum install -y numactl && \
+    yum clean all
+
+COPY benchmarks benchmarks
+COPY models models
+COPY quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/multi_instance_training.sh quickstart/training.sh
+COPY quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/multi_instance_training_demo.sh quickstart/training_demo.sh
+COPY quickstart/common quickstart/common
+
+COPY LICENSE license/LICENSE
+COPY third_party license/third_party
+
+COPY docker/tf-cpu/generate_ssh_key.sh /workspace/generate_ssh_key.sh
+
+RUN cat /workspace/generate_ssh_key.sh >> ${HOME}/.bash_profile && \
+    cat /workspace/generate_ssh_key.sh >> ${HOME}/.bashrc
+
+ENV BASH_ENV=/root/.bash_profile
diff --git a/docker/tf-cpu/tf-resnet50v1-5-training/tf-resnet50v1-5-training.Dockerfile-ubuntu b/docker/tf-cpu/tf-resnet50v1-5-training/tf-resnet50v1-5-training.Dockerfile-ubuntu
new file mode 100644
index 000000000..87d923f3a
--- /dev/null
+++ b/docker/tf-cpu/tf-resnet50v1-5-training/tf-resnet50v1-5-training.Dockerfile-ubuntu
@@ -0,0 +1,110 @@
+# Copyright (c) 2020-2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+#
+# THIS IS A GENERATED DOCKERFILE.
+#
+# This file was assembled from multiple pieces, whose use is documented
+# throughout. Please refer to the TensorFlow dockerfiles documentation
+# for more information.
+
+ARG TF_BASE_IMAGE=intel/intel-optimized-tensorflow-avx512
+
+ARG TF_BASE_TAG=latest
+
+FROM ${TF_BASE_IMAGE}:${TF_BASE_TAG}
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+WORKDIR /workspace/tf-resnet50v1-5-training
+
+ARG PY_VERSION=3.10
+RUN apt-get update && \
+    apt-get install -y software-properties-common && \
+    add-apt-repository ppa:deadsnakes/ppa -y  && \
+    apt install -y python${PY_VERSION} \
+    python${PY_VERSION}-dev 
+
+RUN apt-get update && apt-get install -y build-essential 
+RUN apt-get install --no-install-recommends --fix-missing -y curl ca-certificates python${PY_VERSION}-dev && \
+    apt-get clean all
+
+RUN apt-get update -y && \
+    apt-get install --no-install-recommends --fix-missing -y cmake python3-tk libxext-dev libsm-dev && \
+    apt-get clean all
+
+# Install OpenMPI
+ARG OPENMPI_VERSION="openmpi-4.1.0"
+ARG OPENMPI_DOWNLOAD_URL="https://www.open-mpi.org/software/ompi/v4.1/downloads/openmpi-4.1.0.tar.gz"
+
+RUN mkdir /tmp/openmpi && \
+    cd /tmp/openmpi && \
+    curl -fSsL -O ${OPENMPI_DOWNLOAD_URL} && \
+    tar zxf ${OPENMPI_VERSION}.tar.gz && \
+    cd ${OPENMPI_VERSION} && \
+    ./configure --enable-mpirun-prefix-by-default && \
+    make -j $(nproc) all && \
+    make install && \
+    ldconfig && \
+    cd / && \
+    rm -rf /tmp/openmpi
+
+# Create a wrapper for OpenMPI to allow running as root by default
+RUN mv /usr/local/bin/mpirun /usr/local/bin/mpirun.real && \
+    echo '#!/bin/bash' > /usr/local/bin/mpirun && \
+    echo 'mpirun.real --allow-run-as-root "$@"' >> /usr/local/bin/mpirun && \
+    chmod a+x /usr/local/bin/mpirun
+
+# Configure OpenMPI to run good defaults:
+RUN echo "btl_tcp_if_exclude = lo,docker0" >> /usr/local/etc/openmpi-mca-params.conf
+
+# Install OpenSSH for MPI to communicate between containers
+RUN apt-get update -y && apt-get install --no-install-recommends --fix-missing -y  \
+    openssh-server \
+    openssh-client && \
+    rm  /etc/ssh/ssh_host_*_key \
+    /etc/ssh/ssh_host_*_key.pub && \
+    apt-get clean all
+
+ARG HOROVOD_VERSION=39c8f7c
+ENV HOROVOD_WITHOUT_MXNET=1 \
+    HOROVOD_WITHOUT_PYTORCH=1 \
+    HOROVOD_WITH_TENSORFLOW=1 \
+    HOROVOD_CPU_OPERATIONS=MPI \
+    HOROVOD_WITH_MPI=1 \
+    HOROVOD_WITHOUT_GLOO=1
+
+# Install Horovod
+RUN apt-get update -y && apt-get install --no-install-recommends --fix-missing -y git && \
+    apt-get clean all
+
+RUN python3 -m pip install git+https://github.com/horovod/horovod.git@${HOROVOD_VERSION}
+
+COPY benchmarks benchmarks
+COPY models models
+COPY quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/multi_instance_training.sh quickstart/training.sh
+COPY quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/multi_instance_training_demo.sh quickstart/training_demo.sh
+COPY quickstart/common quickstart/common
+
+RUN apt-get update -y && apt-get install --no-install-recommends --fix-missing -y numactl
+
+COPY LICENSE license/LICENSE
+COPY third_party license/third_party
+
+COPY docker/tf-cpu/generate_ssh_key.sh /workspace/generate_ssh_key.sh
+
+RUN cat /workspace/generate_ssh_key.sh >> ${HOME}/.bash_profile && \
+    cat /workspace/generate_ssh_key.sh >> ${HOME}/.bashrc
+
+ENV BASH_ENV=/root/.bash_profile
diff --git a/docker/tf-cpu/tf-ssd-mobilenet-inference/tests.yaml b/docker/tf-cpu/tf-ssd-mobilenet-inference/tests.yaml
new file mode 100644
index 000000000..c84df4520
--- /dev/null
+++ b/docker/tf-cpu/tf-ssd-mobilenet-inference/tests.yaml
@@ -0,0 +1,153 @@
+SSD-MobileNet Accuracy int8:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-object-detection-tf-ssd-mobilenet-inference
+  cmd: /bin/bash quickstart/accuracy.sh --num-intra-threads 56  --num-inter-threads 1
+  env:
+    DATASET_DIR: /tf_dataset/dataset/SSDMobilenet
+    OUTPUT_DIR: /output/tf-ssd-mobilenet-inference/accuracy_int8
+    PRECISION: int8
+    BATCH_SIZE: '1'
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/SSDMobilenet
+    dst: /tf_dataset/dataset/SSDMobilenet
+  - src: $PWD/output/tf-ssd-mobilenet-inference/accuracy_int8
+    dst: /output/tf-ssd-mobilenet-inference/accuracy_int8
+SSD-MobileNet Accuracy fp32:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-object-detection-tf-ssd-mobilenet-inference
+  cmd: /bin/bash quickstart/accuracy.sh --num-intra-threads 56  --num-inter-threads 1
+  env:
+    DATASET_DIR: /tf_dataset/dataset/SSDMobilenet
+    OUTPUT_DIR: /output/tf-ssd-mobilenet-inference/accuracy_fp32
+    PRECISION: fp32
+    BATCH_SIZE: '1'
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/SSDMobilenet
+    dst: /tf_dataset/dataset/SSDMobilenet
+  - src: $PWD/output/tf-ssd-mobilenet-inference/accuracy_fp32
+    dst: /output/tf-ssd-mobilenet-inference/accuracy_fp32
+SSD-MobileNet Accuracy bf16:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-object-detection-tf-ssd-mobilenet-inference
+  cmd: /bin/bash quickstart/accuracy.sh --num-intra-threads 56  --num-inter-threads 1
+  env:
+    DATASET_DIR: /tf_dataset/dataset/SSDMobilenet
+    OUTPUT_DIR: /output/tf-ssd-mobilenet-inference/accuracy_bf16
+    PRECISION: bfloat16
+    TF_AUTO_MIXED_PRECISION_GRAPH_REWRITE_ALLOWLIST_ADD: BiasAdd,Relu6,Mul,AddV2
+    TF_AUTO_MIXED_PRECISION_GRAPH_REWRITE_INFERLIST_REMOVE: BiasAdd,AddV2,Mul
+    TF_AUTO_MIXED_PRECISION_GRAPH_REWRITE_CLEARLIST_REMOVE: Relu6
+    BATCH_SIZE: '1'
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/SSDMobilenet
+    dst: /tf_dataset/dataset/SSDMobilenet
+  - src: $PWD/output/tf-ssd-mobilenet-inference/accuracy_bf16
+    dst: /output/tf-ssd-mobilenet-inference/accuracy_bf16
+SSD-MobileNet Realtime Inference fp32:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-object-detection-tf-ssd-mobilenet-inference
+  cmd: /bin/bash quickstart/inference_realtime_multi_instance.sh --benchmark-only --num-intra-threads 4 --num-inter-threads 1 --verbose
+  env:
+    DATASET_DIR: ""
+    OUTPUT_DIR: /output/tf-ssd-mobilenet-inference/realtime_fp32
+    BATCH_SIZE: '1'
+    PRECISION: fp32
+    OMP_NUM_THREADS: '4'
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: $PWD/output/tf-ssd-mobilenet-inference/realtime_fp32
+    dst: /output/tf-ssd-mobilenet-inference/realtime_fp32
+SSD-MobileNet Realtime Inference bf16:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-object-detection-tf-ssd-mobilenet-inference
+  cmd: /bin/bash quickstart/inference_realtime_multi_instance.sh --benchmark-only --num-intra-threads 4 --num-inter-threads 1 --verbose
+  env:
+    DATASET_DIR: ""
+    OUTPUT_DIR: /output/tf-ssd-mobilenet-inference/realtime_bf16
+    BATCH_SIZE: '1'
+    PRECISION: bfloat16
+    OMP_NUM_THREADS: '4'
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+    TF_AUTO_MIXED_PRECISION_GRAPH_REWRITE_ALLOWLIST_ADD: BiasAdd,Relu6,Mul,AddV2
+    TF_AUTO_MIXED_PRECISION_GRAPH_REWRITE_INFERLIST_REMOVE: BiasAdd,AddV2,Mul
+    TF_AUTO_MIXED_PRECISION_GRAPH_REWRITE_CLEARLIST_REMOVE: Relu6
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: $PWD/output/tf-ssd-mobilenet-inference/realtime_bf16
+    dst: /output/tf-ssd-mobilenet-inference/realtime_bf16
+SSD-MobileNet Realtime Inference int8:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-object-detection-tf-ssd-mobilenet-inference
+  cmd: /bin/bash quickstart/inference_realtime_multi_instance.sh --benchmark-only --num-intra-threads 4 --num-inter-threads 1 --verbose
+  env:
+    DATASET_DIR: ""
+    OUTPUT_DIR: /output/tf-ssd-mobilenet-inference/realtime_int8
+    BATCH_SIZE: '1'
+    PRECISION: int8
+    OMP_NUM_THREADS: '4'
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: $PWD/output/tf-ssd-mobilenet-inference/realtime_int8
+    dst: /output/tf-ssd-mobilenet-inference/realtime_int8
+SSD-MobileNet Throughput fp32:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-object-detection-tf-ssd-mobilenet-inference
+  cmd: /bin/bash quickstart/inference_throughput_multi_instance.sh --benchmark-only --num-intra-threads 64 --num-inter-threads 1 --verbose
+  env:
+    DATASET_DIR: ""
+    OUTPUT_DIR: /output/tf-ssd-mobilenet-inference/throughput_fp32
+    BATCH_SIZE: '448'
+    PRECISION: fp32
+    OMP_NUM_THREADS: '64'
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: $PWD/output/tf-ssd-mobilenet-inference/throughput_fp32
+    dst: /output/tf-ssd-mobilenet-inference/throughput_fp32
+SSD-MobileNet Throughput bf16:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-object-detection-tf-ssd-mobilenet-inference
+  cmd: /bin/bash quickstart/inference_throughput_multi_instance.sh --benchmark-only --num-intra-threads 64 --num-inter-threads 1 --verbose
+  env:
+    DATASET_DIR: ""
+    OUTPUT_DIR: /output/tf-ssd-mobilenet-inference/throughput_bf16
+    BATCH_SIZE: '448'
+    PRECISION: bfloat16
+    OMP_NUM_THREADS: '64'
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+    TF_AUTO_MIXED_PRECISION_GRAPH_REWRITE_ALLOWLIST_ADD: BiasAdd,Relu6,Mul,AddV2
+    TF_AUTO_MIXED_PRECISION_GRAPH_REWRITE_INFERLIST_REMOVE: BiasAdd,AddV2,Mul
+    TF_AUTO_MIXED_PRECISION_GRAPH_REWRITE_CLEARLIST_REMOVE: Relu6
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: $PWD/output/tf-ssd-mobilenet-inference/throughput_bf16
+    dst: /output/tf-ssd-mobilenet-inference/throughput_bf16
+SSD-MobileNet Throughput int8:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-object-detection-tf-ssd-mobilenet-inference
+  cmd: /bin/bash quickstart/inference_throughput_multi_instance.sh --benchmark-only --num-intra-threads 64 --num-inter-threads 1 --verbose
+  env:
+    DATASET_DIR: ""
+    OUTPUT_DIR: /output/tf-ssd-mobilenet-inference/throughput_int8
+    BATCH_SIZE: '448'
+    PRECISION: int8
+    OMP_NUM_THREADS: '64'
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: $PWD/output/tf-ssd-mobilenet-inference/throughput_int8
+    dst: /output/tf-ssd-mobilenet-inference/throughput_int8
diff --git a/docker/tf-cpu/tf-ssd-mobilenet-inference/tf-ssd-mobilenet-inference.Dockerfile-centos b/docker/tf-cpu/tf-ssd-mobilenet-inference/tf-ssd-mobilenet-inference.Dockerfile-centos
new file mode 100644
index 000000000..72d0a60de
--- /dev/null
+++ b/docker/tf-cpu/tf-ssd-mobilenet-inference/tf-ssd-mobilenet-inference.Dockerfile-centos
@@ -0,0 +1,85 @@
+# Copyright (c) 2020-2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+#
+# THIS IS A GENERATED DOCKERFILE.
+#
+# This file was assembled from multiple pieces, whose use is documented
+# throughout. Please refer to the TensorFlow dockerfiles documentation
+# for more information.
+
+ARG TF_BASE_IMAGE=intel/intel-optimized-tensorflow
+ARG TF_BASE_TAG=2.13-pip-base
+
+FROM ${TF_BASE_IMAGE}:${TF_BASE_TAG}
+
+WORKDIR /workspace/tf-ssd-mobilenet-inference
+
+RUN yum update -y && \
+    yum install -y git \
+        cmake \
+        unzip \
+        ca-certificates \
+        python3-tkinter && \
+    yum clean all
+
+COPY benchmarks benchmarks 
+COPY quickstart/object_detection/tensorflow/ssd-mobilenet/inference/cpu/inference_realtime_multi_instance.sh quickstart/inference_realtime_multi_instance.sh
+COPY quickstart/object_detection/tensorflow/ssd-mobilenet/inference/cpu/inference_throughput_multi_instance.sh quickstart/inference_throughput_multi_instance.sh
+COPY quickstart/object_detection/tensorflow/ssd-mobilenet/inference/cpu/accuracy.sh quickstart/accuracy.sh
+COPY models/common models/common 
+COPY models/object_detection/tensorflow/ssd-mobilenet/inference models/object_detection/tensorflow/ssd-mobilenet/inference
+COPY models/object_detection/tensorflow/ssd-mobilenet/inference/__init__.py  models/object_detection/tensorflow/ssd-mobilenet/inference/__init__.py
+COPY quickstart/common quickstart/common
+
+ARG TF_MODELS_BRANCH
+
+ARG FETCH_PR
+
+ARG CODE_DIR=/tensorflow/models
+
+ENV TF_MODELS_DIR=${CODE_DIR}
+
+RUN git clone https://github.com/tensorflow/models.git ${CODE_DIR} && \
+    ( cd ${CODE_DIR} && \
+    if [ ! -z "${FETCH_PR}" ]; then git fetch origin ${FETCH_PR}; fi && \
+    git checkout ${TF_MODELS_BRANCH} )
+
+# Note pycocotools has to be install after the other requirements
+
+ARG TF_MODELS_DIR=/tensorflow/models
+
+# Downloads protoc and runs it for object detection
+RUN cd ${TF_MODELS_DIR}/research && \
+    wget --quiet -O protobuf.zip https://github.com/google/protobuf/releases/download/v3.3.0/protoc-3.3.0-linux-x86_64.zip && \
+    unzip -o protobuf.zip && \
+    rm protobuf.zip && \
+    ./bin/protoc object_detection/protos/*.proto --python_out=.
+
+RUN python -m pip install Cython \
+        contextlib2 \
+        jupyter \
+        lxml \
+        matplotlib \
+        numpy>=1.17.4 \
+        'pillow>=9.3.0' \
+        pycocotools \
+        requests
+
+COPY LICENSE license/LICENSE
+COPY third_party license/third_party
+
+RUN mkdir -p /workspace/tf-ssd-mobilenet-inference/pretrained_model && \
+    wget -qO /workspace/tf-ssd-mobilenet-inference/pretrained_model/ssdmobilenet_fp32_pretrained_model_combinedNMS.pb https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/ssdmobilenet_fp32_pretrained_model_combinedNMS.pb && \
+    wget -qO /workspace/tf-ssd-mobilenet-inference/pretrained_model/ssdmobilenet_int8_pretrained_model_combinedNMS_s8.pb https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/ssdmobilenet_int8_pretrained_model_combinedNMS_s8.pb
diff --git a/docker/tf-cpu/tf-ssd-mobilenet-inference/tf-ssd-mobilenet-inference.Dockerfile-ubuntu b/docker/tf-cpu/tf-ssd-mobilenet-inference/tf-ssd-mobilenet-inference.Dockerfile-ubuntu
new file mode 100644
index 000000000..6b2ad6ee1
--- /dev/null
+++ b/docker/tf-cpu/tf-ssd-mobilenet-inference/tf-ssd-mobilenet-inference.Dockerfile-ubuntu
@@ -0,0 +1,117 @@
+# Copyright (c) 2020-2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+#
+# THIS IS A GENERATED DOCKERFILE.
+#
+# This file was assembled from multiple pieces, whose use is documented
+# throughout. Please refer to the TensorFlow dockerfiles documentation
+# for more information.
+
+ARG TF_BASE_IMAGE=intel/intel-optimized-tensorflow
+ARG TF_BASE_TAG=2.13-pip-base
+
+FROM ${TF_BASE_IMAGE}:${TF_BASE_TAG}
+
+WORKDIR /workspace/tf-ssd-mobilenet-inference
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+ARG PY_VERSION=3.10
+ARG PYTHON=python3.10
+
+RUN apt-get update && \
+    apt-get install -y software-properties-common && \
+    add-apt-repository ppa:deadsnakes/ppa -y  && \
+    apt install -y python${PY_VERSION} \
+    python${PY_VERSION}-dev \
+    python3-pip
+    
+RUN apt-get update && apt-get install --no-install-recommends --fix-missing -y gcc 
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y git cmake libxext-dev \
+        libsm-dev \
+        ca-certificates \
+        python3-tk && \
+    apt-get clean all
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y build-essential wget
+
+COPY benchmarks benchmarks 
+COPY quickstart/object_detection/tensorflow/ssd-mobilenet/inference/cpu/inference_realtime_multi_instance.sh quickstart/inference_realtime_multi_instance.sh
+COPY quickstart/object_detection/tensorflow/ssd-mobilenet/inference/cpu/inference_throughput_multi_instance.sh quickstart/inference_throughput_multi_instance.sh
+COPY quickstart/object_detection/tensorflow/ssd-mobilenet/inference/cpu/accuracy.sh quickstart/accuracy.sh
+COPY models/common models/common 
+COPY models/object_detection/tensorflow/ssd-mobilenet/inference models/object_detection/tensorflow/ssd-mobilenet/inference
+COPY models/object_detection/tensorflow/ssd-mobilenet/inference/__init__.py  models/object_detection/tensorflow/ssd-mobilenet/inference/__init__.py
+COPY quickstart/common quickstart/common
+
+RUN ln -sf $(which ${PYTHON}) /usr/local/bin/python && \
+    ln -sf $(which ${PYTHON}) /usr/local/bin/python3 && \
+    ln -sf $(which ${PYTHON}) /usr/bin/python
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+        numactl \
+        libxext-dev \
+        libsm-dev \
+        ca-certificates \
+        python3-tk && \
+    python -m pip install requests
+
+ARG TF_MODELS_BRANCH
+
+ARG FETCH_PR
+
+ARG CODE_DIR=/tensorflow/models
+
+ENV TF_MODELS_DIR=${CODE_DIR}
+
+RUN apt-get update -y && apt-get install --no-install-recommends --fix-missing -y git && \
+    git clone https://github.com/tensorflow/models.git ${CODE_DIR} && \
+    ( cd ${CODE_DIR} && \
+    if [ ! -z "${FETCH_PR}" ]; then git fetch origin ${FETCH_PR}; fi && \
+    git checkout ${TF_MODELS_BRANCH} )
+
+# Note pycocotools has to be install after the other requirements
+
+ARG TF_MODELS_DIR=/tensorflow/models
+
+# Downloads protoc and runs it for object detection
+RUN cd ${TF_MODELS_DIR}/research && \
+    apt-get update -y && apt-get install --no-install-recommends --fix-missing -y \
+        unzip \
+        wget && \
+    wget --quiet -O protobuf.zip https://github.com/google/protobuf/releases/download/v3.3.0/protoc-3.3.0-linux-x86_64.zip && \
+    unzip -o protobuf.zip && \
+    rm protobuf.zip && \
+    ./bin/protoc object_detection/protos/*.proto --python_out=.
+
+RUN python -m pip install Cython \
+        contextlib2 \
+        jupyter \
+        lxml \
+        matplotlib \
+        numpy>=1.17.4 \
+        'pillow>=9.3.0' \
+        pycocotools
+
+COPY LICENSE license/LICENSE
+COPY third_party license/third_party
+
+RUN mkdir -p /workspace/tf-ssd-mobilenet-inference/pretrained_model && \
+    wget -qO /workspace/tf-ssd-mobilenet-inference/pretrained_model/ssdmobilenet_fp32_pretrained_model_combinedNMS.pb https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/ssdmobilenet_fp32_pretrained_model_combinedNMS.pb && \
+    wget -qO /workspace/tf-ssd-mobilenet-inference/pretrained_model/ssdmobilenet_int8_pretrained_model_combinedNMS_s8.pb https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/ssdmobilenet_int8_pretrained_model_combinedNMS_s8.pb
diff --git a/docker/tf-cpu/tf-ssd-resnet34-inference/tests.yaml b/docker/tf-cpu/tf-ssd-resnet34-inference/tests.yaml
new file mode 100644
index 000000000..9ad41cf43
--- /dev/null
+++ b/docker/tf-cpu/tf-ssd-resnet34-inference/tests.yaml
@@ -0,0 +1,150 @@
+SSD-ResNet34 Realtime Inference fp32:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-object-detection-tf-ssd-resnet34-inference
+  cmd: /bin/bash quickstart/multi_instance_online_inference_1200.sh --benchmark-only warmup-steps=100 steps=200
+  env:
+    BATCH_SIZE: '1'
+    DATASET_DIR: /tf_dataset/dataset/ssd-resnet34
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+    OUTPUT_DIR: /output/tf-ssd-resnet34-inference/realtime_fp32
+    PRECISION: fp32
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/ssd-resnet34
+    dst: /tf_dataset/dataset/ssd-resnet34
+  - src: $PWD/output/tf-ssd-resnet34-inference/realtime_fp32
+    dst: /output/tf-ssd-resnet34-inference/realtime_fp32
+SSD-ResNet34 Realtime Inference bf16:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-object-detection-tf-ssd-resnet34-inference
+  cmd: /bin/bash quickstart/multi_instance_online_inference_1200.sh --benchmark-only warmup-steps=100 steps=200
+  env:
+    BATCH_SIZE: '1'
+    DATASET_DIR: /tf_dataset/dataset/ssd-resnet34
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+    OUTPUT_DIR: /output/tf-ssd-resnet34-inference/realtime_bf16
+    PRECISION: bfloat16
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/ssd-resnet34
+    dst: /tf_dataset/dataset/ssd-resnet34
+  - src: $PWD/output/tf-ssd-resnet34-inference/realtime_bf16
+    dst: /output/tf-ssd-resnet34-inference/realtime_bf16
+SSD-ResNet34 Realtime Inference int8:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-object-detection-tf-ssd-resnet34-inference
+  cmd: /bin/bash quickstart/multi_instance_online_inference_1200.sh --benchmark-only warmup-steps=100 steps=200
+  env:
+    BATCH_SIZE: '1'
+    DATASET_DIR: /tf_dataset/dataset/ssd-resnet34
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+    OUTPUT_DIR: /output/tf-ssd-resnet34-inference/realtime_int8
+    PRECISION: int8
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/ssd-resnet34
+    dst: /tf_dataset/dataset/ssd-resnet34
+  - src: $PWD/output/tf-ssd-resnet34-inference/realtime_int8
+    dst: /output/tf-ssd-resnet34-inference/realtime_int8
+SSD-ResNet34 Throughput fp32:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-object-detection-tf-ssd-resnet34-inference
+  cmd: /bin/bash quickstart/multi_instance_batch_inference_1200.sh  --benchmark-only warmup-steps=100 steps=200
+  env:
+    DATASET_DIR: /tf_dataset/dataset/ssd-resnet34
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0 
+    OUTPUT_DIR: /output/tf-ssd-resnet34-inference/throughput_fp32
+    BATCH_SIZE: '112'
+    PRECISION: fp32
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/ssd-resnet34
+    dst: /tf_dataset/dataset/ssd-resnet34
+  - src: $PWD/output/tf-ssd-resnet34-inference/throughput_fp32
+    dst: /output/tf-ssd-resnet34-inference/throughput_fp32
+SSD-ResNet34 Throughput bf16:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-object-detection-tf-ssd-resnet34-inference
+  cmd: /bin/bash quickstart/multi_instance_batch_inference_1200.sh --benchmark-only warmup-steps=100 steps=200
+  env:
+    DATASET_DIR: /tf_dataset/dataset/ssd-resnet34
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+    OUTPUT_DIR: /output/tf-ssd-resnet34-inference/throughput_bf16
+    BATCH_SIZE: '112'
+    PRECISION: bfloat16
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/ssd-resnet34
+    dst: /tf_dataset/dataset/ssd-resnet34
+  - src: $PWD/output/tf-ssd-resnet34-inference/throughput_bf16
+    dst: /output/tf-ssd-resnet34-inference/throughput_bf16
+SSD-ResNet34 Throughput int8:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-object-detection-tf-ssd-resnet34-inference
+  cmd: /bin/bash quickstart/multi_instance_batch_inference_1200.sh --benchmark-only warmup-steps=100 steps=200
+  env:
+    DATASET_DIR: /tf_dataset/dataset/ssd-resnet34
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+    OUTPUT_DIR: /output/tf-ssd-resnet34-inference/throughput_int8
+    BATCH_SIZE: '112'
+    PRECISION: int8
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/ssd-resnet34
+    dst: /tf_dataset/dataset/ssd-resnet34
+  - src: $PWD/output/tf-ssd-resnet34-inference/throughput_int8
+    dst: /output/tf-ssd-resnet34-inference/throughput_int8
+SSD-ResNet34 Accuracy fp32:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-object-detection-tf-ssd-resnet34-inference
+  cmd: /bin/bash quickstart/accuracy_1200.sh
+  env:
+    BATCH_SIZE: '1'
+    DATASET_DIR: /tf_dataset/dataset/ssd-resnet34
+    OUTPUT_DIR: /output/tf-ssd-resnet34-inference/accuracy_fp32
+    PRECISION: fp32
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/ssd-resnet34
+    dst: /tf_dataset/dataset/ssd-resnet34
+  - src: $PWD/output/tf-ssd-resnet34-inference/accuracy_fp32
+    dst: /output/tf-ssd-resnet34-inference/accuracy_fp32
+SSD-ResNet34 Accuracy bf16:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-object-detection-tf-ssd-resnet34-inference
+  cmd: /bin/bash quickstart/accuracy_1200.sh
+  env:
+    BATCH_SIZE: '1'
+    DATASET_DIR: /tf_dataset/dataset/ssd-resnet34
+    OUTPUT_DIR: /output/tf-ssd-resnet34-inference/accuracy_bf16
+    PRECISION: bfloat16
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/ssd-resnet34
+    dst: /tf_dataset/dataset/ssd-resnet34
+  - src: $PWD/output/tf-ssd-resnet34-inference/accuracy_bf16
+    dst: /output/tf-ssd-resnet34-inference/accuracy_bf16
+SSD-ResNet34 Accuracy int8:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-object-detection-tf-ssd-resnet34-inference
+  cmd: /bin/bash quickstart/accuracy_1200.sh
+  env:
+    BATCH_SIZE: '1'
+    DATASET_DIR: /tf_dataset/dataset/ssd-resnet34
+    OUTPUT_DIR: /output/tf-ssd-resnet34-inference/accuracy_int8
+    PRECISION: int8
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/ssd-resnet34
+    dst: /tf_dataset/dataset/ssd-resnet34
+  - src: $PWD/output/tf-ssd-resnet34-inference/accuracy_int8
+    dst: /output/tf-ssd-resnet34-inference/accuracy_int8
diff --git a/docker/tf-cpu/tf-ssd-resnet34-inference/tf-ssd-resnet34-inference.Dockerfile-centos b/docker/tf-cpu/tf-ssd-resnet34-inference/tf-ssd-resnet34-inference.Dockerfile-centos
new file mode 100644
index 000000000..0c85a2aea
--- /dev/null
+++ b/docker/tf-cpu/tf-ssd-resnet34-inference/tf-ssd-resnet34-inference.Dockerfile-centos
@@ -0,0 +1,102 @@
+# Copyright (c) 2020-2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+#
+# THIS IS A GENERATED DOCKERFILE.
+#
+# This file was assembled from multiple pieces, whose use is documented
+# throughout. Please refer to the TensorFlow dockerfiles documentation
+# for more information.
+
+ARG TF_BASE_IMAGE=intel/intel-optimized-tensorflow
+ARG TF_BASE_TAG=2.13-pip-base
+
+FROM ${TF_BASE_IMAGE}:${TF_BASE_TAG}
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+WORKDIR /workspace/tf-ssd-resnet34-inference
+
+RUN yum update -y && \
+    yum install -y git \
+        cmake \
+        unzip \
+        ca-certificates \
+        python3-tkinter && \
+    yum clean all
+
+COPY benchmarks benchmarks
+COPY models models
+COPY quickstart/common quickstart/common
+COPY quickstart/object_detection/tensorflow/ssd-resnet34/inference/cpu/multi_instance_online_inference_1200.sh quickstart/multi_instance_online_inference_1200.sh
+COPY quickstart/object_detection/tensorflow/ssd-resnet34/inference/cpu/multi_instance_batch_inference_1200.sh quickstart/multi_instance_batch_inference_1200.sh
+COPY quickstart/object_detection/tensorflow/ssd-resnet34/inference/cpu/accuracy_1200.sh quickstart/accuracy_1200.sh
+
+ARG TF_MODELS_BRANCH
+
+ARG FETCH_PR
+
+ARG CODE_DIR=/workspace/models
+
+ENV TF_MODELS_DIR=${CODE_DIR}
+
+RUN git clone https://github.com/tensorflow/models.git ${CODE_DIR} && \
+    ( cd ${CODE_DIR} && \
+    if [ ! -z "${FETCH_PR}" ]; then git fetch origin ${FETCH_PR}; fi && \
+    git checkout ${TF_MODELS_BRANCH} )
+
+# Note pycocotools has to be install after the other requirements
+
+# Downloads protoc and runs it for object detection
+RUN cd ${TF_MODELS_DIR}/research && \
+    wget --quiet -O protobuf.zip https://github.com/google/protobuf/releases/download/v3.3.0/protoc-3.3.0-linux-x86_64.zip && \
+    unzip -o protobuf.zip && \
+    rm protobuf.zip && \
+    ./bin/protoc object_detection/protos/*.proto --python_out=.
+
+# Note pycocotools has to be install after the other requirements
+RUN python -m pip install Cython \
+        contextlib2 \
+        jupyter \
+        lxml \
+        matplotlib \
+        numpy>=1.17.4 \
+        'pillow>=9.3.0' \
+        pycocotools
+
+RUN yum update -y && \
+    yum install -y \
+       mesa-libGL \
+       glib2-devel && \
+    yum clean all
+
+RUN python -m pip install opencv-python
+
+ARG TF_BENCHMARKS_BRANCH
+
+ARG TF_BENCHMARKS_DIR="/workspace/ssd-resnet-benchmarks"
+
+ENV TF_BENCHMARKS_DIR=${TF_BENCHMARKS_DIR}
+
+RUN yum update && \
+    git clone --single-branch https://github.com/tensorflow/benchmarks.git ${TF_BENCHMARKS_DIR} && \
+    ( cd ${TF_BENCHMARKS_DIR} && \
+    git checkout ${TF_BENCHMARKS_BRANCH} )
+
+COPY LICENSE license/LICENSE
+COPY third_party license/third_party
+
+RUN mkdir -p /workspace/tf-ssd-resnet34-inference/pretrained_model && \
+    wget -qO /workspace/tf-ssd-resnet34-inference/pretrained_model/ssd_resnet34_fp32_1200x1200_pretrained_model.pb https://storage.googleapis.com/intel-optimized-tensorflow/models/2_10_0/ssd_resnet34_fp32_1200x1200_pretrained_model.pb && \
+    wget -qO /workspace/tf-ssd-resnet34-inference/pretrained_model/ssd_resnet34_int8_1200x1200_pretrained_model.pb https://storage.googleapis.com/intel-optimized-tensorflow/models/2_10_0/ssd_resnet34_int8_1200x1200_pretrained_model.pb
diff --git a/docker/tf-cpu/tf-ssd-resnet34-inference/tf-ssd-resnet34-inference.Dockerfile-ubuntu b/docker/tf-cpu/tf-ssd-resnet34-inference/tf-ssd-resnet34-inference.Dockerfile-ubuntu
new file mode 100644
index 000000000..7efd72636
--- /dev/null
+++ b/docker/tf-cpu/tf-ssd-resnet34-inference/tf-ssd-resnet34-inference.Dockerfile-ubuntu
@@ -0,0 +1,131 @@
+# Copyright (c) 2020-2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+#
+# THIS IS A GENERATED DOCKERFILE.
+#
+# This file was assembled from multiple pieces, whose use is documented
+# throughout. Please refer to the TensorFlow dockerfiles documentation
+# for more information.
+
+ARG TF_BASE_IMAGE=intel/intel-optimized-tensorflow
+ARG TF_BASE_TAG=2.13-pip-base
+
+FROM ${TF_BASE_IMAGE}:${TF_BASE_TAG}
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+WORKDIR /workspace/tf-ssd-resnet34-inference
+
+ARG PY_VERSION=3.10
+ARG PYTHON=python3.10
+
+RUN apt-get update && \
+    apt-get install -y software-properties-common && \
+    add-apt-repository ppa:deadsnakes/ppa -y  && \
+    apt install -y python${PY_VERSION} \
+    python${PY_VERSION}-dev \
+    python3-pip
+    
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y build-essential wget
+
+COPY benchmarks benchmarks
+COPY models models
+COPY quickstart/common quickstart/common
+COPY quickstart/object_detection/tensorflow/ssd-resnet34/inference/cpu/multi_instance_online_inference_1200.sh quickstart/multi_instance_online_inference_1200.sh
+COPY quickstart/object_detection/tensorflow/ssd-resnet34/inference/cpu/multi_instance_batch_inference_1200.sh quickstart/multi_instance_batch_inference_1200.sh
+COPY quickstart/object_detection/tensorflow/ssd-resnet34/inference/cpu/accuracy_1200.sh quickstart/accuracy_1200.sh
+RUN ln -sf $(which ${PYTHON}) /usr/local/bin/python && \
+    ln -sf $(which ${PYTHON}) /usr/local/bin/python3 && \
+    ln -sf $(which ${PYTHON}) /usr/bin/python
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y build-essential 
+
+ARG TF_MODELS_BRANCH="f505cecde2d8ebf6fe15f40fb8bc350b2b1ed5dc"
+
+ARG FETCH_PR
+
+ARG CODE_DIR="/workspace/tf_models"
+
+ENV TF_MODELS_DIR=${CODE_DIR}
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y git && \
+    git clone https://github.com/tensorflow/models.git ${CODE_DIR} && \
+    ( cd ${CODE_DIR} && \
+    if [ ! -z "${FETCH_PR}" ]; then git fetch origin ${FETCH_PR}; fi && \
+    git checkout ${TF_MODELS_BRANCH} )
+
+# Note pycocotools has to be install after the other requirements
+RUN python -m pip install Cython \
+        contextlib2 \
+        jupyter \
+        lxml \
+        matplotlib \
+        numpy>=1.17.4 \
+        'pillow>=9.3.0' \
+        pycocotools
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+        numactl \
+        libxext-dev \
+        libsm-dev \
+        ca-certificates \
+        python3-tk && \
+    python -m pip install requests
+
+# ARG TF_MODELS_DIR=${CODE_DIR}
+# WORKDIR ${TF_MODELS_DIR}/research
+
+# Downloads protoc and runs it for object detection
+RUN cd ${CODE_DIR}/research && \
+    apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+    unzip \
+    wget && \
+    wget --quiet -O protobuf.zip https://github.com/google/protobuf/releases/download/v3.3.0/protoc-3.3.0-linux-x86_64.zip && \
+    unzip -o protobuf.zip && \
+    rm protobuf.zip && \
+    ./bin/protoc object_detection/protos/*.proto --python_out=.
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+       libgl1-mesa-dev \
+       glib2.0
+
+RUN python -m pip install opencv-python
+
+RUN python -m pip install tensorflow-addons==0.18.0
+
+ARG TF_BENCHMARKS_BRANCH="509b9d288937216ca7069f31cfb22aaa7db6a4a7"
+
+ARG TF_BENCHMARKS_DIR="/workspace/ssd-resnet-benchmarks"
+
+ENV TF_BENCHMARKS_DIR=${TF_BENCHMARKS_DIR}
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y git && \
+    git clone --single-branch https://github.com/tensorflow/benchmarks.git ${TF_BENCHMARKS_DIR} && \
+    ( cd ${TF_BENCHMARKS_DIR} && \
+    git checkout ${TF_BENCHMARKS_BRANCH} )
+
+COPY LICENSE license/LICENSE
+COPY third_party license/third_party
+
+RUN mkdir -p /workspace/tf-ssd-resnet34-inference/pretrained_model && \
+    wget -qO /workspace/tf-ssd-resnet34-inference/pretrained_model/ssd_resnet34_fp32_1200x1200_pretrained_model.pb https://storage.googleapis.com/intel-optimized-tensorflow/models/2_10_0/ssd_resnet34_fp32_1200x1200_pretrained_model.pb && \
+    wget -qO /workspace/tf-ssd-resnet34-inference/pretrained_model/ssd_resnet34_int8_1200x1200_pretrained_model.pb https://storage.googleapis.com/intel-optimized-tensorflow/models/2_10_0/ssd_resnet34_int8_1200x1200_pretrained_model.pb
diff --git a/docker/tf-cpu/tf-ssd-resnet34-training/tests.yaml b/docker/tf-cpu/tf-ssd-resnet34-training/tests.yaml
new file mode 100644
index 000000000..2957cb5b1
--- /dev/null
+++ b/docker/tf-cpu/tf-ssd-resnet34-training/tests.yaml
@@ -0,0 +1,34 @@
+SSD-ResNet34 FP32 Training:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-object-detection-tf-ssd-resnet34-training
+  cmd: /bin/bash quickstart/training.sh -a 56 --verbose -e 1 
+  env:
+    DATASET_DIR: /tf_dataset/dataset/ssd-resnet34/coco_training_yang
+    OUTPUT_DIR: /output/tf-ssd-resnet34-training/fp32
+    PRECISION: fp32
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+    BATCH_SIZE: '896'
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/ssd-resnet34/coco_training_yang
+    dst: /tf_dataset/dataset/ssd-resnet34/coco_training_yang
+  - src: $PWD/output/tf-ssd-resnet34-training/fp32
+    dst: /output/tf-ssd-resnet34-training/fp32
+SSD-ResNet34 BF16 Training:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-object-detection-tf-ssd-resnet34-training
+  cmd: /bin/bash quickstart/training.sh -a 56 --verbose -e 1
+  env:
+    DATASET_DIR: /tf_dataset/dataset/ssd-resnet34/coco_training_yang
+    OUTPUT_DIR: /output/tf-ssd-resnet34-training/bf16
+    PRECISION: bfloat16
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+    BATCH_SIZE: '896'
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/ssd-resnet34/coco_training_yang
+    dst: /tf_dataset/dataset/ssd-resnet34/coco_training_yang
+  - src: $PWD/output/tf-ssd-resnet34-training/bf16
+    dst: /output/tf-ssd-resnet34-training/bf16
diff --git a/docker/tf-cpu/tf-ssd-resnet34-training/tf-ssd-resnet34-training.Dockerfile-centos b/docker/tf-cpu/tf-ssd-resnet34-training/tf-ssd-resnet34-training.Dockerfile-centos
new file mode 100644
index 000000000..3792e8625
--- /dev/null
+++ b/docker/tf-cpu/tf-ssd-resnet34-training/tf-ssd-resnet34-training.Dockerfile-centos
@@ -0,0 +1,126 @@
+# Copyright (c) 2020-2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+#
+# THIS IS A GENERATED DOCKERFILE.
+#
+# This file was assembled from multiple pieces, whose use is documented
+# throughout. Please refer to the TensorFlow dockerfiles documentation
+# for more information.
+
+ARG TF_BASE_IMAGE=intel/intel-optimized-tensorflow
+ARG TF_BASE_TAG=2.13-pip-base
+
+FROM ${TF_BASE_IMAGE}:${TF_BASE_TAG}
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+WORKDIR /workspace/tf-ssd-resnet34-training  
+
+RUN yum update -y && \
+    yum install -y git cmake \
+        ca-certificates \
+        python3-tkinter && \
+    yum clean all
+
+# Install OpenMPI
+ARG OPENMPI_VERSION="openmpi-4.1.0"
+ARG OPENMPI_DOWNLOAD_URL="https://www.open-mpi.org/software/ompi/v4.1/downloads/openmpi-4.1.0.tar.gz"
+
+RUN mkdir /tmp/openmpi && \
+    cd /tmp/openmpi && \
+    curl -fSsL -O ${OPENMPI_DOWNLOAD_URL} && \
+    tar zxf ${OPENMPI_VERSION}.tar.gz && \
+    cd ${OPENMPI_VERSION} && \
+    ./configure --enable-mpirun-prefix-by-default && \
+    make -j $(nproc) all && \
+    make install && \
+    ldconfig && \
+    cd / && \
+    rm -rf /tmp/openmpi
+
+# Create a wrapper for OpenMPI to allow running as root by default
+RUN mv /usr/local/bin/mpirun /usr/local/bin/mpirun.real && \
+    echo '#!/bin/bash' > /usr/local/bin/mpirun && \
+    echo 'mpirun.real --allow-run-as-root "$@"' >> /usr/local/bin/mpirun && \
+    chmod a+x /usr/local/bin/mpirun
+
+# Configure OpenMPI to run good defaults:
+RUN echo "btl_tcp_if_exclude = lo,docker0" >> /usr/local/etc/openmpi-mca-params.conf
+
+# Install OpenSSH for MPI to communicate between containers
+RUN yum update -y && \
+    yum install -y  \
+    openssh-server \
+    openssh-clients && \
+    yum clean all
+
+
+ARG HOROVOD_VERSION=39c8f7c
+ENV HOROVOD_WITHOUT_MXNET=1 \
+    HOROVOD_WITHOUT_PYTORCH=1 \
+    HOROVOD_WITH_TENSORFLOW=1 \
+    HOROVOD_CPU_OPERATIONS=MPI \
+    HOROVOD_WITH_MPI=1 \
+    HOROVOD_WITHOUT_GLOO=1
+
+# Install Horovod
+
+RUN python3 -m pip install git+https://github.com/horovod/horovod.git@${HOROVOD_VERSION}
+
+RUN pip install opencv-python
+
+RUN yum update && yum install -y numactl
+
+ARG TF_MODELS_BRANCH="8110bb64ca63c48d0caee9d565e5b4274db2220a"
+
+ARG FETCH_PR
+
+ARG CODE_DIR=/workspace/models
+
+ENV TF_MODELS_DIR=${CODE_DIR}
+
+RUN git clone https://github.com/tensorflow/models.git ${CODE_DIR} && \
+    ( cd ${CODE_DIR} && \
+    if [ ! -z "${FETCH_PR}" ]; then git fetch origin ${FETCH_PR}; fi && \
+    git checkout ${TF_MODELS_BRANCH} )
+
+ARG TF_MODELS_DIR=/workspace/models
+
+# Downloads protoc and runs it for object detection
+RUN cd ${TF_MODELS_DIR}/research && \
+    wget --quiet -O protobuf.zip https://github.com/google/protobuf/releases/download/v3.3.0/protoc-3.3.0-linux-x86_64.zip && \
+    unzip -o protobuf.zip && \
+    rm protobuf.zip && \
+    ./bin/protoc object_detection/protos/*.proto --python_out=.
+
+# Note pycocotools has to be install after the other requirements
+RUN pip install \
+        Cython \
+        contextlib2 \
+        jupyter \
+        lxml \
+        matplotlib \
+        numpy>=1.17.4 \
+        'pillow>=9.3.0' && \
+    pip install pycocotools
+
+COPY benchmarks benchmarks
+COPY models models
+COPY quickstart/common quickstart/common
+COPY quickstart/object_detection/tensorflow/ssd-resnet34/training/cpu/training.sh quickstart/training.sh
+
+
+COPY LICENSE license/LICENSE
+COPY third_party license/third_party
diff --git a/docker/tf-cpu/tf-ssd-resnet34-training/tf-ssd-resnet34-training.Dockerfile-ubuntu b/docker/tf-cpu/tf-ssd-resnet34-training/tf-ssd-resnet34-training.Dockerfile-ubuntu
new file mode 100644
index 000000000..8708cb5a0
--- /dev/null
+++ b/docker/tf-cpu/tf-ssd-resnet34-training/tf-ssd-resnet34-training.Dockerfile-ubuntu
@@ -0,0 +1,161 @@
+# Copyright (c) 2020-2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+#
+# THIS IS A GENERATED DOCKERFILE.
+#
+# This file was assembled from multiple pieces, whose use is documented
+# throughout. Please refer to the TensorFlow dockerfiles documentation
+# for more information.
+
+ARG TF_BASE_IMAGE=intel/intel-optimized-tensorflow
+ARG TF_BASE_TAG=2.13-pip-base
+
+FROM ${TF_BASE_IMAGE}:${TF_BASE_TAG}
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+WORKDIR /workspace/tf-ssd-resnet34-training 
+
+ARG PY_VERSION=3.10
+ARG PYTHON=python3.10
+
+RUN apt-get update && \
+    apt-get install -y software-properties-common && \
+    add-apt-repository ppa:deadsnakes/ppa -y  && \
+    apt install -y python${PY_VERSION} \
+    python${PY_VERSION}-dev \
+    python3-pip
+    
+RUN apt-get update && apt-get install --no-install-recommends --fix-missing -y gcc 
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y curl build-essential 
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y git cmake libxext-dev \
+        libsm-dev \
+        ca-certificates \
+        python3-tk && \
+    apt-get clean all
+
+# Install OpenMPI
+ARG OPENMPI_VERSION="openmpi-4.1.0"
+ARG OPENMPI_DOWNLOAD_URL="https://www.open-mpi.org/software/ompi/v4.1/downloads/openmpi-4.1.0.tar.gz"
+
+RUN mkdir /tmp/openmpi && \
+    cd /tmp/openmpi && \
+    curl -fSsL -O ${OPENMPI_DOWNLOAD_URL} && \
+    tar zxf ${OPENMPI_VERSION}.tar.gz && \
+    cd ${OPENMPI_VERSION} && \
+    ./configure --enable-mpirun-prefix-by-default && \
+    make -j $(nproc) all && \
+    make install && \
+    ldconfig && \
+    cd / && \
+    rm -rf /tmp/openmpi
+
+# Create a wrapper for OpenMPI to allow running as root by default
+RUN mv /usr/local/bin/mpirun /usr/local/bin/mpirun.real && \
+    echo '#!/bin/bash' > /usr/local/bin/mpirun && \
+    echo 'mpirun.real --allow-run-as-root "$@"' >> /usr/local/bin/mpirun && \
+    chmod a+x /usr/local/bin/mpirun
+
+# Configure OpenMPI to run good defaults:
+RUN echo "btl_tcp_if_exclude = lo,docker0" >> /usr/local/etc/openmpi-mca-params.conf
+
+# Install OpenSSH for MPI to communicate between containers
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y  \
+    openssh-server \
+    openssh-client && \
+    rm  /etc/ssh/ssh_host_*_key \
+    /etc/ssh/ssh_host_*_key.pub && \
+    apt-get clean all
+
+
+ARG HOROVOD_VERSION=39c8f7c
+ENV HOROVOD_WITHOUT_MXNET=1 \
+    HOROVOD_WITHOUT_PYTORCH=1 \
+    HOROVOD_WITH_TENSORFLOW=1 \
+    HOROVOD_CPU_OPERATIONS=MPI \
+    HOROVOD_WITH_MPI=1 \
+    HOROVOD_WITHOUT_GLOO=1
+
+# Install Horovod
+
+RUN python3 -m pip install git+https://github.com/horovod/horovod.git@${HOROVOD_VERSION}
+
+RUN ln -sf $(which ${PYTHON}) /usr/local/bin/python && \
+    ln -sf $(which ${PYTHON}) /usr/local/bin/python3 && \
+    ln -sf $(which ${PYTHON}) /usr/bin/python
+
+RUN pip install opencv-python
+
+RUN apt-get update && apt-get install --no-install-recommends --fix-missing -y numactl
+
+RUN pip install tensorflow-addons==0.18.0
+
+ARG TF_MODELS_BRANCH="8110bb64ca63c48d0caee9d565e5b4274db2220a"
+
+ARG FETCH_PR
+
+ARG CODE_DIR=/tensorflow/models
+
+ENV TF_MODELS_DIR=${CODE_DIR}
+
+RUN apt-get update && apt-get install --no-install-recommends --fix-missing -y git && \
+    git clone https://github.com/tensorflow/models.git ${CODE_DIR} && \
+    ( cd ${CODE_DIR} && \
+    if [ ! -z "${FETCH_PR}" ]; then git fetch origin ${FETCH_PR}; fi && \
+    git checkout ${TF_MODELS_BRANCH} )
+
+ARG TF_MODELS_DIR=/tensorflow/models
+
+# Downloads protoc and runs it for object detection
+RUN cd ${TF_MODELS_DIR}/research && \
+    apt-get update && apt-get install --no-install-recommends --fix-missing -y \
+        unzip \
+        wget && \
+    wget --quiet -O protobuf.zip https://github.com/google/protobuf/releases/download/v3.3.0/protoc-3.3.0-linux-x86_64.zip && \
+    unzip -o protobuf.zip && \
+    rm protobuf.zip && \
+    ./bin/protoc object_detection/protos/*.proto --python_out=.
+
+# Note pycocotools has to be install after the other requirements
+RUN pip install \
+        Cython \
+        contextlib2 \
+        jupyter \
+        lxml \
+        matplotlib \
+        numpy>=1.17.4 \
+        'pillow>=9.3.0' && \
+    pip install pycocotools
+
+COPY benchmarks benchmarks
+COPY models models
+COPY quickstart/common quickstart/common
+COPY quickstart/object_detection/tensorflow/ssd-resnet34/training/cpu/training.sh quickstart/training.sh
+
+
+COPY LICENSE license/LICENSE
+COPY third_party license/third_party
+
+COPY docker/tf-cpu/generate_ssh_key.sh /workspace/generate_ssh_key.sh
+
+RUN cat /workspace/generate_ssh_key.sh >> ${HOME}/.bash_profile && \
+    cat /workspace/generate_ssh_key.sh >> ${HOME}/.bashrc
+
+ENV BASH_ENV=/root/.bash_profile
diff --git a/docker/tf-cpu/tf-transformer-mlperf-inference/tests.yaml b/docker/tf-cpu/tf-transformer-mlperf-inference/tests.yaml
new file mode 100644
index 000000000..afd9bcd20
--- /dev/null
+++ b/docker/tf-cpu/tf-transformer-mlperf-inference/tests.yaml
@@ -0,0 +1,151 @@
+Transformer Realtime inference fp32:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-language-translation-tf-transformer-mlperf-inference
+  cmd: /bin/bash quickstart/inference_realtime.sh --benchmark-only --num-intra-threads 4 --num-inter-threads 1
+  env:
+    DATASET_DIR: /tf_dataset/pre-trained-models/transformer_mlperf/data
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+    OUTPUT_DIR: /output/tf-transformer-mlperf-inference/realtime_fp32
+    PRECISION: fp32
+    BATCH_SIZE: '1'
+    OMP_NUM_THREADS: '4'
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/pre-trained-models/transformer_mlperf/data
+    dst: /tf_dataset/pre-trained-models/transformer_mlperf/data
+  - src: $PWD/output/tf-transformer-mlperf-inference/realtime_fp32
+    dst: /output/tf-transformer-mlperf-inference/realtime_fp32
+Transformer Realtime inference bf16:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-language-translation-tf-transformer-mlperf-inference
+  cmd: /bin/bash quickstart/inference_realtime.sh --benchmark-only
+  env:
+    DATASET_DIR: /tf_dataset/pre-trained-models/transformer_mlperf/data
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+    OUTPUT_DIR: /output/tf-transformer-mlperf-inference/realtime_bf16
+    PRECISION: bfloat16
+    BATCH_SIZE: '1'
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/pre-trained-models/transformer_mlperf/data
+    dst: /tf_dataset/pre-trained-models/transformer_mlperf/data
+  - src: $PWD/output/tf-transformer-mlperf-inference/realtime_bf16
+    dst: /output/tf-transformer-mlperf-inference/realtime_bf16
+Transformer Realtime inference int8:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-language-translation-tf-transformer-mlperf-inference
+  cmd: /bin/bash quickstart/inference_realtime.sh --benchmark-only
+  env:
+    DATASET_DIR: /tf_dataset/pre-trained-models/transformer_mlperf/data
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+    OUTPUT_DIR: /output/tf-transformer-mlperf-inference/realtime_int8
+    PRECISION: int8
+    BATCH_SIZE: '1'
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/pre-trained-models/transformer_mlperf/data
+    dst: /tf_dataset/pre-trained-models/transformer_mlperf/data
+  - src: $PWD/output/tf-transformer-mlperf-inference/realtime_int8
+    dst: /output/tf-transformer-mlperf-inference/realtime_int8
+Transformer Throughput bf16:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-language-translation-tf-transformer-mlperf-inference
+  cmd: /bin/bash quickstart/inference_throughput.sh
+  env:
+    DATASET_DIR: /tf_dataset/pre-trained-models/transformer_mlperf/data
+    KMP_AFFINITY=granularity: fine,verbose,compact,1,0 
+    OUTPUT_DIR: /output/tf-transformer-mlperf-inference/throughput_bf16
+    PRECISION: bfloat16
+    BATCH_SIZE: '448'
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/pre-trained-models/transformer_mlperf/data
+    dst: /tf_dataset/pre-trained-models/transformer_mlperf/data
+  - src: $PWD/output/tf-transformer-mlperf-inference/throughput_bf16
+    dst: /output/tf-transformer-mlperf-inference/throughput_bf16
+Transformer Throughput fp32:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-language-translation-tf-transformer-mlperf-inference
+  cmd: /bin/bash quickstart/inference_throughput.sh
+  env:
+    DATASET_DIR: /tf_dataset/pre-trained-models/transformer_mlperf/data
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+    OUTPUT_DIR: /output/tf-transformer-mlperf-inference/throughput_fp32
+    PRECISION: fp32
+    BATCH_SIZE: '448'
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/pre-trained-models/transformer_mlperf/data
+    dst: /tf_dataset/pre-trained-models/transformer_mlperf/data
+  - src: $PWD/output/tf-transformer-mlperf-inference/throughput_fp32
+    dst: /output/tf-transformer-mlperf-inference/throughput_fp32
+Transformer Throughput int8:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-language-translation-tf-transformer-mlperf-inference
+  cmd: /bin/bash quickstart/inference_throughput.sh
+  env:
+    DATASET_DIR: /tf_dataset/pre-trained-models/transformer_mlperf/data
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+    OUTPUT_DIR: /output/tf-transformer-mlperf-inference/throughput_int8
+    PRECISION: int8
+    BATCH_SIZE: '448'
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/pre-trained-models/transformer_mlperf/data
+    dst: /tf_dataset/pre-trained-models/transformer_mlperf/data
+  - src: $PWD/output/tf-transformer-mlperf-inference/throughput_int8
+    dst: /output/tf-transformer-mlperf-inference/throughput_int8
+Transformer Accuracy fp32:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-language-translation-tf-transformer-mlperf-inference
+  cmd: /bin/bash quickstart/accuracy.sh
+  env:
+    BATCH_SIZE: '64'
+    DATASET_DIR: /tf_dataset/pre-trained-models/transformer_mlperf/data
+    OUTPUT_DIR: /output/tf-transformer-mlperf-inference/accuracy_fp32
+    PRECISION: fp32
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/pre-trained-models/transformer_mlperf/data
+    dst: /tf_dataset/pre-trained-models/transformer_mlperf/data
+  - src: $PWD/output/tf-transformer-mlperf-inference/accuracy_fp32
+    dst: /output/tf-transformer-mlperf-inference/accuracy_fp32
+Transformer Accuracy bf16:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-language-translation-tf-transformer-mlperf-inference
+  cmd: /bin/bash quickstart/accuracy.sh
+  env:
+    BATCH_SIZE: '64'
+    DATASET_DIR: /tf_dataset/pre-trained-models/transformer_mlperf/data
+    OUTPUT_DIR: /output/tf-transformer-mlperf-inference/accuracy_bf16
+    PRECISION: bfloat16
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/pre-trained-models/transformer_mlperf/data
+    dst: /tf_dataset/pre-trained-models/transformer_mlperf/data
+  - src: $PWD/output/tf-transformer-mlperf-inference/accuracy_bf16
+    dst: /output/tf-transformer-mlperf-inference/accuracy_bf16
+Transformer Accuracy int8:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-language-translation-tf-transformer-mlperf-inference
+  cmd: /bin/bash quickstart/accuracy.sh
+  env:
+    BATCH_SIZE: '64'
+    DATASET_DIR: /tf_dataset/pre-trained-models/transformer_mlperf/data
+    OUTPUT_DIR: /output/tf-transformer-mlperf-inference/accuracy_int8
+    PRECISION: int8
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/pre-trained-models/transformer_mlperf/data
+    dst: /tf_dataset/pre-trained-models/transformer_mlperf/data
+  - src: $PWD/output/tf-transformer-mlperf-inference/accuracy_int8
+    dst: /output/tf-transformer-mlperf-inference/accuracy_int8
diff --git a/docker/tf-cpu/tf-transformer-mlperf-inference/tf-transformer-mlperf-inference.Dockerfile-centos b/docker/tf-cpu/tf-transformer-mlperf-inference/tf-transformer-mlperf-inference.Dockerfile-centos
new file mode 100644
index 000000000..2268fa8fd
--- /dev/null
+++ b/docker/tf-cpu/tf-transformer-mlperf-inference/tf-transformer-mlperf-inference.Dockerfile-centos
@@ -0,0 +1,50 @@
+# Copyright (c) 2020-2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+#
+# THIS IS A GENERATED DOCKERFILE.
+#
+# This file was assembled from multiple pieces, whose use is documented
+# throughout. Please refer to the TensorFlow dockerfiles documentation
+# for more information.
+
+ARG TF_BASE_IMAGE=intel/intel-optimized-tensorflow-avx512
+ARG TF_BASE_TAG=latest
+
+FROM ${TF_BASE_IMAGE}:${TF_BASE_TAG}
+
+WORKDIR /workspace/tf-transformer-mlperf-inference
+
+RUN yum update -y && \
+    yum install -y git \
+        cmake \
+        unzip \
+        ca-certificates \
+        python3-tkinter && \
+    yum clean all
+
+COPY benchmarks benchmarks 
+COPY quickstart/common quickstart/common
+COPY quickstart/language_translation/tensorflow/transformer_mlperf/inference/cpu/accuracy.sh quickstart/accuracy.sh
+COPY quickstart/language_translation/tensorflow/transformer_mlperf/inference/cpu/inference_throughput_multi_instance.sh quickstart/inference_throughput.sh
+COPY quickstart/language_translation/tensorflow/transformer_mlperf/inference/cpu/inference_realtime_multi_instance.sh quickstart/inference_realtime.sh
+COPY models models 
+
+COPY LICENSE license/LICENSE
+COPY third_party license/third_party
+
+RUN mkdir -p /workspace/tf-transformer-mlperf-inference/pretrained_model && \
+    wget -qO /workspace/tf-transformer-mlperf-inference/pretrained_model/transformer_mlperf_int8.pb https://storage.googleapis.com/intel-optimized-tensorflow/models/2_10_0/transformer_mlperf_int8.pb && \
+    wget -qO /workspace/tf-transformer-mlperf-inference/pretrained_model/transformer_mlperf_fp32.pb https://storage.googleapis.com/intel-optimized-tensorflow/models/2_10_0/transformer_mlperf_fp32.pb && \
+    wget -qO /workspace/tf-transformer-mlperf-inference/pretrained_model/transformer_mlperf_bf16.pb https://storage.googleapis.com/intel-optimized-tensorflow/models/2_10_0/transformer_mlperf_bf16.pb
diff --git a/docker/tf-cpu/tf-transformer-mlperf-inference/tf-transformer-mlperf-inference.Dockerfile-ubuntu b/docker/tf-cpu/tf-transformer-mlperf-inference/tf-transformer-mlperf-inference.Dockerfile-ubuntu
new file mode 100644
index 000000000..8c38dd63b
--- /dev/null
+++ b/docker/tf-cpu/tf-transformer-mlperf-inference/tf-transformer-mlperf-inference.Dockerfile-ubuntu
@@ -0,0 +1,63 @@
+# Copyright (c) 2020-2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+#
+# THIS IS A GENERATED DOCKERFILE.
+#
+# This file was assembled from multiple pieces, whose use is documented
+# throughout. Please refer to the TensorFlow dockerfiles documentation
+# for more information.
+
+ARG TF_BASE_IMAGE=intel/intel-optimized-tensorflow-avx512
+ARG TF_BASE_TAG=latest
+
+FROM ${TF_BASE_IMAGE}:${TF_BASE_TAG}
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+WORKDIR /workspace/tf-transformer-mlperf-inference
+
+ARG PY_VERSION=3.10 
+
+RUN apt-get update && \
+    apt-get install -y software-properties-common && \
+    add-apt-repository ppa:deadsnakes/ppa -y  && \
+    apt install -y python${PY_VERSION} \
+    python${PY_VERSION}-dev \
+    python3-pip
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y \
+    numactl \
+    libxext-dev \
+    libsm-dev \
+    ca-certificates \
+    python3-tk \
+    wget && \
+    python -m pip install requests
+
+COPY benchmarks benchmarks 
+COPY quickstart/common quickstart/common
+COPY quickstart/language_translation/tensorflow/transformer_mlperf/inference/cpu/accuracy.sh quickstart/accuracy.sh
+COPY quickstart/language_translation/tensorflow/transformer_mlperf/inference/cpu/inference_throughput_multi_instance.sh quickstart/inference_throughput.sh
+COPY quickstart/language_translation/tensorflow/transformer_mlperf/inference/cpu/inference_realtime_multi_instance.sh quickstart/inference_realtime.sh
+COPY models models 
+
+COPY LICENSE license/LICENSE
+COPY third_party license/third_party
+
+RUN mkdir -p /workspace/tf-transformer-mlperf-inference/pretrained_model && \
+    wget -qO /workspace/tf-transformer-mlperf-inference/pretrained_model/transformer_mlperf_int8.pb https://storage.googleapis.com/intel-optimized-tensorflow/models/2_10_0/transformer_mlperf_int8.pb && \
+    wget -qO /workspace/tf-transformer-mlperf-inference/pretrained_model/transformer_mlperf_fp32.pb https://storage.googleapis.com/intel-optimized-tensorflow/models/2_10_0/transformer_mlperf_fp32.pb && \
+    wget -qO /workspace/tf-transformer-mlperf-inference/pretrained_model/transformer_mlperf_bf16.pb https://storage.googleapis.com/intel-optimized-tensorflow/models/2_10_0/transformer_mlperf_bf16.pb
diff --git a/docker/tf-cpu/tf-transformer-mlperf-training/tests.yaml b/docker/tf-cpu/tf-transformer-mlperf-training/tests.yaml
new file mode 100644
index 000000000..2b2f16f52
--- /dev/null
+++ b/docker/tf-cpu/tf-transformer-mlperf-training/tests.yaml
@@ -0,0 +1,34 @@
+Transformer Bfloat16  training:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-language-translation-tf-transformer-mlperf-training
+  cmd: /bin/bash quickstart/training.sh
+  env:
+    DATASET_DIR: /tf_dataset/dataset/transformer_uniform_data
+    OUTPUT_DIR: /output/tf-transformer-mlperf-training/bf16
+    PRECISION: bfloat16
+    BATCH_SIZE: '12000'
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/transformer_uniform_data
+    dst: /tf_dataset/dataset/transformer_uniform_data
+  - src: $PWD/output/tf-transformer-mlperf-training/bf16
+    dst: /output/tf-transformer-mlperf-training/bf16
+Transformer FP32 training:
+  img: ${REGISTRY}/aiops/mlops-ci:b-${GITHUB_RUN_NUMBER:-0}-${OS:-ubuntu}-language-translation-tf-transformer-mlperf-training
+  cmd: /bin/bash quickstart/training.sh
+  env:
+    DATASET_DIR: /tf_dataset/dataset/transformer_uniform_data
+    OUTPUT_DIR: /output/tf-transformer-mlperf-training/fp32
+    PRECISION: fp32
+    BATCH_SIZE: '5200'
+    KMP_AFFINITY: granularity=fine,verbose,compact,1,0
+  shm_size: 8G
+  privileged: true
+  init: true
+  volumes:
+  - src: /tf_dataset/dataset/transformer_uniform_data
+    dst: /tf_dataset/dataset/transformer_uniform_data
+  - src: $PWD/output/tf-transformer-mlperf-training/fp32
+    dst: /output/tf-transformer-mlperf-training/fp32
diff --git a/docker/tf-cpu/tf-transformer-mlperf-training/tf-transformer-mlperf-training.Dockerfile-centos b/docker/tf-cpu/tf-transformer-mlperf-training/tf-transformer-mlperf-training.Dockerfile-centos
new file mode 100644
index 000000000..2c476a6cd
--- /dev/null
+++ b/docker/tf-cpu/tf-transformer-mlperf-training/tf-transformer-mlperf-training.Dockerfile-centos
@@ -0,0 +1,88 @@
+# Copyright (c) 2020-2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+#
+# THIS IS A GENERATED DOCKERFILE.
+#
+# This file was assembled from multiple pieces, whose use is documented
+# throughout. Please refer to the TensorFlow dockerfiles documentation
+# for more information.
+
+ARG TF_BASE_IMAGE=intel/intel-optimized-tensorflow-avx512
+ARG TF_BASE_TAG=latest
+
+FROM ${TF_BASE_IMAGE}:${TF_BASE_TAG}
+
+RUN yum update -y && \
+    yum install -y git cmake \
+        ca-certificates \
+        python3-tkinter && \
+    yum clean all
+
+# Install OpenMPI
+ARG OPENMPI_VERSION="openmpi-4.1.0"
+ARG OPENMPI_DOWNLOAD_URL="https://www.open-mpi.org/software/ompi/v4.1/downloads/openmpi-4.1.0.tar.gz"
+
+RUN mkdir /tmp/openmpi && \
+    cd /tmp/openmpi && \
+    curl -fSsL -O ${OPENMPI_DOWNLOAD_URL} && \
+    tar zxf ${OPENMPI_VERSION}.tar.gz && \
+    cd ${OPENMPI_VERSION} && \
+    ./configure --enable-mpirun-prefix-by-default && \
+    make -j $(nproc) all && \
+    make install && \
+    ldconfig && \
+    cd / && \
+    rm -rf /tmp/openmpi
+
+# Create a wrapper for OpenMPI to allow running as root by default
+RUN mv /usr/local/bin/mpirun /usr/local/bin/mpirun.real && \
+    echo '#!/bin/bash' > /usr/local/bin/mpirun && \
+    echo 'mpirun.real --allow-run-as-root "$@"' >> /usr/local/bin/mpirun && \
+    chmod a+x /usr/local/bin/mpirun
+
+# Configure OpenMPI to run good defaults:
+RUN echo "btl_tcp_if_exclude = lo,docker0" >> /usr/local/etc/openmpi-mca-params.conf
+
+# Install OpenSSH for MPI to communicate between containers
+RUN yum update -y && \
+    yum install -y  \
+    openssh-server \
+    openssh-clients && \
+    yum clean all
+
+
+ARG HOROVOD_VERSION=39c8f7c
+ENV HOROVOD_WITHOUT_MXNET=1 \
+    HOROVOD_WITHOUT_PYTORCH=1 \
+    HOROVOD_WITH_TENSORFLOW=1 \
+    HOROVOD_CPU_OPERATIONS=MPI \
+    HOROVOD_WITH_MPI=1 \
+    HOROVOD_WITHOUT_GLOO=1
+
+# Install Horovod
+
+RUN python3 -m pip install git+https://github.com/horovod/horovod.git@${HOROVOD_VERSION}
+
+RUN yum update -y && \
+    yum install -y numactl && \
+    yum clean all
+
+COPY benchmarks benchmarks
+COPY quickstart/common quickstart/common
+COPY quickstart/language_translation/tensorflow/transformer_mlperf/training/cpu/training.sh quickstart/training.sh
+COPY models models 
+
+COPY LICENSE license/LICENSE
+COPY third_party license/third_party
diff --git a/docker/tf-cpu/tf-transformer-mlperf-training/tf-transformer-mlperf-training.Dockerfile-ubuntu b/docker/tf-cpu/tf-transformer-mlperf-training/tf-transformer-mlperf-training.Dockerfile-ubuntu
new file mode 100644
index 000000000..5ed2109be
--- /dev/null
+++ b/docker/tf-cpu/tf-transformer-mlperf-training/tf-transformer-mlperf-training.Dockerfile-ubuntu
@@ -0,0 +1,106 @@
+# Copyright (c) 2020-2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ============================================================================
+#
+# THIS IS A GENERATED DOCKERFILE.
+#
+# This file was assembled from multiple pieces, whose use is documented
+# throughout. Please refer to the TensorFlow dockerfiles documentation
+# for more information.
+
+ARG TF_BASE_IMAGE=intel/intel-optimized-tensorflow-avx512
+ARG TF_BASE_TAG=latest
+
+FROM ${TF_BASE_IMAGE}:${TF_BASE_TAG}
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+ARG PY_VERSION=3.10
+
+RUN apt-get update && \
+    apt-get install -y software-properties-common && \
+    add-apt-repository ppa:deadsnakes/ppa -y  && \
+    apt install -y python${PY_VERSION} \
+    python${PY_VERSION}-dev 
+    
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y curl build-essential 
+
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y git cmake libxext-dev \
+        libsm-dev \
+        ca-certificates \
+        python3-tk && \
+    apt-get clean all
+
+# Install OpenMPI
+ARG OPENMPI_VERSION="openmpi-4.1.0"
+ARG OPENMPI_DOWNLOAD_URL="https://www.open-mpi.org/software/ompi/v4.1/downloads/openmpi-4.1.0.tar.gz"
+
+RUN mkdir /tmp/openmpi && \
+    cd /tmp/openmpi && \
+    curl -fSsL -O ${OPENMPI_DOWNLOAD_URL} && \
+    tar zxf ${OPENMPI_VERSION}.tar.gz && \
+    cd ${OPENMPI_VERSION} && \
+    ./configure --enable-mpirun-prefix-by-default && \
+    make -j $(nproc) all && \
+    make install && \
+    ldconfig && \
+    cd / && \
+    rm -rf /tmp/openmpi
+
+# Create a wrapper for OpenMPI to allow running as root by default
+RUN mv /usr/local/bin/mpirun /usr/local/bin/mpirun.real && \
+    echo '#!/bin/bash' > /usr/local/bin/mpirun && \
+    echo 'mpirun.real --allow-run-as-root "$@"' >> /usr/local/bin/mpirun && \
+    chmod a+x /usr/local/bin/mpirun
+
+# Configure OpenMPI to run good defaults:
+RUN echo "btl_tcp_if_exclude = lo,docker0" >> /usr/local/etc/openmpi-mca-params.conf
+
+# Install OpenSSH for MPI to communicate between containers
+RUN apt-get update && \
+    apt-get install --no-install-recommends --fix-missing -y  \
+    openssh-server \
+    openssh-client && \
+    rm  /etc/ssh/ssh_host_*_key \
+    /etc/ssh/ssh_host_*_key.pub && \
+    apt-get clean all
+
+ARG HOROVOD_VERSION=39c8f7c
+ENV HOROVOD_WITHOUT_MXNET=1 \
+    HOROVOD_WITHOUT_PYTORCH=1 \
+    HOROVOD_WITH_TENSORFLOW=1 \
+    HOROVOD_CPU_OPERATIONS=MPI \
+    HOROVOD_WITH_MPI=1 \
+    HOROVOD_WITHOUT_GLOO=1
+
+# Install Horovod
+
+RUN python -m pip install git+https://github.com/horovod/horovod.git@${HOROVOD_VERSION}
+
+COPY benchmarks benchmarks
+COPY quickstart/common quickstart/common
+COPY quickstart/language_translation/tensorflow/transformer_mlperf/training/cpu/training.sh quickstart/training.sh
+COPY models models 
+
+COPY LICENSE license/LICENSE
+COPY third_party license/third_party
+
+COPY docker/tf-cpu/generate_ssh_key.sh /workspace/generate_ssh_key.sh
+
+RUN cat /workspace/generate_ssh_key.sh >> ${HOME}/.bash_profile && \
+    cat /workspace/generate_ssh_key.sh >> ${HOME}/.bashrc
+
+ENV BASH_ENV=/root/.bash_profile
diff --git a/docs/general/EMR_DEVCATALOG.md b/docs/general/EMR_DEVCATALOG.md
new file mode 100644
index 000000000..2c6b95182
--- /dev/null
+++ b/docs/general/EMR_DEVCATALOG.md
@@ -0,0 +1,64 @@
+# Optimized AI Reference Model Workloads on 5th Generation Intel® Xeon® Scalable processors
+
+This document provides links to step-by-step instructions on how to leverage Reference model docker containers to run optimized open-source Deep Learning Training and Inference workloads using PyTorch* and TensorFlow* frameworks on 5th Generation Intel® Xeon® Scalable processors
+
+**Note:** The containers below are finely tuned to demonstrate best performance on Intel® Extension for PyTorch* and Intel®  Optimized TensorFlow*  and are not intended for use in production. 
+## Use cases
+
+The tables below link to documentation on how to run each use case using docker containers. These containers were validated on a host running Linux. 
+
+### Image Recognition
+
+| Framework | Model                                                  | Precisions | Mode |  Dataset |
+| --------| ------------------------------------------------------ | ---------- | ------| --------------------- |
+| PyTorch | [ResNet 50](https://github.com/IntelAI/models/blob/r3.1/quickstart/image_recognition/pytorch/resnet50/training/cpu/DEVCATALOG.md) | FP32,BF32,BF16 | Training | ImageNet 2012 |
+| PyTorch | [ResNet 50](https://github.com/IntelAI/models/blob/r3.1/quickstart/image_recognition/pytorch/resnet50/inference/cpu/DEVCATALOG.md) | FP32,BF32,BF16,INT8 | Inference | ImageNet 2012 |
+| PyTorch | [ResNext-32x16d](https://github.com/IntelAI/models/blob/r3.1/quickstart/image_recognition/pytorch/resnext-32x16d/inference/cpu/DEVCATALOG.md) | FP32,BF32,BF16,INT8 |Inference | ImageNet 2012 |
+| TensorFlow | [ResNet 50v1.5](https://github.com/IntelAI/models/blob/r3.1/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/README_DEV_CAT.md) | FP32,BF16,FP16 | Training | ImageNet 2012 |
+| TensorFlow | [ResNet 50v1.5](https://github.com/IntelAI/models/blob/r3.1/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/README_DEV_CAT.md) | FP32,BF32,BF16,INT8 | Inference | ImageNet 2012 |
+| TensorFlow | [MobileNet V1*](https://github.com/IntelAI/models/blob/r3.1/quickstart/image_recognition/tensorflow/mobilenet_v1/inference/cpu/README_DEV_CAT.md) | FP32,BF32,FP16,INT8 | Inference | ImageNet 2012 |
+
+## Image Segmentation
+
+| Framework | Model                                                  | Precisions | Mode |  Dataset |
+| --------| ------------------------------------------------------ | ---------- | ------| --------------------- |
+| TensorFlow | [3D U-Net MLPerf*](https://github.com/IntelAI/models/blob/r3.1/quickstart/image_segmentation/tensorflow/3d_unet_mlperf/inference/cpu/README_DEV_CAT.md) | FP32,BF16,INT8 Inference | BRATS 2019 |
+
+### Object Detection
+
+| Framework | Model                                                  | Precisions | Mode |  Dataset |
+| --------| ------------------------------------------------------ | ---------- | ------| --------------------- |
+| PyTorch |[Mask R-CNN](https://github.com/IntelAI/models/blob/r3.1/quickstart/object_detection/pytorch/maskrcnn/training/cpu/DEVCATALOG.md) | FP32,BF32,BF16 | Training | COCO 2017 |
+| PyTorch |[Mask R-CNN](https://github.com/IntelAI/models/blob/r3.1/quickstart/object_detection/pytorch/maskrcnn/inference/cpu/DEVCATALOG.md) | FP32,BF32,BF16 | Inference | COCO 2017 |
+| PyTorch |[SSD-ResNet34](https://github.com/IntelAI/models/blob/r3.1/quickstart/object_detection/pytorch/ssd-resnet34/training/cpu/DEVCATALOG.md) | FP32,BF32,BF16 | Training | COCO 2017 |
+| PyTorch |[SSD-ResNet34](https://github.com/IntelAI/models/blob/r3.1/quickstart/object_detection/pytorch/ssd-resnet34/inference/cpu/DEVCATALOG.md) | FP32,BF32,BF16,INT8 | Inference | COCO 2017 |
+| TensorFlow | [SSD-ResNet34](https://github.com/IntelAI/models/blob/r3.1/quickstart/object_detection/tensorflow/ssd-resnet34/training/cpu/README_DEV_CAT.md) | FP32,BF32,BF16 |Training | COCO 2017 |
+| TensorFlow | [SSD-ResNet34](https://github.com/IntelAI/models/blob/r3.1/quickstart/object_detection/tensorflow/ssd-resnet34/inference/cpu/README_DEV_CAT.md) | FP32,BF16,INT8 |Inference | COCO 2017  |
+| TensorFlow | [SSD-MobileNet*](https://github.com/IntelAI/models/blob/r3.1/quickstart/object_detection/tensorflow/ssd-mobilenet/inference/cpu/README_DEV_CAT.md) | FP32,BF32,BF16,INT8 | Inference | COCO 2017 |
+
+### Language Modeling 
+
+| Framework | Model                                                  | Precisions | Mode |  Dataset |
+| --------| ------------------------------------------------------ | ---------- | ------| --------------------- |
+| PyTorch | [BERT large](https://github.com/IntelAI/models/blob/r3.1/quickstart/language_modeling/pytorch/bert_large/training/cpu/DEVCATALOG.md) | FP32,BF32,BF16 | Training | Preprocessed Text dataset |
+| PyTorch |[BERT large](https://github.com/IntelAI/models/blob/r3.1/quickstart/language_modeling/pytorch/bert_large/inference/cpu/DEVCATALOG.md) | FP32,BF32,BF16,INT8 | Inference | SQuAD1.0 |
+| PyTorch | [RNN-T](https://github.com/IntelAI/models/blob/r3.1/quickstart/language_modeling/pytorch/rnnt/inference/cpu/DEVCATALOG.md) | FP32,BF32,BF16,INT8 | Inference | LibriSpeech |
+| PyTorch |[RNN-T](https://github.com/IntelAI/models/blob/r3.1/quickstart/language_modeling/pytorch/rnnt/training/cpu/DEVCATALOG.md) | FP32,BF32,FP16 | Training | LibriSpeech |
+| PyTorch |[DistilBERT base](https://github.com/IntelAI/models/blob/r3.1/quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/DEVCATALOG.md) | FP32,BF32,BF16,INT8-BF16,INT8-BF32 | Inference | SST-2 |
+| TensorFlow | [BERT large](https://github.com/IntelAI/models/blob/r3.1/quickstart/language_modeling/tensorflow/bert_large/training/cpu/README_DEV_CAT.md) | FP32,BF16 | Training |  SQuAD and MRPC |
+| TensorFlow | [BERT large](https://github.com/IntelAI/models/blob/r3.1/quickstart/language_modeling/tensorflow/bert_large/inference/cpu/README_DEV_CAT.md) | FP32,BF32,BF16,INT8 |Inference | SQuAD |
+| TensorFlow | [DistilBERT Base](https://github.com/IntelAI/models/blob/r3.1/quickstart/language_modeling/tensorflow/distilbert_base/inference/cpu/README_DEV_CAT.md) | FP32,BF16,INT8 | Inference | SST-2 | 
+
+## Language Translation
+| Framework | Model                                                  | Precisions | Mode |  Dataset |
+| --------| ------------------------------------------------------ | ---------- | ------| --------------------- |
+| TensorFlow | [Transformer_LT_mlperf*](https://github.com/IntelAI/models/blob/r3.1/quickstart/language_translation/tensorflow/transformer_mlperf/training/cpu/README_DEV_CAT.md) | FP32,BF16 | Training | WMT English-German dataset |
+| TensorFlow | [Transformer_LT_mlperf*](https://github.com/IntelAI/models/blob/r3.1/quickstart/language_translation/tensorflow/transformer_mlperf/inference/cpu/README_DEV_CAT.md) | FP32,BF32,BF16,INT8 | Inference |  WMT English-German dataset |
+
+### Recommendation 
+
+| Framework | Model                                                  | Precisions | Mode |  Dataset |
+| --------| ------------------------------------------------------ | ---------- | ------| --------------------- |
+| PyTorch | [DLRM](https://github.com/IntelAI/models/blob/r3.1/quickstart/recommendation/pytorch/dlrm/training/cpu/DEVCATALOG.md) | FP32,BF32,BF16 | Training | Criteo Terabyte |
+| PyTorch | [DLRM](https://github.com/IntelAI/models/blob/r3.1/quickstart/recommendation/pytorch/dlrm/inference/cpu/DEVCATALOG.md) | FP32,BF32,BF16,INT8 | Inference | Criteo Terabyte |
+| TensorFlow | [DIEN](https://github.com/IntelAI/models/blob/r3.1/quickstart/recommendation/tensorflow/dien/inference/cpu/README_DEV_CAT.md) | FP32,BF32,BF16 | Inference | DIEN dataset |
diff --git a/docs/general/FLEX_DEVCATALOG.md b/docs/general/FLEX_DEVCATALOG.md
index 3f805d148..2aab29124 100644
--- a/docs/general/FLEX_DEVCATALOG.md
+++ b/docs/general/FLEX_DEVCATALOG.md
@@ -1,31 +1,38 @@
-# Model Zoo for Intel® Architecture Workloads Optimized for the Intel® Data Center GPU Flex Series
+# Optimized Intel® Reference Models for the Intel® Data Center GPU Flex Series
 
-This document provides links to step-by-step instructions on how to leverage Model Zoo docker containers to run optimized open-source Deep Learning inference workloads using Intel® Extension for PyTorch* and Intel® Extension for TensorFlow* on the [Intel® Data Center GPU Flex Series](https://www.intel.com/content/www/us/en/products/details/discrete-gpus/data-center-gpu/flex-series.html).
+This document provides links to step-by-step instructions on how to leverage reference model docker containers to run optimized open-source Deep Learning inference workloads using Intel® Extension for PyTorch* and Intel® Extension for TensorFlow* on the [Intel® Data Center GPU Flex Series](https://www.intel.com/content/www/us/en/products/details/discrete-gpus/data-center-gpu/flex-series.html).
 
 ## Base Containers
 
 | AI Framework                 | Extension            | Documentation |
 | -----------------------------| ------------- | ----------------- |
-| PyTorch | Intel® Extension for PyTorch* | [Intel® Extension for PyTorch Container](https://github.com/intel/intel-extension-for-pytorch/blob/v2.0.110%2Bxpu/docker/README.md) |
-| TensorFlow | Intel® Extension for TensorFlow* | [Intel® Extension for TensorFlow Container](https://github.com/intel/intel-extension-for-tensorflow/blob/v2.13.0.0/docker/README.md)|
+| PyTorch | Intel® Extension for PyTorch* | [Intel® Extension for PyTorch Container](https://github.com/intel/intel-extension-for-pytorch/blob/v2.1.10%2Bxpu/docker/README.md) |
+| TensorFlow | Intel® Extension for TensorFlow* | [Intel® Extension for TensorFlow Container](https://github.com/intel/intel-extension-for-tensorflow/blob/v2.14.0.1/docker/README.md)|
 
 ## Optimized Workloads
 
 The table below provides links to run each workload in a docker container. The containers are optimized for Linux*. 
 
 
-| Model                            | Framework                  | Mode and Documentation     |  Dataset |
-| ----------------------------|     ---------- | ----------| ------------ |
-| [ResNet 50 v1.5](https://github.com/tensorflow/models/tree/v2.12.1/official/legacy/image_classification/resnet) | TensorFlow | [INT8 Inference](https://github.com/IntelAI/models/blob/v2.12.1/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/gpu/DEVCATALOG_FLEX.md) | [ImageNet 2012](https://github.com/IntelAI/models/tree/v2.12.1/datasets/imagenet/README.md) |
-| [MaskRCNN](https://arxiv.org/abs/1703.06870) | TensorFlow | [FP16 Inference](https://github.com/IntelAI/models/blob/v2.12.1/quickstart/image_segmentation/tensorflow/maskrcnn/inference/gpu/DEVCATALOG.md) | [COCO 2017](https://github.com/IntelAI/models/blob/v2.12.1/quickstart/image_segmentation/tensorflow/maskrcnn/inference/gpu/DEVCATALOG.md#download-dataset) |
-| [EfficientNet](https://arxiv.org/abs/1905.11946) B0,B3 | TensorFlow| [FP16 Inference](https://github.com/IntelAI/models/blob/v2.12.1/quickstart/image_recognition/tensorflow/efficientnet/inference/gpu/DEVCATALOG.md) | Dummy Image
-| [Stable Diffusion](https://arxiv.org/abs/2112.10752) | TensorFlow | [FP32,FP16 Inference](https://github.com/IntelAI/models/blob/v2.12.1/quickstart/generative-ai/tensorflow/stable_diffusion/inference/gpu/DEVCATALOG.md) | Text prompts |
-| [ResNet 50 v1.5](https://arxiv.org/pdf/1512.03385.pdf) | PyTorch | [INT8 Inference](https://github.com/IntelAI/models/blob/v2.12.1/quickstart/image_recognition/pytorch/resnet50v1_5/inference/gpu/DEVCATALOG_FLEX.md) | [ImageNet 2012](https://github.com/IntelAI/models/tree/v2.12.1/datasets/imagenet/README.md) |
-| [YOLOv5](https://ui.adsabs.harvard.edu/abs/2021zndo...4679653J/abstract) | PyTorch | [FP16 Inference](https://github.com/IntelAI/models/blob/v2.12.1/quickstart/object_detection/pytorch/yolov5/inference/gpu/DEVCATALOG.md) | Dummy Image |
-| [Stable Diffusion](https://arxiv.org/abs/2112.10752) | PyTorch | [FP32,FP16 Inference](https://github.com/IntelAI/models/blob/v2.12.1/quickstart/generative-ai/pytorch/stable_diffusion/inference/gpu/DEVCATALOG.md) | Text prompts
-| [SSD-MobileNet](https://arxiv.org/pdf/1704.04861.pdf) | TensorFlow | [INT8 Inference](https://github.com/IntelAI/models/blob/v2.11.1/quickstart/object_detection/tensorflow/ssd-mobilenet/inference/gpu/DEVCATALOG.md) | [COCO 2017](https://github.com/IntelAI/models/tree/v2.11.1/datasets/coco#download-and-preprocess-the-coco-validation-images) |
-| [SSD-MobileNet v1](https://arxiv.org/pdf/1704.04861.pdf) | PyTorch | [INT8 Inference](https://github.com/IntelAI/models/blob/v2.11.1/quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/DEVCATALOG.md) | [COCO 2017](https://github.com/IntelAI/models/blob/v2.11.1/quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/README.md#datasets)  |
-| [YOLOv4](https://arxiv.org/pdf/1704.04861.pdf) | PyTorch | [INT8 Inference](https://github.com/IntelAI/models/blob/v2.11.1/quickstart/object_detection/pytorch/yolov4/inference/gpu/DEVCATALOG.md) | [COCO 2017](https://github.com/IntelAI/models/blob/v2.11.1/quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/README.md#datasets) |
-
-
-**Note**: SSD-MobileNet and YOLOv4 models are supported on older Intel® Extension for TensorFlow* v2.12 and Intel® Extension for PyTorch* 1.13.100+xpu versions. The other models in the list are validated on Intel® Extension for TensorFlow* v2.13 and Intel® Extension for PyTorch* 2.0.100+xpu versions.
+| Model                            | Framework                  | Mode and Documentation     |
+| ----------------------------|     ---------- | ----------|
+| [DistilBERT](https://arxiv.org/abs/1910.01108) | PyTorch | [FP16 and FP32  Inference](../../models_v2/pytorch/distilbert/inference/gpu/CONTAINER_FLEX.md) |
+| [DLRM-v1](https://arxiv.org/abs/1906.00091) | PyTorch | [ FP16 Inference](../../models_v2/pytorch/dlrm/inference/gpu/CONTAINER.md) | 
+| [EfficientNet](https://arxiv.org/abs/1905.11946) B0,B3,B4 | TensorFlow | [FP16 Inference](../../models_v2/tensorflow/efficientnet/inference/gpu/CONTAINER.md) |
+| [Mask R-CNN](https://arxiv.org/abs/1703.06870) | TensorFlow | [FP16 Inference](../../models_v2/tensorflow/maskrcnn/inference/gpu/CONTAINER.md) |
+| [ResNet50 v1.5](https://arxiv.org/pdf/1512.03385.pdf) | PyTorch | [INT8 Inference](../../quickstart/image_recognition/pytorch/resnet50v1_5/inference/gpu/DEVCATALOG_FLEX.md) | 
+| [ResNet50 v1.5](https://arxiv.org/pdf/1512.03385.pdf) | TensorFlow | [INT8 Inference](../../models_v2/tensorflow/resnet50v1_5/inference/gpu/CONTAINER_FLEX.md) | 
+| [SSD-MobileNet v1](https://arxiv.org/pdf/1704.04861.pdf) | TensorFlow | [INT8 Inference](../../quickstart/object_detection/tensorflow/ssd-mobilenet/inference/gpu/DEVCATALOG.md) | 
+| [SSD-MobileNet v1](https://arxiv.org/pdf/1704.04861.pdf) | PyTorch | [INT8 Inference](../../quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/DEVCATALOG.md) | 
+| [Stable Diffusion](https://arxiv.org/abs/2112.10752) | PyTorch | [Flex GPU FP16 Inference](../../models_v2/pytorch/stable_diffusion/inference/gpu/CONTAINER_FLEX.md) and [Arc GPU FP16,FP32 Inference](../../quickstart/generative-ai/pytorch/stable_diffusion/inference/gpu/WSL2_ARC_STABLE_DIFFUSION.md) |
+| [Stable Diffusion](https://arxiv.org/abs/2112.10752) | TensorFlow | [FP32,FP16 Inference](../../models_v2/tensorflow/stable_diffusion/inference/gpu/CONTAINER.md) | 
+| [Wide and Deep](https://arxiv.org/abs/1606.07792) | TensorFlow | [FP16 Inference](../../models_v2/tensorflow/wide_deep_large_ds/inference/gpu/CONTAINER.md) |
+| [YOLOv4](https://arxiv.org/pdf/1704.04861.pdf) | PyTorch | [INT8 Inference](../../quickstart/object_detection/pytorch/yolov4/inference/gpu/DEVCATALOG.md) |
+| [YOLOv5](https://ui.adsabs.harvard.edu/abs/2021zndo...4679653J/abstract) | PyTorch | [FP16 Inference](../../quickstart/object_detection/pytorch/yolov5/inference/gpu/DEVCATALOG.md) | 
+
+
+**Note**: 
+* SSD-MobileNetv1, and YOLOv4 models are supported on older Intel® Extension for TensorFlow* v2.12 and Intel® Extension for PyTorch* 1.13.120+xpu versions. 
+* ResNet50 v1.5(PyTorch) and YOLOv5 models are supported on older Intel® Extension for PyTorch* 2.0.110+xpu versions.
+* Stable Diffusion for Arc GPU was validated on PyTorch* 2.0.110+xpu version.
+* The other models in the list are validated on Intel® Extension for TensorFlow* v2.14 and Intel® Extension for PyTorch* 2.1.10+xpu versions.
diff --git a/docs/general/MAX_DEVCATALOG.md b/docs/general/MAX_DEVCATALOG.md
index 8e2ac6776..940114809 100644
--- a/docs/general/MAX_DEVCATALOG.md
+++ b/docs/general/MAX_DEVCATALOG.md
@@ -1,30 +1,35 @@
-# Model Zoo for Intel® Architecture Workloads Optimized for the Intel® Data Center GPU Max Series
+# Optimized Intel® Reference Models for Intel® Data Center GPU Max Series
 
-This document provides links to step-by-step instructions on how to leverage Model Zoo docker containers to run optimized open-source Deep Learning training and inference workloads using Intel® Extension for PyTorch* and Intel® Extension for TensorFlow* on the [Intel® Data Center GPU Max Series](https://www.intel.com/content/www/us/en/products/details/discrete-gpus/data-center-gpu/max-series.html).
+This document provides links to step-by-step instructions on how to leverage reference model docker containers to run optimized open-source Deep Learning training and inference workloads using Intel® Extension for PyTorch* and Intel® Extension for TensorFlow* on the [Intel® Data Center GPU Max Series](https://www.intel.com/content/www/us/en/products/details/discrete-gpus/data-center-gpu/max-series.html).
 
 ## Base Containers
 
 | AI Framework                 | Extension            | Documentation |
 | -----------------------------| ------------- | ----------------- |
-| PyTorch | Intel® Extension for PyTorch* | [Intel® Extension for PyTorch Container](https://github.com/intel/intel-extension-for-pytorch/blob/v2.0.110%2Bxpu/docker/README.md) |
-| TensorFlow | Intel® Extension for TensorFlow* | [Intel® Extension for TensorFlow Container](https://github.com/intel/intel-extension-for-tensorflow/blob/v2.13.0.0/docker/README.md)|
+| PyTorch | Intel® Extension for PyTorch* | [Intel® Extension for PyTorch Container](https://github.com/intel/intel-extension-for-pytorch/blob/v2.1.10%2Bxpu/docker/README.md) |
+| TensorFlow | Intel® Extension for TensorFlow* | [Intel® Extension for TensorFlow Container](https://github.com/intel/intel-extension-for-tensorflow/blob/v2.14.0.1/docker/README.md)|
 
 ## Optimized Workloads
 
 The table below provides links to run each workload in a docker container. The containers are optimized for Linux*. 
 
-| Model                            | Framework                  | Mode | Precisions | Dataset |
-| ----------------------------|     ---------- | ------------------- | ------------ | ------------ |
-| [ResNet50v1-5](https://arxiv.org/pdf/1512.03385.pdf) | PyTorch | [Inference](https://github.com/IntelAI/models/blob/v3.0.0/quickstart/image_recognition/pytorch/resnet50v1_5/inference/gpu/DEVCATALOG_MAX.md) | INT8 | [ImageNet 2012](https://github.com/IntelAI/models/tree/v3.0.0/datasets/imagenet/README.md) |
-| [BERT Large](https://arxiv.org/pdf/1810.04805.pdf)                                           | PyTorch | [Inference](https://github.com/IntelAI/models/blob/v3.0.0/quickstart/language_modeling/pytorch/bert_large/inference/gpu/DEVCATALOG.md) | FP16 | [SQuAD](https://github.com/IntelAI/models/blob/v3.0.0/datasets/bert_data/README.md#inference) |
-| [DLRM](https://arxiv.org/abs/1906.00091) | PyTorch | [Inference](https://github.com/IntelAI/models/blob/v3.0.0/quickstart/recommendation/pytorch/torchrec_dlrm/inference/gpu/DEVCATALOG.md) | FP16 | [Criteo](https://ailab.criteo.com/ressources/criteo-1tb-click-logs-dataset-for-mlperf) |
-| [DLRM](https://arxiv.org/abs/1906.00091) | PyTorch | [Training](https://github.com/IntelAI/models/blob/v3.0.0/quickstart/recommendation/pytorch/torchrec_dlrm/training/gpu/DEVCATALOG.md) | BF16 | [Criteo](https://ailab.criteo.com/ressources/criteo-1tb-click-logs-dataset-for-mlperf) | 
-| [ResNet50v1-5](https://arxiv.org/pdf/1512.03385.pdf) | TensorFlow| [Inference](https://github.com/IntelAI/models/blob/v3.0.0/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/gpu/DEVCATALOG_MAX.md) | INT8,FP16,FP32 | [ImageNet 2012](https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md) |
-| [BERT Large](https://arxiv.org/pdf/1810.04805.pdf)                                           | TensorFlow | [Inference](https://github.com/IntelAI/models/blob/v3.0.0/quickstart/language_modeling/tensorflow/bert_large/inference/gpu/DEVCATALOG.md) | FP16,FP32 | [SQuAD ](https://github.com/IntelAI/models/blob/v3.0.0/datasets/bert_data/README.md#inference) |
-| [ResNet50v1-5](https://arxiv.org/pdf/1512.03385.pdf) | TensorFlow | [Training](https://github.com/IntelAI/models/blob/r2.12/quickstart/image_recognition/tensorflow/resnet50v1_5/training/gpu/DEVCATALOG.md) |  BF16 | [ImageNet 2012](https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md) |
-| [ResNet50v1-5](https://arxiv.org/pdf/1512.03385.pdf) | PyTorch | [Training](https://github.com/IntelAI/models/blob/r2.12/quickstart/image_recognition/pytorch/resnet50v1_5/training/gpu/DEVCATALOG.md) | BF16 |  [ImageNet 2012](https://github.com/IntelAI/models/tree/master/datasets/imagenet/README.md) |
-| [BERT Large](https://arxiv.org/pdf/1810.04805.pdf)                                           | TensorFlow | [Training](https://github.com/IntelAI/models/blob/r2.12/quickstart/language_modeling/tensorflow/bert_large/training/gpu/DEVCATALOG.md) | BF16 | Dummy dataset |
-| [BERT Large](https://arxiv.org/pdf/1810.04805.pdf)                                           | PyTorch | [Training](https://github.com/IntelAI/models/blob/r2.12/quickstart/language_modeling/pytorch/bert_large/training/gpu/DEVCATALOG.md) | BF16 | [MLCommons](https://github.com/IntelAI/models/blob/master/quickstart/language_modeling/pytorch/bert_large/training/gpu/DEVCATALOG.md#datasets) |
-
-
-**Note**: ResNet50v1.5 and BERT-Large training workloads are supported on older Intel® Extension for TensorFlow* v2.12 and Intel® Extension for PyTorch* 1.13.100+xpu versions. The other models in the list are validated on Intel® Extension for TensorFlow* v2.13 and Intel® Extension for PyTorch* 2.0.100+xpu versions.
+| Model                            | Framework                  | Mode | Precisions | 
+| ----------------------------|     ---------- | ------------------- | ------------ |
+| [3D-UNet](https://arxiv.org/abs/1606.06650) | TensorFlow | [Training](../../models_v2/tensorflow/3d_unet/training/gpu/CONTAINER.md) | BF16 |
+| [BERT Large](https://arxiv.org/pdf/1810.04805.pdf)                                           | PyTorch | [Inference](../../models_v2/pytorch/bert_large/inference/gpu/CONTAINER.md) | FP16, BF16 and FP32 | 
+| [BERT Large](https://arxiv.org/pdf/1810.04805.pdf)                                           | PyTorch | [Training](../../models_v2/pytorch/bert_large/training/gpu) | BF16,TF32 and FP32 |
+| [BERT Large](https://arxiv.org/pdf/1810.04805.pdf)                                           | TensorFlow | [Inference](../../quickstart/language_modeling/tensorflow/bert_large/inference/gpu/DEVCATALOG.md) | FP16,FP32 |
+| [BERT Large](https://arxiv.org/pdf/1810.04805.pdf)                                           | TensorFlow | [Training](../../models_v2/tensorflow/bert_large/training/gpu/CONTAINER.md) | BF16 |
+| [DistilBERT](https://arxiv.org/abs/1910.01108) | PyTorch | [Inference](../../models_v2/pytorch/distilbert/inference/gpu/CONTAINER_MAX.md) | FP16,BF16 and FP32 |
+| [DLRM](https://arxiv.org/abs/1906.00091) | PyTorch | [Inference](../../quickstart/recommendation/pytorch/torchrec_dlrm/inference/gpu/DEVCATALOG.md) | FP16 | 
+| [DLRM](https://arxiv.org/abs/1906.00091) | PyTorch | [Training](../../quickstart/recommendation/pytorch/torchrec_dlrm/training/gpu/DEVCATALOG.md) | BF16 | 
+| [Mask R-CNN](https://arxiv.org/abs/1703.06870) | TensorFlow | [Training](../../models_v2/tensorflow/maskrcnn/training/gpu/CONTAINER.md) | BF16 |
+| [ResNet50 v1.5](https://arxiv.org/pdf/1512.03385.pdf) | PyTorch | [Inference](../../models_v2/pytorch/resnet50v1_5/inference/gpu/CONTAINER_MAX.md) | INT8,FP16,BF16,FP32 and TF32 | 
+| [ResNet50 v1.5](https://arxiv.org/pdf/1512.03385.pdf) | PyTorch | [Training](../../models_v2/pytorch/resnet50v1_5/training/gpu/CONTAINER.md) | BF16,FP32 and TF32 | 
+| [ResNet50 v1.5](https://arxiv.org/pdf/1512.03385.pdf) | TensorFlow | [Training](../../models_v2/tensorflow/resnet50v1_5/training/gpu/CONTAINER.md) |  BF16 | 
+| [ResNet50 v1.5](https://arxiv.org/pdf/1512.03385.pdf) | TensorFlow | [Inference](../../quickstart/image_recognition/tensorflow/resnet50v1_5/inference/gpu/DEVCATALOG_MAX.md) | INT8,FP16,FP32 | 
+| [Stable Diffusion](https://arxiv.org/abs/2112.10752) | PyTorch | [Inference](../../models_v2/pytorch/stable_diffusion/inference/gpu/CONTAINER_MAX.md) | FP16 |
+
+**Note**: 
+* DLRM(PyTorch) training and inference, BERT-Large(TensorFlow) inference and ResNet50v1.5(TensorFlow)    inference workloads are supported on older Intel® Extension for TensorFlow* v2.13 and Intel® Extension for PyTorch* 2.0.110+xpu versions. 
+* The other models in the list are validated on Intel® Extension for TensorFlow* v2.14 and Intel® Extension for PyTorch* 2.1.10+xpu versions.
diff --git a/docs/general/pytorch/BareMetalSetup.md b/docs/general/pytorch/BareMetalSetup.md
index 4734f3fb9..884fb90bd 100644
--- a/docs/general/pytorch/BareMetalSetup.md
+++ b/docs/general/pytorch/BareMetalSetup.md
@@ -6,11 +6,23 @@ Prepare the environment, you may create a Python virtual enviromment `virtualenv
     # Install torch,torchvision
     python -m pip install torch torchvision
 
-## The following components are required by some PyTorch workloads. Only build them if indicated in the documentation for that workload.
+## Install generic dependencies
+Make sure the following components are installed in your environment :
 
     # Requirements:
     gcc >= 5
     Cmake >= 3.19.6
+    build-essential 
+    ca-certificates 
+    git 
+    wget 
+    make 
+    cmake 
+    autoconf 
+    bzip2 
+    tar
+    numactl 
+    libegl1-mesa 
 
 ### Install jemalloc
     Install jemalloc either using conda or from source
@@ -40,7 +52,8 @@ Prepare the environment, you may create a Python virtual enviromment `virtualenv
 ### Build torch-ccl 
     cd ..
     git clone https://github.com/intel-innersource/frameworks.ai.pytorch.torch-ccl.git
-    cd frameworks.ai.pytorch.torch-ccl && git checkout public_master
+    # torch-ccl branch refer to https://github.com/intel-innersource/frameworks.ai.pytorch.ipex-cpu/blob/cpu-device/dependency_version.yml
+    cd frameworks.ai.pytorch.torch-ccl && git checkout $target_branch
     git submodule sync 
     git submodule update --init --recursive
     python setup.py install 
diff --git a/docs/notebooks/transfer_learning/README.md b/docs/notebooks/transfer_learning/README.md
index 03875c4d7..de55a5890 100644
--- a/docs/notebooks/transfer_learning/README.md
+++ b/docs/notebooks/transfer_learning/README.md
@@ -13,17 +13,17 @@ and [Intel® Extension for PyTorch](https://github.com/intel/intel-extension-for
 
 | Notebook | Domain: Use Case | Framework| Description |
 | ---------| ---------|----------|-------------|
-| [BERT SQuAD fine tuning with TF Hub](/docs/notebooks/transfer_learning/tfhub_bert/question_answering) | NLP: Question Answering | TensorFlow | Demonstrates BERT fine tuning using scripts from the [TensorFlow Model Garden](https://github.com/tensorflow/models) and the [SQuAD dataset](https://rajpurkar.github.io/SQuAD-explorer/). The notebook allows for selecting a BERT large or BERT base model from [TF Hub](https://tfhub.dev). The fine tuned model is evaluated and exported as a saved model. |
-| [BERT Text Classification with TF Hub](/docs/notebooks/transfer_learning/tfhub_bert/text_classification) | NLP: Text Classification | TensorFlow | Demonstrates BERT binary text classification fine tuning using the [IMDb movie review dataset](https://www.tensorflow.org/datasets/catalog/imdb_reviews) and multiclass text classification fine tuning using the [AG News datasets](https://www.tensorflow.org/datasets/catalog/ag_news_subset) from [TensorFlow Datasets](https://www.tensorflow.org/datasets) or a custom dataset (for binary classification). The notebook allows for selecting a BERT encoder (BERT large, BERT base, or small BERT) to use along with a preprocessor from [TF Hub](https://tfhub.dev). The fine tuned model is evaluated and exported as a saved model. |
-| [Text Classifier fine tuning with PyTorch & Hugging Face](/docs/notebooks/transfer_learning/pytorch_text_classification) | NLP: Text Classification | PyTorch |Demonstrates fine tuning [Hugging Face models](https://huggingface.co/models) to do sentiment analysis using the [IMDb movie review dataset from Hugging Face Datasets](https://huggingface.co/datasets/imdb) or a custom dataset with [Intel® Extension for PyTorch*](https://github.com/intel/intel-extension-for-pytorch) |
-| [LLM instruction tuning with PyTorch & Hugging Face](/docs/notebooks/transfer_learning/pytorch_text_generation) | NLP: Text Generation | PyTorch | Demonstrates instruction tuning [Hugging Face Causal Language Models](https://huggingface.co/docs/transformers/tasks/language_modeling) to do text generation using [Hugging Face Datasets](https://huggingface.co/datasets/imdb) or a custom dataset |
+| [BERT SQuAD fine tuning with TF Hub](/docs/notebooks/transfer_learning/text_classification/bert_classifier_fine_tuning) | NLP: Question Answering | TensorFlow | Demonstrates BERT fine tuning using scripts from the [TensorFlow Model Garden](https://github.com/tensorflow/models) and the [SQuAD dataset](https://rajpurkar.github.io/SQuAD-explorer/). The notebook allows for selecting a BERT large or BERT base model from [TF Hub](https://tfhub.dev). The fine tuned model is evaluated and exported as a saved model. |
+| [BERT Text Classification with TF Hub](/docs/notebooks/transfer_learning/text_classification/tfhub_bert_text_classification/) | NLP: Text Classification | TensorFlow | Demonstrates BERT binary text classification fine tuning using the [IMDb movie review dataset](https://www.tensorflow.org/datasets/catalog/imdb_reviews) and multiclass text classification fine tuning using the [AG News datasets](https://www.tensorflow.org/datasets/catalog/ag_news_subset) from [TensorFlow Datasets](https://www.tensorflow.org/datasets) or a custom dataset (for binary classification). The notebook allows for selecting a BERT encoder (BERT large, BERT base, or small BERT) to use along with a preprocessor from [TF Hub](https://tfhub.dev). The fine tuned model is evaluated and exported as a saved model. |
+| [Text Classifier fine tuning with PyTorch & Hugging Face](/docs/notebooks/transfer_learning/text_classification/pytorch_text_classification) | NLP: Text Classification | PyTorch |Demonstrates fine tuning [Hugging Face models](https://huggingface.co/models) to do sentiment analysis using the [IMDb movie review dataset from Hugging Face Datasets](https://huggingface.co/datasets/imdb) or a custom dataset with [Intel® Extension for PyTorch*](https://github.com/intel/intel-extension-for-pytorch) |
+| [LLM instruction tuning with PyTorch & Hugging Face](/docs/notebooks/transfer_learning/text_generation/pytorch_text_generation) | NLP: Text Generation | PyTorch | Demonstrates instruction tuning [Hugging Face Causal Language Models](https://huggingface.co/docs/transformers/tasks/language_modeling) to do text generation using [Hugging Face Datasets](https://huggingface.co/datasets/imdb) or a custom dataset |
 
 
 ## Computer Vision
 
-| Notebook | Framework | Description |
-| ---------| ----------|-------------|
-| [Image Classification with TF Hub](/docs/notebooks/transfer_learning/tf_image_classification) | CV: Image Classification | TensorFlow | Demonstrates transfer learning with multiple [TF Hub](https://tfhub.dev) image classifiers, TF datasets, and custom image datasets |
-| [Image Classification with PyTorch & Torchvision](/docs/notebooks/transfer_learning/pytorch_image_classification) | CV: Image Classification | PyTorch | Demonstrates transfer learning with multiple [Torchvision](https://pytorch.org/vision/stable/index.html) image classification models, Torchvision datasets, and custom datasets |
-| [Image Classification with Hugging Face](/docs/notebooks/transfer_learning/huggingface_image_classification) | CV: Image Classification | PyTorch, TensorFlow | Demonstrates transfer learning with [Hugging Face models](https://huggingface.co/models) for image classification using either TensorFlow or PyTorch as a backend. |
-| [Object Detection with PyTorch & Torchvision](/docs/notebooks/transfer_learning/pytorch_object_detection) | CV: Object Detection | PyTorch |Demonstrates transfer learning with multiple [Torchvision](https://pytorch.org/vision/stable/index.html) object detection models, a public image dataset, and a customized Torchvision dataset |
+| Notebook | Domain: Use Case | Framework | Description |
+| ---------| ----------| ----------|-------------|
+| [Image Classification with TF Hub](/docs/notebooks/transfer_learning/image_classification/tf_image_classification) | CV: Image Classification | TensorFlow | Demonstrates transfer learning with multiple [TF Hub](https://tfhub.dev) image classifiers, TF datasets, and custom image datasets |
+| [Image Classification with PyTorch & Torchvision](/docs/notebooks/transfer_learning/image_classification/pytorch_image_classification) | CV: Image Classification | PyTorch | Demonstrates transfer learning with multiple [Torchvision](https://pytorch.org/vision/stable/index.html) image classification models, Torchvision datasets, and custom datasets |
+| [Image Classification with Hugging Face](/docs/notebooks/transfer_learning/image_classification/huggingface_image_classification) | CV: Image Classification | PyTorch, TensorFlow | Demonstrates transfer learning with [Hugging Face models](https://huggingface.co/models) for image classification using either TensorFlow or PyTorch as a backend. |
+| [Object Detection with PyTorch & Torchvision](/docs/notebooks/transfer_learning/object_detection/pytorch_object_detection) | CV: Object Detection | PyTorch |Demonstrates transfer learning with multiple [Torchvision](https://pytorch.org/vision/stable/index.html) object detection models, a public image dataset, and a customized Torchvision dataset |
diff --git a/docs/notebooks/transfer_learning/huggingface_image_classification/HuggingFace_Image_Classification_Transfer_Learning.ipynb b/docs/notebooks/transfer_learning/image_classification/huggingface_image_classification/HuggingFace_Image_Classification_Transfer_Learning.ipynb
similarity index 100%
rename from docs/notebooks/transfer_learning/huggingface_image_classification/HuggingFace_Image_Classification_Transfer_Learning.ipynb
rename to docs/notebooks/transfer_learning/image_classification/huggingface_image_classification/HuggingFace_Image_Classification_Transfer_Learning.ipynb
diff --git a/docs/notebooks/transfer_learning/huggingface_image_classification/README.md b/docs/notebooks/transfer_learning/image_classification/huggingface_image_classification/README.md
similarity index 100%
rename from docs/notebooks/transfer_learning/huggingface_image_classification/README.md
rename to docs/notebooks/transfer_learning/image_classification/huggingface_image_classification/README.md
diff --git a/docs/notebooks/transfer_learning/pytorch_image_classification/PyTorch_Image_Classification_Transfer_Learning.ipynb b/docs/notebooks/transfer_learning/image_classification/pytorch_image_classification/PyTorch_Image_Classification_Transfer_Learning.ipynb
similarity index 96%
rename from docs/notebooks/transfer_learning/pytorch_image_classification/PyTorch_Image_Classification_Transfer_Learning.ipynb
rename to docs/notebooks/transfer_learning/image_classification/pytorch_image_classification/PyTorch_Image_Classification_Transfer_Learning.ipynb
index 524e17935..02f477bec 100644
--- a/docs/notebooks/transfer_learning/pytorch_image_classification/PyTorch_Image_Classification_Transfer_Learning.ipynb
+++ b/docs/notebooks/transfer_learning/image_classification/pytorch_image_classification/PyTorch_Image_Classification_Transfer_Learning.ipynb
@@ -48,7 +48,7 @@
     "from pydoc import locate\n",
     "import warnings\n",
     "import wget\n",
-    "from zipfile import ZipFile\n",
+    "import tarfile\n",
     "\n",
     "import intel_extension_for_pytorch as ipex\n",
     "import matplotlib.pyplot as plt\n",
@@ -161,7 +161,11 @@
    "cell_type": "code",
    "execution_count": null,
    "id": "537d83e1",
-   "metadata": {},
+   "metadata": {
+    "tags": [
+     "remove_for_custom_dataset"
+    ]
+   },
    "outputs": [],
    "source": [
     "dataset = torchvision.datasets.Food101(dataset_directory, split='train',\n",
@@ -193,7 +197,11 @@
   {
    "cell_type": "markdown",
    "id": "f235fa0f",
-   "metadata": {},
+   "metadata": {
+    "tags": [
+     "remove_for_pyt_dataset"
+    ]
+   },
    "source": [
     "To use your own image dataset for transfer learning with the rest of this notebook, format your images as `.jpg` files and save them in folders named after the classes that you want the model to predict. To provide a working example using the correct layout, we will download and extract a flower species dataset. After downloading and extracting, you will have the following  subdirectories in your dataset directory. Each species subfolder will contain numerous `.jpg` files:\n",
     "\n",
@@ -214,7 +222,11 @@
    "cell_type": "code",
    "execution_count": null,
    "id": "29105a9e",
-   "metadata": {},
+   "metadata": {
+    "tags": [
+     "remove_for_pyt_dataset"
+    ]
+   },
    "outputs": [],
    "source": [
     "# When you have your own properly organized subdirectory of images, adjust this variable\n",
@@ -225,7 +237,11 @@
    "cell_type": "code",
    "execution_count": null,
    "id": "13e96b9e",
-   "metadata": {},
+   "metadata": {
+    "tags": [
+     "remove_for_pyt_dataset"
+    ]
+   },
    "outputs": [],
    "source": [
     "# Only run this if you want to use the example flowers dataset\n",
@@ -234,17 +250,21 @@
     "    dataset_url = \"https://storage.googleapis.com/download.tensorflow.org/example_images/flower_photos.tgz\"\n",
     "    \n",
     "    # Download and extract the tar\n",
-    "    zip_file = wget.download(dataset_url, dataset_directory)\n",
-    "    print(\"Extracting {} to {}\".format(zip_file, dataset_directory))\n",
-    "    with ZipFile(zip_file, \"r\") as zipfile:\n",
-    "        zipfile.extractall(path=dataset_directory)\n"
+    "    tar_file = wget.download(dataset_url, dataset_directory)\n",
+    "    print(\"Extracting {} to {}\".format(tar_file, dataset_directory))\n",
+    "    with tarfile.open(tar_file, \"r:gz\") as tar:\n",
+    "        tar.extractall(path=dataset_directory)\n"
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
    "id": "a5482b8a",
-   "metadata": {},
+   "metadata": {
+    "tags": [
+     "remove_for_pyt_dataset"
+    ]
+   },
    "outputs": [],
    "source": [
     "dataset = datasets.ImageFolder(dataset_subdir, get_transform(True))\n",
@@ -256,7 +276,11 @@
    "cell_type": "code",
    "execution_count": null,
    "id": "e791a208",
-   "metadata": {},
+   "metadata": {
+    "tags": [
+     "remove_for_pyt_dataset"
+    ]
+   },
    "outputs": [],
    "source": [
     "# Use 25% for validation and 75% for training\n",
diff --git a/docs/notebooks/transfer_learning/pytorch_image_classification/README.md b/docs/notebooks/transfer_learning/image_classification/pytorch_image_classification/README.md
similarity index 100%
rename from docs/notebooks/transfer_learning/pytorch_image_classification/README.md
rename to docs/notebooks/transfer_learning/image_classification/pytorch_image_classification/README.md
diff --git a/docs/notebooks/transfer_learning/pytorch_image_classification/model_utils.py b/docs/notebooks/transfer_learning/image_classification/pytorch_image_classification/model_utils.py
similarity index 98%
rename from docs/notebooks/transfer_learning/pytorch_image_classification/model_utils.py
rename to docs/notebooks/transfer_learning/image_classification/pytorch_image_classification/model_utils.py
index 290e48aaa..f7e0d8288 100644
--- a/docs/notebooks/transfer_learning/pytorch_image_classification/model_utils.py
+++ b/docs/notebooks/transfer_learning/image_classification/pytorch_image_classification/model_utils.py
@@ -1,4 +1,3 @@
-#
 # -*- coding: utf-8 -*-
 #
 # Copyright (c) 2022 Intel Corporation
@@ -66,7 +65,7 @@ def get_retrainable_model(model_name, num_classes, do_fine_tuning=False):
         for param in model.parameters():
             param.requires_grad = False
 
-    if type(classifier_layer) == list:
+    if isinstance(classifier_layer, list):
         classifier = getattr(model, classifier_layer[0])[classifier_layer[1]]
         num_features = classifier.in_features
         model.classifier[classifier_layer[1]] = torch.nn.Linear(num_features, num_classes)
diff --git a/docs/notebooks/transfer_learning/tf_image_classification/Image_Classification_Transfer_Learning.ipynb b/docs/notebooks/transfer_learning/image_classification/tf_image_classification/Image_Classification_Transfer_Learning.ipynb
similarity index 97%
rename from docs/notebooks/transfer_learning/tf_image_classification/Image_Classification_Transfer_Learning.ipynb
rename to docs/notebooks/transfer_learning/image_classification/tf_image_classification/Image_Classification_Transfer_Learning.ipynb
index ab5e4edb9..c12f0a08d 100644
--- a/docs/notebooks/transfer_learning/tf_image_classification/Image_Classification_Transfer_Learning.ipynb
+++ b/docs/notebooks/transfer_learning/image_classification/tf_image_classification/Image_Classification_Transfer_Learning.ipynb
@@ -122,7 +122,11 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {},
+   "metadata": {
+    "tags": [
+        "remove_for_tf_dataset"
+       ]
+   },
    "outputs": [],
    "source": [
     "# When you have your own properly organized subdirectory of images, adjust this variable\n",
@@ -132,7 +136,11 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {},
+   "metadata": {
+    "tags": [
+        "remove_for_tf_dataset"
+       ]
+   },
    "outputs": [],
    "source": [
     "# Only run this if you want to use the example flowers dataset\n",
@@ -148,7 +156,11 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {},
+   "metadata": {
+    "tags": [
+        "remove_for_tf_dataset"
+       ]
+   },
    "outputs": [],
    "source": [
     "import pathlib\n",
@@ -161,7 +173,11 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {},
+   "metadata": {
+    "tags": [
+        "remove_for_tf_dataset"
+       ]
+   },
    "outputs": [],
    "source": [
     "# Create an image data generator and partition the data into train and test sets\n",
@@ -202,7 +218,11 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {},
+   "metadata": {
+    "tags": [
+        "remove_for_custom_dataset"
+       ]
+   },
    "outputs": [],
    "source": [
     "# Options include: \"food101\", cats_vs_dogs\", \"horses_or_humans\", \"rock_paper_scissors\", and \"tf_flowers\"\n",
diff --git a/docs/notebooks/transfer_learning/tf_image_classification/README.md b/docs/notebooks/transfer_learning/image_classification/tf_image_classification/README.md
similarity index 100%
rename from docs/notebooks/transfer_learning/tf_image_classification/README.md
rename to docs/notebooks/transfer_learning/image_classification/tf_image_classification/README.md
diff --git a/docs/notebooks/transfer_learning/tf_image_classification/model_util.py b/docs/notebooks/transfer_learning/image_classification/tf_image_classification/model_util.py
similarity index 100%
rename from docs/notebooks/transfer_learning/tf_image_classification/model_util.py
rename to docs/notebooks/transfer_learning/image_classification/tf_image_classification/model_util.py
diff --git a/docs/notebooks/transfer_learning/pytorch_object_detection/.gitignore b/docs/notebooks/transfer_learning/object_detection/pytorch_object_detection/.gitignore
similarity index 100%
rename from docs/notebooks/transfer_learning/pytorch_object_detection/.gitignore
rename to docs/notebooks/transfer_learning/object_detection/pytorch_object_detection/.gitignore
diff --git a/docs/notebooks/transfer_learning/pytorch_object_detection/PyTorch_Object_Detection_Transfer_Learning.ipynb b/docs/notebooks/transfer_learning/object_detection/pytorch_object_detection/PyTorch_Object_Detection_Transfer_Learning.ipynb
similarity index 98%
rename from docs/notebooks/transfer_learning/pytorch_object_detection/PyTorch_Object_Detection_Transfer_Learning.ipynb
rename to docs/notebooks/transfer_learning/object_detection/pytorch_object_detection/PyTorch_Object_Detection_Transfer_Learning.ipynb
index 1b525b7e4..f8f21c5fd 100644
--- a/docs/notebooks/transfer_learning/pytorch_object_detection/PyTorch_Object_Detection_Transfer_Learning.ipynb
+++ b/docs/notebooks/transfer_learning/object_detection/pytorch_object_detection/PyTorch_Object_Detection_Transfer_Learning.ipynb
@@ -155,7 +155,11 @@
    "cell_type": "code",
    "execution_count": null,
    "id": "48c7f147",
-   "metadata": {},
+   "metadata": {
+    "tags": [
+        "remove_for_pyt_dataset"
+       ]
+   },
    "outputs": [],
    "source": [
     "LABELS = COCO_LABELS\n",
@@ -182,7 +186,11 @@
    "cell_type": "code",
    "execution_count": null,
    "id": "11ed8f8d",
-   "metadata": {},
+   "metadata": {
+    "tags": [
+        "remove_for_pyt_dataset"
+       ]
+   },
    "outputs": [],
    "source": [
     "dataset = PennFudanDataset(dataset_path, get_transform(train=True))\n",
@@ -223,7 +231,11 @@
    "cell_type": "code",
    "execution_count": null,
    "id": "16b73b54",
-   "metadata": {},
+   "metadata": {
+    "tags": [
+        "remove_for_custom_dataset"
+       ]
+   },
    "outputs": [],
    "source": [
     "LABELS = KITTI_LABELS\n",
diff --git a/docs/notebooks/transfer_learning/pytorch_object_detection/README.md b/docs/notebooks/transfer_learning/object_detection/pytorch_object_detection/README.md
similarity index 100%
rename from docs/notebooks/transfer_learning/pytorch_object_detection/README.md
rename to docs/notebooks/transfer_learning/object_detection/pytorch_object_detection/README.md
diff --git a/docs/notebooks/transfer_learning/pytorch_object_detection/dataset_utils.py b/docs/notebooks/transfer_learning/object_detection/pytorch_object_detection/dataset_utils.py
similarity index 100%
rename from docs/notebooks/transfer_learning/pytorch_object_detection/dataset_utils.py
rename to docs/notebooks/transfer_learning/object_detection/pytorch_object_detection/dataset_utils.py
diff --git a/docs/notebooks/transfer_learning/pytorch_object_detection/model_utils.py b/docs/notebooks/transfer_learning/object_detection/pytorch_object_detection/model_utils.py
similarity index 100%
rename from docs/notebooks/transfer_learning/pytorch_object_detection/model_utils.py
rename to docs/notebooks/transfer_learning/object_detection/pytorch_object_detection/model_utils.py
diff --git a/docs/notebooks/transfer_learning/pytorch_text_classification/README.md b/docs/notebooks/transfer_learning/pytorch_text_classification/README.md
deleted file mode 100644
index c5064ea4c..000000000
--- a/docs/notebooks/transfer_learning/pytorch_text_classification/README.md
+++ /dev/null
@@ -1,48 +0,0 @@
-# Text Classifier fine tuning with PyTorch
-
-This notebook demonstrates fine tuning [pretrained models from Hugging Face](https://huggingface.co/models)
-using text classification datasets from the [Hugging Face Datasets catalog](https://huggingface.co/datasets) or
-a custom dataset. The [IMDb Larget Movie Review dataset](https://ai.stanford.edu/~amaas/data/sentiment/) is used
-from the Hugging Face Datasets catalog, and the [SMS Spam Collection dataset](https://archive.ics.uci.edu/ml/datasets/sms+spam+collection)
-is used as an example of a custom dataset being loaded from a csv file.
-
-The notebook uses
-[Intel® Extension for PyTorch\*](https://intel.github.io/intel-extension-for-pytorch) which extends PyTorch
-with optimizations for extra performance boost on Intel hardware.
-
-The notebook performs the following steps:
-1. Import dependencies and setup parameters
-2. Prepare the dataset
-3. Prepare the Model for Fine Tuning and Evaluation
-4. Export the model
-5. Reload the model and make predictions
-
-## Running the notebook
-
-To run the notebook, follow the instructions to install the [Transfer Learning Notebook requirements](setup.md).
-
-## References
-
-Dataset Citations
-```
-@InProceedings{maas-EtAl:2011:ACL-HLT2011,
-  author    = {Maas, Andrew L.  and  Daly, Raymond E.  and  Pham, Peter T.  and  Huang, Dan  and  Ng, Andrew Y.  and  Potts, Christopher},
-  title     = {Learning Word Vectors for Sentiment Analysis},
-  booktitle = {Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies},
-  month     = {June},
-  year      = {2011},
-  address   = {Portland, Oregon, USA},
-  publisher = {Association for Computational Linguistics},
-  pages     = {142--150},
-  url       = {http://www.aclweb.org/anthology/P11-1015}
-}
-
-@misc{misc_sms_spam_collection_228,
-  author       = {Almeida, Tiago},
-  title        = {{SMS Spam Collection}},
-  year         = {2012},
-  howpublished = {UCI Machine Learning Repository}
-}
-```
-Please see this dataset's applicable license for terms and conditions. Intel Corporation does not own the rights to this data set and does not confer any rights to it.
-
diff --git a/docs/notebooks/transfer_learning/pytorch_text_generation/README.md b/docs/notebooks/transfer_learning/pytorch_text_generation/README.md
deleted file mode 100644
index ef3c038e1..000000000
--- a/docs/notebooks/transfer_learning/pytorch_text_generation/README.md
+++ /dev/null
@@ -1,39 +0,0 @@
-# Large language model instruction tuning with PyTorch
-
-This notebook demonstrates instruction tuning [pretrained causal language models from Hugging Face](https://huggingface.co/models)
-using text generation datasets from the [Hugging Face Datasets catalog](https://huggingface.co/datasets) or
-a custom dataset. The [databricks-dolly-15k](https://huggingface.co/datasets/databricks/databricks-dolly-15k) dataset is used
-from the Hugging Face Datasets catalog, and an [Intel domain dataset](https://raw.githubusercontent.com/intel/intel-extension-for-transformers/1.0.1/examples/optimization/pytorch/huggingface/language-modeling/chatbot/intel_domain.json)
-is used as an example of a custom dataset being loaded from a json file.
-
-The notebook includes options for bfloat16 precision training,
-[Intel® Extension for PyTorch\*](https://intel.github.io/intel-extension-for-pytorch) which extends PyTorch
-with optimizations for extra performance boost on Intel hardware, and [SmoothQuant quantization with Intel® Neural Compressor](https://github.com/intel/neural-compressor/blob/v2.1.1/docs/source/smooth_quant.md).
-
-The notebook performs the following steps:
-1. Import dependencies and setup parameters
-2. Prepare the dataset
-3. Prepare the model and test domain knowledge
-4. Transfer Learning
-5. Retest domain knowledge
-6. Quantize the model
-
-## Running the notebook
-
-To run the notebook, follow the instructions in `setup.md`.
-
-## References
-
-Dataset Citations
-
-<b>[databricks-dolly-15k](https://huggingface.co/datasets/databricks/databricks-dolly-15k)</b> - Copyright (2023) Databricks, Inc. This dataset was developed at Databricks (https://www.databricks.com) and its use is subject to the CC BY-SA 3.0 license. Certain categories of material in the dataset include materials from the following sources, licensed under the CC BY-SA 3.0 license: Wikipedia (various pages) - https://www.wikipedia.org/ Copyright © Wikipedia editors and contributors.
-
-```
-@software{together2023redpajama,
-  author = {Together Computer},
-  title = {RedPajama: An Open Source Recipe to Reproduce LLaMA training dataset},
-  month = April,
-  year = 2023,
-  url = {https://github.com/togethercomputer/RedPajama-Data}
-}
-```
diff --git a/docs/notebooks/transfer_learning/tfhub_bert/question_answering/BERT_Question_Answering.ipynb b/docs/notebooks/transfer_learning/question_answering/BERT_Question_Answering.ipynb
similarity index 97%
rename from docs/notebooks/transfer_learning/tfhub_bert/question_answering/BERT_Question_Answering.ipynb
rename to docs/notebooks/transfer_learning/question_answering/BERT_Question_Answering.ipynb
index 71e767b6e..1b9eda7bb 100644
--- a/docs/notebooks/transfer_learning/tfhub_bert/question_answering/BERT_Question_Answering.ipynb
+++ b/docs/notebooks/transfer_learning/question_answering/BERT_Question_Answering.ipynb
@@ -1,7 +1,6 @@
 {
  "cells": [
   {
-   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -17,7 +16,6 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -34,7 +32,6 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "import json\n",
     "import os\n",
     "import pandas as pd\n",
     "import tensorflow as tf\n",
@@ -48,7 +45,6 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -137,7 +133,6 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -263,7 +258,6 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -327,7 +321,6 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
@@ -339,13 +332,16 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {},
+   "metadata": {
+    "scrolled": true
+   },
    "outputs": [],
    "source": [
     "import tensorflow as tf\n",
-    "from official.legacy.bert import bert_models\n",
-    "from official.legacy.bert import configs as bert_configs\n",
-    "from official.legacy.bert import model_saving_utils\n",
+    "\n",
+    "from official.nlp.bert import bert_models\n",
+    "from official.nlp.bert import configs as bert_configs\n",
+    "from official.nlp.bert import model_saving_utils\n",
     "\n",
     "tf.keras.mixed_precision.set_global_policy('float32')\n",
     "bert_config_obj = bert_configs.BertConfig.from_json_file(bert_config)\n",
@@ -362,7 +358,6 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "metadata": {},
    "source": [
diff --git a/docs/notebooks/transfer_learning/tfhub_bert/question_answering/README.md b/docs/notebooks/transfer_learning/question_answering/README.md
similarity index 100%
rename from docs/notebooks/transfer_learning/tfhub_bert/question_answering/README.md
rename to docs/notebooks/transfer_learning/question_answering/README.md
diff --git a/docs/notebooks/transfer_learning/tfhub_bert/question_answering/bert_qa_utils.py b/docs/notebooks/transfer_learning/question_answering/bert_qa_utils.py
similarity index 100%
rename from docs/notebooks/transfer_learning/tfhub_bert/question_answering/bert_qa_utils.py
rename to docs/notebooks/transfer_learning/question_answering/bert_qa_utils.py
diff --git a/docs/notebooks/transfer_learning/tfhub_bert/question_answering/tfhub_bert_model_map_qa.json b/docs/notebooks/transfer_learning/question_answering/tfhub_bert_model_map_qa.json
similarity index 100%
rename from docs/notebooks/transfer_learning/tfhub_bert/question_answering/tfhub_bert_model_map_qa.json
rename to docs/notebooks/transfer_learning/question_answering/tfhub_bert_model_map_qa.json
diff --git a/docs/notebooks/transfer_learning/requirements.txt b/docs/notebooks/transfer_learning/requirements.txt
index 51a8785d3..341eff8a3 100644
--- a/docs/notebooks/transfer_learning/requirements.txt
+++ b/docs/notebooks/transfer_learning/requirements.txt
@@ -1,7 +1,9 @@
-Pillow~=10.0.1 # upgraded to fix CVEs
+accelerate>=0.20.1
+Pillow~=10.2.0 # upgraded to fix CVEs
 PyYAML~=6.0
 charset-normalizer~=3.1.0
 datasets~=2.12.0
+fsspec~=2023.9.2
 gin-config~=0.5.0
 intel-extension-for-pytorch==1.13.100
 intel-tensorflow==2.12.0
@@ -12,7 +14,7 @@ ipywidgets~=8.0.6
 jmespath~=1.0.1
 matplotlib-inline~=0.1.6
 matplotlib~=3.7.1
-notebook~=6.5.4
+notebook~=7.0.7
 numpy~=1.23.5
 opencv-python~=4.7.0.72
 pandas~=2.0.1
@@ -24,9 +26,10 @@ sentencepiece~=0.1.99
 tensorflow-addons~=0.20.0
 tensorflow-datasets~=4.9.2
 tensorflow-hub~=0.13.0
+tf-models-official
 torch==1.13.1
 torchvision==0.14.1
-transformers~=4.30.0
+transformers~=4.36.0
 urllib3~=2.0.2
 wget~=3.2
 evaluate==0.4.0
diff --git a/docs/notebooks/transfer_learning/run_tl_notebooks.sh b/docs/notebooks/transfer_learning/run_tl_notebooks.sh
new file mode 100644
index 000000000..2c49863a2
--- /dev/null
+++ b/docs/notebooks/transfer_learning/run_tl_notebooks.sh
@@ -0,0 +1,103 @@
+#!/usr/bin/env bash
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+
+tf_notebooks=(
+    "docs/notebooks/transfer_learning/image_classification/tf_image_classification/Image_Classification_Transfer_Learning.ipynb"
+    "docs/notebooks/transfer_learning/image_classification/huggingface_image_classification/HuggingFace_Image_Classification_Transfer_Learning.ipynb"
+    "docs/notebooks/transfer_learning/question_answering/BERT_Question_Answering.ipynb"
+    "docs/notebooks/transfer_learning/text_classification/tfhub_bert_text_classification/BERT_Binary_Text_Classification.ipynb"
+    "docs/notebooks/transfer_learning/text_classification/tfhub_bert_text_classification/BERT_Multi_Text_Classification.ipynb"
+)
+
+pyt_notebooks=(
+    "docs/notebooks/transfer_learning/image_classification/pytorch_image_classification/PyTorch_Image_Classification_Transfer_Learning.ipynb"
+    "docs/notebooks/transfer_learning/object_detection/pytorch_object_detection/PyTorch_Object_Detection_Transfer_Learning.ipynb"
+    "docs/notebooks/transfer_learning/text_classification/pytorch_text_classification/PyTorch_Text_Classifier_fine_tuning.ipynb"
+)
+
+if [[ $# -eq 0 ]] ; then
+    echo "No argument supplied. Please input tensorflow, pytorch, a notebook path, or a directory containing one or more notebooks."
+    exit 1
+fi
+
+CURDIR=$PWD
+INPUT=$1
+SUCCESS=0
+
+# Set to an error exit code, if any notebook fails
+exit_code_summary=${SUCCESS}
+
+# Array tracking the notebooks with errors
+failed_notebooks=()
+
+if [[ $INPUT == "tensorflow" ]] ; then
+    notebooks=${tf_notebooks[*]}
+elif [[ $INPUT == "pytorch" ]] ; then
+    notebooks=${pyt_notebooks[*]}
+else
+    # Parse the filename from the path
+    DIR=${INPUT%/*}
+    FILE="${INPUT##*/}"
+
+    # If no file was given, find all notebooks in the directory
+    if [ -z "$FILE" ] ; then
+        readarray -d '' notebooks < <(find ${DIR} -maxdepth 1 -name *.ipynb -print0)
+    else
+        notebooks=($1)
+    fi
+fi
+
+echo "Notebooks: ${notebooks[*]}"
+for notebook in ${notebooks[*]}; do
+    DIR=${notebook%/*}
+    echo "Running ${notebook}..."
+
+    if [[ $# -eq 2 ]] ; then
+        echo "Stripping tag ${2}..."
+        jupyter nbconvert --to script \
+            --TagRemovePreprocessor.enabled=True \
+            --TagRemovePreprocessor.remove_cell_tags $2 \
+            --output notebook_test ${notebook}
+    else
+        jupyter nbconvert --to script --output notebook_test ${notebook}
+    fi
+
+    pushd ${DIR}
+    PYTHONPATH=${CURDIR} ipython notebook_test.py
+    script_exit_code=$?
+
+    if [ ${script_exit_code} != ${SUCCESS} ]; then
+        failed_notebooks+=(${notebook})
+        exit_code_summary=${script_exit_code}
+    fi
+
+    rm notebook_test.py
+    popd
+done
+
+# If any notebook failed, print out the failing notebook(s)
+if [ ${exit_code_summary} != ${SUCCESS} ]; then
+    echo ""
+    echo "Failed notebooks:"
+    for failed_nb in "${failed_notebooks[@]}"
+    do
+        echo ${failed_nb}
+    done
+fi
+
+exit ${exit_code_summary}
diff --git a/docs/notebooks/transfer_learning/bert_classifier_fine_tuning/BERT_Classifier_fine_tuning_using_IMDB.ipynb b/docs/notebooks/transfer_learning/text_classification/bert_classifier_fine_tuning/BERT_Classifier_fine_tuning_using_IMDB.ipynb
similarity index 100%
rename from docs/notebooks/transfer_learning/bert_classifier_fine_tuning/BERT_Classifier_fine_tuning_using_IMDB.ipynb
rename to docs/notebooks/transfer_learning/text_classification/bert_classifier_fine_tuning/BERT_Classifier_fine_tuning_using_IMDB.ipynb
diff --git a/docs/notebooks/transfer_learning/bert_classifier_fine_tuning/README.md b/docs/notebooks/transfer_learning/text_classification/bert_classifier_fine_tuning/README.md
similarity index 100%
rename from docs/notebooks/transfer_learning/bert_classifier_fine_tuning/README.md
rename to docs/notebooks/transfer_learning/text_classification/bert_classifier_fine_tuning/README.md
diff --git a/docs/notebooks/transfer_learning/bert_classifier_fine_tuning/imdb.yaml b/docs/notebooks/transfer_learning/text_classification/bert_classifier_fine_tuning/imdb.yaml
similarity index 100%
rename from docs/notebooks/transfer_learning/bert_classifier_fine_tuning/imdb.yaml
rename to docs/notebooks/transfer_learning/text_classification/bert_classifier_fine_tuning/imdb.yaml
diff --git a/docs/notebooks/transfer_learning/bert_classifier_fine_tuning/quantize.py b/docs/notebooks/transfer_learning/text_classification/bert_classifier_fine_tuning/quantize.py
similarity index 100%
rename from docs/notebooks/transfer_learning/bert_classifier_fine_tuning/quantize.py
rename to docs/notebooks/transfer_learning/text_classification/bert_classifier_fine_tuning/quantize.py
diff --git a/docs/notebooks/transfer_learning/pytorch_text_classification/PyTorch_Text_Classifier_fine_tuning.ipynb b/docs/notebooks/transfer_learning/text_classification/pytorch_text_classification/PyTorch_Text_Classifier_fine_tuning.ipynb
similarity index 97%
rename from docs/notebooks/transfer_learning/pytorch_text_classification/PyTorch_Text_Classifier_fine_tuning.ipynb
rename to docs/notebooks/transfer_learning/text_classification/pytorch_text_classification/PyTorch_Text_Classifier_fine_tuning.ipynb
index 1c6fc7cf1..d7db8140c 100644
--- a/docs/notebooks/transfer_learning/pytorch_text_classification/PyTorch_Text_Classifier_fine_tuning.ipynb
+++ b/docs/notebooks/transfer_learning/text_classification/pytorch_text_classification/PyTorch_Text_Classifier_fine_tuning.ipynb
@@ -1,7 +1,6 @@
 {
  "cells": [
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "b65f0c82",
    "metadata": {},
@@ -21,7 +20,6 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "454a6685",
    "metadata": {},
@@ -105,7 +103,6 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "2f258d4f",
    "metadata": {},
@@ -194,7 +191,6 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "fd8512e1",
    "metadata": {},
@@ -203,7 +199,6 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "640e5611",
    "metadata": {},
@@ -221,7 +216,11 @@
    "cell_type": "code",
    "execution_count": null,
    "id": "9a1d5fc0",
-   "metadata": {},
+   "metadata": {
+    "tags": [
+     "remove_for_custom_dataset"
+    ]
+   },
    "outputs": [],
    "source": [
     "class HFDSTextClassificationData(TextClassificationData):\n",
@@ -285,7 +284,6 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "362625ec",
    "metadata": {},
@@ -294,7 +292,6 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "28c0ba36",
    "metadata": {},
@@ -317,7 +314,11 @@
    "cell_type": "code",
    "execution_count": null,
    "id": "c706bfbf",
-   "metadata": {},
+   "metadata": {
+    "tags": [
+     "remove_for_pyt_dataset"
+    ]
+   },
    "outputs": [],
    "source": [
     "class CustomCsvTextClassificationData(TextClassificationData):\n",
@@ -387,7 +388,9 @@
     "# Modify the variables below to use a different dataset or a csv file on your local system.\n",
     "# The csv_path variable should be pointing to a csv file with 2 columns (the label and the text)\n",
     "dataset_url = \"https://archive.ics.uci.edu/static/public/228/sms+spam+collection.zip\"\n",
-    "dataset_dir = os.path.join(dataset_dir, \"sms_spam_collection\")\n",
+    "if not os.path.exists(os.path.join(dataset_dir, \"sms_spam_collection\")):\n",
+    "    os.makedirs(os.path.join(dataset_dir, \"sms_spam_collection\"))\n",
+    "dataset_directory = os.path.join(dataset_dir, \"sms_spam_collection\")\n",
     "csv_name = \"SMSSpamCollection\"\n",
     "delimiter = \"\\t\"\n",
     "label_names = [\"ham\", \"spam\"]\n",
@@ -396,15 +399,15 @@
     "renamed_csv = \"{}.csv\".format(csv_name)\n",
     "\n",
     "# If we don't already have the csv file, download and extract the zip file to get it.\n",
-    "if not os.path.exists(os.path.join(dataset_dir, csv_name)) and \\\n",
-    "                      not os.path.exists(os.path.join(dataset_dir, renamed_csv)):\n",
-    "    zip_file = wget.download(dataset_url, dataset_dir)\n",
-    "    print(\"Extracting {} to {}\".format(zip_file, dataset_dir))\n",
+    "if not os.path.exists(os.path.join(dataset_directory, csv_name)) and \\\n",
+    "                      not os.path.exists(os.path.join(dataset_directory, renamed_csv)):\n",
+    "    zip_file = wget.download(dataset_url, dataset_directory)\n",
+    "    print(\"Extracting {} to {}\".format(zip_file, dataset_directory))\n",
     "    with ZipFile(zip_file, \"r\") as zipfile:\n",
-    "        zipfile.extractall(path=dataset_dir)\n",
+    "        zipfile.extractall(path=dataset_directory)\n",
     "\n",
-    "if not os.path.exists(os.path.join(dataset_dir, renamed_csv)):\n",
-    "    os.rename(os.path.join(dataset_dir, csv_name), os.path.join(dataset_dir, renamed_csv))\n",
+    "if not os.path.exists(os.path.join(dataset_directory, renamed_csv)):\n",
+    "    os.rename(os.path.join(dataset_directory, csv_name), os.path.join(dataset_directory, renamed_csv))\n",
     "    \n",
     "# Columns\n",
     "sentence1_key = \"text\"\n",
@@ -416,7 +419,7 @@
     "    example[\"label\"] = int(example[\"label\"] == \"spam\")\n",
     "    return example\n",
     "\n",
-    "dataset = CustomCsvTextClassificationData(tokenizer, \"smsspamcollection\", dataset_dir, [renamed_csv], delimiter,\n",
+    "dataset = CustomCsvTextClassificationData(tokenizer, \"smsspamcollection\", dataset_directory, [renamed_csv], delimiter,\n",
     "                                          label_names, sentence1_key, sentence2_key, label_key, train_size=1000,\n",
     "                                          eval_size=1000, map_function=map_spam)\n",
     "\n",
@@ -425,7 +428,6 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "e3a24bcd",
    "metadata": {},
@@ -604,7 +606,6 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "3e8f1edd",
    "metadata": {},
@@ -613,7 +614,6 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "1a606f16",
    "metadata": {},
@@ -665,7 +665,6 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "5fcabd97",
    "metadata": {},
@@ -700,7 +699,6 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "cce10679",
    "metadata": {},
@@ -738,7 +736,6 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "f4821e46",
    "metadata": {},
@@ -772,7 +769,6 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "b83b873f",
    "metadata": {},
@@ -792,7 +788,6 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "49449342",
    "metadata": {},
@@ -815,7 +810,6 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "c70a5386",
    "metadata": {},
@@ -853,7 +847,6 @@
    ]
   },
   {
-   "attachments": {},
    "cell_type": "markdown",
    "id": "bee40324",
    "metadata": {},
diff --git a/docs/notebooks/transfer_learning/text_classification/pytorch_text_classification/README.md b/docs/notebooks/transfer_learning/text_classification/pytorch_text_classification/README.md
new file mode 100644
index 000000000..9fe6d386f
--- /dev/null
+++ b/docs/notebooks/transfer_learning/text_classification/pytorch_text_classification/README.md
@@ -0,0 +1,48 @@
+# Text Classifier fine tuning with PyTorch
+
+This notebook demonstrates fine tuning [pretrained models from Hugging Face](https://huggingface.co/models)
+using text classification datasets from the [Hugging Face Datasets catalog](https://huggingface.co/datasets) or
+a custom dataset. The [IMDb Larget Movie Review dataset](https://ai.stanford.edu/~amaas/data/sentiment/) is used
+from the Hugging Face Datasets catalog, and the [SMS Spam Collection dataset](https://archive.ics.uci.edu/ml/datasets/sms+spam+collection)
+is used as an example of a custom dataset being loaded from a csv file.
+
+The notebook uses
+[Intel® Extension for PyTorch\*](https://intel.github.io/intel-extension-for-pytorch) which extends PyTorch
+with optimizations for extra performance boost on Intel hardware.
+
+The notebook performs the following steps:
+1. Import dependencies and setup parameters
+2. Prepare the dataset
+3. Prepare the Model for Fine Tuning and Evaluation
+4. Export the model
+5. Reload the model and make predictions
+
+## Running the notebook
+
+To run the notebook, follow the instructions to install the `setup.md`.
+
+## References
+
+Dataset Citations
+```
+@InProceedings{maas-EtAl:2011:ACL-HLT2011,
+  author    = {Maas, Andrew L.  and  Daly, Raymond E.  and  Pham, Peter T.  and  Huang, Dan  and  Ng, Andrew Y.  and  Potts, Christopher},
+  title     = {Learning Word Vectors for Sentiment Analysis},
+  booktitle = {Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies},
+  month     = {June},
+  year      = {2011},
+  address   = {Portland, Oregon, USA},
+  publisher = {Association for Computational Linguistics},
+  pages     = {142--150},
+  url       = {http://www.aclweb.org/anthology/P11-1015}
+}
+
+@misc{misc_sms_spam_collection_228,
+  author       = {Almeida, Tiago},
+  title        = {{SMS Spam Collection}},
+  year         = {2012},
+  howpublished = {UCI Machine Learning Repository}
+}
+```
+Please see this dataset's applicable license for terms and conditions. Intel Corporation does not own the rights to this data set and does not confer any rights to it.
+
diff --git a/docs/notebooks/transfer_learning/tfhub_bert/text_classification/BERT_Binary_Text_Classification.ipynb b/docs/notebooks/transfer_learning/text_classification/tfhub_bert_text_classification/BERT_Binary_Text_Classification.ipynb
similarity index 96%
rename from docs/notebooks/transfer_learning/tfhub_bert/text_classification/BERT_Binary_Text_Classification.ipynb
rename to docs/notebooks/transfer_learning/text_classification/tfhub_bert_text_classification/BERT_Binary_Text_Classification.ipynb
index 93485bf38..3b0e16c34 100644
--- a/docs/notebooks/transfer_learning/tfhub_bert/text_classification/BERT_Binary_Text_Classification.ipynb
+++ b/docs/notebooks/transfer_learning/text_classification/tfhub_bert_text_classification/BERT_Binary_Text_Classification.ipynb
@@ -195,7 +195,11 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {},
+   "metadata": {
+    "tags": [
+     "remove_for_custom_dataset"
+    ]
+   },
    "outputs": [],
    "source": [
     "class TFDSBinaryTextClassificationData(BinaryTextClassificationData):\n",
@@ -282,7 +286,11 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {},
+   "metadata": {
+    "tags": [
+     "remove_for_tf_dataset"
+    ]
+   },
    "outputs": [],
    "source": [
     "class CustomCsvBinaryTextClassificationData(BinaryTextClassificationData):\n",
@@ -356,22 +364,34 @@
     "        self.test_ds = self.test_ds.apply(tf.data.experimental.assert_cardinality(test_size))\n",
     "\n",
     "\n",
-    "# Modify the variables below to use a different dataset or a csv file on your local system.\n",
-    "# The csv_path variable should be pointing to a csv file with 2 columns (the label and the text)\n",
     "dataset_url = \"https://archive.ics.uci.edu/static/public/228/sms+spam+collection.zip\"\n",
     "dataset_directory = os.path.join(dataset_directory, \"sms_spam_collection\")\n",
+    "if not os.path.exists(dataset_directory):\n",
+    "    os.makedirs(dataset_directory)\n",
+    "\n",
     "csv_name = \"SMSSpamCollection\"\n",
     "delimiter = \"\\t\"\n",
     "header = False  # Set to true if the csv file has a header row\n",
-    "csv_path = os.path.join(dataset_directory, csv_name)\n",
+    "\n",
+    "# Rename the file to include the csv extension so that the dataset API knows how to load the file\n",
+    "renamed_csv = \"{}.csv\".format(csv_name)\n",
     "\n",
     "# If we don't already have the csv file, download and extract the zip file to get it.\n",
-    "if not os.path.exists(csv_path):\n",
+    "if not os.path.exists(os.path.join(dataset_directory, csv_name)) and \\\n",
+    "                      not os.path.exists(os.path.join(dataset_directory, renamed_csv)):\n",
     "    zip_file = wget.download(dataset_url, dataset_directory)\n",
     "    print(\"Extracting {} to {}\".format(zip_file, dataset_directory))\n",
     "    with ZipFile(zip_file, \"r\") as zipfile:\n",
     "        zipfile.extractall(path=dataset_directory)\n",
     "\n",
+    "if not os.path.exists(os.path.join(dataset_directory, renamed_csv)):\n",
+    "    os.rename(os.path.join(dataset_directory, csv_name), os.path.join(dataset_directory, renamed_csv))\n",
+    "\n",
+    "# Modify the variables below to use a different dataset or a csv file on your local system.\n",
+    "# The csv_path variable should be pointing to a csv file with 2 columns (the label and the text)\n",
+    "\n",
+    "csv_path = os.path.join(dataset_directory, renamed_csv)\n",
+    "\n",
     "# Define the label map for your dataset. The label map below is for the SMS Spam Collection dataset.\n",
     "# The labels defined in this dictionary should match the labels in the csv file.\n",
     "label_map = {\n",
@@ -447,7 +467,7 @@
     "%%time\n",
     "\n",
     "# The number of training epochs to run\n",
-    "num_train_epochs = 2\n",
+    "num_train_epochs = 1\n",
     "\n",
     "# Learning rate\n",
     "learning_rate = 3e-5\n",
diff --git a/docs/notebooks/transfer_learning/tfhub_bert/text_classification/BERT_Multi_Text_Classification.ipynb b/docs/notebooks/transfer_learning/text_classification/tfhub_bert_text_classification/BERT_Multi_Text_Classification.ipynb
similarity index 99%
rename from docs/notebooks/transfer_learning/tfhub_bert/text_classification/BERT_Multi_Text_Classification.ipynb
rename to docs/notebooks/transfer_learning/text_classification/tfhub_bert_text_classification/BERT_Multi_Text_Classification.ipynb
index 6e08dfd89..5289840cd 100644
--- a/docs/notebooks/transfer_learning/tfhub_bert/text_classification/BERT_Multi_Text_Classification.ipynb
+++ b/docs/notebooks/transfer_learning/text_classification/tfhub_bert_text_classification/BERT_Multi_Text_Classification.ipynb
@@ -183,7 +183,11 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {},
+   "metadata": {
+    "tags": [
+        "remove_for_custom_dataset"
+       ]
+   },
    "outputs": [],
    "source": [
     "class TFDSMultiTextClassificationData(MultiTextClassificationData):\n",
@@ -273,7 +277,11 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {},
+   "metadata": {
+    "tags": [
+        "remove_for_tf_dataset"
+       ]
+   },
    "outputs": [],
    "source": [
     "class CustomCsvMultiTextClassificationData(MultiTextClassificationData):\n",
diff --git a/docs/notebooks/transfer_learning/tfhub_bert/text_classification/README.md b/docs/notebooks/transfer_learning/text_classification/tfhub_bert_text_classification/README.md
similarity index 100%
rename from docs/notebooks/transfer_learning/tfhub_bert/text_classification/README.md
rename to docs/notebooks/transfer_learning/text_classification/tfhub_bert_text_classification/README.md
diff --git a/docs/notebooks/transfer_learning/tfhub_bert/text_classification/bert_utils.py b/docs/notebooks/transfer_learning/text_classification/tfhub_bert_text_classification/bert_utils.py
similarity index 100%
rename from docs/notebooks/transfer_learning/tfhub_bert/text_classification/bert_utils.py
rename to docs/notebooks/transfer_learning/text_classification/tfhub_bert_text_classification/bert_utils.py
diff --git a/docs/notebooks/transfer_learning/tfhub_bert/text_classification/tfhub_bert_model_map_classifier.json b/docs/notebooks/transfer_learning/text_classification/tfhub_bert_text_classification/tfhub_bert_model_map_classifier.json
similarity index 100%
rename from docs/notebooks/transfer_learning/tfhub_bert/text_classification/tfhub_bert_model_map_classifier.json
rename to docs/notebooks/transfer_learning/text_classification/tfhub_bert_text_classification/tfhub_bert_model_map_classifier.json
diff --git a/docs/notebooks/transfer_learning/pytorch_text_generation/PyTorch_LLM_Instruction_Tuning.ipynb b/docs/notebooks/transfer_learning/text_generation/pytorch_text_generation/PyTorch_LLM_Instruction_Tuning.ipynb
similarity index 96%
rename from docs/notebooks/transfer_learning/pytorch_text_generation/PyTorch_LLM_Instruction_Tuning.ipynb
rename to docs/notebooks/transfer_learning/text_generation/pytorch_text_generation/PyTorch_LLM_Instruction_Tuning.ipynb
index 87739953e..72089eb04 100644
--- a/docs/notebooks/transfer_learning/pytorch_text_generation/PyTorch_LLM_Instruction_Tuning.ipynb
+++ b/docs/notebooks/transfer_learning/text_generation/pytorch_text_generation/PyTorch_LLM_Instruction_Tuning.ipynb
@@ -249,7 +249,7 @@
    "outputs": [],
    "source": [
     "# Choose a URL to download or skip this cell and provide a local path in the next cell\n",
-    "url = \"https://raw.githubusercontent.com/intel/intel-extension-for-transformers/1.0.1/examples/optimization/pytorch/huggingface/language-modeling/chatbot/intel_domain.json\"\n",
+    "url = \"https://raw.githubusercontent.com/sahil280114/codealpaca/master/data/code_alpaca_2k.json\"\n",
     "\n",
     "filename = os.path.basename(url)\n",
     "destination = os.path.join(dataset_dir, filename)\n",
@@ -542,10 +542,10 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "# For intel-domain custom dataset\n",
+    "# For code generation custom dataset\n",
     "prompt_template = PROMPT_DICT[\"prompt_with_context\"]\n",
-    "test_example = {dataset_schema['instruction_key']: 'Write a short marketing paragraph about the following product.',\n",
-    "               dataset_schema['context_key']: 'Intel Xeon processor',\n",
+    "test_example = {dataset_schema['instruction_key']: 'Write a Python function that sorts the following list.',\n",
+    "               dataset_schema['context_key']: '[3, 2, 1]',\n",
     "               dataset_schema['response_key']: ''}"
    ]
   },
@@ -619,25 +619,20 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "if model_info['lora_target_modules']:\n",
-    "    lora_rank = 8  # Rank parameter \n",
-    "    lora_alpha = 32  # Alpha parameter\n",
-    "    lora_dropout = 0.05  # Dropout parameter \n",
-    "    lora_target_modules = model_info['lora_target_modules']\n",
+    "lora_rank = 8  # Rank parameter \n",
+    "lora_alpha = 32  # Alpha parameter\n",
+    "lora_dropout = 0.05  # Dropout parameter \n",
     "\n",
-    "    # PEFT settings\n",
-    "    peft_config = LoraConfig(\n",
-    "        r=lora_rank,\n",
-    "        lora_alpha=lora_alpha,\n",
-    "        lora_dropout=lora_dropout,\n",
-    "        target_modules=lora_target_modules,\n",
-    "        bias=\"none\",\n",
-    "        task_type=TaskType.CAUSAL_LM,\n",
-    "    )\n",
-    "    model = get_peft_model(model, peft_config)\n",
-    "    model.print_trainable_parameters()\n",
-    "else:\n",
-    "    print('The {} model is not compatible with LoRA'.format(model_name))"
+    "# PEFT settings\n",
+    "peft_config = LoraConfig(\n",
+    "    r=lora_rank,\n",
+    "    lora_alpha=lora_alpha,\n",
+    "    lora_dropout=lora_dropout,\n",
+    "    bias=\"none\",\n",
+    "    task_type=TaskType.CAUSAL_LM,\n",
+    ")\n",
+    "model = get_peft_model(model, peft_config)\n",
+    "model.print_trainable_parameters()"
    ]
   },
   {
@@ -975,6 +970,17 @@
     "  year = 2023,\n",
     "  url = {https://github.com/togethercomputer/RedPajama-Data}\n",
     "}\n",
+    "```\n",
+    "\n",
+    "```\n",
+    "@misc{codealpaca,\n",
+    "  author = {Sahil Chaudhary},\n",
+    "  title = {Code Alpaca: An Instruction-following LLaMA model for code generation},\n",
+    "  year = {2023},\n",
+    "  publisher = {GitHub},\n",
+    "  journal = {GitHub repository},\n",
+    "  howpublished = {\\url{https://github.com/sahil280114/codealpaca}},\n",
+    "}\n",
     "```"
    ]
   },
@@ -1003,7 +1009,7 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.10.6"
+   "version": "3.10.12"
   }
  },
  "nbformat": 4,
diff --git a/docs/notebooks/transfer_learning/text_generation/pytorch_text_generation/README.md b/docs/notebooks/transfer_learning/text_generation/pytorch_text_generation/README.md
new file mode 100644
index 000000000..5bf2514a6
--- /dev/null
+++ b/docs/notebooks/transfer_learning/text_generation/pytorch_text_generation/README.md
@@ -0,0 +1,50 @@
+# Large language model instruction tuning with PyTorch
+
+This notebook demonstrates instruction tuning [pretrained causal language models from Hugging Face](https://huggingface.co/models)
+using text generation datasets from the [Hugging Face Datasets catalog](https://huggingface.co/datasets) or
+a custom dataset. The [databricks-dolly-15k](https://huggingface.co/datasets/databricks/databricks-dolly-15k) dataset is used
+from the Hugging Face Datasets catalog, and a subset of the [Code Alpaca](https://github.com/sahil280114/codealpaca) dataset
+is used as an example of a custom dataset being loaded from a json file.
+
+The notebook includes options for bfloat16 precision training,
+[Intel® Extension for PyTorch\*](https://intel.github.io/intel-extension-for-pytorch) which extends PyTorch
+with optimizations for extra performance boost on Intel hardware, and [SmoothQuant quantization with Intel® Neural Compressor](https://github.com/intel/neural-compressor/blob/v2.1.1/docs/source/smooth_quant.md).
+
+The notebook performs the following steps:
+1. Import dependencies and setup parameters
+2. Prepare the dataset
+3. Prepare the model and test domain knowledge
+4. Transfer Learning
+5. Retest domain knowledge
+6. Quantize the model
+
+## Running the notebook
+
+To run the notebook, follow the instructions to install the `setup.md`.
+
+## References
+
+Dataset Citations
+
+<b>[databricks-dolly-15k](https://huggingface.co/datasets/databricks/databricks-dolly-15k)</b> - Copyright (2023) Databricks, Inc. This dataset was developed at Databricks (https://www.databricks.com) and its use is subject to the CC BY-SA 3.0 license. Certain categories of material in the dataset include materials from the following sources, licensed under the CC BY-SA 3.0 license: Wikipedia (various pages) - https://www.wikipedia.org/ Copyright © Wikipedia editors and contributors.
+
+```
+@software{together2023redpajama,
+  author = {Together Computer},
+  title = {RedPajama: An Open Source Recipe to Reproduce LLaMA training dataset},
+  month = April,
+  year = 2023,
+  url = {https://github.com/togethercomputer/RedPajama-Data}
+}
+```
+
+```
+@misc{codealpaca,
+  author = {Sahil Chaudhary},
+  title = {Code Alpaca: An Instruction-following LLaMA model for code generation},
+  year = {2023},
+  publisher = {GitHub},
+  journal = {GitHub repository},
+  howpublished = {\url{https://github.com/sahil280114/codealpaca}},
+}
+```
diff --git a/docs/notebooks/transfer_learning/pytorch_text_generation/llm_utils.py b/docs/notebooks/transfer_learning/text_generation/pytorch_text_generation/llm_utils.py
similarity index 100%
rename from docs/notebooks/transfer_learning/pytorch_text_generation/llm_utils.py
rename to docs/notebooks/transfer_learning/text_generation/pytorch_text_generation/llm_utils.py
diff --git a/models/common/pytorch/common/utils.py b/models/common/pytorch/common/utils.py
new file mode 100644
index 000000000..e835616fc
--- /dev/null
+++ b/models/common/pytorch/common/utils.py
@@ -0,0 +1,68 @@
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#! /usr/bin/env python3
+# coding: utf-8
+
+"""
+Utils functions for parsers
+"""
+
+import sys
+from email.message import EmailMessage
+from smtplib import SMTP, SMTPException
+
+
+def is_digit(value):
+    """Check if value is digit."""
+    try:
+        float(value)
+        return True
+    except ValueError:
+        return False
+
+
+def get_sorted_filelist(filelist, expected_result_dict):
+    """Get sorted filelist."""
+    sorted_filelist = []
+    if filelist is None:
+        for key in expected_result_dict:
+            file_name = key + ".log"
+            sorted_filelist.append(file_name)
+    else:
+        sorted_filelist = filelist
+    return sorted_filelist
+
+
+def send_report(sender, recipients, subject, html):
+    """Send report."""
+    msg = EmailMessage()
+    msg['Subject'] = subject
+    msg['From'] = sender
+    msg['To'] = recipients
+    msg.set_content(html, subtype="html")
+    try:
+        with SMTP('smtp.intel.com') as session:
+            session.sendmail(sender, recipients, msg.as_string())
+    except SMTPException as e:
+        raise e
+
+
+def check_python_version():
+    """Check python version."""
+    if not (sys.version_info.major == 3 and sys.version_info.minor >= 6):
+        print("Python 3.6 or higher is required.")
+        print("You are using Python {}.{}.".format(sys.version_info.major, sys.version_info.minor))
+        sys.exit(1)
diff --git a/models/language_translation/tensorflow/mlperf_gnmt/__init__.py b/models/common/pytorch/consts/__init__.py
similarity index 100%
rename from models/language_translation/tensorflow/mlperf_gnmt/__init__.py
rename to models/common/pytorch/consts/__init__.py
diff --git a/models/common/pytorch/consts/patterns/common.py b/models/common/pytorch/consts/patterns/common.py
new file mode 100644
index 000000000..8448896d0
--- /dev/null
+++ b/models/common/pytorch/consts/patterns/common.py
@@ -0,0 +1,41 @@
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Common pattern consts."""
+
+UNITS = {
+    "higher_better": [
+        "BLEU4",
+        "PSNR",
+        "acc@1",
+        "accuracy",
+        "angles/s",
+        "epoch/s",
+        "f1",
+        "fps",
+        "image/s",
+        "inst/s",
+        "items/s",
+        "mAP",
+        "mean",
+        "sample/s",
+        "sent/s",
+    ],
+    "lower_better": [
+        "loss",
+        "perplexity",
+        "s",
+    ],
+}
diff --git a/models/common/pytorch/consts/patterns/ddp/inf/ddp-dlrm-terabyte.py b/models/common/pytorch/consts/patterns/ddp/inf/ddp-dlrm-terabyte.py
new file mode 100644
index 000000000..a1d70b807
--- /dev/null
+++ b/models/common/pytorch/consts/patterns/ddp/inf/ddp-dlrm-terabyte.py
@@ -0,0 +1,37 @@
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Consts for ddp-dlrm-terabyte inference."""
+
+ACC = {
+    "type": "total",
+    "pattern": r"AUROC over test set: (\d+.\d+)",
+    "unit": "AUROC",
+    "number_of_partials": 2,
+}
+
+
+PERF = {
+    "type": "max",
+    "pattern": r"avg eval time per iter at ITER: 45, (\d+.\d+) s",
+    "inverse": True,
+    "multiply": False,
+    "use_batch_size": True,
+    "unit": "samples/s",
+}
+
+FUNCTIONAL = {
+    "pattern": r"AUROC over test set: (\d+.\d+)",
+}
diff --git a/models/common/pytorch/consts/patterns/ddp/train/ddp-bert.py b/models/common/pytorch/consts/patterns/ddp/train/ddp-bert.py
new file mode 100644
index 000000000..adfa47ea0
--- /dev/null
+++ b/models/common/pytorch/consts/patterns/ddp/train/ddp-bert.py
@@ -0,0 +1,35 @@
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Consts for ddp-bert training."""
+
+ACC = {
+    "type": "mean",
+    "pattern": r"perplexity = (\d+.\d+)",
+    "unit": "perplexity",
+}
+
+PERF = {
+    "type": "total",
+    "pattern": r"bert_train throughput:  (\d+.\d+)  sentences/s",
+    "inverse": False,
+    "multiply": False,
+    "use_batch_size": False,
+    "unit": "sent/s",
+}
+
+FUNCTIONAL = {
+    "pattern": r"perplexity = (.*)",
+}
diff --git a/models/common/pytorch/consts/patterns/ddp/train/ddp-dlrm-terabyte.py b/models/common/pytorch/consts/patterns/ddp/train/ddp-dlrm-terabyte.py
new file mode 100644
index 000000000..509b410c4
--- /dev/null
+++ b/models/common/pytorch/consts/patterns/ddp/train/ddp-dlrm-terabyte.py
@@ -0,0 +1,37 @@
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Consts for ddp-dlrm-terabyte training."""
+
+ACC = {
+    "type": "total",
+    "pattern": r"AUROC over test set: (\d+.\d+)",
+    "unit": "AUROC",
+    "number_of_partials": 2
+}
+
+
+PERF = {
+    "type": "max",
+    "pattern": r"avg training time per iter at ITER: 45, (\d+.\d+) s",
+    "inverse": True,
+    "multiply": False,
+    "use_batch_size": True,
+    "unit": "samples/s",
+}
+
+FUNCTIONAL = {
+    "pattern": r"avg training time per iter at ITER: 45, (\d+.\d+) s",
+}
diff --git a/models/common/pytorch/consts/patterns/ddp/train/ddp-resnet50.py b/models/common/pytorch/consts/patterns/ddp/train/ddp-resnet50.py
new file mode 100644
index 000000000..70cccd90b
--- /dev/null
+++ b/models/common/pytorch/consts/patterns/ddp/train/ddp-resnet50.py
@@ -0,0 +1,37 @@
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Consts for ddp-resnet50 training."""
+
+ACC = {
+    "type": "mean",
+    "pattern": r"Loss (\d+.\d+)",
+    "unit": "loss",
+    "number_of_partials": 2,
+}
+
+
+PERF = {
+    "type": "total",
+    "pattern": r"throughput:(\d+.\d+) image/sec",
+    "inverse": False,
+    "multiply": False,
+    "use_batch_size": False,
+    "unit": "image/s",
+}
+
+FUNCTIONAL = {
+    "pattern": r"Training performance: batch size:\d+, throughput:\d+.\d+ image/sec",
+}
diff --git a/models/common/pytorch/consts/patterns/ddp/train/ddp-rnnt.py b/models/common/pytorch/consts/patterns/ddp/train/ddp-rnnt.py
new file mode 100644
index 000000000..144405989
--- /dev/null
+++ b/models/common/pytorch/consts/patterns/ddp/train/ddp-rnnt.py
@@ -0,0 +1,35 @@
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Consts for ddp-rnnt training."""
+
+ACC = {
+    "type": "mean",
+    "pattern": r"t_loss_t=== \[[0-9]\] tensor\(\[(\d+.\d+|\w+)\]",
+    "unit": "loss",
+}
+
+PERF = {
+    "type": "total",
+    "pattern": r"throughput: (\d+.\d+) sample per second",
+    "inverse": False,
+    "multiply": False,
+    "use_batch_size": False,
+    "unit": "fps",
+}
+
+FUNCTIONAL = {
+    "pattern": r"throughput: \d+.\d+ sample per second"
+}
diff --git a/models/common/pytorch/consts/patterns/por/inf/3dunet.py b/models/common/pytorch/consts/patterns/por/inf/3dunet.py
new file mode 100644
index 000000000..99d8b1da4
--- /dev/null
+++ b/models/common/pytorch/consts/patterns/por/inf/3dunet.py
@@ -0,0 +1,35 @@
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Consts for 3dunet inference."""
+
+ACC = {
+    "type": "total",
+    "pattern": r"Accuracy: mean = (\d.\d+)",
+    "unit": "mean",
+}
+
+PERF = {
+    "type": "total",
+    "pattern": r"3dunet_inf throughput:  (\d+.\d+)  sample/s",
+    "inverse": False,
+    "multiply": False,
+    "use_batch_size": False,
+    "unit": "sample/s",
+}
+
+FUNCTIONAL = {
+    "pattern": r"No warnings encountered during test."
+}
diff --git a/models/common/pytorch/consts/patterns/por/inf/bertsquad.py b/models/common/pytorch/consts/patterns/por/inf/bertsquad.py
new file mode 100644
index 000000000..ee3a77222
--- /dev/null
+++ b/models/common/pytorch/consts/patterns/por/inf/bertsquad.py
@@ -0,0 +1,35 @@
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Consts for bertsquad inference."""
+
+ACC = {
+    "type": "total",
+    "pattern": r"\'f1\': (\d+.\d+)",
+    "unit": "f1",
+}
+
+PERF = {
+    "type": "total",
+    "pattern": r"bert_inf throughput:  (\d+.\d+)  sentences/s",
+    "inverse": False,
+    "multiply": False,
+    "use_batch_size": False,
+    "unit": "sent/s",
+}
+
+FUNCTIONAL = {
+    "pattern": r"\'f1\': \d+.\d+"
+}
diff --git a/models/common/pytorch/consts/patterns/por/inf/distilbert.py b/models/common/pytorch/consts/patterns/por/inf/distilbert.py
new file mode 100644
index 000000000..1ca6db589
--- /dev/null
+++ b/models/common/pytorch/consts/patterns/por/inf/distilbert.py
@@ -0,0 +1,35 @@
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Consts for distilbert inf."""
+
+ACC = {
+    "type": "total",
+    "pattern": r"'acc': (.*),",
+    "unit": "acc",
+}
+
+PERF = {
+    "type": "total",
+    "pattern": r"perf (\d+.\d+) sentences/s",
+    "inverse": False,
+    "multiply": False,
+    "use_batch_size": False,
+    "unit": "sent/s",
+}
+
+FUNCTIONAL = {
+    "pattern": r"perf (\d+.\d+) sentences/s",
+}
diff --git a/models/common/pytorch/consts/patterns/por/inf/dlrm-kaggle.py b/models/common/pytorch/consts/patterns/por/inf/dlrm-kaggle.py
new file mode 100644
index 000000000..3bd07e916
--- /dev/null
+++ b/models/common/pytorch/consts/patterns/por/inf/dlrm-kaggle.py
@@ -0,0 +1,35 @@
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Consts for dlrm-kaggle inference."""
+
+ACC = {
+    "type": "total",
+    "pattern": r"accuracy (\d+.\d+) %",
+    "unit": "accuracy",
+}
+
+PERF = {
+    "type": "total",
+    "pattern": r"dlrm_inf throughput:  (\d+.\d+)  samples/s",
+    "inverse": False,
+    "multiply": False,
+    "use_batch_size": False,
+    "unit": "inst/s",
+}
+
+FUNCTIONAL = {
+    "pattern": r"dlrm_inf throughput:  \d+.\d+  samples/s"
+}
diff --git a/models/common/pytorch/consts/patterns/por/inf/fastpitch.py b/models/common/pytorch/consts/patterns/por/inf/fastpitch.py
new file mode 100644
index 000000000..aee5d6045
--- /dev/null
+++ b/models/common/pytorch/consts/patterns/por/inf/fastpitch.py
@@ -0,0 +1,35 @@
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Consts for fastpitch inference."""
+
+ACC = {
+    "type": "total",
+    "pattern": r"Acc=\s*(\d+.\d+) %",
+    "unit": "Acc",
+}
+
+PERF = {
+    "type": "total",
+    "pattern": r"throughput:\s*(\d+.\d+)",
+    "inverse": False,
+    "multiply": False,
+    "use_batch_size": False,
+    "unit": "fps",
+}
+
+FUNCTIONAL = {
+        "pattern": r"throughput:\s*(\d+.\d+)"
+}
diff --git a/models/common/pytorch/consts/patterns/por/inf/hifi-gan.py b/models/common/pytorch/consts/patterns/por/inf/hifi-gan.py
new file mode 100644
index 000000000..3e7d1b515
--- /dev/null
+++ b/models/common/pytorch/consts/patterns/por/inf/hifi-gan.py
@@ -0,0 +1,35 @@
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Consts for hifi-gan inference."""
+
+ACC = {
+    "type": "total",
+    "pattern": r"accuracy (\d+.\d+) %",
+    "unit": "accuracy",
+}
+
+PERF = {
+    "type": "total",
+    "pattern": r"throughput:\s*(\d+.\d+)",
+    "inverse": False,
+    "multiply": False,
+    "use_batch_size": False,
+    "unit": "imgs/s",
+}
+
+FUNCTIONAL = {
+    "pattern": r"throughput:\s*(\d+.\d+)"
+}
diff --git a/models/common/pytorch/consts/patterns/por/inf/quartznet.py b/models/common/pytorch/consts/patterns/por/inf/quartznet.py
new file mode 100644
index 000000000..724824342
--- /dev/null
+++ b/models/common/pytorch/consts/patterns/por/inf/quartznet.py
@@ -0,0 +1,35 @@
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Consts for quartznet inference."""
+
+ACC = {
+    "type": "total",
+    "pattern": r"WER=\s*(\d+.\d+) %",
+    "unit": "WER",
+}
+
+PERF = {
+    "type": "total",
+    "pattern": r"throughput=\s*(\d+.\d+)",
+    "inverse": False,
+    "multiply": False,
+    "use_batch_size": False,
+    "unit": "fps",
+}
+
+FUNCTIONAL = {
+    "pattern": r"throughput=\s*(\d+.\d+)"
+}
diff --git a/models/common/pytorch/consts/patterns/por/inf/resnet50.py b/models/common/pytorch/consts/patterns/por/inf/resnet50.py
new file mode 100644
index 000000000..606bef621
--- /dev/null
+++ b/models/common/pytorch/consts/patterns/por/inf/resnet50.py
@@ -0,0 +1,35 @@
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Consts for resnet50 inference."""
+
+ACC = {
+    "type": "total",
+    "pattern": r" * Acc@1:(\d+.\d+)",
+    "unit": "acc@1",
+}
+
+PERF = {
+    "type": "total",
+    "pattern": r"throughput:(\d+.\d+) image/sec",
+    "inverse": False,
+    "multiply": False,
+    "use_batch_size": False,
+    "unit": "fps",
+}
+
+FUNCTIONAL = {
+    "pattern": r"Evalution performance: batch size:\d+, throughput:\d+.\d+ image/sec"
+}
diff --git a/models/common/pytorch/consts/patterns/por/inf/rnnt.py b/models/common/pytorch/consts/patterns/por/inf/rnnt.py
new file mode 100644
index 000000000..8b0e589a8
--- /dev/null
+++ b/models/common/pytorch/consts/patterns/por/inf/rnnt.py
@@ -0,0 +1,35 @@
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Consts for rnnt inference."""
+
+ACC = {
+    "type": "total",
+    "pattern": r"Accuracy: (.*)",
+    "unit": "accuracy",
+}
+
+PERF = {
+    "type": "total",
+    "pattern": r"rnnt_inf throughput: (\d+.\d+) sample per second",
+    "inverse": False,
+    "multiply": False,
+    "use_batch_size": False,
+    "unit": "fps",
+}
+
+FUNCTIONAL = {
+    "pattern": r"rnnt_inf throughput: (\d+.\d+) sample per second",
+}
diff --git a/models/common/pytorch/consts/patterns/por/inf/ssd-mobilenetv1.py b/models/common/pytorch/consts/patterns/por/inf/ssd-mobilenetv1.py
new file mode 100644
index 000000000..44c447224
--- /dev/null
+++ b/models/common/pytorch/consts/patterns/por/inf/ssd-mobilenetv1.py
@@ -0,0 +1,40 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+
+"""Consts for ssd-mobilenetv1 inference."""
+
+ACC = {
+    "type": "total",
+    "pattern": r"Average Precision Across All Classes:(\d+.\d+)",
+    "unit": "accuracy",
+}
+
+PERF = {
+    "type": "mean",
+    "number_of_partials": 10,
+    "pattern": r"Inference time:  (\d+.\d+)",
+    "inverse": True,
+    "multiply": False,
+    "use_batch_size": True,
+    "unit": "inst/s",
+}
+
+FUNCTIONAL = {
+    "pattern": r"Average Precision Across All Classes:\d+.\d+"
+}
diff --git a/models/common/pytorch/consts/patterns/por/inf/stable-diffusion-coco.py b/models/common/pytorch/consts/patterns/por/inf/stable-diffusion-coco.py
new file mode 100644
index 000000000..164953ee0
--- /dev/null
+++ b/models/common/pytorch/consts/patterns/por/inf/stable-diffusion-coco.py
@@ -0,0 +1,35 @@
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Consts for stable-diffusion inf."""
+
+ACC = {
+    "type": "total",
+    "pattern": r"FID Scores:  tensor\((.*)\)",
+    "unit": "accuracy",
+}
+
+PERF = {
+    "type": "total",
+    "pattern": r"Thoughput:  (\d+.\d+)",
+    "inverse": False,
+    "multiply": False,
+    "use_batch_size": False,
+    "unit": "fps",
+}
+
+FUNCTIONAL = {
+    "pattern": r"FID Scores:  tensor(.*)",
+}
diff --git a/models/common/pytorch/consts/patterns/por/inf/stable-diffusion.py b/models/common/pytorch/consts/patterns/por/inf/stable-diffusion.py
new file mode 100644
index 000000000..fe91ad01b
--- /dev/null
+++ b/models/common/pytorch/consts/patterns/por/inf/stable-diffusion.py
@@ -0,0 +1,35 @@
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Consts for stable-diffusion inf."""
+
+ACC = {
+    "type": "total",
+    "pattern": r"CLIP score: (\d+.\d+)",
+    "unit": "accuracy",
+}
+
+PERF = {
+    "type": "total",
+    "pattern": r"50/50 .* (\d+.\d+)it/s",
+    "inverse": False,
+    "multiply": False,
+    "use_batch_size": False,
+    "unit": "it/s",
+}
+
+FUNCTIONAL = {
+    "pattern": r"CLIP score: (\d+.\d+)",
+}
diff --git a/models/common/pytorch/consts/patterns/por/inf/swin-transformer.py b/models/common/pytorch/consts/patterns/por/inf/swin-transformer.py
new file mode 100644
index 000000000..c776d8ce4
--- /dev/null
+++ b/models/common/pytorch/consts/patterns/por/inf/swin-transformer.py
@@ -0,0 +1,35 @@
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Consts for swin-transformer training."""
+
+ACC = {
+    "type": "total",
+    "pattern": r"Loss (\d+.\d+)",
+    "unit": "loss",
+}
+
+PERF = {
+    "type": "total",
+    "pattern": r"Throughput: (\d+.\d+)",
+    "inverse": False,
+    "multiply": False,
+    "use_batch_size": False,
+    "unit": "fps",
+}
+
+FUNCTIONAL = {
+    "pattern": r"Throughput: (\d+.\d+)",
+}
diff --git a/models/common/pytorch/consts/patterns/por/inf/unetpp.py b/models/common/pytorch/consts/patterns/por/inf/unetpp.py
new file mode 100644
index 000000000..7ec6593b7
--- /dev/null
+++ b/models/common/pytorch/consts/patterns/por/inf/unetpp.py
@@ -0,0 +1,35 @@
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Consts for unet++ inference."""
+
+ACC = {
+    "type": "total",
+    "pattern": r"Acc=\s*(\d+.\d+) %",
+    "unit": "Acc",
+}
+
+PERF = {
+    "type": "total",
+    "pattern": r"Throughput:\s*(\d+.\d+)",
+    "inverse": False,
+    "multiply": False,
+    "use_batch_size": False,
+    "unit": "fps",
+}
+
+FUNCTIONAL = {
+        "pattern": r"Throughput:\s*(\d+.\d+)"
+}
diff --git a/models/common/pytorch/consts/patterns/por/train/bert.py b/models/common/pytorch/consts/patterns/por/train/bert.py
new file mode 100644
index 000000000..5683fe20d
--- /dev/null
+++ b/models/common/pytorch/consts/patterns/por/train/bert.py
@@ -0,0 +1,37 @@
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+
+"""Consts for bert training."""
+
+ACC = {
+    "type": "total",
+    "pattern": r"perplexity = (.*)",
+    "unit": "perplexity",
+}
+
+PERF = {
+    "type": "total",
+    "pattern": r"bert_train throughput:  (\d+.\d+)  sentences/s",
+    "inverse": False,
+    "multiply": False,
+    "use_batch_size": False,
+    "unit": "sent/s",
+}
+
+FUNCTIONAL = {
+    "pattern": r"perplexity = (.*)",
+}
diff --git a/models/common/pytorch/consts/patterns/por/train/resnet50.py b/models/common/pytorch/consts/patterns/por/train/resnet50.py
new file mode 100644
index 000000000..3e8cc76ad
--- /dev/null
+++ b/models/common/pytorch/consts/patterns/por/train/resnet50.py
@@ -0,0 +1,36 @@
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+"""Consts for resnet50 training."""
+
+ACC = {
+    "type": "total",
+    "pattern": r"Loss (\d+.\d+)",
+    "unit": "loss",
+}
+
+PERF = {
+    "type": "total",
+    "pattern": r"throughput:(\d+.\d+) image/sec",
+    "inverse": False,
+    "multiply": False,
+    "use_batch_size": False,
+    "unit": "image/s",
+}
+
+FUNCTIONAL = {
+    "pattern": r"Training performance: batch size:\d+, throughput:\d+.\d+ image/sec",
+}
diff --git a/models/common/pytorch/consts/patterns/por/train/stable-diffusion-coco.py b/models/common/pytorch/consts/patterns/por/train/stable-diffusion-coco.py
new file mode 100644
index 000000000..46c208e60
--- /dev/null
+++ b/models/common/pytorch/consts/patterns/por/train/stable-diffusion-coco.py
@@ -0,0 +1,35 @@
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Consts for stable-diffusion training."""
+
+ACC = {
+    "type": "total",
+    "pattern": r"loss: (.*)",
+    "unit": "loss",
+}
+
+PERF = {
+    "type": "total",
+    "pattern": r"Thoughput:  (\d+.\d+)",
+    "inverse": False,
+    "multiply": False,
+    "use_batch_size": False,
+    "unit": "fps",
+}
+
+FUNCTIONAL = {
+    "pattern": r"Thoughput:  (\d+.\d+)",
+}
diff --git a/models/diffusion/pytorch/stable_diffusion/inference.py b/models/diffusion/pytorch/stable_diffusion/inference.py
new file mode 100755
index 000000000..a268133c9
--- /dev/null
+++ b/models/diffusion/pytorch/stable_diffusion/inference.py
@@ -0,0 +1,419 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import argparse
+import copy
+import logging
+import os
+import time
+import threading
+import numpy as np
+import pathlib
+from tqdm import tqdm
+
+import torch
+from PIL import Image
+from diffusers import DiffusionPipeline
+from torchmetrics.image.fid import FrechetInceptionDistance
+import torchvision.datasets as dset
+import torchvision.transforms as transforms
+
+logging.getLogger().setLevel(logging.INFO)
+
+
+def parse_args():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--model_name_or_path", type=str, default="stabilityai/stable-diffusion-2-1", help="Model path")
+    parser.add_argument("--int8_model_path", type=str, default="sd_v2_1.pt", help="INT8 model path")
+    parser.add_argument("--dataset_path", type=str, default=None, help="COCO2017 dataset path")
+    parser.add_argument("--prompt", type=str, default="A big burly grizzly bear is show with grass in the background.", help="input text")
+    parser.add_argument("--output_dir", type=str, default=None,help="output path")
+    parser.add_argument("--seed", type=int, default=0, help="random seed")
+    parser.add_argument('--precision', type=str, default="fp32", help='precision: fp32, bf32, bf16, fp16, int8-bf16, int8-fp32')
+    parser.add_argument('--ipex', action='store_true', default=False, help='ipex')
+    parser.add_argument('--jit', action='store_true', default=False, help='jit trace')
+    parser.add_argument('--compile_ipex', action='store_true', default=False, help='compile with ipex backend')
+    parser.add_argument('--compile_inductor', action='store_true', default=False, help='compile with inductor backend')
+    parser.add_argument('--profile', action='store_true', default=False, help='profile')
+    parser.add_argument('--benchmark', action='store_true', default=False, help='test performance')
+    parser.add_argument('--accuracy', action='store_true', default=False, help='test accuracy')
+    parser.add_argument('-w', '--warmup_iterations', default=-1, type=int, help='number of warmup iterations to run')
+    parser.add_argument('-i', '--iterations', default=-1, type=int, help='number of total iterations to run')
+    parser.add_argument('--world-size', default=-1, type=int, help='number of nodes for distributed training')
+    parser.add_argument('--rank', default=-1, type=int, help='node rank for distributed training')
+    parser.add_argument('--dist-url', default='env://', type=str, help='url used to set up distributed training')
+    parser.add_argument('--dist-backend', default='ccl', type=str, help='distributed backend')
+    parser.add_argument("--weight-sharing", action='store_true', default=False, help="using weight_sharing to test the performance of inference")
+    parser.add_argument("--number-instance", default=0, type=int, help="the instance numbers for test the performance of latcy, only works when enable weight-sharing")
+
+    args = parser.parse_args()
+    return args
+
+def run_weights_sharing_model(pipe, tid, args):
+    total_time = 0
+    for i in range(args.iterations + args.warmup_iterations):
+        # run model
+        start = time.time()
+        if args.precision == "bf16" or args.precision == "fp16" or args.precision == "int8-bf16":
+            with torch.cpu.amp.autocast(dtype=args.dtype), torch.no_grad():
+                output = pipe(args.prompt, generator=torch.manual_seed(args.seed)).images
+        else:
+            with torch.no_grad():
+                output = pipe(args.prompt, generator=torch.manual_seed(args.seed)).images
+        end = time.time()
+        print('time per prompt(s): {:.2f}'.format((end - start)))
+        if i >= args.warmup_iterations:
+            total_time += end - start
+
+    print("Instance num: ", tid)
+    print("Latency: {:.2f} s".format(total_time / args.iterations))
+    print("Throughput: {:.5f} samples/sec".format(args.iterations / total_time))
+
+def main():
+
+    args = parse_args()
+    logging.info(f"Parameters {args}")
+
+    # CCL related
+    os.environ['MASTER_ADDR'] = str(os.environ.get('MASTER_ADDR', '127.0.0.1'))
+    os.environ['MASTER_PORT'] = '29500'
+    os.environ['RANK'] = str(os.environ.get('PMI_RANK', 0))
+    os.environ['WORLD_SIZE'] = str(os.environ.get('PMI_SIZE', 1))
+
+    if args.dist_url == "env://" and args.world_size == -1:
+        args.world_size = int(os.environ["WORLD_SIZE"])
+        print("World size: ", args.world_size)
+
+    args.distributed = args.world_size > 1 
+    if args.distributed:
+        if args.dist_url == "env://" and args.rank == -1:
+            args.rank = int(os.environ["RANK"])
+
+    # load model
+    pipe = DiffusionPipeline.from_pretrained(args.model_name_or_path)
+
+    # data type
+    if args.precision == "fp32":
+        print("Running fp32 ...")
+        args.dtype=torch.float32
+    elif args.precision == "bf32":
+        print("Running bf32 ...")
+        args.dtype=torch.float32
+    elif args.precision == "bf16":
+        print("Running bf16 ...")
+        args.dtype=torch.bfloat16
+    elif args.precision == "fp16":
+        print("Running fp16 ...")
+        args.dtype=torch.half
+    elif args.precision == "int8-bf16":
+        print("Running int8-bf16 ...")
+        args.dtype=torch.bfloat16
+    elif args.precision == "int8-fp32":
+        print("Running int8-fp32 ...")
+        args.dtype=torch.float32
+    else:
+        raise ValueError("--precision needs to be the following: fp32, bf32, bf16, fp16, int8-bf16, int8-fp32")
+
+    if args.compile_inductor:
+        pipe.precision = torch.float32
+    else:
+        pipe.precision = args.dtype
+    if args.model_name_or_path == "stabilityai/stable-diffusion-2-1":
+        text_encoder_input = torch.ones((1, 77), dtype=torch.int64)
+        input = torch.randn(2, 4, 96, 96).to(memory_format=torch.channels_last).to(dtype=pipe.precision), torch.tensor(921), torch.randn(2, 77, 1024).to(dtype=pipe.precision)
+    elif args.model_name_or_path == "SimianLuo/LCM_Dreamshaper_v7":
+        text_encoder_input = torch.ones((1, 77), dtype=torch.int64)
+        input = torch.randn(1, 4, 96, 96).to(memory_format=torch.channels_last).to(dtype=pipe.precision), torch.tensor(921), torch.randn(1, 77, 768).to(dtype=pipe.precision), torch.randn(1, 256).to(dtype=pipe.precision)
+    else:
+         raise ValueError("This script currently only supports stabilityai/stable-diffusion-2-1 and SimianLuo/LCM_Dreamshaper_v7.")
+
+    # ipex
+    if args.ipex:
+        print("Running IPEX ...")
+        import intel_extension_for_pytorch as ipex
+        if args.precision == "fp32":
+            pipe.text_encoder = ipex.optimize(pipe.text_encoder.eval(), inplace=True, sample_input=text_encoder_input)
+            pipe.unet = ipex.optimize(pipe.unet.eval(), inplace=True)
+            pipe.vae = ipex.optimize(pipe.vae.eval(), inplace=True)
+        elif args.precision == "bf32":
+            ipex.set_fp32_math_mode(mode=ipex.FP32MathMode.BF32, device="cpu")
+            pipe.text_encoder = ipex.optimize(pipe.text_encoder.eval(), inplace=True, auto_kernel_selection=True, sample_input=text_encoder_input)
+            pipe.unet = ipex.optimize(pipe.unet.eval(), inplace=True, auto_kernel_selection=True)
+            pipe.vae = ipex.optimize(pipe.vae.eval(), inplace=True, auto_kernel_selection=True)
+        elif args.precision == "bf16" or args.precision == "fp16":
+            pipe.text_encoder = ipex.optimize(pipe.text_encoder.eval(), dtype=args.dtype, inplace=True, sample_input=text_encoder_input)
+            pipe.unet = ipex.optimize(pipe.unet.eval(), dtype=args.dtype, inplace=True)
+            pipe.vae = ipex.optimize(pipe.vae.eval(), dtype=args.dtype, inplace=True)
+        elif args.precision == "int8-bf16" or args.precision == "int8-fp32":
+            pipe.text_encoder = ipex.optimize(pipe.text_encoder.eval(), dtype=args.dtype, inplace=True, sample_input=text_encoder_input)
+            pipe.unet_highprecision = copy.deepcopy(pipe.unet)
+            pipe.unet_highprecision = ipex.optimize(pipe.unet_highprecision.eval(), dtype=args.dtype, inplace=True)
+            pipe.HIGH_PRECISION_STEPS = 5
+            from quantization_modules import load_int8_model, convert_to_fp32model
+            pipe.unet = load_int8_model(pipe.unet, args.int8_model_path)
+            pipe.unet = convert_to_fp32model(pipe.unet)
+            pipe.vae = ipex.optimize(pipe.vae.eval(), dtype=args.dtype, inplace=True)
+        else:
+            raise ValueError("--precision needs to be the following:: fp32, bf32, bf16, fp16, int8-bf16, int8-fp32")
+
+    # jit trace
+    if args.jit:
+        print("JIT trace ...")
+        # from utils_vis import make_dot, draw
+        if args.precision == "bf16" or args.precision == "fp16":
+            with torch.cpu.amp.autocast(dtype=args.dtype), torch.no_grad():
+                pipe.traced_unet = torch.jit.trace(pipe.unet, input, strict=False)
+                pipe.traced_unet = torch.jit.freeze(pipe.traced_unet)
+                pipe.traced_unet(*input)
+                pipe.traced_unet(*input)
+                # print(pipe.traced_unet.graph_for(input))
+        elif args.precision == "int8-bf16":
+            with torch.cpu.amp.autocast(dtype=args.dtype), torch.no_grad():
+                pipe.traced_unet = torch.jit.trace(pipe.unet, input, strict=False)
+                pipe.traced_unet = torch.jit.freeze(pipe.traced_unet)
+                pipe.traced_unet(*input)
+                pipe.traced_unet(*input)
+                # print(pipe.traced_unet.graph_for(input))
+                pipe.unet_highprecision = torch.jit.trace(pipe.unet_highprecision, input, strict=False)
+                pipe.unet_highprecision = torch.jit.freeze(pipe.unet_highprecision)
+                pipe.unet_highprecision(*input)
+                pipe.unet_highprecision(*input)
+                # print(pipe.unet_highprecision.graph_for(input))
+        elif args.precision == "int8-fp32":
+            with torch.no_grad():
+                pipe.traced_unet = torch.jit.trace(pipe.unet, input, strict=False)
+                pipe.traced_unet = torch.jit.freeze(pipe.traced_unet)
+                pipe.traced_unet(*input)
+                pipe.traced_unet(*input)
+                # print(pipe.traced_unet.graph_for(input))
+                pipe.unet_highprecision = torch.jit.trace(pipe.unet_highprecision, input, strict=False)
+                pipe.unet_highprecision = torch.jit.freeze(pipe.unet_highprecision)
+                pipe.unet_highprecision(*input)
+                pipe.unet_highprecision(*input)
+                # print(pipe.unet_highprecision.graph_for(input))
+        else:
+            with torch.no_grad():
+                pipe.traced_unet = torch.jit.trace(pipe.unet, input, strict=False)
+                pipe.traced_unet = torch.jit.freeze(pipe.traced_unet)
+                pipe.traced_unet(*input)
+                pipe.traced_unet(*input)
+                # print(pipe.traced_unet.graph_for(input))
+
+    # torch.compile with ipex backend
+    if args.compile_ipex:
+        print("torch.compile with ipex backend ...")
+        if args.precision == "fp32" or args.precision == "bf16":
+            import intel_extension_for_pytorch as ipex
+            pipe.text_encoder = ipex.optimize(pipe.text_encoder.eval(), dtype=args.dtype, weights_prepack=False)
+            pipe.unet = ipex.optimize(pipe.unet.eval(), dtype=args.dtype, weights_prepack=False)
+            pipe.vae = ipex.optimize(pipe.vae.eval(), dtype=args.dtype, weights_prepack=False)
+            ipex._set_compiler_backend("torchscript")
+            pipe.text_encoder = torch.compile(pipe.text_encoder, backend="ipex")
+            pipe.unet = torch.compile(pipe.unet, backend="ipex")
+            pipe.vae.decode = torch.compile(pipe.vae.decode, backend="ipex")
+        else:
+            raise ValueError("If you want to use torch.compile with ipex backend, --precision needs to be the following: fp32, bf16")
+    # torch.compile with inductor backend
+    if args.compile_inductor:
+        print("torch.compile with inductor backend ...")
+        from torch._inductor import config as inductor_config
+        inductor_config.cpp_wrapper = True
+        if args.precision == "fp32":
+            with torch.no_grad():
+                pipe.unet = torch.compile(pipe.unet)
+                pipe.unet(*input)
+                pipe.unet(*input)
+        elif args.precision == "bf16":
+            with torch.cpu.amp.autocast(), torch.no_grad():
+                pipe.unet = torch.compile(pipe.unet)
+                pipe.unet(*input)
+                pipe.unet(*input)
+        elif args.precision == "fp16":
+            with torch.cpu.amp.autocast(dtype=torch.half), torch.no_grad():
+                pipe.unet = torch.compile(pipe.unet)
+                pipe.unet(*input)
+                pipe.unet(*input)
+        elif args.precision == "int8-fp32":
+            pipe.HIGH_PRECISION_STEPS = 5
+            pipe.unet_highprecision = copy.deepcopy(pipe.unet)
+            with torch.no_grad():
+                pipe.unet_highprecision = torch.compile(pipe.unet_highprecision)
+                pipe.unet_highprecision(*input)
+                pipe.unet_highprecision(*input)
+            from torch.ao.quantization.quantize_pt2e import prepare_pt2e, convert_pt2e
+            import torch.ao.quantization.quantizer.x86_inductor_quantizer as xiq
+            from torch.ao.quantization.quantizer.x86_inductor_quantizer import X86InductorQuantizer
+            from torch._export import capture_pre_autograd_graph
+            with torch.no_grad():
+                pipe.traced_unet = capture_pre_autograd_graph(pipe.unet, input)
+                quantizer = X86InductorQuantizer()
+                quantizer.set_global(xiq.get_default_x86_inductor_quantization_config())
+                pipe.traced_unet = prepare_pt2e(pipe.traced_unet, quantizer)
+                # calibration
+                pipe(args.prompt)
+                pipe.traced_unet = convert_pt2e(pipe.traced_unet)
+                torch.ao.quantization.move_exported_model_to_eval(pipe.traced_unet)
+                pipe.traced_unet = torch.compile(pipe.traced_unet)
+                pipe.traced_unet(*input)
+                pipe.traced_unet(*input)
+        elif args.precision == "int8-bf16":
+            pipe.HIGH_PRECISION_STEPS = 5
+            pipe.unet_highprecision = copy.deepcopy(pipe.unet)
+            with torch.cpu.amp.autocast(), torch.no_grad():
+                pipe.unet_highprecision = torch.compile(pipe.unet_highprecision)
+                pipe.unet_highprecision(*input)
+                pipe.unet_highprecision(*input)
+            from torch.ao.quantization.quantize_pt2e import prepare_pt2e, convert_pt2e
+            import torch.ao.quantization.quantizer.x86_inductor_quantizer as xiq
+            from torch.ao.quantization.quantizer.x86_inductor_quantizer import X86InductorQuantizer
+            from torch._export import capture_pre_autograd_graph
+            with torch.no_grad():
+                pipe.traced_unet = capture_pre_autograd_graph(pipe.unet, input)
+                quantizer = X86InductorQuantizer()
+                quantizer.set_global(xiq.get_default_x86_inductor_quantization_config())
+                pipe.traced_unet = prepare_pt2e(pipe.traced_unet, quantizer)
+                # calibration
+                pipe(args.prompt)
+                pipe.traced_unet = convert_pt2e(pipe.traced_unet)
+                torch.ao.quantization.move_exported_model_to_eval(pipe.traced_unet)
+            with torch.cpu.amp.autocast(), torch.no_grad():
+                pipe.traced_unet = torch.compile(pipe.traced_unet)
+                pipe.traced_unet(*input)
+                pipe.traced_unet(*input)
+        else:
+            raise ValueError("If you want to use torch.compile with inductor backend, --precision needs to be the following: fp32, bf16, int8-bf16, int8-fp32")
+
+    if args.distributed:
+        import oneccl_bindings_for_pytorch
+        torch.distributed.init_process_group(backend=args.dist_backend,
+                                init_method=args.dist_url,
+                                world_size=args.world_size,
+                                rank=args.rank)
+        print("Rank and world size: ", torch.distributed.get_rank()," ", torch.distributed.get_world_size())              
+        # print("Create DistributedDataParallel in CPU")
+        # pipe = torch.nn.parallel.DistributedDataParallel(pipe)
+
+    if args.accuracy:
+        # prepare dataloader
+        val_coco = dset.CocoCaptions(root = '{}/val2017'.format(args.dataset_path),
+                                    annFile = '{}/annotations/captions_val2017.json'.format(args.dataset_path),
+                                    transform=transforms.Compose([transforms.Resize((512, 512)), transforms.PILToTensor(), ]))
+
+        if args.distributed:
+            val_sampler = torch.utils.data.distributed.DistributedSampler(val_coco, shuffle=False)
+        else:
+            val_sampler = None
+
+        val_dataloader = torch.utils.data.DataLoader(val_coco,
+                                                    batch_size=1,
+                                                    shuffle=False,
+                                                    num_workers=0,
+                                                    sampler=val_sampler)
+
+    # benchmark
+    if args.benchmark:
+        print("Running benchmark ...")
+        if args.weight_sharing:
+            print("weight sharing ...")
+            threads = []
+            for i in range(1, args.number_instance+1):
+                thread = threading.Thread(target=run_weights_sharing_model, args=(pipe, i, args))
+                threads.append(thread)
+                thread.start()
+            for thread in threads:
+                thread.join()
+            exit()
+        else:
+            total_time = 0
+            for i in range(args.iterations + args.warmup_iterations):
+                # run model
+                start = time.time()
+                if args.precision == "bf16" or args.precision == "fp16" or args.precision == "int8-bf16":
+                    with torch.cpu.amp.autocast(dtype=args.dtype), torch.no_grad():
+                        output = pipe(args.prompt, generator=torch.manual_seed(args.seed)).images
+                else:
+                    with torch.no_grad():
+                        output = pipe(args.prompt, generator=torch.manual_seed(args.seed)).images
+                end = time.time()
+                print('time per prompt(s): {:.2f}'.format((end - start)))
+                if i >= args.warmup_iterations:
+                    total_time += end - start
+
+            print("Latency: {:.2f} s".format(total_time / args.iterations))
+            print("Throughput: {:.5f} samples/sec".format(args.iterations / total_time))
+
+    if args.accuracy:
+        print("Running accuracy ...")
+        # run model
+        if args.distributed:
+            torch.distributed.barrier()
+        fid = FrechetInceptionDistance(normalize=True)
+        for i, (images, prompts) in enumerate(tqdm(val_dataloader)):
+            prompt = prompts[0][0]
+            real_image = images[0]
+            print("prompt: ", prompt)
+            if args.precision == "bf16" or args.precision == "fp16" or args.precision == "int8-bf16":
+                with torch.cpu.amp.autocast(dtype=args.dtype), torch.no_grad():
+                    output = pipe(prompt, generator=torch.manual_seed(args.seed), output_type="numpy").images
+            else:
+                with torch.no_grad():
+                    output = pipe(prompt, generator=torch.manual_seed(args.seed), output_type="numpy").images
+
+            if args.output_dir:
+                if not os.path.exists(args.output_dir):
+                    os.mkdir(args.output_dir)
+                image_name = time.strftime("%Y%m%d_%H%M%S")
+                Image.fromarray((output[0] * 255).round().astype("uint8")).save(f"{args.output_dir}/fake_image_{image_name}.png")
+                Image.fromarray(real_image.permute(1, 2, 0).numpy()).save(f"{args.output_dir}/real_image_{image_name}.png")
+
+            fake_image = torch.tensor(output[0]).unsqueeze(0).permute(0, 3, 1, 2)
+            real_image = real_image.unsqueeze(0) / 255.0
+            
+            fid.update(real_image, real=True)
+            fid.update(fake_image, real=False)
+
+            if args.iterations > 0 and i == args.iterations - 1:
+                break
+
+        print(f"FID: {float(fid.compute())}")
+
+    # profile
+    if args.profile:
+        print("Running profiling ...")
+        with torch.profiler.profile(activities=[torch.profiler.ProfilerActivity.CPU], record_shapes=True) as p:
+            if args.precision == "bf16" or args.precision == "fp16" or args.precision == "int8-bf16":
+                with torch.cpu.amp.autocast(dtype=args.dtype), torch.no_grad():
+                    pipe(args.prompt, generator=torch.manual_seed(args.seed)).images
+            else:
+                with torch.no_grad():
+                    pipe(args.prompt, generator=torch.manual_seed(args.seed)).images
+
+        output = p.key_averages().table(sort_by="self_cpu_time_total")
+        print(output)
+        timeline_dir = str(pathlib.Path.cwd()) + '/timeline/'
+        if not os.path.exists(timeline_dir):
+            try:
+                os.makedirs(timeline_dir)
+            except:
+                pass
+        timeline_file = timeline_dir + 'timeline-' + str(torch.backends.quantized.engine) + '-' + \
+                    'stable_diffusion-' + '-' + str(os.getpid()) + '.json'
+        p.export_chrome_trace(timeline_file)
+
+if __name__ == '__main__':
+    main()
diff --git a/models/diffusion/pytorch/stable_diffusion/quantization_modules.py b/models/diffusion/pytorch/stable_diffusion/quantization_modules.py
new file mode 100644
index 000000000..e241d696b
--- /dev/null
+++ b/models/diffusion/pytorch/stable_diffusion/quantization_modules.py
@@ -0,0 +1,205 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import torch
+import torch.nn.functional as F
+from torch.ao.quantization import FakeQuantize, FakeQuantizeBase, default_fake_quant, default_per_channel_weight_fake_quant, default_fused_act_fake_quant, default_fused_per_channel_wt_fake_quant
+from torch.ao.nn.quantized import Linear, Conv2d, Quantize
+try:
+    from peft.tuners.lora import Linear as LoRALinear
+    from peft.utils import transpose
+    peft_available = True
+except:
+    peft_available = False
+
+ACT_REDUCE_RANGE = False
+WEIGHT_REDUCE_RANGE = False
+
+
+class QuantizedLinear(Linear):
+    def forward(self, x):
+        return super().forward(
+            self.input_quant(x.float())
+        ).dequantize().bfloat16()
+
+
+class FakeQuantLinear(torch.nn.Linear):
+    def __init__(self, module: torch.nn.Linear):
+        self.__dict__.update(module.__dict__.copy())
+        self.add_module('activation_pre_process', default_fake_quant(reduce_range=ACT_REDUCE_RANGE))
+        self.add_module('weight_fake_quant', default_per_channel_weight_fake_quant(reduce_range=WEIGHT_REDUCE_RANGE))
+        self.add_module('activation_post_process', default_fake_quant(reduce_range=ACT_REDUCE_RANGE))
+        self.is_lora_layer = True if peft_available and isinstance(module, LoRALinear) else False
+
+    def forward(self, x):
+        x = self.activation_pre_process(x)
+        weight = self.weight
+        if self.is_lora_layer and not self.disable_adapters and self.r[self.active_adapter] > 0:
+            lora_weight = transpose(
+                    self.lora_B[self.active_adapter].weight @ self.lora_A[self.active_adapter].weight,
+                    self.fan_in_fan_out,
+                ) * self.scaling[self.active_adapter]
+            weight = weight + lora_weight
+        x = F.linear(x, self.weight_fake_quant(weight), self.bias)
+        x = self.activation_post_process(x)
+        return x
+
+    def convert(self):
+        if self.is_lora_layer and not self.disable_adapters and self.r[self.active_adapter] > 0:
+            lora_weight = transpose(
+                    self.lora_B[self.active_adapter].weight @ self.lora_A[self.active_adapter].weight,
+                    self.fan_in_fan_out,
+                ) * self.scaling[self.active_adapter]
+            self.weight.data += lora_weight.data
+        module = QuantizedLinear.from_float(self)
+        input_quant = torch.quantization.QuantStub()
+        input_quant.add_module('activation_post_process', self.activation_pre_process)
+        input_quant = Quantize.from_float(input_quant)
+        module.add_module('input_quant', input_quant)
+        return module
+
+
+class QuantizedConv2d(Conv2d):
+    def forward(self, x):
+        return super().forward(
+            self.input_quant(x.float())
+        ).dequantize().bfloat16()
+
+
+class FakeQuantConv2d(torch.nn.Conv2d):
+    def __init__(self, module: torch.nn.Conv2d):
+        self.__dict__.update(module.__dict__.copy())
+        self.add_module('activation_pre_process', default_fake_quant(reduce_range=ACT_REDUCE_RANGE))
+        self.add_module('weight_fake_quant', default_per_channel_weight_fake_quant(reduce_range=WEIGHT_REDUCE_RANGE))
+        self.add_module('activation_post_process', default_fake_quant(reduce_range=ACT_REDUCE_RANGE))
+
+    def forward(self, x):
+        x = self.activation_pre_process(x)
+        x = self._conv_forward(x, self.weight_fake_quant(self.weight), self.bias)
+        x = self.activation_post_process(x)
+        return x
+
+    def convert(self):
+        module = QuantizedConv2d.from_float(self)
+        input_quant = torch.quantization.QuantStub()
+        input_quant.add_module('activation_post_process', self.activation_pre_process)
+        input_quant = Quantize.from_float(input_quant)
+        module.add_module('input_quant', input_quant)
+        return module
+
+
+def get_submodules(model, key):
+    parent = model.get_submodule(".".join(key.split(".")[:-1]))
+    target_name = key.split(".")[-1]
+    target = model.get_submodule(key)
+    return parent, target, target_name
+
+def find_and_replace(model, fake_quant=True):
+    assert isinstance(model, torch.nn.Module), "Only support torch Module."
+    key_list = [key for key, _ in model.named_modules()]
+    for key in key_list:
+        try:
+            parent, target, target_name = get_submodules(model, key)
+        except:
+            continue
+        if fake_quant:
+            if isinstance(target, torch.nn.Linear):
+                setattr(parent, target_name, FakeQuantLinear(target))
+            elif isinstance(target, torch.nn.Conv2d):
+                setattr(parent, target_name, FakeQuantConv2d(target))
+        else:
+            if isinstance(target, FakeQuantLinear):
+                setattr(parent, target_name, target.convert())
+            elif isinstance(target, FakeQuantConv2d):
+                setattr(parent, target_name, target.convert())
+
+def convert2quantized_model(model):
+    model.to(torch.device("cpu"))
+    find_and_replace(model, fake_quant=False)
+    return model
+
+class FP32Conv2d(torch.nn.Module):
+    def __init__(self, module: Conv2d):
+        super(FP32Conv2d, self).__init__()
+        self.quant_weight = module.weight()
+        self.bias = module.bias()
+        self.stride = module.stride
+        self.padding = module.padding
+        self.dilation = module.dilation
+        self.groups = module.groups
+        self.input_quant = module.input_quant
+
+    def forward(self, x):
+        x = self.input_quant(x.float().to(memory_format=torch.channels_last)).dequantize()
+        return F.conv2d(x, self.quant_weight.dequantize(), self.bias, self.stride,
+                        self.padding, self.dilation, self.groups)
+
+class FP32Linear(torch.nn.Module):
+    def __init__(self, module: Linear):
+        super(FP32Linear, self).__init__()
+        self.quant_weight = module.weight()
+        self.bias = module.bias()
+        self.input_quant = module.input_quant
+
+    def forward(self, x):
+        x = self.input_quant(x.float()).dequantize()
+        return F.linear(x, self.quant_weight.dequantize(), self.bias)
+
+def convert_to_fp32model(model):
+    assert isinstance(model, torch.nn.Module), "Only support torch Module."
+    key_list = [key for key, _ in model.named_modules()]
+    for key in key_list:
+        try:
+            parent, target, target_name = get_submodules(model, key)
+        except:
+            continue
+        if isinstance(target, QuantizedLinear):
+            setattr(parent, target_name, FP32Linear(target))
+        elif isinstance(target, QuantizedConv2d):
+            setattr(parent, target_name, FP32Conv2d(target))
+    return model
+
+def disable_all_observers(model):
+    assert isinstance(model, torch.nn.Module), "Only support torch Module."
+    for name, module in model.named_modules():
+        if isinstance(module, FakeQuantizeBase):
+            module.disable_observer()
+
+def sync_all_observers(model):
+    assert isinstance(model, torch.nn.Module), "Only support torch Module."
+    for name, module in model.named_modules():
+        if isinstance(module, FakeQuantize):
+            _scale, _zero_point = module.calculate_qparams()
+            _scale, _zero_point = _scale.to(module.scale.device), _zero_point.to(module.zero_point.device)
+            if module.scale.shape != _scale.shape:
+                module.scale.resize_(_scale.shape)
+                module.zero_point.resize_(_zero_point.shape)
+            module.scale.copy_(_scale)
+            module.zero_point.copy_(_zero_point)
+
+def load_int8_model(fp32_model, int8_model_path, fake_quantize_model=False):
+    find_and_replace(fp32_model)
+    if fake_quantize_model:
+        fp32_model.load_state_dict(torch.load(int8_model_path))
+        disable_all_observers(fp32_model)
+        sync_all_observers(fp32_model)
+    int8_model = convert2quantized_model(fp32_model)
+    print('Converted to quantized model.')
+    if not fake_quantize_model:
+        int8_model.load_state_dict(torch.load(int8_model_path))
+    return int8_model
diff --git a/models/diffusion/pytorch/stable_diffusion/textual_inversion.py b/models/diffusion/pytorch/stable_diffusion/textual_inversion.py
new file mode 100644
index 000000000..61017febb
--- /dev/null
+++ b/models/diffusion/pytorch/stable_diffusion/textual_inversion.py
@@ -0,0 +1,1041 @@
+#!/usr/bin/env python
+# coding=utf-8
+# Copyright 2023 The HuggingFace Inc. team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+
+import argparse
+import logging
+import math
+import os
+import random
+import shutil
+import warnings
+from pathlib import Path
+
+import numpy as np
+import PIL
+import safetensors
+import torch
+import torch.nn.functional as F
+import torch.utils.checkpoint
+import transformers
+from accelerate import Accelerator
+from accelerate.logging import get_logger
+from accelerate.utils import ProjectConfiguration, set_seed
+from huggingface_hub import create_repo, upload_folder
+
+# TODO: remove and import from diffusers.utils when the new version of diffusers is released
+from packaging import version
+from PIL import Image
+from torch.utils.data import Dataset
+from torchvision import transforms
+from tqdm.auto import tqdm
+from transformers import CLIPTextModel, CLIPTokenizer
+
+import diffusers
+from diffusers import (
+    AutoencoderKL,
+    DDPMScheduler,
+    DiffusionPipeline,
+    DPMSolverMultistepScheduler,
+    StableDiffusionPipeline,
+    UNet2DConditionModel,
+)
+from diffusers.optimization import get_scheduler
+from diffusers.utils import check_min_version, is_wandb_available
+from diffusers.utils.import_utils import is_xformers_available
+
+import time
+
+
+if is_wandb_available():
+    import wandb
+
+if version.parse(version.parse(PIL.__version__).base_version) >= version.parse("9.1.0"):
+    PIL_INTERPOLATION = {
+        "linear": PIL.Image.Resampling.BILINEAR,
+        "bilinear": PIL.Image.Resampling.BILINEAR,
+        "bicubic": PIL.Image.Resampling.BICUBIC,
+        "lanczos": PIL.Image.Resampling.LANCZOS,
+        "nearest": PIL.Image.Resampling.NEAREST,
+    }
+else:
+    PIL_INTERPOLATION = {
+        "linear": PIL.Image.LINEAR,
+        "bilinear": PIL.Image.BILINEAR,
+        "bicubic": PIL.Image.BICUBIC,
+        "lanczos": PIL.Image.LANCZOS,
+        "nearest": PIL.Image.NEAREST,
+    }
+# ------------------------------------------------------------------------------
+
+
+# Will error if the minimal version of diffusers is not installed. Remove at your own risks.
+check_min_version("0.23.0")
+
+logger = get_logger(__name__)
+
+
+def save_model_card(repo_id: str, images=None, base_model=str, repo_folder=None):
+    img_str = ""
+    for i, image in enumerate(images):
+        image.save(os.path.join(repo_folder, f"image_{i}.png"))
+        img_str += f"![img_{i}](./image_{i}.png)\n"
+
+    yaml = f"""
+---
+license: creativeml-openrail-m
+base_model: {base_model}
+tags:
+- stable-diffusion
+- stable-diffusion-diffusers
+- text-to-image
+- diffusers
+- textual_inversion
+inference: true
+---
+    """
+    model_card = f"""
+# Textual inversion text2image fine-tuning - {repo_id}
+These are textual inversion adaption weights for {base_model}. You can find some example images in the following. \n
+{img_str}
+"""
+    with open(os.path.join(repo_folder, "README.md"), "w") as f:
+        f.write(yaml + model_card)
+
+
+def log_validation(text_encoder, tokenizer, unet, vae, args, accelerator, weight_dtype, epoch):
+    logger.info(
+        f"Running validation... \n Generating {args.num_validation_images} images with prompt:"
+        f" {args.validation_prompt}."
+    )
+    # create pipeline (note: unet and vae are loaded again in float32)
+    pipeline = DiffusionPipeline.from_pretrained(
+        args.pretrained_model_name_or_path,
+        text_encoder=accelerator.unwrap_model(text_encoder),
+        tokenizer=tokenizer,
+        unet=unet,
+        vae=vae,
+        safety_checker=None,
+        revision=args.revision,
+        torch_dtype=weight_dtype,
+    )
+    pipeline.scheduler = DPMSolverMultistepScheduler.from_config(pipeline.scheduler.config)
+    pipeline = pipeline.to(accelerator.device)
+    pipeline.set_progress_bar_config(disable=True)
+
+    # run inference
+    generator = None if args.seed is None else torch.Generator(device=accelerator.device).manual_seed(args.seed)
+    images = []
+    for _ in range(args.num_validation_images):
+        with torch.autocast("cuda"):
+            image = pipeline(args.validation_prompt, num_inference_steps=25, generator=generator).images[0]
+        images.append(image)
+
+    for tracker in accelerator.trackers:
+        if tracker.name == "tensorboard":
+            np_images = np.stack([np.asarray(img) for img in images])
+            tracker.writer.add_images("validation", np_images, epoch, dataformats="NHWC")
+        if tracker.name == "wandb":
+            tracker.log(
+                {
+                    "validation": [
+                        wandb.Image(image, caption=f"{i}: {args.validation_prompt}") for i, image in enumerate(images)
+                    ]
+                }
+            )
+
+    del pipeline
+    torch.cuda.empty_cache()
+    return images
+
+
+def save_progress(text_encoder, placeholder_token_ids, accelerator, args, save_path, safe_serialization=True):
+    logger.info("Saving embeddings")
+    learned_embeds = (
+        accelerator.unwrap_model(text_encoder)
+        .get_input_embeddings()
+        .weight[min(placeholder_token_ids) : max(placeholder_token_ids) + 1]
+    )
+    learned_embeds_dict = {args.placeholder_token: learned_embeds.detach().cpu()}
+
+    if safe_serialization:
+        safetensors.torch.save_file(learned_embeds_dict, save_path, metadata={"format": "pt"})
+    else:
+        torch.save(learned_embeds_dict, save_path)
+
+
+def parse_args():
+    parser = argparse.ArgumentParser(description="Simple example of a training script.")
+    parser.add_argument(
+        "--save_steps",
+        type=int,
+        default=500,
+        help="Save learned_embeds.bin every X updates steps.",
+    )
+    parser.add_argument(
+        "--save_as_full_pipeline",
+        action="store_true",
+        help="Save the complete stable diffusion pipeline.",
+    )
+    parser.add_argument(
+        "--num_vectors",
+        type=int,
+        default=1,
+        help="How many textual inversion vectors shall be used to learn the concept.",
+    )
+    parser.add_argument(
+        "--pretrained_model_name_or_path",
+        type=str,
+        default=None,
+        required=True,
+        help="Path to pretrained model or model identifier from huggingface.co/models.",
+    )
+    parser.add_argument(
+        "--revision",
+        type=str,
+        default=None,
+        required=False,
+        help="Revision of pretrained model identifier from huggingface.co/models.",
+    )
+    parser.add_argument(
+        "--tokenizer_name",
+        type=str,
+        default=None,
+        help="Pretrained tokenizer name or path if not the same as model_name",
+    )
+    parser.add_argument(
+        "--train_data_dir", type=str, default=None, required=True, help="A folder containing the training data."
+    )
+    parser.add_argument(
+        "--placeholder_token",
+        type=str,
+        default=None,
+        required=True,
+        help="A token to use as a placeholder for the concept.",
+    )
+    parser.add_argument(
+        "--initializer_token", type=str, default=None, required=True, help="A token to use as initializer word."
+    )
+    parser.add_argument("--learnable_property", type=str, default="object", help="Choose between 'object' and 'style'")
+    parser.add_argument("--repeats", type=int, default=100, help="How many times to repeat the training data.")
+    parser.add_argument(
+        "--output_dir",
+        type=str,
+        default="text-inversion-model",
+        help="The output directory where the model predictions and checkpoints will be written.",
+    )
+    parser.add_argument("--seed", type=int, default=0, help="A seed for reproducible training.")
+    parser.add_argument(
+        "--resolution",
+        type=int,
+        default=512,
+        help=(
+            "The resolution for input images, all the images in the train/validation dataset will be resized to this"
+            " resolution"
+        ),
+    )
+    parser.add_argument(
+        "--center_crop", action="store_true", help="Whether to center crop images before resizing to resolution."
+    )
+    parser.add_argument(
+        "--train_batch_size", type=int, default=16, help="Batch size (per device) for the training dataloader."
+    )
+    parser.add_argument("--num_train_epochs", type=int, default=100)
+    parser.add_argument(
+        "--max_train_steps",
+        type=int,
+        default=None,
+        help="Total number of training steps to perform.  If provided, overrides num_train_epochs.",
+    )
+    parser.add_argument(
+        "--gradient_accumulation_steps",
+        type=int,
+        default=1,
+        help="Number of updates steps to accumulate before performing a backward/update pass.",
+    )
+    parser.add_argument(
+        "--gradient_checkpointing",
+        action="store_true",
+        help="Whether or not to use gradient checkpointing to save memory at the expense of slower backward pass.",
+    )
+    parser.add_argument(
+        "--learning_rate",
+        type=float,
+        default=1e-4,
+        help="Initial learning rate (after the potential warmup period) to use.",
+    )
+    parser.add_argument(
+        "--scale_lr",
+        action="store_true",
+        default=False,
+        help="Scale the learning rate by the number of GPUs, gradient accumulation steps, and batch size.",
+    )
+    parser.add_argument(
+        "--lr_scheduler",
+        type=str,
+        default="constant",
+        help=(
+            'The scheduler type to use. Choose between ["linear", "cosine", "cosine_with_restarts", "polynomial",'
+            ' "constant", "constant_with_warmup"]'
+        ),
+    )
+    parser.add_argument(
+        "--lr_warmup_steps", type=int, default=500, help="Number of steps for the warmup in the lr scheduler."
+    )
+    parser.add_argument(
+        "--lr_num_cycles",
+        type=int,
+        default=1,
+        help="Number of hard resets of the lr in cosine_with_restarts scheduler.",
+    )
+    parser.add_argument(
+        "--dataloader_num_workers",
+        type=int,
+        default=0,
+        help=(
+            "Number of subprocesses to use for data loading. 0 means that the data will be loaded in the main process."
+        ),
+    )
+    parser.add_argument("--adam_beta1", type=float, default=0.9, help="The beta1 parameter for the Adam optimizer.")
+    parser.add_argument("--adam_beta2", type=float, default=0.999, help="The beta2 parameter for the Adam optimizer.")
+    parser.add_argument("--adam_weight_decay", type=float, default=1e-2, help="Weight decay to use.")
+    parser.add_argument("--adam_epsilon", type=float, default=1e-08, help="Epsilon value for the Adam optimizer")
+    parser.add_argument("--push_to_hub", action="store_true", help="Whether or not to push the model to the Hub.")
+    parser.add_argument("--hub_token", type=str, default=None, help="The token to use to push to the Model Hub.")
+    parser.add_argument(
+        "--hub_model_id",
+        type=str,
+        default=None,
+        help="The name of the repository to keep in sync with the local `output_dir`.",
+    )
+    parser.add_argument(
+        "--logging_dir",
+        type=str,
+        default="logs",
+        help=(
+            "[TensorBoard](https://www.tensorflow.org/tensorboard) log directory. Will default to"
+            " *output_dir/runs/**CURRENT_DATETIME_HOSTNAME***."
+        ),
+    )
+    parser.add_argument(
+        "--precision",
+        type=str,
+        default="fp32",
+        choices=["fp32", "bf32", "bf16", "fp16"],
+        help=(
+            "precision, only support fp32, bf32, bf16, fp16."
+        ),
+    )
+    parser.add_argument(
+        "--allow_tf32",
+        action="store_true",
+        help=(
+            "Whether or not to allow TF32 on Ampere GPUs. Can be used to speed up training. For more information, see"
+            " https://pytorch.org/docs/stable/notes/cuda.html#tensorfloat-32-tf32-on-ampere-devices"
+        ),
+    )
+    parser.add_argument(
+        "--report_to",
+        type=str,
+        default="tensorboard",
+        help=(
+            'The integration to report the results and logs to. Supported platforms are `"tensorboard"`'
+            ' (default), `"wandb"` and `"comet_ml"`. Use `"all"` to report to all integrations.'
+        ),
+    )
+    parser.add_argument(
+        "--validation_prompt",
+        type=str,
+        default=None,
+        help="A prompt that is used during validation to verify that the model is learning.",
+    )
+    parser.add_argument(
+        "--num_validation_images",
+        type=int,
+        default=4,
+        help="Number of images that should be generated during validation with `validation_prompt`.",
+    )
+    parser.add_argument(
+        "--validation_steps",
+        type=int,
+        default=100,
+        help=(
+            "Run validation every X steps. Validation consists of running the prompt"
+            " `args.validation_prompt` multiple times: `args.num_validation_images`"
+            " and logging the images."
+        ),
+    )
+    parser.add_argument(
+        "--validation_epochs",
+        type=int,
+        default=None,
+        help=(
+            "Deprecated in favor of validation_steps. Run validation every X epochs. Validation consists of running the prompt"
+            " `args.validation_prompt` multiple times: `args.num_validation_images`"
+            " and logging the images."
+        ),
+    )
+    parser.add_argument("--local_rank", type=int, default=-1, help="For distributed training: local_rank")
+    parser.add_argument(
+        "--checkpointing_steps",
+        type=int,
+        default=500,
+        help=(
+            "Save a checkpoint of the training state every X updates. These checkpoints are only suitable for resuming"
+            " training using `--resume_from_checkpoint`."
+        ),
+    )
+    parser.add_argument(
+        "--checkpoints_total_limit",
+        type=int,
+        default=None,
+        help=("Max number of checkpoints to store."),
+    )
+    parser.add_argument(
+        "--resume_from_checkpoint",
+        type=str,
+        default=None,
+        help=(
+            "Whether training should be resumed from a previous checkpoint. Use a path saved by"
+            ' `--checkpointing_steps`, or `"latest"` to automatically select the last available checkpoint.'
+        ),
+    )
+    parser.add_argument(
+        "--enable_xformers_memory_efficient_attention", action="store_true", help="Whether or not to use xformers."
+    )
+    parser.add_argument(
+        "--no_safe_serialization",
+        action="store_true",
+        help="If specified save the checkpoint not in `safetensors` format, but in original PyTorch format instead.",
+    )
+    parser.add_argument('-w', '--warmup_iterations', default=0, type=int, help='number of warmup iterations to run')
+    parser.add_argument('--train-no-eval', action='store_true', default=False, help='only train, but not evaluate model on validation set')
+    parser.add_argument('--ipex', action='store_true', default=False, help='ipex')
+
+    args = parser.parse_args()
+    env_local_rank = int(os.environ.get("LOCAL_RANK", -1))
+    if env_local_rank != -1 and env_local_rank != args.local_rank:
+        args.local_rank = env_local_rank
+
+    if args.train_data_dir is None:
+        raise ValueError("You must specify a train data directory.")
+
+    return args
+
+
+imagenet_templates_small = [
+    "a photo of a {}",
+    "a rendering of a {}",
+    "a cropped photo of the {}",
+    "the photo of a {}",
+    "a photo of a clean {}",
+    "a photo of a dirty {}",
+    "a dark photo of the {}",
+    "a photo of my {}",
+    "a photo of the cool {}",
+    "a close-up photo of a {}",
+    "a bright photo of the {}",
+    "a cropped photo of a {}",
+    "a photo of the {}",
+    "a good photo of the {}",
+    "a photo of one {}",
+    "a close-up photo of the {}",
+    "a rendition of the {}",
+    "a photo of the clean {}",
+    "a rendition of a {}",
+    "a photo of a nice {}",
+    "a good photo of a {}",
+    "a photo of the nice {}",
+    "a photo of the small {}",
+    "a photo of the weird {}",
+    "a photo of the large {}",
+    "a photo of a cool {}",
+    "a photo of a small {}",
+]
+
+imagenet_style_templates_small = [
+    "a painting in the style of {}",
+    "a rendering in the style of {}",
+    "a cropped painting in the style of {}",
+    "the painting in the style of {}",
+    "a clean painting in the style of {}",
+    "a dirty painting in the style of {}",
+    "a dark painting in the style of {}",
+    "a picture in the style of {}",
+    "a cool painting in the style of {}",
+    "a close-up painting in the style of {}",
+    "a bright painting in the style of {}",
+    "a cropped painting in the style of {}",
+    "a good painting in the style of {}",
+    "a close-up painting in the style of {}",
+    "a rendition in the style of {}",
+    "a nice painting in the style of {}",
+    "a small painting in the style of {}",
+    "a weird painting in the style of {}",
+    "a large painting in the style of {}",
+]
+
+
+class TextualInversionDataset(Dataset):
+    def __init__(
+        self,
+        data_root,
+        tokenizer,
+        learnable_property="object",  # [object, style]
+        size=512,
+        repeats=100,
+        interpolation="bicubic",
+        flip_p=0.5,
+        set="train",
+        placeholder_token="*",
+        center_crop=False,
+    ):
+        self.data_root = data_root
+        self.tokenizer = tokenizer
+        self.learnable_property = learnable_property
+        self.size = size
+        self.placeholder_token = placeholder_token
+        self.center_crop = center_crop
+        self.flip_p = flip_p
+
+        self.image_paths = [os.path.join(self.data_root, file_path) for file_path in os.listdir(self.data_root)]
+
+        self.num_images = len(self.image_paths)
+        self._length = self.num_images
+
+        if set == "train":
+            self._length = self.num_images * repeats
+
+        self.interpolation = {
+            "linear": PIL_INTERPOLATION["linear"],
+            "bilinear": PIL_INTERPOLATION["bilinear"],
+            "bicubic": PIL_INTERPOLATION["bicubic"],
+            "lanczos": PIL_INTERPOLATION["lanczos"],
+        }[interpolation]
+
+        self.templates = imagenet_style_templates_small if learnable_property == "style" else imagenet_templates_small
+        self.flip_transform = transforms.RandomHorizontalFlip(p=self.flip_p)
+
+    def __len__(self):
+        return self._length
+
+    def __getitem__(self, i):
+        example = {}
+        image = Image.open(self.image_paths[i % self.num_images])
+
+        if not image.mode == "RGB":
+            image = image.convert("RGB")
+
+        placeholder_string = self.placeholder_token
+        text = random.choice(self.templates).format(placeholder_string)
+
+        example["input_ids"] = self.tokenizer(
+            text,
+            padding="max_length",
+            truncation=True,
+            max_length=self.tokenizer.model_max_length,
+            return_tensors="pt",
+        ).input_ids[0]
+
+        # default to score-sde preprocessing
+        img = np.array(image).astype(np.uint8)
+
+        if self.center_crop:
+            crop = min(img.shape[0], img.shape[1])
+            (
+                h,
+                w,
+            ) = (
+                img.shape[0],
+                img.shape[1],
+            )
+            img = img[(h - crop) // 2 : (h + crop) // 2, (w - crop) // 2 : (w + crop) // 2]
+
+        image = Image.fromarray(img)
+        image = image.resize((self.size, self.size), resample=self.interpolation)
+
+        image = self.flip_transform(image)
+        image = np.array(image).astype(np.uint8)
+        image = (image / 127.5 - 1.0).astype(np.float32)
+
+        example["pixel_values"] = torch.from_numpy(image).permute(2, 0, 1)
+        return example
+
+
+def main():
+    args = parse_args()
+    print(args)
+    logging_dir = os.path.join(args.output_dir, args.logging_dir)
+    accelerator_project_config = ProjectConfiguration(project_dir=args.output_dir, logging_dir=logging_dir)
+
+    if args.precision == "fp32":
+        print("Running fp32 ...")
+        weight_dtype = torch.float32
+        mixed_precision = "no"
+    elif args.precision == "bf32":
+        print("Running bf32 ...")
+        weight_dtype = torch.float32
+        mixed_precision = "no"
+    elif args.precision == "bf16":
+        print("Running bf16 ...")
+        weight_dtype = torch.bfloat16
+        mixed_precision = "bf16"
+    elif args.precision == "fp16":
+        print("Running fp16 ...")
+        weight_dtype = torch.float16
+        mixed_precision = "fp16"
+    else:
+        raise ValueError("--precision needs to be the following: fp32, bf32, bf16, fp16.")
+
+    accelerator = Accelerator(
+        gradient_accumulation_steps=args.gradient_accumulation_steps,
+        mixed_precision=mixed_precision,
+        log_with=args.report_to,
+        project_config=accelerator_project_config,
+    )
+
+    if args.report_to == "wandb":
+        if not is_wandb_available():
+            raise ImportError("Make sure to install wandb if you want to use it for logging during training.")
+
+    # Make one log on every process with the configuration for debugging.
+    logging.basicConfig(
+        format="%(asctime)s - %(levelname)s - %(name)s - %(message)s",
+        datefmt="%m/%d/%Y %H:%M:%S",
+        level=logging.INFO,
+    )
+    logger.info(accelerator.state, main_process_only=False)
+    if accelerator.is_local_main_process:
+        transformers.utils.logging.set_verbosity_warning()
+        diffusers.utils.logging.set_verbosity_info()
+    else:
+        transformers.utils.logging.set_verbosity_error()
+        diffusers.utils.logging.set_verbosity_error()
+
+    # If passed along, set the training seed now.
+    if args.seed is not None:
+        set_seed(args.seed)
+
+    # Handle the repository creation
+    if accelerator.is_main_process:
+        if args.output_dir is not None:
+            os.makedirs(args.output_dir, exist_ok=True)
+
+        if args.push_to_hub:
+            repo_id = create_repo(
+                repo_id=args.hub_model_id or Path(args.output_dir).name, exist_ok=True, token=args.hub_token
+            ).repo_id
+
+    # Load tokenizer
+    if args.tokenizer_name:
+        tokenizer = CLIPTokenizer.from_pretrained(args.tokenizer_name)
+    elif args.pretrained_model_name_or_path:
+        tokenizer = CLIPTokenizer.from_pretrained(args.pretrained_model_name_or_path, subfolder="tokenizer")
+
+    # Load scheduler and models
+    noise_scheduler = DDPMScheduler.from_pretrained(args.pretrained_model_name_or_path, subfolder="scheduler")
+    text_encoder = CLIPTextModel.from_pretrained(
+        args.pretrained_model_name_or_path, subfolder="text_encoder", revision=args.revision
+    )
+    vae = AutoencoderKL.from_pretrained(args.pretrained_model_name_or_path, subfolder="vae", revision=args.revision)
+    unet = UNet2DConditionModel.from_pretrained(
+        args.pretrained_model_name_or_path, subfolder="unet", revision=args.revision
+    )
+
+    # Add the placeholder token in tokenizer
+    placeholder_tokens = [args.placeholder_token]
+
+    if args.num_vectors < 1:
+        raise ValueError(f"--num_vectors has to be larger or equal to 1, but is {args.num_vectors}")
+
+    # add dummy tokens for multi-vector
+    additional_tokens = []
+    for i in range(1, args.num_vectors):
+        additional_tokens.append(f"{args.placeholder_token}_{i}")
+    placeholder_tokens += additional_tokens
+
+    num_added_tokens = tokenizer.add_tokens(placeholder_tokens)
+    if num_added_tokens != args.num_vectors:
+        raise ValueError(
+            f"The tokenizer already contains the token {args.placeholder_token}. Please pass a different"
+            " `placeholder_token` that is not already in the tokenizer."
+        )
+
+    # Convert the initializer_token, placeholder_token to ids
+    token_ids = tokenizer.encode(args.initializer_token, add_special_tokens=False)
+    # Check if initializer_token is a single token or a sequence of tokens
+    if len(token_ids) > 1:
+        raise ValueError("The initializer token must be a single token.")
+
+    initializer_token_id = token_ids[0]
+    placeholder_token_ids = tokenizer.convert_tokens_to_ids(placeholder_tokens)
+
+    # Resize the token embeddings as we are adding new special tokens to the tokenizer
+    text_encoder.resize_token_embeddings(len(tokenizer))
+
+    # Initialise the newly added placeholder token with the embeddings of the initializer token
+    token_embeds = text_encoder.get_input_embeddings().weight.data
+    with torch.no_grad():
+        for token_id in placeholder_token_ids:
+            token_embeds[token_id] = token_embeds[initializer_token_id].clone()
+
+    # Freeze vae and unet
+    vae.requires_grad_(False)
+    unet.requires_grad_(False)
+    # Freeze all parameters except for the token embeddings in text encoder
+    text_encoder.text_model.encoder.requires_grad_(False)
+    text_encoder.text_model.final_layer_norm.requires_grad_(False)
+    text_encoder.text_model.embeddings.position_embedding.requires_grad_(False)
+
+    if args.gradient_checkpointing:
+        # Keep unet in train mode if we are using gradient checkpointing to save memory.
+        # The dropout cannot be != 0 so it doesn't matter if we are in eval or train mode.
+        unet.train()
+        text_encoder.gradient_checkpointing_enable()
+        unet.enable_gradient_checkpointing()
+
+    if args.enable_xformers_memory_efficient_attention:
+        if is_xformers_available():
+            import xformers
+
+            xformers_version = version.parse(xformers.__version__)
+            if xformers_version == version.parse("0.0.16"):
+                logger.warn(
+                    "xFormers 0.0.16 cannot be used for training in some GPUs. If you observe problems during training, please update xFormers to at least 0.0.17. See https://huggingface.co/docs/diffusers/main/en/optimization/xformers for more details."
+                )
+            unet.enable_xformers_memory_efficient_attention()
+        else:
+            raise ValueError("xformers is not available. Make sure it is installed correctly")
+
+    # Enable TF32 for faster training on Ampere GPUs,
+    # cf https://pytorch.org/docs/stable/notes/cuda.html#tensorfloat-32-tf32-on-ampere-devices
+    if args.allow_tf32:
+        torch.backends.cuda.matmul.allow_tf32 = True
+
+    if args.scale_lr:
+        args.learning_rate = (
+            args.learning_rate * args.gradient_accumulation_steps * args.train_batch_size * accelerator.num_processes
+        )
+
+    # Initialize the optimizer
+    optimizer = torch.optim.AdamW(
+        text_encoder.get_input_embeddings().parameters(),  # only optimize the embeddings
+        lr=args.learning_rate,
+        betas=(args.adam_beta1, args.adam_beta2),
+        weight_decay=args.adam_weight_decay,
+        eps=args.adam_epsilon,
+    )
+
+    # Dataset and DataLoaders creation:
+    train_dataset = TextualInversionDataset(
+        data_root=args.train_data_dir,
+        tokenizer=tokenizer,
+        size=args.resolution,
+        placeholder_token=(" ".join(tokenizer.convert_ids_to_tokens(placeholder_token_ids))),
+        repeats=args.repeats,
+        learnable_property=args.learnable_property,
+        center_crop=args.center_crop,
+        set="train",
+    )
+    train_dataloader = torch.utils.data.DataLoader(
+        train_dataset, batch_size=args.train_batch_size, shuffle=True, num_workers=args.dataloader_num_workers
+    )
+    if args.validation_epochs is not None:
+        warnings.warn(
+            f"FutureWarning: You are doing logging with validation_epochs={args.validation_epochs}."
+            " Deprecated validation_epochs in favor of `validation_steps`"
+            f"Setting `args.validation_steps` to {args.validation_epochs * len(train_dataset)}",
+            FutureWarning,
+            stacklevel=2,
+        )
+        args.validation_steps = args.validation_epochs * len(train_dataset)
+
+    # Scheduler and math around the number of training steps.
+    overrode_max_train_steps = False
+    num_update_steps_per_epoch = math.ceil(len(train_dataloader) / args.gradient_accumulation_steps)
+    if args.max_train_steps is None:
+        args.max_train_steps = args.num_train_epochs * num_update_steps_per_epoch
+        overrode_max_train_steps = True
+
+    lr_scheduler = get_scheduler(
+        args.lr_scheduler,
+        optimizer=optimizer,
+        num_warmup_steps=args.lr_warmup_steps * accelerator.num_processes,
+        num_training_steps=args.max_train_steps * accelerator.num_processes,
+        num_cycles=args.lr_num_cycles,
+    )
+
+    # Prepare everything with our `accelerator`.
+    text_encoder, optimizer, train_dataloader, lr_scheduler = accelerator.prepare(
+        text_encoder, optimizer, train_dataloader, lr_scheduler
+    )
+
+    # Move vae and unet to device and cast to weight_dtype
+    unet.to(accelerator.device, dtype=weight_dtype)
+    vae.to(accelerator.device, dtype=weight_dtype)
+
+    if args.ipex:
+        import intel_extension_for_pytorch as ipex
+        vae_input = torch.randn(1, 3, 512, 512).to(dtype=weight_dtype)
+        unet_input = torch.randn(1, 4, 64, 64).to(dtype=weight_dtype), torch.tensor(280), torch.randn(1, 77, 1024).to(dtype=weight_dtype)
+        if args.precision == "bf32":
+            ipex.set_fp32_math_mode(mode=ipex.FP32MathMode.BF32, device="cpu")
+        unet = ipex.optimize(unet, dtype=weight_dtype, sample_input=unet_input, auto_kernel_selection=True)
+        vae = ipex.optimize(vae, dtype=weight_dtype, sample_input=vae_input, auto_kernel_selection=True)
+
+    # We need to recalculate our total training steps as the size of the training dataloader may have changed.
+    num_update_steps_per_epoch = math.ceil(len(train_dataloader) / args.gradient_accumulation_steps)
+    if overrode_max_train_steps:
+        args.max_train_steps = args.num_train_epochs * num_update_steps_per_epoch
+    # Afterwards we recalculate our number of training epochs
+    args.num_train_epochs = math.ceil(args.max_train_steps / num_update_steps_per_epoch)
+
+    # We need to initialize the trackers we use, and also store our configuration.
+    # The trackers initializes automatically on the main process.
+    if accelerator.is_main_process:
+        accelerator.init_trackers("textual_inversion", config=vars(args))
+
+    # Train!
+    total_batch_size = args.train_batch_size * accelerator.num_processes * args.gradient_accumulation_steps
+
+    logger.info("***** Running training *****")
+    logger.info(f"  Num examples = {len(train_dataset)}")
+    logger.info(f"  Num Epochs = {args.num_train_epochs}")
+    logger.info(f"  Instantaneous batch size per device = {args.train_batch_size}")
+    logger.info(f"  Total train batch size (w. parallel, distributed & accumulation) = {total_batch_size}")
+    logger.info(f"  Gradient Accumulation steps = {args.gradient_accumulation_steps}")
+    logger.info(f"  Total optimization steps = {args.max_train_steps}")
+    global_step = 0
+    first_epoch = 0
+    # Potentially load in the weights and states from a previous save
+    if args.resume_from_checkpoint:
+        if args.resume_from_checkpoint != "latest":
+            path = os.path.basename(args.resume_from_checkpoint)
+        else:
+            # Get the most recent checkpoint
+            dirs = os.listdir(args.output_dir)
+            dirs = [d for d in dirs if d.startswith("checkpoint")]
+            dirs = sorted(dirs, key=lambda x: int(x.split("-")[1]))
+            path = dirs[-1] if len(dirs) > 0 else None
+
+        if path is None:
+            accelerator.print(
+                f"Checkpoint '{args.resume_from_checkpoint}' does not exist. Starting a new training run."
+            )
+            args.resume_from_checkpoint = None
+            initial_global_step = 0
+        else:
+            accelerator.print(f"Resuming from checkpoint {path}")
+            accelerator.load_state(os.path.join(args.output_dir, path))
+            global_step = int(path.split("-")[1])
+
+            initial_global_step = global_step
+            first_epoch = global_step // num_update_steps_per_epoch
+
+    else:
+        initial_global_step = 0
+
+    progress_bar = tqdm(
+        range(0, args.max_train_steps),
+        initial=initial_global_step,
+        desc="Steps",
+        # Only show the progress bar once on each machine.
+        disable=not accelerator.is_local_main_process,
+    )
+
+    # keep original embeddings as reference
+    orig_embeds_params = accelerator.unwrap_model(text_encoder).get_input_embeddings().weight.data.clone()
+
+    start = time.time()
+    train_time = 0
+    for epoch in range(first_epoch, args.num_train_epochs):
+        text_encoder.train()
+        for step, batch in enumerate(train_dataloader):
+            with accelerator.accumulate(text_encoder):
+                # , torch.profiler.profile(activities=[torch.profiler.ProfilerActivity.CPU], record_shapes=True) as p:
+                start1 = time.time()
+                # Convert images to latent space
+                latents = vae.encode(batch["pixel_values"].to(dtype=weight_dtype)).latent_dist.sample().detach()
+                latents = latents * vae.config.scaling_factor
+
+                # Sample noise that we'll add to the latents
+                noise = torch.randn_like(latents)
+                bsz = latents.shape[0]
+                # Sample a random timestep for each image
+                timesteps = torch.randint(0, noise_scheduler.config.num_train_timesteps, (bsz,), device=latents.device)
+                timesteps = timesteps.long()
+
+                # Add noise to the latents according to the noise magnitude at each timestep
+                # (this is the forward diffusion process)
+                noisy_latents = noise_scheduler.add_noise(latents, noise, timesteps)
+
+                # Get the text embedding for conditioning
+                encoder_hidden_states = text_encoder(batch["input_ids"])[0].to(dtype=weight_dtype)
+
+                # Predict the noise residual
+                model_pred = unet(noisy_latents, timesteps, encoder_hidden_states, return_dict=False)[0]
+
+                # Get the target for loss depending on the prediction type
+                if noise_scheduler.config.prediction_type == "epsilon":
+                    target = noise
+                elif noise_scheduler.config.prediction_type == "v_prediction":
+                    target = noise_scheduler.get_velocity(latents, noise, timesteps)
+                else:
+                    raise ValueError(f"Unknown prediction type {noise_scheduler.config.prediction_type}")
+
+                loss = F.mse_loss(model_pred.float(), target.float(), reduction="mean")
+
+                accelerator.backward(loss)
+
+                optimizer.step()
+                lr_scheduler.step()
+                optimizer.zero_grad()
+
+                end1 = time.time()
+                if step >= args.warmup_iterations:
+                    train_time += end1 - start1
+
+                # Let's make sure we don't update any embedding weights besides the newly added token
+                index_no_updates = torch.ones((len(tokenizer),), dtype=torch.bool)
+                index_no_updates[min(placeholder_token_ids) : max(placeholder_token_ids) + 1] = False
+
+                with torch.no_grad():
+                    accelerator.unwrap_model(text_encoder).get_input_embeddings().weight[
+                        index_no_updates
+                    ] = orig_embeds_params[index_no_updates]
+
+            # output = p.key_averages().table(sort_by="self_cpu_time_total")
+            # print(output)
+
+            # Checks if the accelerator has performed an optimization step behind the scenes
+            if accelerator.sync_gradients:
+                images = []
+                progress_bar.update(1)
+                global_step += 1
+                if global_step % args.save_steps == 0:
+                    weight_name = (
+                        f"learned_embeds-steps-{global_step}.bin"
+                        if args.no_safe_serialization
+                        else f"learned_embeds-steps-{global_step}.safetensors"
+                    )
+                    save_path = os.path.join(args.output_dir, weight_name)
+                    save_progress(
+                        text_encoder,
+                        placeholder_token_ids,
+                        accelerator,
+                        args,
+                        save_path,
+                        safe_serialization=not args.no_safe_serialization,
+                    )
+
+                if accelerator.is_main_process:
+                    if global_step % args.checkpointing_steps == 0:
+                        # _before_ saving state, check if this save would set us over the `checkpoints_total_limit`
+                        if args.checkpoints_total_limit is not None:
+                            checkpoints = os.listdir(args.output_dir)
+                            checkpoints = [d for d in checkpoints if d.startswith("checkpoint")]
+                            checkpoints = sorted(checkpoints, key=lambda x: int(x.split("-")[1]))
+
+                            # before we save the new checkpoint, we need to have at _most_ `checkpoints_total_limit - 1` checkpoints
+                            if len(checkpoints) >= args.checkpoints_total_limit:
+                                num_to_remove = len(checkpoints) - args.checkpoints_total_limit + 1
+                                removing_checkpoints = checkpoints[0:num_to_remove]
+
+                                logger.info(
+                                    f"{len(checkpoints)} checkpoints already exist, removing {len(removing_checkpoints)} checkpoints"
+                                )
+                                logger.info(f"removing checkpoints: {', '.join(removing_checkpoints)}")
+
+                                for removing_checkpoint in removing_checkpoints:
+                                    removing_checkpoint = os.path.join(args.output_dir, removing_checkpoint)
+                                    shutil.rmtree(removing_checkpoint)
+
+                        save_path = os.path.join(args.output_dir, f"checkpoint-{global_step}")
+                        accelerator.save_state(save_path)
+                        logger.info(f"Saved state to {save_path}")
+
+                    if args.validation_prompt is not None and global_step % args.validation_steps == 0:
+                        images = log_validation(
+                            text_encoder, tokenizer, unet, vae, args, accelerator, weight_dtype, epoch
+                        )
+
+            logs = {"loss": loss.detach().item(), "lr": lr_scheduler.get_last_lr()[0]}
+            progress_bar.set_postfix(**logs)
+            accelerator.log(logs, step=global_step)
+
+            if global_step >= args.max_train_steps:
+                break
+
+    end = time.time()
+    print("\ntime_to_train(s): {:.2f}".format((end - start)))
+
+    throughput = (args.max_train_steps * args.gradient_accumulation_steps - args.warmup_iterations) / (train_time)
+    print("Throughput: {:.5f}".format(throughput))
+
+    # Create the pipeline using the trained modules and save it.
+    accelerator.wait_for_everyone()
+    if accelerator.is_main_process:
+        if args.push_to_hub and not args.save_as_full_pipeline:
+            logger.warn("Enabling full model saving because --push_to_hub=True was specified.")
+            save_full_model = True
+        else:
+            save_full_model = args.save_as_full_pipeline
+        if save_full_model:
+            pipeline = StableDiffusionPipeline.from_pretrained(
+                args.pretrained_model_name_or_path,
+                text_encoder=accelerator.unwrap_model(text_encoder),
+                vae=vae,
+                unet=unet,
+                tokenizer=tokenizer,
+            )
+            pipeline.save_pretrained(args.output_dir, safe_serialization=not args.no_safe_serialization)
+        # Save the newly trained embeddings
+        weight_name = "learned_embeds.bin" if args.no_safe_serialization else "learned_embeds.safetensors"
+        save_path = os.path.join(args.output_dir, weight_name)
+        save_progress(
+            text_encoder,
+            placeholder_token_ids,
+            accelerator,
+            args,
+            save_path,
+            safe_serialization=not args.no_safe_serialization,
+        )
+
+        if args.push_to_hub:
+            save_model_card(
+                repo_id,
+                images=images,
+                base_model=args.pretrained_model_name_or_path,
+                repo_folder=args.output_dir,
+            )
+            upload_folder(
+                repo_id=repo_id,
+                folder_path=args.output_dir,
+                commit_message="End of training",
+                ignore_patterns=["step_*", "epoch_*"],
+            )
+
+    accelerator.end_training()
+
+    if not args.train_no_eval and accelerator.is_main_process:
+        # run test
+        pipe = StableDiffusionPipeline.from_pretrained(args.output_dir)
+        prompt = "a yellow <cat-toy> robot at the beach, high quality"
+        image = pipe(prompt, num_inference_steps=50, guidance_scale=7.5, generator=torch.manual_seed(args.seed)).images[0]
+        filename = "output_" + args.precision + ".png"
+        image.save(filename)
+
+
+if __name__ == "__main__":
+    main()
diff --git a/models/diffusion/tensorflow/__init__.py b/models/diffusion/tensorflow/__init__.py
new file mode 100644
index 000000000..ce0a27f82
--- /dev/null
+++ b/models/diffusion/tensorflow/__init__.py
@@ -0,0 +1,20 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+#
+
diff --git a/models/diffusion/tensorflow/stable_diffusion/__init__.py b/models/diffusion/tensorflow/stable_diffusion/__init__.py
new file mode 100644
index 000000000..f07dbaf88
--- /dev/null
+++ b/models/diffusion/tensorflow/stable_diffusion/__init__.py
@@ -0,0 +1,19 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+#
diff --git a/models/diffusion/tensorflow/stable_diffusion/inference/__init__.py b/models/diffusion/tensorflow/stable_diffusion/inference/__init__.py
new file mode 100644
index 000000000..f07dbaf88
--- /dev/null
+++ b/models/diffusion/tensorflow/stable_diffusion/inference/__init__.py
@@ -0,0 +1,19 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+#
diff --git a/models/diffusion/tensorflow/stable_diffusion/inference/eval_inference.py b/models/diffusion/tensorflow/stable_diffusion/inference/eval_inference.py
new file mode 100644
index 000000000..605af44fd
--- /dev/null
+++ b/models/diffusion/tensorflow/stable_diffusion/inference/eval_inference.py
@@ -0,0 +1,214 @@
+import os
+import time
+import keras_cv
+from tensorflow import keras
+from argparse import ArgumentParser
+import matplotlib.pyplot as plt
+import pandas as pd
+import json
+import glob
+
+import numpy as np
+from numpy import cov, trace, iscomplexobj
+from numpy.random import randint
+from scipy.linalg import sqrtm
+from keras.applications.inception_v3 import InceptionV3
+from keras.applications.inception_v3 import preprocess_input
+from keras.datasets.mnist import load_data
+from skimage.transform import resize
+
+from PIL import Image
+import tensorflow as tf
+
+class eval_stable_diffusion:
+    """Evaluate Stable Diffusion model"""
+
+    def __init__(self):
+
+        arg_parser = ArgumentParser(description='Parse args')
+
+        arg_parser.add_argument('-b', "--batch-size",
+                                help='Specify the batch size. If this '
+                                     'parameter is not specified, then '
+                                     'it will run with batch size of 1 ',
+                                dest='batch_size', type=int, default=1)
+
+        arg_parser.add_argument("-p", "--precision",
+                                help="Specify the model precision to use: fp32, bfloat16 or fp16",
+                                required=True, choices=["fp32", "bfloat16", "fp16"],
+                                dest="precision")
+        
+        arg_parser.add_argument("-d", "--data-location",
+                                help='Specify the location of the data. ',
+                                dest="data_location", type=str, default=None)
+        
+        arg_parser.add_argument('-s', "--steps", type=int, default=50,
+                                help="number of steps for diffusion model")
+        
+        arg_parser.add_argument("-o", "--output-dir",
+                                help="Specify the location of the output " + \
+                                     "directory for logs and saved model",
+                                dest='output_dir', required=True)
+
+        arg_parser.add_argument('-r', "--accuracy-only",
+                                help='For accuracy measurement only.',
+                                dest='accuracy_only', action='store_true')
+
+        # parse the arguments
+        self.args = arg_parser.parse_args()
+
+    def plot_images(self, images):
+        plt.figure(figsize=(20, 20))
+        for i in range(len(images)):
+            ax = plt.subplot(1, len(images), i + 1)
+            plt.imshow(images[i])
+            plt.axis("off")
+
+    # scale an array of images to a new size
+    def scale_images(self, images, new_shape):
+        images_list = list()
+        for image in images:
+            # resize with nearest neighbor interpolation
+            new_image = resize(image, new_shape, 0)
+            # store
+            images_list.append(new_image)
+        return np.asarray(images_list)
+
+    # calculate frechet inception distance
+    def calculate_fid(self, model, images1, images2):
+        # calculate activations
+        act1 = model.predict(images1)
+        act2 = model.predict(images2)
+        # calculate mean and covariance statistics
+        mu1, sigma1 = act1.mean(axis=0), cov(act1, rowvar=False)
+        mu2, sigma2 = act2.mean(axis=0), cov(act2, rowvar=False)
+        # calculate sum squared difference between means
+        ssdiff = np.sum((mu1 - mu2)**2.0)
+        # calculate sqrt of product between cov
+        covmean = sqrtm(sigma1.dot(sigma2))
+        # check and correct imaginary numbers from sqrt
+        if iscomplexobj(covmean):
+            covmean = covmean.real
+        # calculate score
+        fid = ssdiff + trace(sigma1 + sigma2 - 2.0 * covmean)
+        return fid
+
+    def run(self):
+
+        print("Run inference with " + str(self.args.precision) + " precision with a batch size of " + str(self.args.batch_size))
+
+        if self.args.precision == "bfloat16":
+            print("Enabling auto-mixed precision for bfloat16")
+            tf.config.optimizer.set_experimental_options({'auto_mixed_precision_onednn_bfloat16': True})
+            print(tf.config.optimizer.get_experimental_options())
+        elif self.args.precision == "fp16":
+            print("Enabling auto-mixed precision for fp16")
+            tf.config.optimizer.set_experimental_options({'auto_mixed_precision': True})
+            print(tf.config.optimizer.get_experimental_options())
+
+        model = keras_cv.models.StableDiffusionV2(img_width=512, img_height=512)
+
+        if (not self.args.accuracy_only):
+            print("Performance Benchmarking")
+
+            print("First run for warming up the model")
+            model.text_to_image("warming up the model", batch_size=self.args.batch_size)
+
+            start = time.time()
+            images = model.text_to_image(
+                "a cute magical flying dog, fantasy art, "
+                "golden color, high quality, highly detailed, elegant, sharp focus, "
+                "concept art, character concepts, digital painting, mystery, adventure",
+                batch_size=self.args.batch_size,
+                num_steps=self.args.steps,
+            )
+            end = time.time()
+
+            total_time = end - start
+            print('Batch size = %d' % self.args.batch_size)
+            if (self.args.batch_size == 1):
+                print('Latency: %.3f s' % (total_time))
+            # print throughput for both batch size 1 and batch_size
+            print("Avg Throughput: " + str(self.args.batch_size / total_time) + " examples/sec")
+
+            self.plot_images(images)
+
+            print(f"Inference time: {(end - start):.2f} seconds")
+            keras.backend.clear_session()
+        else:  # accuracy check
+            print("Accuracy Check")
+
+            if not self.args.data_location:
+                exit("Please provide a path to the COCO dataset to compute accuracy using --data-location arg.")
+
+            with open(f'{self.args.data_location}/annotations/captions_val2017.json', 'r') as f:
+                data = json.load(f)
+                data = data['annotations']
+
+            img_cap_pairs = []
+
+            for sample in data:
+                img_name = '%012d.jpg' % sample['image_id']
+                img_cap_pairs.append([img_name, sample['caption']])
+
+            captions = pd.DataFrame(img_cap_pairs, columns=['image', 'caption'])
+            captions['image'] = captions['image'].apply(
+                lambda x: f'{self.args.data_location}/val2017/{x}'
+            )
+            captions = captions.drop_duplicates(subset=['image'])
+            captions = captions.sample(10, random_state=2023)
+            captions = captions.reset_index(drop=True)
+            print("Dataset shape: ", captions.shape)
+            print("Data: ", captions.head())
+
+            if not os.path.isdir(os.path.join(self.args.output_dir, "original")):
+                os.makedirs(os.path.join(self.args.output_dir, "original"))
+            else:
+                files = glob.glob(os.path.join(self.args.output_dir, "original", "*"))
+                for f in files:
+                    os.remove(f)
+            if not os.path.isdir(os.path.join(self.args.output_dir, "generated")):
+                os.makedirs(os.path.join(self.args.output_dir, "generated"))
+            else:
+                files = glob.glob(os.path.join(self.args.output_dir, "generated", "*"))
+                for f in files:
+                    os.remove(f)
+
+            original_imgs, generated_imgs = [], []
+            for index, row in captions.iterrows():
+                
+                img = Image.open(row["image"])
+                np_img = np.asarray(img)
+                np_img = np_img.reshape(1, np_img.shape[0], np_img.shape[1], np_img.shape[2])
+                scaled_img = self.scale_images(np_img, (512, 512, 3))
+                original_imgs.append(scaled_img)
+                save_img = scaled_img.reshape(scaled_img.shape[1], scaled_img.shape[2], scaled_img.shape[3])
+                im = Image.fromarray(save_img)
+                im.save(self.args.output_dir + "/original/" + row["image"].split('/')[-1])
+
+                print("Generating image with Stable Diffusion for the prompt: ", row["caption"])
+                generated_img = model.text_to_image(row["caption"], seed=2023, num_steps=self.args.steps)
+                generated_imgs.append(generated_img)
+                save_img = generated_img.reshape(generated_img.shape[1], generated_img.shape[2], generated_img.shape[3])
+                im = Image.fromarray(save_img)
+                im.save(self.args.output_dir + "/generated/" + row["image"].split('/')[-1])
+                print("Saved the generated image at: ", str(self.args.output_dir) + "/generated/" + row["image"].split('/')[-1])
+
+            original_imgs_np = np.array(original_imgs)
+            original_imgs_np = original_imgs_np.reshape(original_imgs_np.shape[0], original_imgs_np.shape[2], original_imgs_np.shape[3], original_imgs_np.shape[4])
+
+            generated_imgs_np = np.array(generated_imgs)
+            generated_imgs_np = generated_imgs_np.reshape(generated_imgs_np.shape[0], generated_imgs_np.shape[2], generated_imgs_np.shape[3], generated_imgs_np.shape[4])
+
+            original_imgs_np = preprocess_input(original_imgs_np)
+            generated_imgs_np = preprocess_input(generated_imgs_np)
+
+            inception_model = InceptionV3(include_top=False, pooling='avg', input_shape=(512,512,3))
+            fid = self.calculate_fid(inception_model, original_imgs_np, generated_imgs_np)
+            print('FID score between original and generated images: %.3f' % fid)
+
+
+
+if __name__ == "__main__":
+    evaluate_stable_diffusion = eval_stable_diffusion()
+    evaluate_stable_diffusion.run()
diff --git a/models/diffusion/tensorflow/stable_diffusion/inference/patch b/models/diffusion/tensorflow/stable_diffusion/inference/patch
new file mode 100644
index 000000000..ba2e78a83
--- /dev/null
+++ b/models/diffusion/tensorflow/stable_diffusion/inference/patch
@@ -0,0 +1,296 @@
+diff --git a/keras_cv/models/stable_diffusion/__internal__/layers/attention_block.py b/keras_cv/models/stable_diffusion/__internal__/layers/attention_block.py
+index e7e1896..d9f9a3b 100644
+--- a/keras_cv/models/stable_diffusion/__internal__/layers/attention_block.py
++++ b/keras_cv/models/stable_diffusion/__internal__/layers/attention_block.py
+@@ -14,6 +14,11 @@
+ 
+ import tensorflow as tf
+ from tensorflow import keras
++try:
++    import intel_extension_for_tensorflow as itex
++    keras.layers.GroupNormalization = itex.ops.GroupNormalization
++except:
++    pass
+ 
+ from keras_cv.models.stable_diffusion.__internal__.layers.padded_conv2d import (
+     PaddedConv2D,
+diff --git a/keras_cv/models/stable_diffusion/__internal__/layers/resnet_block.py b/keras_cv/models/stable_diffusion/__internal__/layers/resnet_block.py
+index 29aeaaa..f280430 100644
+--- a/keras_cv/models/stable_diffusion/__internal__/layers/resnet_block.py
++++ b/keras_cv/models/stable_diffusion/__internal__/layers/resnet_block.py
+@@ -13,6 +13,11 @@
+ # limitations under the License.
+ 
+ from tensorflow import keras
++try:
++    import intel_extension_for_tensorflow as itex
++    keras.layers.GroupNormalization = itex.ops.GroupNormalization
++except:
++    pass
+ 
+ from keras_cv.models.stable_diffusion.__internal__.layers.padded_conv2d import (
+     PaddedConv2D,
+diff --git a/keras_cv/models/stable_diffusion/decoder.py b/keras_cv/models/stable_diffusion/decoder.py
+index fe619d3..6944bc6 100644
+--- a/keras_cv/models/stable_diffusion/decoder.py
++++ b/keras_cv/models/stable_diffusion/decoder.py
+@@ -13,6 +13,11 @@
+ # limitations under the License.
+ 
+ from tensorflow import keras
++try:
++    import intel_extension_for_tensorflow as itex
++    keras.layers.GroupNormalization = itex.ops.GroupNormalization
++except:
++    pass
+ 
+ from keras_cv.models.stable_diffusion.__internal__.layers.attention_block import (  # noqa: E501
+     AttentionBlock,
+diff --git a/keras_cv/models/stable_diffusion/diffusion_model.py b/keras_cv/models/stable_diffusion/diffusion_model.py
+index 25b5241..d3e8803 100644
+--- a/keras_cv/models/stable_diffusion/diffusion_model.py
++++ b/keras_cv/models/stable_diffusion/diffusion_model.py
+@@ -14,6 +14,12 @@
+ 
+ import tensorflow as tf
+ from tensorflow import keras
++try:
++    import intel_extension_for_tensorflow as itex
++    keras.layers.GroupNormalization = itex.ops.GroupNormalization
++    keras.layers.LayerNormalization = itex.ops.LayerNormalization
++except:
++    pass
+ 
+ from keras_cv.models.stable_diffusion.__internal__.layers.padded_conv2d import (
+     PaddedConv2D,
+@@ -303,6 +309,27 @@ class CrossAttention(keras.layers.Layer):
+         self.head_size = head_size
+         self.out_proj = keras.layers.Dense(num_heads * head_size)
+ 
++
++    def naive_scaled_dot_product_attention(self, query, key, value):
++        i_dtype = query.dtype
++        atten_scores = tf.matmul(query, key, transpose_b=True)
++        atten_scores = tf.multiply(atten_scores, tf.cast(self.scale, i_dtype))
++        atten_probs = tf.nn.softmax(atten_scores)
++        # `atten_output` = [B, N, F, H]
++        atten_output = tf.matmul(atten_probs, value)
++        # `atten_output` = [B, F, N, H]
++        atten_output  = tf.transpose(a=atten_output, perm=[0, 2, 1, 3])
++        return atten_output
++
++
++    def sdp(self, q, k, v):
++        try:
++            from intel_extension_for_tensorflow.python.ops.multi_head_attention import scaled_dot_product_attention
++            output = scaled_dot_product_attention(q, k, v, use_fast_attention=True)
++        except ImportError:
++            output = self.naive_scaled_dot_product_attention(q, k, v)
++        return output
++
+     def call(self, inputs):
+         inputs, context = inputs
+         context = inputs if context is None else context
+@@ -316,17 +343,10 @@ class CrossAttention(keras.layers.Layer):
+         )
+ 
+         q = tf.transpose(q, (0, 2, 1, 3))  # (bs, num_heads, time, head_size)
+-        k = tf.transpose(k, (0, 2, 3, 1))  # (bs, num_heads, head_size, time)
++        k = tf.transpose(k, (0, 2, 1, 3))  # (bs, num_heads, head_size, time)
+         v = tf.transpose(v, (0, 2, 1, 3))  # (bs, num_heads, time, head_size)
+ 
+-        score = td_dot(q, k) * self.scale
+-        weights = keras.activations.softmax(
+-            score
+-        )  # (bs, num_heads, time, time)
+-        attn = td_dot(weights, v)
+-        attn = tf.transpose(
+-            attn, (0, 2, 1, 3)
+-        )  # (bs, time, num_heads, head_size)
++        attn = self.sdp(q, k, v)
+         out = tf.reshape(
+             attn, (-1, inputs.shape[1], self.num_heads * self.head_size)
+         )
+@@ -352,10 +372,7 @@ class GEGLU(keras.layers.Layer):
+     def call(self, inputs):
+         x = self.dense(inputs)
+         x, gate = x[..., : self.output_dim], x[..., self.output_dim :]
+-        tanh_res = keras.activations.tanh(
+-            gate * 0.7978845608 * (1 + 0.044715 * (gate**2))
+-        )
+-        return x * 0.5 * gate * (1 + tanh_res)
++        return x * tf.keras.activations.gelu(gate, approximate=True)
+ 
+ 
+ def td_dot(a, b):
+diff --git a/keras_cv/models/stable_diffusion/image_encoder.py b/keras_cv/models/stable_diffusion/image_encoder.py
+index 614b11d..6ec83ae 100644
+--- a/keras_cv/models/stable_diffusion/image_encoder.py
++++ b/keras_cv/models/stable_diffusion/image_encoder.py
+@@ -13,6 +13,11 @@
+ # limitations under the License.
+ 
+ from tensorflow import keras
++try:
++    import intel_extension_for_tensorflow as itex
++    keras.layers.GroupNormalization = itex.ops.GroupNormalization
++except:
++    pass
+ 
+ from keras_cv.models.stable_diffusion.__internal__.layers.attention_block import (  # noqa: E501
+     AttentionBlock,
+diff --git a/keras_cv/models/stable_diffusion/stable_diffusion.py b/keras_cv/models/stable_diffusion/stable_diffusion.py
+index 5ef7471..06312e5 100644
+--- a/keras_cv/models/stable_diffusion/stable_diffusion.py
++++ b/keras_cv/models/stable_diffusion/stable_diffusion.py
+@@ -29,6 +29,8 @@ import math
+ import numpy as np
+ import tensorflow as tf
+ from tensorflow import keras
++from keras import backend as K
++import os
+ 
+ from keras_cv.models.stable_diffusion.clip_tokenizer import SimpleTokenizer
+ from keras_cv.models.stable_diffusion.constants import _ALPHAS_CUMPROD
+@@ -51,6 +53,7 @@ class StableDiffusionBase:
+         img_height=512,
+         img_width=512,
+         jit_compile=False,
++        precision="fp32",
+     ):
+         # UNet requires multiples of 2**7 = 128
+         img_height = round(img_height / 128) * 128
+@@ -66,6 +69,7 @@ class StableDiffusionBase:
+         self._tokenizer = None
+ 
+         self.jit_compile = jit_compile
++        self.to_fp16 = (precision == "fp16")
+ 
+     def text_to_image(
+         self,
+@@ -76,6 +80,9 @@ class StableDiffusionBase:
+         unconditional_guidance_scale=7.5,
+         seed=None,
+     ):
++        if self.to_fp16:
++            K.set_floatx('float16')
++
+         encoded_text = self.encode_text(prompt)
+ 
+         return self.generate_image(
+@@ -207,18 +214,21 @@ class StableDiffusionBase:
+ 
+         # Iterative reverse diffusion stage
+         timesteps = tf.range(1, 1000, 1000 // num_steps)
++        t_embs_lst = self._get_timesteps_embedding(timesteps, batch_size)
++        contexts = tf.concat((unconditional_context, context), 0)
++
+         alphas, alphas_prev = self._get_initial_alphas(timesteps)
+         progbar = keras.utils.Progbar(len(timesteps))
+         iteration = 0
+         for index, timestep in list(enumerate(timesteps))[::-1]:
+             latent_prev = latent  # Set aside the previous latent vector
+-            t_emb = self._get_timestep_embedding(timestep, batch_size)
+-            unconditional_latent = self.diffusion_model.predict_on_batch(
+-                [latent, t_emb, unconditional_context]
+-            )
+-            latent = self.diffusion_model.predict_on_batch(
+-                [latent, t_emb, context]
+-            )
++            latents = tf.concat((latent, latent), 0)
++            t_embs = t_embs_lst[index]
++
++            pred_latent = self.diffusion_model.predict_on_batch(
++                [latents, t_embs, contexts])
++            unconditional_latent, latent = tf.split(pred_latent, 2)
++
+             latent = unconditional_latent + unconditional_guidance_scale * (
+                 latent - unconditional_latent
+             )
+@@ -303,6 +313,22 @@ class StableDiffusionBase:
+         if self._tokenizer is None:
+             self._tokenizer = SimpleTokenizer()
+         return self._tokenizer
++    def _get_timesteps_embedding(
++        self, timesteps, batch_size, dim=320, max_period=10000
++    ):
++        half = dim // 2
++        freqs = tf.math.exp(
++            -math.log(max_period) * tf.range(0, half, dtype=tf.float32) / half
++        )
++        # timesteps shape: [num_steps]
++        args = tf.cast(tf.reshape(timesteps, [-1, 1]), dtype=tf.float32) * freqs
++        # embeddings shape:(steps, half)
++        embeddings = tf.concat([tf.math.cos(args), tf.math.sin(args)], axis=1)
++        embeddings = tf.expand_dims(embeddings, axis=1)
++        if self.to_fp16:
++            embeddings = tf.cast(embeddings, tf.float16)
++        #  2 is to concatenate the embedding of two forward pass
++        return tf.repeat(embeddings, batch_size * 2, axis=1)
+ 
+     def _get_timestep_embedding(
+         self, timestep, batch_size, dim=320, max_period=10000
+@@ -314,6 +340,8 @@ class StableDiffusionBase:
+         args = tf.convert_to_tensor([timestep], dtype=tf.float32) * freqs
+         embedding = tf.concat([tf.math.cos(args), tf.math.sin(args)], 0)
+         embedding = tf.reshape(embedding, [1, -1])
++        if self.to_fp16:
++            embedding = tf.cast(embedding, tf.float16)
+         return tf.repeat(embedding, batch_size, axis=0)
+ 
+     def _get_initial_alphas(self, timesteps):
+@@ -324,14 +352,25 @@ class StableDiffusionBase:
+ 
+     def _get_initial_diffusion_noise(self, batch_size, seed):
+         if seed is not None:
+-            return tf.random.stateless_normal(
+-                (batch_size, self.img_height // 8, self.img_width // 8, 4),
+-                seed=[seed, seed],
+-            )
++            if self.to_fp16:
++              return tf.random.stateless_normal(
++                  (batch_size, self.img_height // 8, self.img_width // 8, 4),
++                  seed=[seed, seed], dtype=tf.float16
++              )
++            else:
++                return tf.random.stateless_normal(
++                  (batch_size, self.img_height // 8, self.img_width // 8, 4),
++                  seed=[seed, seed],
++              )
+         else:
+-            return tf.random.normal(
+-                (batch_size, self.img_height // 8, self.img_width // 8, 4)
+-            )
++            if self.to_fp16:
++                return tf.random.normal(
++                    (batch_size, self.img_height // 8, self.img_width // 8, 4), dtype=tf.float16
++                )
++            else:
++                return tf.random.normal(
++                    (batch_size, self.img_height // 8, self.img_width // 8, 4)
++                )
+ 
+     @staticmethod
+     def _get_pos_ids():
+@@ -390,8 +429,9 @@ class StableDiffusion(StableDiffusionBase):
+         img_height=512,
+         img_width=512,
+         jit_compile=False,
++        precision="fp32",
+     ):
+-        super().__init__(img_height, img_width, jit_compile)
++        super().__init__(img_height, img_width, jit_compile, precision)
+         print(
+             "By using this model checkpoint, you acknowledge that its usage is "
+             "subject to the terms of the CreativeML Open RAIL-M license at "
+@@ -475,8 +515,9 @@ class StableDiffusionV2(StableDiffusionBase):
+         img_height=512,
+         img_width=512,
+         jit_compile=False,
++        precision="fp32",
+     ):
+-        super().__init__(img_height, img_width, jit_compile)
++        super().__init__(img_height, img_width, jit_compile, precision)
+         print(
+             "By using this model checkpoint, you acknowledge that its usage is "
+             "subject to the terms of the CreativeML Open RAIL++-M license at "
diff --git a/models/diffusion/tensorflow/stable_diffusion/inference/patch_for_stockTF b/models/diffusion/tensorflow/stable_diffusion/inference/patch_for_stockTF
new file mode 100644
index 000000000..3cca02472
--- /dev/null
+++ b/models/diffusion/tensorflow/stable_diffusion/inference/patch_for_stockTF
@@ -0,0 +1,138 @@
+diff --git a/keras_cv/models/object_detection/predict_utils.py b/keras_cv/models/object_detection/predict_utils.py
+index 8eb6e10..48fb78d 100644
+--- a/keras_cv/models/object_detection/predict_utils.py
++++ b/keras_cv/models/object_detection/predict_utils.py
+@@ -15,13 +15,13 @@
+ import tensorflow as tf
+ 
+ try:
+-    from keras.src.engine.training import _minimum_control_deps
+-    from keras.src.engine.training import reduce_per_replica
+-    from keras.src.utils import tf_utils
++    from tf_keras.src.engine.training import _minimum_control_deps
++    from tf_keras.src.engine.training import reduce_per_replica
++    from tf_keras.src.utils import tf_utils
+ except ImportError:
+-    from keras.engine.training import _minimum_control_deps
+-    from keras.engine.training import reduce_per_replica
+-    from keras.utils import tf_utils
++    from tf_keras.engine.training import _minimum_control_deps
++    from tf_keras.engine.training import reduce_per_replica
++    from tf_keras.utils import tf_utils
+ 
+ 
+ def make_predict_function(model, force=False):
+diff --git a/keras_cv/models/stable_diffusion/diffusion_model.py b/keras_cv/models/stable_diffusion/diffusion_model.py
+index 25b5241..599f136 100644
+--- a/keras_cv/models/stable_diffusion/diffusion_model.py
++++ b/keras_cv/models/stable_diffusion/diffusion_model.py
+@@ -303,6 +303,22 @@ class CrossAttention(keras.layers.Layer):
+         self.head_size = head_size
+         self.out_proj = keras.layers.Dense(num_heads * head_size)
+ 
++
++    def naive_scaled_dot_product_attention(self, query, key, value):
++        i_dtype = query.dtype
++        atten_scores = tf.matmul(query, key, transpose_b=True)
++        atten_scores = tf.multiply(atten_scores, tf.cast(self.scale, i_dtype))
++        atten_probs = tf.nn.softmax(atten_scores)
++        # `atten_output` = [B, N, F, H]
++        atten_output = tf.matmul(atten_probs, value)
++        # `atten_output` = [B, F, N, H]
++        atten_output  = tf.transpose(a=atten_output, perm=[0, 2, 1, 3])
++        return atten_output
++
++
++    def sdp(self, q, k, v):
++        return self.naive_scaled_dot_product_attention(q, k, v)
++
+     def call(self, inputs):
+         inputs, context = inputs
+         context = inputs if context is None else context
+@@ -316,17 +332,10 @@ class CrossAttention(keras.layers.Layer):
+         )
+ 
+         q = tf.transpose(q, (0, 2, 1, 3))  # (bs, num_heads, time, head_size)
+-        k = tf.transpose(k, (0, 2, 3, 1))  # (bs, num_heads, head_size, time)
++        k = tf.transpose(k, (0, 2, 1, 3))  # (bs, num_heads, head_size, time)
+         v = tf.transpose(v, (0, 2, 1, 3))  # (bs, num_heads, time, head_size)
+ 
+-        score = td_dot(q, k) * self.scale
+-        weights = keras.activations.softmax(
+-            score
+-        )  # (bs, num_heads, time, time)
+-        attn = td_dot(weights, v)
+-        attn = tf.transpose(
+-            attn, (0, 2, 1, 3)
+-        )  # (bs, time, num_heads, head_size)
++        attn = self.sdp(q, k, v)
+         out = tf.reshape(
+             attn, (-1, inputs.shape[1], self.num_heads * self.head_size)
+         )
+@@ -352,10 +361,7 @@ class GEGLU(keras.layers.Layer):
+     def call(self, inputs):
+         x = self.dense(inputs)
+         x, gate = x[..., : self.output_dim], x[..., self.output_dim :]
+-        tanh_res = keras.activations.tanh(
+-            gate * 0.7978845608 * (1 + 0.044715 * (gate**2))
+-        )
+-        return x * 0.5 * gate * (1 + tanh_res)
++        return x * tf.keras.activations.gelu(gate, approximate=True)
+ 
+ 
+ def td_dot(a, b):
+diff --git a/keras_cv/models/stable_diffusion/stable_diffusion.py b/keras_cv/models/stable_diffusion/stable_diffusion.py
+index 5ef7471..bafe9c5 100644
+--- a/keras_cv/models/stable_diffusion/stable_diffusion.py
++++ b/keras_cv/models/stable_diffusion/stable_diffusion.py
+@@ -207,18 +207,21 @@ class StableDiffusionBase:
+ 
+         # Iterative reverse diffusion stage
+         timesteps = tf.range(1, 1000, 1000 // num_steps)
++        t_embs_lst = self._get_timesteps_embedding(timesteps, batch_size)
++        contexts = tf.concat((unconditional_context, context), 0)
++
+         alphas, alphas_prev = self._get_initial_alphas(timesteps)
+         progbar = keras.utils.Progbar(len(timesteps))
+         iteration = 0
+         for index, timestep in list(enumerate(timesteps))[::-1]:
+             latent_prev = latent  # Set aside the previous latent vector
+-            t_emb = self._get_timestep_embedding(timestep, batch_size)
+-            unconditional_latent = self.diffusion_model.predict_on_batch(
+-                [latent, t_emb, unconditional_context]
+-            )
+-            latent = self.diffusion_model.predict_on_batch(
+-                [latent, t_emb, context]
+-            )
++            latents = tf.concat((latent, latent), 0)
++            t_embs = t_embs_lst[index]
++
++            pred_latent = self.diffusion_model.predict_on_batch(
++                [latents, t_embs, contexts])
++            unconditional_latent, latent = tf.split(pred_latent, 2)
++
+             latent = unconditional_latent + unconditional_guidance_scale * (
+                 latent - unconditional_latent
+             )
+@@ -303,6 +306,21 @@ class StableDiffusionBase:
+         if self._tokenizer is None:
+             self._tokenizer = SimpleTokenizer()
+         return self._tokenizer
++    def _get_timesteps_embedding(
++        self, timesteps, batch_size, dim=320, max_period=10000
++    ):
++        half = dim // 2
++        freqs = tf.math.exp(
++            -math.log(max_period) * tf.range(0, half, dtype=tf.float32) / half
++        )
++        # timesteps shape: [num_steps]
++        args = tf.cast(tf.reshape(timesteps, [-1, 1]), dtype=tf.float32) * freqs
++        # embeddings shape:(steps, half)
++        embeddings = tf.concat([tf.math.cos(args), tf.math.sin(args)], axis=1)
++        embeddings = tf.expand_dims(embeddings, axis=1)
++
++        #  2 is to concatenate the embedding of two forward pass
++        return tf.repeat(embeddings, batch_size * 2, axis=1)
+ 
+     def _get_timestep_embedding(
+         self, timestep, batch_size, dim=320, max_period=10000
diff --git a/models/diffusion/tensorflow/stable_diffusion/inference/requirements.txt b/models/diffusion/tensorflow/stable_diffusion/inference/requirements.txt
new file mode 100644
index 000000000..3fe2b7c98
--- /dev/null
+++ b/models/diffusion/tensorflow/stable_diffusion/inference/requirements.txt
@@ -0,0 +1,5 @@
+pandas
+matplotlib
+scipy==1.11.1
+scikit-image
+numpy==1.25.2
diff --git a/models/image_recognition/pytorch/common/main.py b/models/image_recognition/pytorch/common/main.py
index 9f3f1ce6c..0ea74d190 100644
--- a/models/image_recognition/pytorch/common/main.py
+++ b/models/image_recognition/pytorch/common/main.py
@@ -167,6 +167,8 @@
                          'multi node data parallel training')
 parser.add_argument('--ipex', action='store_true', default=False,
                     help='use intel pytorch extension')
+parser.add_argument('--inductor', action='store_true', default=False,
+                    help='use torch.compile() default inductor backend')
 parser.add_argument('--no-cuda', action='store_true', default=False,
                     help='disable CUDA')
 parser.add_argument('--int8', action='store_true', default=False,
@@ -283,10 +285,14 @@ def main_worker(gpu, ngpus_per_node, args):
     num_classes = model.fc.out_features
     if args.ipex:
         import intel_extension_for_pytorch as ipex
+    elif args.inductor:
+        args.jit = False
     # for ipex path, always convert model to channels_last for bf16, fp32.
     # TODO: int8 path: https://jira.devtools.intel.com/browse/MFDNN-6103
     if args.ipex and not args.int8:
         model = model.to(memory_format=torch.channels_last)
+    if args.inductor:
+        model = model.to(memory_format=torch.channels_last)
 
     if args.ipex and args.bf32:
         ipex.set_fp32_math_mode(mode=ipex.FP32MathMode.BF32, device="cpu")
@@ -386,8 +392,9 @@ def main_worker(gpu, ngpus_per_node, args):
         assert args.int8, "please enable int8 path if you want to do int8 calibration path"
     if args.dummy:
         assert args.evaluate, "please using real dataset if you want run training path"
-    if not args.ipex:
+    if not args.ipex and not args.inductor:
         # for offical pytorch, int8 and jit path is not enabled.
+        # for torch.compile(backend=inductor) INT8 quantization is been supported.
         assert not args.int8, "int8 path is not enabled for offical pytorch"
         assert not args.jit, "jit path is not enabled for offical pytorch"
 
@@ -437,6 +444,7 @@ def main_worker(gpu, ngpus_per_node, args):
         else:
             print("using offical pytorch model to do inference\n")
 
+        # IPEX Path
         if args.ipex:
             model.eval()
             if args.int8:
@@ -478,6 +486,65 @@ def main_worker(gpu, ngpus_per_node, args):
                         with torch.no_grad():
                             model = torch.jit.trace(model, x).eval()
                     model = torch.jit.freeze(model)
+        # torch.compile() inductor path
+        elif args.inductor:
+            model.eval()
+            from torch._inductor import config as inductor_config
+            inductor_config.cpp_wrapper = True
+            x = torch.randn(args.batch_size, 3, 224, 224).contiguous(memory_format=torch.channels_last)
+            if args.int8:
+                from torch.ao.quantization.quantize_pt2e import prepare_pt2e, convert_pt2e
+                import torch.ao.quantization.quantizer.x86_inductor_quantizer as xiq
+                from torch.ao.quantization.quantizer.x86_inductor_quantizer import X86InductorQuantizer
+                from torch._export import capture_pre_autograd_graph, dynamic_dim
+                print('[Info] Running torch.compile() INT8 quantization')
+                with torch.no_grad():
+                    example_inputs = (x,)
+                    exported_model = capture_pre_autograd_graph(
+                        model,
+                        example_inputs
+                    )
+                    quantizer = X86InductorQuantizer()
+                    quantizer.set_global(xiq.get_default_x86_inductor_quantization_config())
+                    prepared_model = prepare_pt2e(exported_model, quantizer)
+                    prepared_model(x)
+                    converted_model = convert_pt2e(prepared_model)
+                    torch.ao.quantization.move_exported_model_to_eval(converted_model)
+                    if args.ipex:
+                        print('[Info] Running torch.compile() with IPEX backend')
+                        model = torch.compile(converted_model, backend="ipex")
+                    else:
+                        print('[Info] Running torch.compile() with default backend')
+                        model = torch.compile(converted_model)
+            elif args.bf16:
+                with torch.no_grad(), torch.cpu.amp.autocast(dtype=torch.bfloat16):
+                    x = x.to(torch.bfloat16)
+                    if args.ipex:
+                        print('[Info] Running torch.compile() BFloat16 with IPEX backend')
+                        model = torch.compile(model, backend="ipex")
+                    else:
+                        print('[Info] Running torch.compile() BFloat16 with default backend')
+                        model = torch.compile(model)
+            elif args.fp16:
+                with torch.no_grad(), torch.cpu.amp.autocast(dtype=torch.half):
+                    x = x.to(torch.half)
+                    if args.ipex:
+                        print('[Info] Running torch.compile() FPloat16 with IPEX backend')
+                        model = torch.compile(model, backend="ipex")
+                    else:
+                        print('[Info] Running torch.compile() FPloat16 with default backend')
+                        model = torch.compile(model)
+            else:
+                with torch.no_grad():
+                    if args.ipex:
+                        print('[Info] Running torch.compile() Float32 with IPEX backend')
+                        model = torch.compile(model, backend="ipex")
+                    else:
+                        print('[Info] Running torch.compile() Float32 with default backend')
+                        model = torch.compile(model)
+            with torch.no_grad(), torch.cpu.amp.autocast(enabled=args.bf16 or args.fp16, dtype=torch.half if args.fp16 else torch.bfloat16):
+                y = model(x)
+                y = model(x)
         validate(val_loader, model, criterion, args)
         return
 
@@ -498,6 +565,15 @@ def main_worker(gpu, ngpus_per_node, args):
         if args.ipex and args.fp16:
             scaler = torch.cpu.amp.GradScaler()
             model, optimizer = ipex.optimize(model, optimizer=optimizer, dtype=torch.half, auto_kernel_selection=True, fuse_update_step=False)
+        
+        if args.inductor:
+            with torch.cpu.amp.autocast(enabled=args.bf16 or args.fp16, dtype=torch.half if args.fp16 else torch.bfloat16):
+                if args.ipex:
+                    print('[Info] Running training steps torch.compile() with IPEX backend')
+                    model = torch.compile(model, backend="ipex")
+                else:
+                    print('[Info] Running training steps torch.compile() with default backend')
+                    model = torch.compile(model)
 
     # parallelize
     if args.distributed and not args.cuda and args.gpu is None:
@@ -532,7 +608,7 @@ def train(train_loader, val_loader, model, criterion, optimizer, lr_scheduler, a
 
     if args.bf16:
         print("running bfloat16 training step\n")
-    elif args.ipex and args.fp16:
+    elif args.fp16:
         print("running float16 training step\n")
     else:
         print("running fp32 training step\n")
@@ -552,7 +628,7 @@ def train(train_loader, val_loader, model, criterion, optimizer, lr_scheduler, a
                 images = images.contiguous(memory_format=torch.channels_last)
             if args.bf16:
                 images = images.to(torch.bfloat16)
-            if args.ipex and args.fp16:
+            if args.fp16:
                 images = images.to(torch.half)
 
             if (epoch - args.start_epoch) * len(train_loader) + i == args.warmup_iterations:
@@ -569,7 +645,7 @@ def train(train_loader, val_loader, model, criterion, optimizer, lr_scheduler, a
                 with torch.cpu.amp.autocast(dtype=torch.bfloat16):
                     output = model(images)
                 output = output.to(torch.float32)
-            elif args.ipex and args.fp16:
+            elif args.fp16:
                 with torch.cpu.amp.autocast(dtype=torch.half):
                     output = model(images)
                 output = output.to(torch.float32)
@@ -580,7 +656,7 @@ def train(train_loader, val_loader, model, criterion, optimizer, lr_scheduler, a
             # compute gradient and do SGD step
             if not args.distributed:
                 optimizer.zero_grad(set_to_none=True)
-            if args.ipex and args.fp16:
+            if args.fp16:
                 scaler.scale(loss).backward()
                 # TODO check it works for fp16 dtype
                 if lr_scheduler:
@@ -638,10 +714,9 @@ def run_weights_sharing_model(m, tid, args):
     x = torch.randn(args.batch_size, 3, 224, 224)
     if args.bf16:
         x = x.to(torch.bfloat16)
-    if args.ipex and args.fp16:
+    if args.fp16:
         x = x.to(torch.half)
-    if args.ipex:
-        x = x.contiguous(memory_format=torch.channels_last)
+    x = x.contiguous(memory_format=torch.channels_last)
 
     with torch.no_grad():
         while num_images < steps:
@@ -649,7 +724,7 @@ def run_weights_sharing_model(m, tid, args):
             if not args.jit and args.bf16:
                 with torch.cpu.amp.autocast(dtype=torch.bfloat16):
                     y = m(x)
-            elif not args.jit and args.ipex and args.fp16:
+            elif not args.jit and args.fp16:
                 with torch.cpu.amp.autocast(dtype=torch.half):
                     y = m(x)
             else:
@@ -687,10 +762,9 @@ def validate(val_loader, model, criterion, args):
         prefix='Test: ')
     print('Evaluating RESNET: total Steps: {}'.format(number_iter))
 
-
-
     # switch to evaluate mode
-    model.eval()
+    if not (args.inductor and args.int8):
+        model.eval()
 
     if args.ipex and args.int8 and args.calibration:
         print("runing int8 calibration step\n")
@@ -724,13 +798,11 @@ def validate(val_loader, model, criterion, args):
                         thread.join()
                     exit()
                 else:
-                    images = torch.randn(args.batch_size, 3, 224, 224)
-                    if args.ipex:
-                        images = images.contiguous(memory_format=torch.channels_last)
+                    images = torch.randn(args.batch_size, 3, 224, 224).contiguous(memory_format=torch.channels_last)
                     target = (torch.rand(args.batch_size) * num_classes).long()
                     if args.bf16:
                         images = images.to(torch.bfloat16)
-                    if args.ipex and args.fp16:
+                    if args.fp16:
                         images = images.to(torch.half)
                     for i in range(number_iter):
                         if i >= args.warmup_iterations:
@@ -738,7 +810,7 @@ def validate(val_loader, model, criterion, args):
                         if not args.jit and args.bf16:
                             with torch.cpu.amp.autocast(dtype=torch.bfloat16):
                                 output = model(images)
-                        elif not args.jit and args.ipex and args.fp16:
+                        elif not args.jit and args.fp16:
                             with torch.cpu.amp.autocast(dtype=torch.half):
                                 output = model(images)
                         else:
@@ -747,7 +819,7 @@ def validate(val_loader, model, criterion, args):
                         if i >= args.warmup_iterations:
                             batch_time.update(time.time() - end)
 
-                        if args.bf16 or (args.ipex and args.fp16):
+                        if args.bf16 or args.fp16:
                             output = output.to(torch.float32)
 
                         loss = criterion(output, target)
@@ -767,7 +839,7 @@ def validate(val_loader, model, criterion, args):
                         images = images.contiguous(memory_format=torch.channels_last)
                     if args.bf16:
                         images = images.to(torch.bfloat16)
-                    if args.ipex and args.fp16:
+                    if args.fp16:
                         images = images.to(torch.half)
                     if args.gpu is not None:
                         images = images.cuda(args.gpu, non_blocking=True)
@@ -777,7 +849,7 @@ def validate(val_loader, model, criterion, args):
                     if not args.jit and args.bf16:
                         with torch.cpu.amp.autocast(dtype=torch.bfloat16):
                             output = model(images)
-                    elif not args.jit and args.ipex and args.fp16:
+                    elif not args.jit and args.fp16:
                         with torch.cpu.amp.autocast(dtype=torch.half):
                             output = model(images)
 
@@ -787,7 +859,7 @@ def validate(val_loader, model, criterion, args):
                     # compute output
                     batch_time.update(time.time() - end)
                     #print(output)
-                    if args.bf16 or (args.ipex and args.fp16):
+                    if args.bf16 or args.fp16:
                         output = output.to(torch.float32)
                     loss = criterion(output, target)
                     # measure accuracy and record loss
diff --git a/models/image_recognition/pytorch/common/train.py b/models/image_recognition/pytorch/common/train.py
index b66d873cd..7d25c5972 100644
--- a/models/image_recognition/pytorch/common/train.py
+++ b/models/image_recognition/pytorch/common/train.py
@@ -195,7 +195,7 @@ def main_worker(args):
         print("Using global batch size: ", int(args.local_batch_size * args.world_size))
         print("Create DistributedDataParallel in CPU")
         model = torch.nn.parallel.DistributedDataParallel(model, find_unused_parameters=True, broadcast_buffers=False,
-                                                                 gradient_as_bucket_view=True, bucket_cap_mb=50)
+                                                                 gradient_as_bucket_view=False, bucket_cap_mb=50)
 
     # optionally resume from a checkpoint
     if args.resume:
@@ -270,8 +270,6 @@ def main_worker(args):
         scaler = torch.cpu.amp.GradScaler()
     else:
         scaler = None
-    # Train mode
-    model.train()
 
     if args.ipex:
         if args.bf16:
@@ -309,6 +307,9 @@ def main_worker(args):
             [batch_time, data_time, losses, top1, top5],
             prefix="Epoch: [{}]".format(epoch))
 
+        # Train mode
+        model.train()
+
         start = time.time()
         for i, (images, target) in enumerate(train_loader):
 
diff --git a/models/image_recognition/tensorflow/inceptionv4/inference/__init__.py b/models/image_recognition/tensorflow/inceptionv4/inference/__init__.py
deleted file mode 100644
index c4fdb7d61..000000000
--- a/models/image_recognition/tensorflow/inceptionv4/inference/__init__.py
+++ /dev/null
@@ -1,19 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
diff --git a/models/image_recognition/tensorflow/inceptionv4/inference/accuracy.py b/models/image_recognition/tensorflow/inceptionv4/inference/accuracy.py
deleted file mode 100644
index eacb42212..000000000
--- a/models/image_recognition/tensorflow/inceptionv4/inference/accuracy.py
+++ /dev/null
@@ -1,170 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
-# Copyright 2017 The TensorFlow Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-
-import argparse
-import sys
-import os
-import time
-import numpy as np
-
-from google.protobuf import text_format
-import tensorflow as tf
-import preprocessing
-import datasets
-
-NUM_TEST_IMAGES = 50000
-
-
-def load_graph(model_file):
-    graph = tf.Graph()
-    graph_def = tf.compat.v1.GraphDef()
-
-    import os
-    file_ext = os.path.splitext(model_file)[1]
-
-    with open(model_file, "rb") as f:
-        if file_ext == '.pbtxt':
-            text_format.Merge(f.read(), graph_def)
-        else:
-            graph_def.ParseFromString(f.read())
-    with graph.as_default():
-        tf.import_graph_def(graph_def, name='')
-
-    return graph
-
-
-if __name__ == "__main__":
-    parser = argparse.ArgumentParser()
-    parser.add_argument("--input_graph", default=None,
-                        help="graph/model to be executed")
-    parser.add_argument("--data_location", default=None,
-                        help="full path to the validation data")
-    parser.add_argument("--input_height", default=None,
-                        type=int, help="input height")
-    parser.add_argument("--input_width", default=None,
-                        type=int, help="input width")
-    parser.add_argument("--batch_size", default=32,
-                        type=int, help="batch size")
-    parser.add_argument("--input_layer", default="input",
-                        help="name of input layer")
-    parser.add_argument("--output_layer",
-                        default="InceptionV4/Logits/Predictions",
-                        help="name of output layer")
-    parser.add_argument(
-        '--num_inter_threads',
-        help='number threads across operators',
-        type=int, default=1)
-    parser.add_argument(
-        '--num_intra_threads',
-        help='number threads for an operator',
-        type=int, default=1)
-    args = parser.parse_args()
-
-    if args.input_graph:
-        model_file = args.input_graph
-    else:
-        sys.exit("Please provide a graph file.")
-    if args.input_height:
-        input_height = args.input_height
-    else:
-        input_height = 299
-    if args.input_width:
-        input_width = args.input_width
-    else:
-        input_width = 299
-    batch_size = args.batch_size
-    input_layer = args.input_layer
-    output_layer = args.output_layer
-    num_inter_threads = args.num_inter_threads
-    num_intra_threads = args.num_intra_threads
-    data_location = args.data_location
-
-
-    data_graph = tf.Graph() ###
-    with data_graph.as_default(): ###
-      dataset = datasets.ImagenetData(data_location)
-      preprocessor = dataset.get_image_preprocessor()(
-          input_height, input_width, batch_size,
-          1,  # device count
-          tf.float32,  # data_type for input fed to the graph
-          train=False,  # doing inference
-          resize_method='bilinear')
-      images, labels = preprocessor.minibatch(dataset, subset='validation',
-                                            use_datasets=True,
-                                            cache_data=False)
-    graph = load_graph(model_file)
-    input_tensor = graph.get_tensor_by_name(input_layer + ":0")
-    output_tensor = graph.get_tensor_by_name(output_layer + ":0")
-
-    config = tf.compat.v1.ConfigProto()
-    config.inter_op_parallelism_threads = num_inter_threads
-    config.intra_op_parallelism_threads = num_intra_threads
-
-    total_accuracy1, total_accuracy5 = (0.0, 0.0)
-    num_processed_images = 0
-    num_remaining_images = dataset.num_examples_per_epoch(subset='validation') \
-                           - num_processed_images
-
-    with tf.compat.v1.Session(graph=data_graph) as sess: ###
-        sess_graph = tf.compat.v1.Session(graph=graph, config=config)
-        while num_remaining_images >= batch_size:
-            # Reads and preprocess data
-            np_images, np_labels = sess.run([images[0], labels[0]])
-            num_processed_images += batch_size
-            num_remaining_images -= batch_size
-            start_time = time.time()
-            # Compute inference on the preprocessed data
-            predictions = sess_graph.run(output_tensor,
-                                         {input_tensor: np_images})
-            elapsed_time = time.time() - start_time
-            accuracy1 = tf.reduce_sum(
-                input_tensor=tf.cast(tf.nn.in_top_k(predictions=tf.constant(predictions),
-                                       targets=tf.constant(np_labels), k=1), tf.float32))
-
-            accuracy5 = tf.reduce_sum(
-                input_tensor=tf.cast(tf.nn.in_top_k(predictions=tf.constant(predictions),
-                                       targets=tf.constant(np_labels), k=5), tf.float32))
-            np_accuracy1, np_accuracy5 = sess.run([accuracy1, accuracy5])
-            total_accuracy1 += np_accuracy1
-            total_accuracy5 += np_accuracy5
-            print("Iteration time: %0.4f ms" % elapsed_time)
-            print(
-                "Processed %d images. (Top1 accuracy, Top5 accuracy) = (%0.4f, %0.4f)" \
-                % (
-                num_processed_images, total_accuracy1 / num_processed_images,
-                total_accuracy5 / num_processed_images))
diff --git a/models/image_recognition/tensorflow/inceptionv4/inference/benchmark.py b/models/image_recognition/tensorflow/inceptionv4/inference/benchmark.py
deleted file mode 100644
index ae97b9466..000000000
--- a/models/image_recognition/tensorflow/inceptionv4/inference/benchmark.py
+++ /dev/null
@@ -1,168 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
-# Copyright 2017 The TensorFlow Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-
-import argparse
-import sys
-import os
-import time
-import numpy as np
-
-from google.protobuf import text_format
-import tensorflow as tf
-
-def load_graph(model_file):
-  graph = tf.Graph()
-  graph_def = tf.compat.v1.GraphDef()
-
-  import os
-  file_ext = os.path.splitext(model_file)[1]
-
-  with open(model_file, "rb") as f:
-    if file_ext == '.pbtxt':
-      text_format.Merge(f.read(), graph_def)
-    else:
-      graph_def.ParseFromString(f.read())
-  with graph.as_default():
-    tf.import_graph_def(graph_def, name='')
-
-  return graph
-
-if __name__ == "__main__":
-  parser = argparse.ArgumentParser()
-  parser.add_argument("--input_graph", default=None,
-                      help="graph/model to be executed")
-  parser.add_argument("--input_height", default=None,
-                      type=int, help="input height")
-  parser.add_argument("--input_width", default=None,
-                      type=int, help="input width")
-  parser.add_argument("--batch_size", default=32,
-                      type=int, help="batch size")
-  parser.add_argument("--input_layer", default="input",
-                      help="name of input layer")
-  parser.add_argument("--output_layer", default="InceptionV4/Logits/Predictions",
-                      help="name of output layer")
-  parser.add_argument(
-      '--num_inter_threads',
-      help='number threads across operators',
-      type=int, default=1)
-  parser.add_argument(
-      '--num_intra_threads',
-      help='number threads for an operator',
-      type=int, default=1)
-  parser.add_argument("--warmup_steps", type=int, default=10,
-                      help="number of warmup steps")
-  parser.add_argument("--steps", type=int, default=50, help="number of steps")
-  args = parser.parse_args()
-
-  if args.input_graph:
-    model_file = args.input_graph
-  else:
-    sys.exit("Please provide a graph file.")
-  if args.input_height:
-    input_height = args.input_height
-  else:
-    input_height = 299
-  if args.input_width:
-    input_width = args.input_width
-  else:
-    input_width = 299
-  batch_size = args.batch_size
-  input_layer = args.input_layer
-  output_layer = args.output_layer
-  warmup_steps = args.warmup_steps
-  steps = args.steps
-  assert steps > 10, "Benchmark steps should be at least 10."
-  num_inter_threads = args.num_inter_threads
-  num_intra_threads = args.num_intra_threads
-
-  data_graph = tf.Graph() ##
-  with data_graph.as_default():##
-    input_shape = [batch_size, input_height, input_width, 3]
-    images = tf.random.truncated_normal(
-          input_shape,
-          dtype=tf.float32,
-          stddev=10,
-          name='synthetic_images')
-
-  #image_data = None
-  #with tf.compat.v1.Session() as sess:
-  #  image_data = sess.run(images)
-
-  graph = load_graph(model_file)
-
-  input_tensor = graph.get_tensor_by_name(input_layer + ":0");
-  output_tensor = graph.get_tensor_by_name(output_layer + ":0");
-  tf.compat.v1.global_variables_initializer()###
-
-  config = tf.compat.v1.ConfigProto()
-  config.inter_op_parallelism_threads = num_inter_threads
-  config.intra_op_parallelism_threads = num_intra_threads
-
-  data_config = tf.compat.v1.ConfigProto()###
-  data_config.inter_op_parallelism_threads = num_inter_threads ###
-  data_config.intra_op_parallelism_threads = num_intra_threads ###
-  
-  data_sess = tf.compat.v1.Session(graph=data_graph, config=data_config) ###
-
-  with tf.compat.v1.Session(graph=graph, config=config) as sess:
-    sys.stdout.flush()
-    print("[Running warmup steps...]")
-    image_data = data_sess.run(images) ###
-    for t in range(warmup_steps):
-      start_time = time.perf_counter()
-      sess.run(output_tensor, {input_tensor: image_data})
-      elapsed_time = time.perf_counter() - start_time
-      if((t+1) % 10 == 0):
-        print("steps = {0}, {1} images/sec"
-              "".format(t+1, batch_size/elapsed_time))
-
-    print("[Running benchmark steps...]")
-    total_time   = 0;
-    total_images = 0;
-    for t in range(steps):
-      start_time = time.perf_counter()
-      results = sess.run(output_tensor, {input_tensor: image_data})
-      elapsed_time = time.perf_counter() - start_time
-      if((t+1) % 10 == 0):
-        print("steps = {0}, {1} images/sec"
-              "".format(t+1, batch_size/elapsed_time));
-      total_time += elapsed_time
-      total_images += batch_size
-    average_time = total_time / total_images
-    if batch_size == 1:
-      print('Latency: %.3f ms' % (average_time * 1000))
diff --git a/models/image_recognition/tensorflow/inceptionv4/inference/cnn_util.py b/models/image_recognition/tensorflow/inceptionv4/inference/cnn_util.py
deleted file mode 100644
index e74df9a41..000000000
--- a/models/image_recognition/tensorflow/inceptionv4/inference/cnn_util.py
+++ /dev/null
@@ -1,48 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
-# Copyright 2017 The TensorFlow Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-
-"""Utilities for CNN benchmarks."""
-
-import tensorflow as tf
-
-
-def tensorflow_version_tuple():
-  v = tf.__version__
-  major, minor, patch = v.split('.')
-  return (int(major), int(minor), patch)
-
-
-def tensorflow_version():
-  vt = tensorflow_version_tuple()
-  return vt[0] * 1000 + vt[1]
diff --git a/models/image_recognition/tensorflow/inceptionv4/inference/datasets.py b/models/image_recognition/tensorflow/inceptionv4/inference/datasets.py
deleted file mode 100644
index 93148f888..000000000
--- a/models/image_recognition/tensorflow/inceptionv4/inference/datasets.py
+++ /dev/null
@@ -1,194 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
-# Copyright 2017 The TensorFlow Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-
-"""Benchmark dataset utilities.
-"""
-
-from abc import abstractmethod
-import os
-
-import numpy as np
-from six.moves import cPickle
-from six.moves import xrange  # pylint: disable=redefined-builtin
-import tensorflow as tf
-
-from tensorflow.python.platform import gfile
-import preprocessing
-
-
-IMAGENET_NUM_TRAIN_IMAGES = 1281167
-IMAGENET_NUM_VAL_IMAGES = 50000
-
-
-def create_dataset(data_dir, data_name):
-  """Create a Dataset instance based on data_dir and data_name."""
-  supported_datasets = {
-      'imagenet': ImagenetData,
-      'cifar10': Cifar10Data,
-  }
-  if not data_dir and not data_name:
-    # When using synthetic data, use synthetic imagenet images by default.
-    data_name = 'imagenet'
-
-  if data_name is None:
-    for supported_name in supported_datasets:
-      if supported_name in data_dir:
-        data_name = supported_name
-        break
-
-  if data_name is None:
-    raise ValueError('Could not identify name of dataset. '
-                     'Please specify with --data_name option.')
-
-  if data_name not in supported_datasets:
-    raise ValueError('Unknown dataset. Must be one of %s', ', '.join(
-        [key for key in sorted(supported_datasets.keys())]))
-
-  return supported_datasets[data_name](data_dir)
-
-
-class Dataset(object):
-  """Abstract class for cnn benchmarks dataset."""
-
-  def __init__(self, name, height=None, width=None, depth=None, data_dir=None,
-               queue_runner_required=False, num_classes=1000):
-    self.name = name
-    self.height = height
-    self.width = width
-    self.depth = depth or 3
-
-    self.data_dir = data_dir
-    self._queue_runner_required = queue_runner_required
-    self._num_classes = num_classes
-
-  def tf_record_pattern(self, subset):
-    return os.path.join(self.data_dir, '%s-*-of-*' % subset)
-
-  def reader(self):
-    return tf.compat.v1.TFRecordReader()
-
-  @property
-  def num_classes(self):
-    return self._num_classes
-
-  @num_classes.setter
-  def num_classes(self, val):
-    self._num_classes = val
-
-  @abstractmethod
-  def num_examples_per_epoch(self, subset):
-    pass
-
-  def __str__(self):
-    return self.name
-
-  def get_image_preprocessor(self):
-    return None
-
-  def queue_runner_required(self):
-    return self._queue_runner_required
-
-  def use_synthetic_gpu_images(self):
-    return not self.data_dir
-
-
-class ImagenetData(Dataset):
-  """Configuration for Imagenet dataset."""
-
-  def __init__(self, data_dir=None):
-    super(ImagenetData, self).__init__('imagenet', 300, 300, data_dir=data_dir)
-
-  def num_examples_per_epoch(self, subset='train'):
-    if subset == 'train':
-      return IMAGENET_NUM_TRAIN_IMAGES
-    elif subset == 'validation':
-      return IMAGENET_NUM_VAL_IMAGES
-    else:
-      raise ValueError('Invalid data subset "%s"' % subset)
-
-  def get_image_preprocessor(self):
-    if self.use_synthetic_gpu_images():
-      return preprocessing.SyntheticImagePreprocessor
-    else:
-      return preprocessing.RecordInputImagePreprocessor
-
-
-class Cifar10Data(Dataset):
-  """Configuration for cifar 10 dataset.
-
-  It will mount all the input images to memory.
-  """
-
-  def __init__(self, data_dir=None):
-    super(Cifar10Data, self).__init__('cifar10', 32, 32, data_dir=data_dir,
-                                      queue_runner_required=True,
-                                      num_classes=10)
-
-  def read_data_files(self, subset='train'):
-    """Reads from data file and returns images and labels in a numpy array."""
-    assert self.data_dir, ('Cannot call `read_data_files` when using synthetic '
-                           'data')
-    if subset == 'train':
-      filenames = [os.path.join(self.data_dir, 'data_batch_%d' % i)
-                   for i in xrange(1, 6)]
-    elif subset == 'validation':
-      filenames = [os.path.join(self.data_dir, 'test_batch')]
-    else:
-      raise ValueError('Invalid data subset "%s"' % subset)
-
-    inputs = []
-    for filename in filenames:
-      with gfile.Open(filename, 'r') as f:
-        inputs.append(cPickle.load(f))
-    # See http://www.cs.toronto.edu/~kriz/cifar.html for a description of the
-    # input format.
-    all_images = np.concatenate(
-        [each_input['data'] for each_input in inputs]).astype(np.float32)
-    all_labels = np.concatenate(
-        [each_input['labels'] for each_input in inputs])
-    return all_images, all_labels
-
-  def num_examples_per_epoch(self, subset='train'):
-    if subset == 'train':
-      return 50000
-    elif subset == 'validation':
-      return 10000
-    else:
-      raise ValueError('Invalid data subset "%s"' % subset)
-
-  def get_image_preprocessor(self):
-    if self.use_synthetic_gpu_images():
-      return preprocessing.SyntheticImagePreprocessor
-    else:
-      return preprocessing.Cifar10ImagePreprocessor
diff --git a/models/image_recognition/tensorflow/inceptionv4/inference/preprocessing.py b/models/image_recognition/tensorflow/inceptionv4/inference/preprocessing.py
deleted file mode 100644
index d398f1030..000000000
--- a/models/image_recognition/tensorflow/inceptionv4/inference/preprocessing.py
+++ /dev/null
@@ -1,644 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
-# Copyright 2017 The TensorFlow Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-
-"""Image pre-processing utilities.
-"""
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-import math
-from six.moves import xrange  # pylint: disable=redefined-builtin
-import tensorflow as tf
-
-#from tensorflow.contrib.data.python.ops import batching
-#from tensorflow.contrib.data.python.ops import interleave_ops
-#from tensorflow.contrib.image.python.ops import distort_image_ops
-
-from tensorflow.python.data.experimental import parallel_interleave###
-from tensorflow.python.data.experimental import map_and_batch###
-
-from tensorflow.python.layers import utils
-from tensorflow.python.ops import data_flow_ops
-from tensorflow.python.platform import gfile
-import cnn_util
-
-from tensorflow.python.ops import control_flow_ops
-
-
-def parse_example_proto(example_serialized):
-    """Parses an Example proto containing a training example of an image.
-  
-    The output of the build_image_data.py image preprocessing script is a dataset
-    containing serialized Example protocol buffers. Each Example proto contains
-    the following fields:
-  
-      image/height: 462
-      image/width: 581
-      image/colorspace: 'RGB'
-      image/channels: 3
-      image/class/label: 615
-      image/class/synset: 'n03623198'
-      image/class/text: 'knee pad'
-      image/object/bbox/xmin: 0.1
-      image/object/bbox/xmax: 0.9
-      image/object/bbox/ymin: 0.2
-      image/object/bbox/ymax: 0.6
-      image/object/bbox/label: 615
-      image/format: 'JPEG'
-      image/filename: 'ILSVRC2012_val_00041207.JPEG'
-      image/encoded: <JPEG encoded string>
-  
-    Args:
-      example_serialized: scalar Tensor tf.string containing a serialized
-        Example protocol buffer.
-  
-    Returns:
-      image_buffer: Tensor tf.string containing the contents of a JPEG file.
-      label: Tensor tf.int32 containing the label.
-      bbox: 3-D float Tensor of bounding boxes arranged [1, num_boxes, coords]
-        where each coordinate is [0, 1) and the coordinates are arranged as
-        [ymin, xmin, ymax, xmax].
-      text: Tensor tf.string containing the human-readable label.
-    """
-    # Dense features in Example proto.
-    feature_map = {
-        'image/encoded': tf.io.FixedLenFeature([], dtype=tf.string,
-                                            default_value=''),
-        'image/class/label': tf.io.FixedLenFeature([1], dtype=tf.int64,
-                                                default_value=-1),
-        'image/class/text': tf.io.FixedLenFeature([], dtype=tf.string,
-                                               default_value=''),
-    }
-    sparse_float32 = tf.io.VarLenFeature(dtype=tf.float32)
-    # Sparse features in Example proto.
-    feature_map.update(
-        {k: sparse_float32 for k in ['image/object/bbox/xmin',
-                                     'image/object/bbox/ymin',
-                                     'image/object/bbox/xmax',
-                                     'image/object/bbox/ymax']})
-
-    features = tf.io.parse_single_example(serialized=example_serialized, features=feature_map)
-    label = tf.cast(features['image/class/label'], dtype=tf.int32)
-
-    xmin = tf.expand_dims(features['image/object/bbox/xmin'].values, 0)
-    ymin = tf.expand_dims(features['image/object/bbox/ymin'].values, 0)
-    xmax = tf.expand_dims(features['image/object/bbox/xmax'].values, 0)
-    ymax = tf.expand_dims(features['image/object/bbox/ymax'].values, 0)
-
-    # Note that we impose an ordering of (y, x) just to make life difficult.
-    bbox = tf.concat([ymin, xmin, ymax, xmax], 0)
-
-    # Force the variable number of bounding boxes into the shape
-    # [1, num_boxes, coords].
-    bbox = tf.expand_dims(bbox, 0)
-    bbox = tf.transpose(a=bbox, perm=[0, 2, 1])
-
-    return features['image/encoded'], label, bbox, features['image/class/text']
-
-
-def get_image_resize_method(resize_method, batch_position=0):
-    """Get tensorflow resize method.
-  
-    If resize_method is 'round_robin', return different methods based on batch
-    position in a round-robin fashion. NOTE: If the batch size is not a multiple
-    of the number of methods, then the distribution of methods will not be
-    uniform.
-  
-    Args:
-      resize_method: (string) nearest, bilinear, bicubic, area, or round_robin.
-      batch_position: position of the image in a batch. NOTE: this argument can
-        be an integer or a tensor
-    Returns:
-      one of resize type defined in tf.image.ResizeMethod.
-    """
-    resize_methods_map = {
-        'nearest': tf.image.ResizeMethod.NEAREST_NEIGHBOR,
-        'bilinear': tf.image.ResizeMethod.BILINEAR,
-        'bicubic': tf.image.ResizeMethod.BICUBIC,
-        'area': tf.image.ResizeMethod.AREA
-    }
-
-    if resize_method != 'round_robin':
-        return resize_methods_map[resize_method]
-
-    # return a resize method based on batch position in a round-robin fashion.
-    resize_methods = resize_methods_map.values()
-
-    def lookup(index):
-        return resize_methods[index]
-
-    def resize_method_0():
-        return utils.smart_cond(batch_position % len(resize_methods) == 0,
-                                lambda: lookup(0), resize_method_1)
-
-    def resize_method_1():
-        return utils.smart_cond(batch_position % len(resize_methods) == 1,
-                                lambda: lookup(1), resize_method_2)
-
-    def resize_method_2():
-        return utils.smart_cond(batch_position % len(resize_methods) == 2,
-                                lambda: lookup(2), lambda: lookup(3))
-
-    # NOTE(jsimsa): Unfortunately, we cannot use a single recursive function here
-    # because TF would not be able to construct a finite graph.
-
-    return resize_method_0()
-
-
-def decode_jpeg(image_buffer, scope=None):  # , dtype=tf.float32):
-    """Decode a JPEG string into one 3-D float image Tensor.
-  
-    Args:
-      image_buffer: scalar string Tensor.
-      scope: Optional scope for op_scope.
-    Returns:
-      3-D float Tensor with values ranging from [0, 1).
-    """
-    # with tf.op_scope([image_buffer], scope, 'decode_jpeg'):
-    # with tf.name_scope(scope, 'decode_jpeg', [image_buffer]):
-    with tf.compat.v1.name_scope(scope or 'decode_jpeg'):
-        # Decode the string as an RGB JPEG.
-        # Note that the resulting image contains an unknown height and width
-        # that is set dynamically by decode_jpeg. In other words, the height
-        # and width of image is unknown at compile-time.
-        image = tf.image.decode_jpeg(image_buffer, channels=3)  # ,
-        #     fancy_upscaling=False,
-        #     dct_method='INTEGER_FAST')
-
-        # image = tf.Print(image, [tf.shape(image)], 'Image shape: ')
-        image = tf.image.convert_image_dtype(image, dtype=tf.float32)
-
-        return image
-
-
-def preprocess_for_eval(image, height, width,
-                        central_fraction=0.875, scope=None):
-    """Prepare one image for evaluation.
-  
-    If height and width are specified it would output an image with that size by
-    applying resize_bilinear.
-  
-    If central_fraction is specified it would crop the central fraction of the
-    input image.
-  
-    Args:
-      image: 3-D Tensor of image. If dtype is tf.float32 then the range should be
-        [0, 1], otherwise it would converted to tf.float32 assuming that the range
-        is [0, MAX], where MAX is largest positive representable number for
-        int(8/16/32) data type (see `tf.image.convert_image_dtype` for details).
-      height: integer
-      width: integer
-      central_fraction: Optional Float, fraction of the image to crop.
-      scope: Optional scope for name_scope.
-    Returns:
-      3-D float Tensor of prepared image.
-    """
-    with tf.compat.v1.name_scope(scope, 'eval_image', [image, height, width]):
-        if image.dtype != tf.float32:
-            image = tf.image.convert_image_dtype(image, dtype=tf.float32)
-        # Crop the central region of the image with an area containing 87.5% of
-        # the original image.
-        if central_fraction:
-            image = tf.image.central_crop(image,
-                                          central_fraction=central_fraction)
-
-        if height and width:
-            # Resize the image to the specified height and width.
-            image = tf.expand_dims(image, 0)
-            image = tf.image.resize(image, [height, width],
-                                             method=tf.image.ResizeMethod.BILINEAR)
-            image = tf.squeeze(image, [0])
-        image = tf.subtract(image, 0.5)
-        image = tf.multiply(image, 2.0)
-        return image
-
-
-def apply_with_random_selector(x, func, num_cases):
-    """Computes func(x, sel), with sel sampled from [0...num_cases-1].
-  
-    Args:
-      x: input Tensor.
-      func: Python function to apply.
-      num_cases: Python int32, number of cases to sample sel from.
-  
-    Returns:
-      The result of func(x, sel), where func receives the value of the
-      selector as a python integer, but sel is sampled dynamically.
-    """
-    sel = tf.random.uniform([], maxval=num_cases, dtype=tf.int32)
-    # Pass the real x only to one of the func calls.
-    return control_flow_ops.merge([
-        func(control_flow_ops.switch(x, tf.equal(sel, case))[1], case)
-        for case in range(num_cases)])[0]
-
-
-def distort_color(image, color_ordering=0, fast_mode=True, scope=None):
-    """Distort the color of a Tensor image.
-  
-    Each color distortion is non-commutative and thus ordering of the color ops
-    matters. Ideally we would randomly permute the ordering of the color ops.
-    Rather then adding that level of complication, we select a distinct ordering
-    of color ops for each preprocessing thread.
-  
-    Args:
-      image: 3-D Tensor containing single image in [0, 1].
-      color_ordering: Python int, a type of distortion (valid values: 0-3).
-      fast_mode: Avoids slower ops (random_hue and random_contrast)
-      scope: Optional scope for name_scope.
-    Returns:
-      3-D Tensor color-distorted image on range [0, 1]
-    Raises:
-      ValueError: if color_ordering not in [0, 3]
-    """
-    with tf.compat.v1.name_scope(scope, 'distort_color', [image]):
-        if fast_mode:
-            if color_ordering == 0:
-                image = tf.image.random_brightness(image, max_delta=32. / 255.)
-                image = tf.image.random_saturation(image, lower=0.5, upper=1.5)
-            else:
-                image = tf.image.random_saturation(image, lower=0.5, upper=1.5)
-                image = tf.image.random_brightness(image, max_delta=32. / 255.)
-        else:
-            if color_ordering == 0:
-                image = tf.image.random_brightness(image, max_delta=32. / 255.)
-                image = tf.image.random_saturation(image, lower=0.5, upper=1.5)
-                image = tf.image.random_hue(image, max_delta=0.2)
-                image = tf.image.random_contrast(image, lower=0.5, upper=1.5)
-            elif color_ordering == 1:
-                image = tf.image.random_saturation(image, lower=0.5, upper=1.5)
-                image = tf.image.random_brightness(image, max_delta=32. / 255.)
-                image = tf.image.random_contrast(image, lower=0.5, upper=1.5)
-                image = tf.image.random_hue(image, max_delta=0.2)
-            elif color_ordering == 2:
-                image = tf.image.random_contrast(image, lower=0.5, upper=1.5)
-                image = tf.image.random_hue(image, max_delta=0.2)
-                image = tf.image.random_brightness(image, max_delta=32. / 255.)
-                image = tf.image.random_saturation(image, lower=0.5, upper=1.5)
-            elif color_ordering == 3:
-                image = tf.image.random_hue(image, max_delta=0.2)
-                image = tf.image.random_saturation(image, lower=0.5, upper=1.5)
-                image = tf.image.random_contrast(image, lower=0.5, upper=1.5)
-                image = tf.image.random_brightness(image, max_delta=32. / 255.)
-            else:
-                raise ValueError('color_ordering must be in [0, 3]')
-
-        # The random_* ops do not necessarily clamp.
-        return tf.clip_by_value(image, 0.0, 1.0)
-
-
-def distorted_bounding_box_crop(image,
-                                bbox,
-                                min_object_covered=0.1,
-                                aspect_ratio_range=(0.75, 1.33),
-                                area_range=(0.05, 1.0),
-                                max_attempts=100,
-                                scope=None):
-    """Generates cropped_image using a one of the bboxes randomly distorted.
-  
-    See `tf.image.sample_distorted_bounding_box` for more documentation.
-  
-    Args:
-      image: 3-D Tensor of image (it will be converted to floats in [0, 1]).
-      bbox: 3-D float Tensor of bounding boxes arranged [1, num_boxes, coords]
-        where each coordinate is [0, 1) and the coordinates are arranged
-        as [ymin, xmin, ymax, xmax]. If num_boxes is 0 then it would use the whole
-        image.
-      min_object_covered: An optional `float`. Defaults to `0.1`. The cropped
-        area of the image must contain at least this fraction of any bounding box
-        supplied.
-      aspect_ratio_range: An optional list of `floats`. The cropped area of the
-        image must have an aspect ratio = width / height within this range.
-      area_range: An optional list of `floats`. The cropped area of the image
-        must contain a fraction of the supplied image within in this range.
-      max_attempts: An optional `int`. Number of attempts at generating a cropped
-        region of the image of the specified constraints. After `max_attempts`
-        failures, return the entire image.
-      scope: Optional scope for name_scope.
-    Returns:
-      A tuple, a 3-D Tensor cropped_image and the distorted bbox
-    """
-    with tf.compat.v1.name_scope(scope, 'distorted_bounding_box_crop', [image, bbox]):
-        # Each bounding box has shape [1, num_boxes, box coords] and
-        # the coordinates are ordered [ymin, xmin, ymax, xmax].
-
-        # A large fraction of image datasets contain a human-annotated bounding
-        # box delineating the region of the image containing the object of interest.
-        # We choose to create a new bounding box for the object which is a randomly
-        # distorted version of the human-annotated bounding box that obeys an
-        # allowed range of aspect ratios, sizes and overlap with the human-annotated
-        # bounding box. If no box is supplied, then we assume the bounding box is
-        # the entire image.
-        sample_distorted_bounding_box = tf.image.sample_distorted_bounding_box(
-            image_size=tf.shape(input=image),
-            bounding_boxes=bbox,
-            min_object_covered=min_object_covered,
-            aspect_ratio_range=aspect_ratio_range,
-            area_range=area_range,
-            max_attempts=max_attempts,
-            use_image_if_no_bounding_boxes=True)
-        bbox_begin, bbox_size, distort_bbox = sample_distorted_bounding_box
-
-        # Crop the image to the specified bounding box.
-        cropped_image = tf.slice(image, bbox_begin, bbox_size)
-        return cropped_image, distort_bbox
-
-
-def preprocess_for_train(image, height, width, bbox,
-                         batch_position,
-                         fast_mode=True,
-                         scope=None,
-                         add_image_summaries=True):
-    """Distort one image for training a network.
-  
-    Distorting images provides a useful technique for augmenting the data
-    set during training in order to make the network invariant to aspects
-    of the image that do not effect the label.
-  
-    Args:
-      image: 3-D Tensor of image. If dtype is tf.float32 then the range should be
-        [0, 1], otherwise it would converted to tf.float32 assuming that the range
-        is [0, MAX], where MAX is largest positive representable number for
-        int(8/16/32) data type (see `tf.image.convert_image_dtype` for details).
-      height: integer
-      width: integer
-      bbox: 3-D float Tensor of bounding boxes arranged [1, num_boxes, coords]
-        where each coordinate is [0, 1) and the coordinates are arranged
-        as [ymin, xmin, ymax, xmax].
-      batch_position: position of the image in a batch, which affects how images
-        are distorted and resized. NOTE: this argument can be an integer or a
-        tensor
-      scope: Optional scope for op_scope.
-      add_image_summaries: Enable image summaries.
-    Returns:
-      3-D float Tensor of distorted image used for training with range [-1, 1].
-    """
-
-    with tf.compat.v1.name_scope(scope, 'distort_image', [image, height, width, bbox]):
-        if bbox is None:
-            bbox = tf.constant([0.0, 0.0, 1.0, 1.0],
-                               dtype=tf.float32,
-                               shape=[1, 1, 4])
-        if image.dtype != tf.float32:
-            image = tf.image.convert_image_dtype(image, dtype=tf.float32)
-        # Each bounding box has shape [1, num_boxes, box coords] and
-        # the coordinates are ordered [ymin, xmin, ymax, xmax].
-        image_with_box = tf.image.draw_bounding_boxes(tf.expand_dims(image, 0),
-                                                      bbox)
-        if add_image_summaries:
-            tf.compat.v1.summary.image('image_with_bounding_boxes', image_with_box)
-
-        distorted_image, distorted_bbox = distorted_bounding_box_crop(image,
-                                                                      bbox)
-        # Restore the shape since the dynamic slice based upon the bbox_size loses
-        # the third dimension.
-        distorted_image.set_shape([None, None, 3])
-        image_with_distorted_box = tf.image.draw_bounding_boxes(
-            tf.expand_dims(image, 0), distorted_bbox)
-        if add_image_summaries:
-            tf.compat.v1.summary.image('images_with_distorted_bounding_box',
-                             image_with_distorted_box)
-
-        # This resizing operation may distort the images because the aspect
-        # ratio is not respected. We select a resize method in a round robin
-        # fashion based on the thread number.
-        # Note that ResizeMethod contains 4 enumerated resizing methods.
-
-        # We select only 1 case for fast_mode bilinear.
-        num_resize_cases = 1 if fast_mode else 4
-        distorted_image = apply_with_random_selector(
-            distorted_image,
-            lambda x, method: tf.image.resize(x, [height, width],
-                                                     method),
-            num_cases=num_resize_cases)
-
-        if add_image_summaries:
-            tf.compat.v1.summary.image('cropped_resized_image',
-                             tf.expand_dims(distorted_image, 0))
-
-        # Randomly flip the image horizontally.
-        distorted_image = tf.image.random_flip_left_right(distorted_image)
-        # Randomly distort the colors. There are 1 or 4 ways to do it.
-        num_distort_cases = 1 if fast_mode else 4
-        distorted_image = apply_with_random_selector(
-            distorted_image,
-            lambda x, ordering: distort_color(x, ordering, fast_mode),
-            num_cases=num_distort_cases)
-
-        if add_image_summaries:
-            tf.compat.v1.summary.image('final_distorted_image',
-                             tf.expand_dims(distorted_image, 0))
-        distorted_image = tf.subtract(distorted_image, 0.5)
-        distorted_image = tf.multiply(distorted_image, 2.0)
-        return distorted_image
-
-
-def distort_color(image, batch_position=0, distort_color_in_yiq=False,
-                  scope=None):
-    """Distort the color of the image.
-  
-    Each color distortion is non-commutative and thus ordering of the color ops
-    matters. Ideally we would randomly permute the ordering of the color ops.
-    Rather then adding that level of complication, we select a distinct ordering
-    of color ops based on the position of the image in a batch.
-  
-    Args:
-      image: float32 Tensor containing single image. Tensor values should be in
-        range [0, 1].
-      batch_position: the position of the image in a batch. NOTE: this argument
-        can be an integer or a tensor
-      distort_color_in_yiq: distort color of input images in YIQ space.
-      scope: Optional scope for op_scope.
-    Returns:
-      color-distorted image
-    """
-    with tf.compat.v1.name_scope(scope or 'distort_color'):
-        def distort_fn_0(image=image):
-            """Variant 0 of distort function."""
-            image = tf.image.random_brightness(image, max_delta=32. / 255.)
-            # if distort_color_in_yiq:
-            #  image = distort_image_ops.random_hsv_in_yiq(
-            #      image, lower_saturation=0.5, upper_saturation=1.5,
-            #      max_delta_hue=0.2 * math.pi)
-            # else:
-            image = tf.image.random_saturation(image, lower=0.5, upper=1.5)
-            image = tf.image.random_hue(image, max_delta=0.2)
-            image = tf.image.random_contrast(image, lower=0.5, upper=1.5)
-            return image
-
-        def distort_fn_1(image=image):
-            """Variant 1 of distort function."""
-            image = tf.image.random_brightness(image, max_delta=32. / 255.)
-            image = tf.image.random_contrast(image, lower=0.5, upper=1.5)
-            # if distort_color_in_yiq:
-            #  image = distort_image_ops.random_hsv_in_yiq(
-            #      image, lower_saturation=0.5, upper_saturation=1.5,
-            #      max_delta_hue=0.2 * math.pi)
-            # else:
-            image = tf.image.random_saturation(image, lower=0.5, upper=1.5)
-            image = tf.image.random_hue(image, max_delta=0.2)
-            return image
-
-        image = utils.smart_cond(batch_position % 2 == 0, distort_fn_0,
-                                 distort_fn_1)
-        # The random_* ops do not necessarily clamp.
-        image = tf.clip_by_value(image, 0.0, 1.0)
-        return image
-
-
-class RecordInputImagePreprocessor(object):
-    """Preprocessor for images with RecordInput format."""
-
-    def __init__(self,
-                 height,
-                 width,
-                 batch_size,
-                 num_splits,
-                 dtype,
-                 train,
-                 distortions=False,
-                 resize_method="bilinear",
-                 shift_ratio=0,
-                 summary_verbosity=1,
-                 distort_color_in_yiq=False,
-                 fuse_decode_and_crop=False):
-        self.height = height
-        self.width = width
-        self.batch_size = batch_size
-        self.num_splits = num_splits
-        self.dtype = dtype
-        self.train = train
-        self.resize_method = resize_method
-        self.shift_ratio = shift_ratio
-        self.distortions = distortions
-        self.distort_color_in_yiq = distort_color_in_yiq
-        self.fuse_decode_and_crop = fuse_decode_and_crop
-        if self.batch_size % self.num_splits != 0:
-            raise ValueError(
-                ('batch_size must be a multiple of num_splits: '
-                 'batch_size %d, num_splits: %d') %
-                (self.batch_size, self.num_splits))
-        self.batch_size_per_split = self.batch_size // self.num_splits
-        self.summary_verbosity = summary_verbosity
-
-    def image_preprocess(self, image_buffer, bbox, batch_position):
-        """Preprocessing image_buffer as a function of its batch position."""
-        if self.train:
-            image_buffer = tf.image.decode_jpeg(
-                image_buffer, channels=3, dct_method='INTEGER_FAST')
-            image = preprocess_for_train(image_buffer, self.height, self.width,
-                                         bbox,
-                                         batch_position)
-        else:
-            image = tf.image.decode_jpeg(
-                image_buffer, channels=3, dct_method='INTEGER_FAST')
-            image = preprocess_for_eval(image, self.height, self.width)
-        return image
-
-    def parse_and_preprocess(self, value, batch_position):
-        image_buffer, label_index, bbox, _ = parse_example_proto(value)
-        image = self.image_preprocess(image_buffer, bbox, batch_position)
-        return (label_index, image)
-
-    def minibatch(self, dataset, subset, use_datasets, cache_data,
-                  shift_ratio=-1):
-        if shift_ratio < 0:
-            shift_ratio = self.shift_ratio
-        with tf.compat.v1.name_scope('batch_processing'):
-            # Build final results per split.
-            images = [[] for _ in range(self.num_splits)]
-            labels = [[] for _ in range(self.num_splits)]
-            if use_datasets:
-                glob_pattern = dataset.tf_record_pattern(subset)
-                file_names = gfile.Glob(glob_pattern)
-                if not file_names:
-                    raise ValueError(
-                        'Found no files in --data_dir matching: {}'
-                        .format(glob_pattern))
-                ds = tf.data.TFRecordDataset.list_files(file_names)
-                ds = ds.apply(
-                    #interleave_ops.parallel_interleave(
-                    parallel_interleave( #
-                        tf.data.TFRecordDataset, cycle_length=10))
-                if cache_data:
-                    ds = ds.take(1).cache().repeat()
-                counter = tf.data.Dataset.range(self.batch_size)
-                counter = counter.repeat()
-                ds = tf.data.Dataset.zip((ds, counter))
-                ds = ds.prefetch(buffer_size=self.batch_size)
-                ds = ds.shuffle(buffer_size=10000)
-                ds = ds.repeat()
-                ds = ds.apply(
-                    #batching.map_and_batch(
-                    map_and_batch( ###
-                        map_func=self.parse_and_preprocess,
-                        batch_size=self.batch_size_per_split,
-                        num_parallel_batches=self.num_splits))
-                ds = ds.prefetch(buffer_size=self.num_splits)
-                ds_iterator = tf.compat.v1.data.make_one_shot_iterator(ds)
-                for d in xrange(self.num_splits):
-                    labels[d], images[d] = ds_iterator.get_next()
-
-            else:
-                record_input = data_flow_ops.RecordInput(
-                    file_pattern=dataset.tf_record_pattern(subset),
-                    seed=301,
-                    parallelism=64,
-                    buffer_size=10000,
-                    batch_size=self.batch_size,
-                    shift_ratio=shift_ratio,
-                    name='record_input')
-                records = record_input.get_yield_op()
-                records = tf.split(records, self.batch_size, 0)
-                records = [tf.reshape(record, []) for record in records]
-                for idx in xrange(self.batch_size):
-                    value = records[idx]
-                    (label, image) = self.parse_and_preprocess(value, idx)
-                    split_index = idx % self.num_splits
-                    labels[split_index].append(label)
-                    images[split_index].append(image)
-
-            for split_index in xrange(self.num_splits):
-                if not use_datasets:
-                    images[split_index] = tf.parallel_stack(
-                        images[split_index])
-                    labels[split_index] = tf.concat(labels[split_index], 0)
-                images[split_index] = tf.cast(images[split_index], self.dtype)
-                depth = 3
-                images[split_index] = tf.reshape(
-                    images[split_index],
-                    shape=[self.batch_size_per_split, self.height, self.width,
-                           depth])
-                labels[split_index] = tf.reshape(labels[split_index],
-                                                 [self.batch_size_per_split])
-            return images, labels
diff --git a/models/image_segmentation/tensorflow/maskrcnn/inference/gpu/EnableInference.patch b/models/image_segmentation/tensorflow/maskrcnn/inference/gpu/EnableInference.patch
index 7c95b6625..b2364a955 100644
--- a/models/image_segmentation/tensorflow/maskrcnn/inference/gpu/EnableInference.patch
+++ b/models/image_segmentation/tensorflow/maskrcnn/inference/gpu/EnableInference.patch
@@ -32,6 +32,21 @@ index 53b59d6e..859396a6 100644
  
      # setup dataset
      dataset = Dataset(params)
+diff --git a/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/ops/roi_ops.py b/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/ops/roi_ops.py
+index 8670753e..250198b7 100644
+--- a/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/ops/roi_ops.py
++++ b/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/ops/roi_ops.py
+@@ -96,8 +96,8 @@ def _propose_rois_gpu(scores,
+         pre_nms_boxes = tf.cast(pre_nms_boxes, dtype=tf.float32)
+         pre_nms_scores = tf.cast(pre_nms_scores, dtype=tf.float32)
+
+-        with tf.device('CPU:0'):
+-            boxes, scores, _, _ = tf.image.combined_non_max_suppression(
++        # with tf.device('CPU:0'):
++        boxes, scores, _, _ = tf.image.combined_non_max_suppression(
+                 pre_nms_boxes,
+                 pre_nms_scores,
+                 max_output_size_per_class=topk_limit,
 diff --git a/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/runtime/callbacks.py b/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/runtime/callbacks.py
 index 4e7b56c9..8ffbcdc5 100644
 --- a/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/runtime/callbacks.py
@@ -39,7 +54,7 @@ index 4e7b56c9..8ffbcdc5 100644
 @@ -23,7 +23,7 @@ class DLLoggerMetricsCallback(KerasCallback):
      Keras callback that saves metrics using DLLogger.
      """
- 
+
 -    def __init__(self, dllogger, log_every=10, log_learning_rate=False):
 +    def __init__(self, log_every=10, log_learning_rate=False):
          """
@@ -53,21 +68,21 @@ index 4e7b56c9..8ffbcdc5 100644
 +        # self._dllogger = dllogger
          self._log_every = log_every
          self._log_learning_rate = log_learning_rate
- 
+
 @@ -68,7 +68,8 @@ class DLLoggerMetricsCallback(KerasCallback):
          if not logs:
              return
- 
+
 -        self._dllogger.log(step=step, data=logs)
 +        # self._dllogger.log(step=step, data=logs)
 +        print(step, {k: v for k, v in logs.items()}, flush=True)
- 
- 
+
+
  class DLLoggerPerfCallback(KerasCallback):
 @@ -76,9 +77,9 @@ class DLLoggerPerfCallback(KerasCallback):
      Keras callback that measures performance and logs it using DLLogger.
      """
- 
+
 -    def __init__(self, dllogger, batch_sizes, warmup_steps=0, log_every=None):
 +    def __init__(self, batch_sizes, warmup_steps=0, log_every=None):
          super().__init__()
@@ -77,7 +92,7 @@ index 4e7b56c9..8ffbcdc5 100644
          self._warmup_steps = warmup_steps
          self._log_every = log_every
 @@ -124,6 +125,17 @@ class DLLoggerPerfCallback(KerasCallback):
- 
+
      def _log_perf(self, deltas, mode, step=tuple()):
          deltas = np.array(deltas)
 +        data={
@@ -99,7 +114,7 @@ index 4e7b56c9..8ffbcdc5 100644
              }
          )
 +        '''
- 
+
      @staticmethod
      def _calculate_throughput(deltas, batch_size):
 diff --git a/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/runtime/run.py b/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/runtime/run.py
@@ -108,17 +123,17 @@ index d7b001b3..2fecc57c 100644
 +++ b/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/runtime/run.py
 @@ -2,7 +2,7 @@ import logging
  import os
- 
+
  import tensorflow as tf
 -import dllogger
 +# import dllogger
- 
+
  from mrcnn_tf2.model.mask_rcnn import MaskRCNN
  from mrcnn_tf2.runtime.callbacks import DLLoggerMetricsCallback, DLLoggerPerfCallback, PretrainedWeightsLoadingCallback
 @@ -80,10 +80,14 @@ def run_evaluation(dataset, params):
          include_mask=params.include_mask
      )
- 
+
 +    '''
      dllogger.log(
          step=tuple(),
@@ -127,32 +142,32 @@ index d7b001b3..2fecc57c 100644
 +    '''
 +
 +    print(tuple(), {k: float(v) for k, v in eval_results.items()}, flush=True)
- 
- 
+
+
  def run_inference(dataset, params):
 @@ -112,8 +116,8 @@ def setup(params):
          logging.info('XLA is activated')
- 
+
      if params.amp:
 -        policy = tf.keras.mixed_precision.experimental.Policy("mixed_float16", loss_scale="dynamic")
 -        tf.keras.mixed_precision.experimental.set_policy(policy)
 +        policy = tf.keras.mixed_precision.Policy("mixed_float16")
 +        tf.keras.mixed_precision.set_global_policy(policy)
          logging.info('AMP is activated')
- 
- 
+
+
 @@ -143,12 +147,12 @@ def create_model(params):
- 
+
  def create_callbacks(params):
      yield DLLoggerMetricsCallback(
 -        dllogger=dllogger,
 +        # dllogger=dllogger,
          log_every=params.log_every
      )
- 
+
      yield DLLoggerPerfCallback(
 -        dllogger=dllogger,
 +        # dllogger=dllogger,
          batch_sizes={
              'train': params.train_batch_size * getattr(params, 'replicas', 1),
-             'test': params.eval_batch_size * getattr(params, 'replicas', 1),
+             'test': params.eval_batch_size * getattr(params, 'replicas', 1),
\ No newline at end of file
diff --git a/models/image_segmentation/tensorflow/maskrcnn/inference/gpu/README.md b/models/image_segmentation/tensorflow/maskrcnn/inference/gpu/README.md
new file mode 100644
index 000000000..ad5f65164
--- /dev/null
+++ b/models/image_segmentation/tensorflow/maskrcnn/inference/gpu/README.md
@@ -0,0 +1,59 @@
+# Mask RCNN Inference
+
+Mask RCNN Inference BKC.
+
+## Model Information
+
+| **Use Case** | **Framework** | **Model Repo** | **Branch/Commit/Tag** | **Optional Patch** 
+| :---: | :---: | :---: | :---: |:----------:|
+|   training   |  Tensorflow   | [DeepLearningExamples/MaskRCNN](https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow2/Segmentation/MaskRCNN) |        5be8a3cae21ee2d80e3935a4746827cb3367bcac         |  [EnableInference.patch](#EnableInference.patch)       |
+
+# Pre-Requisite
+* Host has Intel® Data Center GPU MAX or FLEX
+* Host has installed latest Intel® Data Center GPU Max & Flex Series Drivers https://dgpu-docs.intel.com/driver/installation.html
+
+# Dataset 
+Download & preprocess COCO 2017 dataset. 
+```
+export DATASET_DIR=/path/to/dataset/dir
+git clone https://github.com/NVIDIA/DeepLearningExamples.git
+cd ./DeepLearningExamples/TensorFlow2/Segmentation/MaskRCNN/dataset
+bash download_and_preprocess_coco.sh $DATASET_DIR
+```
+
+## Inference
+1. `git clone https://github.com/IntelAI/models.git`
+2. `cd models/models/image_segmentation/tensorflow/maskrcnn/training/gpu`
+3. Run `setup.sh` this will install all the required dependencies & create virtual environment `venv`.
+4. Download weights
+    ```
+        pushd .
+        cd ./DeepLearningExamples/TensorFlow2/Segmentation/MaskRCNN
+        python scripts/download_weights.py --save_dir=./weights
+        popd 
+    ```
+5. Setup required environment paramaters
+
+    |   **Parameter**    |                   **export command**                   |
+    |:------------------------------------------------------:| :--- |
+    |  **DATASET_DIR**   |       `export DATASET_DIR=/the/path/to/dataset`        |
+    |   **PRETRAINED_DIR**   | `export PRETRAINED_DIR=/the/path/to/pretrained_model/` |
+    |   **PRECISION**   |         `export PRECISION=fp16` (fp16 or fp32)         |
+    |   **BATCH_SIZE** (optional)  |                 `export BATCH_SIZE=16`                 |
+6. Run `run_model.sh`
+
+## Output
+
+Output will typicall looks like:
+```
+2023-09-15 12:45:12.198281: I tensorflow/core/framework/local_rendezvous.cc:421] Local rendezvous recv item cancelled. Key hash: 15253968942006736824
+() {'predict_throughput': 32.896633477338334, 'predict_latency': 0.4863719569062287, 'predict_latency_90': 0.48666301561606345, 'predict_latency_95': 0.48671875026262756, 'predict_latency_99': 0.48682774246035293, 'predict_time': 166.06602716445923}
+```
+
+Final results of the training run can be found in `results.yaml` file.
+```
+results:
+ - key: throughput
+   value: 32.896633477338334
+   unit: images/sec
+```
diff --git a/models/image_segmentation/tensorflow/maskrcnn/inference/gpu/requirements.txt b/models/image_segmentation/tensorflow/maskrcnn/inference/gpu/requirements.txt
new file mode 100644
index 000000000..dae9fe407
--- /dev/null
+++ b/models/image_segmentation/tensorflow/maskrcnn/inference/gpu/requirements.txt
@@ -0,0 +1 @@
+opencv-python-headless
\ No newline at end of file
diff --git a/models/image_segmentation/tensorflow/maskrcnn/inference/gpu/run_model.sh b/models/image_segmentation/tensorflow/maskrcnn/inference/gpu/run_model.sh
new file mode 100644
index 000000000..b40a77f4b
--- /dev/null
+++ b/models/image_segmentation/tensorflow/maskrcnn/inference/gpu/run_model.sh
@@ -0,0 +1,79 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2019 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# SPDX-License-Identifier: EPL-2.0
+#
+
+#!/bin/bash
+
+# Create an array of input directories that are expected and then verify that they exist
+declare -A input_envs
+input_envs[DATASET_DIR]=${DATASET_DIR}
+input_envs[PRETRAINED_DIR]=${PRETRAINED_DIR}
+input_envs[PRECISION]=${PRECISION}
+
+for i in "${!input_envs[@]}"; do
+  var_name=$i
+  env_param=${input_envs[$i]}
+ 
+  if [[ -z $env_param ]]; then
+    echo "The required environment variable $var_name is not set" >&2
+    exit 1
+  fi
+done
+
+if [ -d ${DATASET_DIR} ];then
+  echo "DATASET_DIR is: "${DATASET_DIR}
+else
+  echo "Error: the path of dataset does not exist!"
+  exit 1
+fi
+
+BATCH_SIZE=${BATCH_SIZE:-16}
+
+if [ ${PRECISION} == "fp16" ];then
+  AMP=" --amp "
+else
+  AMP=""
+fi
+
+echo 'Running with parameters:'
+echo " DATASET_PATH: ${DATASET_DIR}"
+echo " PRETRAINED_DIR: ${PRETRAINED_DIR}"
+echo " PRECISION: ${PRECISION}"
+echo " BATCH_SIZE: $BATCH_SIZE"
+
+cd ./DeepLearningExamples/TensorFlow2/Segmentation/MaskRCNN
+
+python scripts/inference.py \
+  --data_dir=$DATASET_DIR \
+  --batch_size=$BATCH_SIZE \
+  --no_xla \
+  --weights_dir=$PRETRAINED_DIR $AMP |& tee maskrcnn_inference_${PRECISION}_BS${BATCH_SIZE}.log
+
+throughput=$(cat maskrcnn_inference_${PRECISION}_BS${BATCH_SIZE}.log | grep Throughput | awk -F ' ' '{print $5}')
+yaml_content=$(cat <<EOF
+results:
+ - key: throughput
+   value: $throughput
+   unit: records/sec
+EOF
+)
+
+# Write the content to a YAML file
+echo "$yaml_content" >  ./results.yaml
+echo "YAML file created."
diff --git a/models/image_segmentation/tensorflow/maskrcnn/inference/gpu/setup.sh b/models/image_segmentation/tensorflow/maskrcnn/inference/gpu/setup.sh
new file mode 100644
index 000000000..a0ac1ee91
--- /dev/null
+++ b/models/image_segmentation/tensorflow/maskrcnn/inference/gpu/setup.sh
@@ -0,0 +1,33 @@
+#!/usr/bin/env bash
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# setup.sh
+#  - install OS pkgs
+#  - should create virtual env & install pip  requirement.txt
+#  - git clones & applying patches
+
+set -e
+apt-get update && apt-get install -y python3-venv protobuf-compiler
+python3 -m venv $PWD/venv
+. ./venv/bin/activate
+pip install -r requirements.txt
+
+git clone https://github.com/NVIDIA/DeepLearningExamples.git
+cd DeepLearningExamples
+git checkout 5be8a3cae21ee2d80e3935a4746827cb3367bca
+git apply ../EnableInference.patch
+cd -
diff --git a/models/language_modeling/pytorch/bert_large/inference/gpu/requirements.txt b/models/language_modeling/pytorch/bert_large/inference/gpu/requirements.txt
index c8b0afcef..33d46be5c 100644
--- a/models/language_modeling/pytorch/bert_large/inference/gpu/requirements.txt
+++ b/models/language_modeling/pytorch/bert_large/inference/gpu/requirements.txt
@@ -1,2 +1,2 @@
 tensorboardX
-transformers==4.30.0 # fixed for Snyk scans to avoid vulnerability
+transformers==4.36.0 # fixed for Snyk scans to avoid vulnerability
diff --git a/models/language_modeling/pytorch/bert_large/training/run_pretrain_mlperf.py b/models/language_modeling/pytorch/bert_large/training/run_pretrain_mlperf.py
index 48f8a6224..141aa5c3a 100644
--- a/models/language_modeling/pytorch/bert_large/training/run_pretrain_mlperf.py
+++ b/models/language_modeling/pytorch/bert_large/training/run_pretrain_mlperf.py
@@ -16,7 +16,6 @@
 """
 Fine-tuning the library models for masked language modeling (BERT, ALBERT, RoBERTa...)
 on a text file or a dataset without using HuggingFace Trainer.
-
 Here is the full list of checkpoints on the hub that can be fine-tuned by this script:
 https://huggingface.co/models?filter=masked-lm
 """
@@ -86,7 +85,6 @@
 
 except ImportError as e:
     oneccl_bindings_for_pytorch = False
-import intel_extension_for_pytorch as ipex
 
 
 logger = logging.getLogger(__name__)
@@ -396,6 +394,8 @@ def parse_args():
                         help="Total batch size for training.")
 
     parser.add_argument("--profile", action="store_true", help="Whether to enable profiling")
+    parser.add_argument("--ipex", action="store_true", default=False)
+    parser.add_argument("--inductor", action="store_true", default=False)
 
     args = parser.parse_args()
 
@@ -589,6 +589,9 @@ def calc_accuracy(outputs, masked_lm_labels, next_sentence_label, args):
 
 def main():
     args = parse_args()
+    if args.ipex:
+        print('Using ipex')
+        import intel_extension_for_pytorch as ipex
     status = 'aborted'  # later set to 'success' if termination criteria met
     device, args = setup_training(args)
     total_batch_size = global_batch_size(args) 
@@ -606,13 +609,21 @@ def main():
     model, optimizer, lr_scheduler, checkpoint, global_step = prepare_model_and_optimizer(args, device)
     model.train()
     if args.bf32:
-        ipex.set_fp32_math_mode(mode=ipex.FP32MathMode.BF32, device="cpu")
-        model, optimizer = ipex.optimize(model, dtype=torch.float32, optimizer=optimizer, auto_kernel_selection=True)
+        if args.ipex:
+            ipex.set_fp32_math_mode(mode=ipex.FP32MathMode.BF32, device="cpu")
+            model, optimizer = ipex.optimize(model, dtype=torch.float32, optimizer=optimizer, auto_kernel_selection=True)
     elif args.fp16:
-        scaler = torch.cpu.amp.GradScaler()
-        model, optimizer = ipex.optimize(model, optimizer=optimizer, dtype=torch.half, auto_kernel_selection=True, weights_prepack=True, fuse_update_step=False)
+        if args.ipex:
+            scaler = torch.cpu.amp.GradScaler()
+            model, optimizer = ipex.optimize(model, optimizer=optimizer, dtype=torch.half, auto_kernel_selection=True, weights_prepack=True, fuse_update_step=False)
     else:
-        model, optimizer = ipex.optimize(model, optimizer=optimizer, dtype=torch.bfloat16 if args.bf16 else torch.float32)
+        if args.ipex:
+            model, optimizer = ipex.optimize(model, optimizer=optimizer, dtype=torch.bfloat16 if args.bf16 else torch.float32)
+    with torch.cpu.amp.autocast(enabled=args.bf16):
+        if args.inductor:
+            print('[Info] Running training steps torch.compile() with default backend')
+            model = torch.compile(model)
+
     worker_seeds, shuffling_seeds = utils.setup_seeds(args.seed, args.num_epochs_to_generate_seeds_for, device)
     worker_seed = worker_seeds[args.local_rank]
 
diff --git a/models/language_modeling/pytorch/common/enable_ipex_for_transformers.diff b/models/language_modeling/pytorch/common/enable_ipex_for_transformers.diff
index 8befaad5f..f833e48d3 100644
--- a/models/language_modeling/pytorch/common/enable_ipex_for_transformers.diff
+++ b/models/language_modeling/pytorch/common/enable_ipex_for_transformers.diff
@@ -1,5 +1,5 @@
 diff --git a/examples/legacy/question-answering/run_squad.py b/examples/legacy/question-answering/run_squad.py
-index fc9411e95..ce73205cd 100644
+index fc9411e95..2579f42b8 100644
 --- a/examples/legacy/question-answering/run_squad.py
 +++ b/examples/legacy/question-answering/run_squad.py
 @@ -22,6 +22,9 @@ import logging
@@ -12,12 +12,11 @@ index fc9411e95..ce73205cd 100644
  
  import numpy as np
  import torch
-@@ -48,7 +51,11 @@ from transformers.data.metrics.squad_metrics import (
+@@ -48,7 +51,10 @@ from transformers.data.metrics.squad_metrics import (
  from transformers.data.processors.squad import SquadResult, SquadV1Processor, SquadV2Processor
  from transformers.trainer_utils import is_main_process
  
 -
-+import intel_extension_for_pytorch as ipex
 +def trace_handler(prof):
 +    print(prof.key_averages().table(
 +        sort_by="self_cpu_time_total", row_limit=-1))
@@ -25,19 +24,29 @@ index fc9411e95..ce73205cd 100644
  try:
      from torch.utils.tensorboard import SummaryWriter
  except ImportError:
-@@ -265,6 +272,136 @@ def train(args, train_dataset, model, tokenizer):
+@@ -265,6 +271,205 @@ def train(args, train_dataset, model, tokenizer):
  
      return global_step, tr_loss / global_step
  
-+def wrap_model(model, args):
++def wrap_model(model, args, eval_dataloader):
++    for it, batch in enumerate(eval_dataloader):
++        break
++    torch_compile_inputs = {
++        "input_ids": batch[0],
++        "attention_mask": batch[1],
++        "token_type_ids": batch[2],
++    }
 +    model.eval()
-+    ipex.nn.utils._model_convert.replace_dropout_with_identity(model)
++    if args.ipex:
++        ipex.nn.utils._model_convert.replace_dropout_with_identity(model)
++    else:
++        torch._C._jit_set_texpr_fuser_enabled(False)
 +    dumpy_tensor = torch.ones((args.eval_batch_size, 384), dtype=torch.long) \
 +                    if not args.use_multi_stream_module \
 +                    else torch.ones((args.eval_batch_size//args.num_streams, 384), dtype=torch.long)
 +    jit_inputs = (dumpy_tensor, dumpy_tensor, dumpy_tensor)
 +    print(args)
-+    if args.int8:
++    if args.int8 and args.ipex:
 +        from intel_extension_for_pytorch.quantization import prepare, convert
 +        from torch.ao.quantization import MinMaxObserver, PerChannelMinMaxObserver, QConfig
 +        qconfig = QConfig(activation=MinMaxObserver.with_args(qscheme=torch.per_tensor_affine, dtype=torch.quint8), weight=PerChannelMinMaxObserver.with_args(dtype=torch.qint8, qscheme=torch.per_channel_symmetric))
@@ -48,7 +57,7 @@ index fc9411e95..ce73205cd 100644
 +        if args.int8_fp32:
 +            model = convert(prepared_model)
 +            model = torch.jit.trace(model, jit_inputs, strict=False)
-+        else: #int8_bf16 mix
++        elif args.int8_bf16:
 +            with torch.cpu.amp.autocast():
 +                model = convert(prepared_model)
 +                model = torch.jit.trace(model, jit_inputs, strict=False)
@@ -68,17 +77,24 @@ index fc9411e95..ce73205cd 100644
 +            #y = model(dumpy_tensor, dumpy_tensor, dumpy_tensor)
 +            #print("############################################")
 +    elif args.bf16:
-+        model = ipex.optimize(model, dtype=torch.bfloat16)
++        if args.ipex:
++            model = ipex.optimize(model, dtype=torch.bfloat16)
++        else:
++            torch._C._jit_set_autocast_mode(False)
++            model = model.to(torch.bfloat16)
 +        with torch.cpu.amp.autocast(),torch.no_grad():
-+            model = torch.jit.trace(model, jit_inputs, strict=False)
-+            model = torch.jit.freeze(model)
-+            #model = torch.jit._recursive.wrap_cpp_module(torch._C._freeze_module(model._c, preserveParameters=True))
-+            print(model.graph)
++            if args.use_jit:
++                model = torch.jit.trace(model, jit_inputs, strict=False)
++                model = torch.jit.freeze(model)
++                #model = torch.jit._recursive.wrap_cpp_module(torch._C._freeze_module(model._c, preserveParameters=True))
++                print(model.graph)
 +    elif args.fp16_cpu:
-+        model = ipex.optimize(model, dtype=torch.half, conv_bn_folding=False, auto_kernel_selection=True, weights_prepack=True)
++        if args.ipex:
++            model = ipex.optimize(model, dtype=torch.half, conv_bn_folding=False, auto_kernel_selection=True, weights_prepack=True)
 +        with torch.cpu.amp.autocast(enabled=True, dtype=torch.half):
-+            model = torch.jit.trace(model, jit_inputs, strict=False)
-+            model = torch.jit.freeze(model)
++            if args.use_jit:
++                model = torch.jit.trace(model, jit_inputs, strict=False)
++                model = torch.jit.freeze(model)
 +    elif args.bf32:
 +        ipex.set_fp32_math_mode(mode=ipex.FP32MathMode.BF32, device="cpu")
 +        model = ipex.optimize(model, dtype=torch.float32, level="O1", auto_kernel_selection=True)
@@ -90,6 +106,58 @@ index fc9411e95..ce73205cd 100644
 +            model = torch.jit.trace(model, jit_inputs, strict=False)
 +            #model = torch.jit._recursive.wrap_cpp_module(torch._C._freeze_module(model._c, preserveParameters=True)) 
 +            model = torch.jit.freeze(model)
++    # torch.compile() path
++    if args.inductor:
++        from torch._inductor import config as inductor_config
++        inductor_config.cpp_wrapper = True
++        if args.int8:
++            from torch.ao.quantization.quantize_pt2e import prepare_pt2e, convert_pt2e
++            import torch.ao.quantization.quantizer.x86_inductor_quantizer as xiq
++            from torch.ao.quantization.quantizer.x86_inductor_quantizer import X86InductorQuantizer
++            from torch._export import capture_pre_autograd_graph, dynamic_dim
++            print('[Info] Running torch.compile() for INT8 quantization')
++            print('[Info] Running int8_bf16 is: {}'.format(args.int8_bf16))
++            with torch.no_grad():
++                constraints = None
++                if args.eval_batch_size != 1:
++                    constraints = [
++                        dynamic_dim(torch_compile_inputs["input_ids"], 0),
++                        dynamic_dim(torch_compile_inputs["attention_mask"], 0),
++                        dynamic_dim(torch_compile_inputs["token_type_ids"], 0),
++                        dynamic_dim(torch_compile_inputs["input_ids"], 0) == dynamic_dim(torch_compile_inputs["attention_mask"], 0),
++                        dynamic_dim(torch_compile_inputs["input_ids"], 0) == dynamic_dim(torch_compile_inputs["token_type_ids"], 0),
++                    ]                    
++                exported_model = capture_pre_autograd_graph(
++                    model,
++                    (),
++                    kwargs=torch_compile_inputs,
++                    constraints=constraints,
++                )
++                quantizer = X86InductorQuantizer()
++                quantizer.set_global(xiq.get_default_x86_inductor_quantization_config())
++                prepared_model = prepare_pt2e(exported_model, quantizer)
++                prepared_model(**torch_compile_inputs)
++                converted_model = convert_pt2e(prepared_model)
++                torch.ao.quantization.move_exported_model_to_eval(converted_model)
++                with torch.cpu.amp.autocast(enabled=args.int8_bf16):
++                    if args.ipex:
++                        model = torch.compile(converted_model, backend="ipex")
++                    else:
++                        model = torch.compile(converted_model, dynamic=(args.eval_batch_size != 1))
++                    model(**torch_compile_inputs)
++                    model(**torch_compile_inputs)
++        else:
++            with torch.no_grad(), torch.cpu.amp.autocast(enabled=args.bf16 or args.fp16_cpu, dtype=torch.half if args.fp16_cpu else torch.bfloat16):
++                if args.ipex:
++                    print('[Info] Running torch.compile() with IPEX backend')
++                    model = torch.compile(model, backend="ipex")
++                else:
++                    print('[Info] Running torch.compile() with default backend')
++                    model = torch.compile(model)
++                # warmup run before threading to compile the model
++                model(**torch_compile_inputs)
++                model(**torch_compile_inputs)
++
 +    if args.use_multi_stream_module:
 +        print("Use multi stream module on numa node:{0}, num of streams:{1}, batch per stream:{2}".format(args.instance_number, args.num_streams, args.eval_batch_size//args.num_streams))
 +        input_hint_object = {"input_ids": 0, "attention_mask": 0, "token_type_ids": 0}
@@ -134,7 +202,7 @@ index fc9411e95..ce73205cd 100644
 +                    print('P99 Latency {:.2f} ms'.format(p99*1000))
 +                    print("Throughput: {:.3f} sentence/s".format(throughput))     
 +                    break
-+                with torch.no_grad():
++                with torch.no_grad(), torch.cpu.amp.autocast(enabled=args.bf16 or args.int8_bf16 or args.fp16_cpu, dtype=torch.half if args.fp16_cpu else torch.bfloat16):
 +                    inputs = {
 +                        "input_ids": batch[0],
 +                        "attention_mask": batch[1],
@@ -162,15 +230,16 @@ index fc9411e95..ce73205cd 100644
  
  def evaluate(args, model, tokenizer, prefix=""):
      dataset, examples, features = load_and_cache_examples(args, tokenizer, evaluate=True, output_examples=True)
-@@ -274,6 +411,7 @@ def evaluate(args, model, tokenizer, prefix=""):
- 
-     args.eval_batch_size = args.per_gpu_eval_batch_size * max(1, args.n_gpu)
- 
-+    model = wrap_model(model, args)
-     # Note that DistributedSampler samples randomly
+@@ -278,6 +483,8 @@ def evaluate(args, model, tokenizer, prefix=""):
      eval_sampler = SequentialSampler(dataset)
      eval_dataloader = DataLoader(dataset, sampler=eval_sampler, batch_size=args.eval_batch_size)
-@@ -286,6 +424,40 @@ def evaluate(args, model, tokenizer, prefix=""):
+ 
++    model = wrap_model(model, args, eval_dataloader)
++
+     # multi-gpu evaluate
+     if args.n_gpu > 1 and not isinstance(model, torch.nn.DataParallel):
+         model = torch.nn.DataParallel(model)
+@@ -286,12 +493,47 @@ def evaluate(args, model, tokenizer, prefix=""):
      logger.info("***** Running evaluation {} *****".format(prefix))
      logger.info("  Num examples = %d", len(dataset))
      logger.info("  Batch size = %d", args.eval_batch_size)
@@ -211,7 +280,26 @@ index fc9411e95..ce73205cd 100644
  
      all_results = []
      start_time = timeit.default_timer()
-@@ -320,7 +492,7 @@ def evaluate(args, model, tokenizer, prefix=""):
+ 
+     for batch in tqdm(eval_dataloader, desc="Evaluating"):
+-        model.eval()
++        if not (args.inductor and args.int8):
++            model.eval()
+         batch = tuple(t.to(args.device) for t in batch)
+ 
+         with torch.no_grad():
+@@ -314,13 +556,17 @@ def evaluate(args, model, tokenizer, prefix=""):
+                     inputs.update(
+                         {"langs": (torch.ones(batch[0].shape, dtype=torch.int64) * args.lang_id).to(args.device)}
+                     )
+-            outputs = model(**inputs)
++            if args.inductor:
++                with torch.cpu.amp.autocast(enabled=args.bf16 or args.int8_bf16 or args.fp16_cpu, dtype=torch.half if args.fp16_cpu else torch.bfloat16):
++                    outputs = model(**inputs)
++            else:
++                outputs = model(**inputs)
+ 
+         for i, feature_index in enumerate(feature_indices):
              eval_feature = features[feature_index.item()]
              unique_id = int(eval_feature.unique_id)
  
@@ -220,7 +308,7 @@ index fc9411e95..ce73205cd 100644
  
              # Some models (XLNet, XLM) use 5 arguments for their predictions, while the other "simpler"
              # models only use two.
-@@ -671,6 +843,47 @@ def main():
+@@ -671,7 +917,57 @@ def main():
      parser.add_argument("--server_port", type=str, default="", help="Can be used for distant debugging.")
  
      parser.add_argument("--threads", type=int, default=1, help="multiple threads for converting example to features")
@@ -246,6 +334,8 @@ index fc9411e95..ce73205cd 100644
 +                        help='use llga int8 in pytorch jit model')
 +    parser.add_argument('--int8_fp32', dest='int8_fp32', action='store_true',
 +                        help='use int8 fp32 mix precision')
++    parser.add_argument('--int8_bf16', dest='int8_bf16', action='store_true',
++                        help='use int8 bf16 mix precision')
 +    parser.add_argument("--int8_config", type=str, default="config.json", 
 +                        help="quantization config file for int8 mode")
 +    parser.add_argument("--do_calibration", action='store_true',
@@ -264,11 +354,19 @@ index fc9411e95..ce73205cd 100644
 +                        help="The number of stream to use, used for multi stream module")
 +    parser.add_argument("--instance_number", type=int, default=0,
 +                        help="The socket to run multi stream module, used for multi stream module with multi sockets")
++    parser.add_argument("--ipex", action='store_true', default=False,
++                        help='use intel pytorch extension')
++    parser.add_argument('--inductor', action='store_true', default=False,
++                        help='using torch.compile() inductor backend')
 +
      args = parser.parse_args()
++    if args.ipex:
++        import intel_extension_for_pytorch as ipex
++        global ipex
  
      if args.doc_stride >= args.max_seq_length - args.max_query_length:
-@@ -742,11 +955,13 @@ def main():
+         logger.warning(
+@@ -742,11 +1038,13 @@ def main():
      args.model_type = args.model_type.lower()
      config = AutoConfig.from_pretrained(
          args.config_name if args.config_name else args.model_name_or_path,
@@ -282,7 +380,7 @@ index fc9411e95..ce73205cd 100644
          cache_dir=args.cache_dir if args.cache_dir else None,
          use_fast=False,  # SquadDataset is not compatible with Fast tokenizers which have a smarter overflow handeling
      )
-@@ -824,7 +1039,7 @@ def main():
+@@ -824,7 +1122,7 @@ def main():
          for checkpoint in checkpoints:
              # Reload the model
              global_step = checkpoint.split("-")[-1] if len(checkpoints) > 1 else ""
@@ -364,7 +462,7 @@ index 587dc2e59..b4e331e28 100644
  class GELUActivation(nn.Module):
      """
 diff --git a/src/transformers/generation/utils.py b/src/transformers/generation/utils.py
-index ae12ae293..1fff54a49 100644
+index ae12ae293..041fc2f23 100644
 --- a/src/transformers/generation/utils.py
 +++ b/src/transformers/generation/utils.py
 @@ -16,6 +16,8 @@
@@ -748,20 +846,33 @@ index ae12ae293..1fff54a49 100644
                      sequences=sequence_outputs["sequences"],
                      sequences_scores=sequence_outputs["sequence_scores"],
                      scores=scores,
-@@ -2931,7 +3139,12 @@ class GenerationMixin:
+@@ -2931,7 +3139,9 @@ class GenerationMixin:
                      hidden_states=decoder_hidden_states,
                  )
          else:
 -            return sequence_outputs["sequences"]
 +            output_result = sequence_outputs["sequences"]
 +        # result
-+        if self.token_latency is not None:
-+            return (output_result, latency_list)
-+        else:
-+            return output_result
++        return (output_result, latency_list)
  
      def beam_sample(
          self,
+diff --git a/src/transformers/modeling_utils.py b/src/transformers/modeling_utils.py
+index 0df8d7e25..99d107a81 100644
+--- a/src/transformers/modeling_utils.py
++++ b/src/transformers/modeling_utils.py
+@@ -1831,7 +1831,10 @@ class PreTrainedModel(nn.Module, ModuleUtilsMixin, GenerationMixin, PushToHubMix
+                 and is_main_process
+                 and reg.fullmatch(filename_no_suffix) is not None
+             ):
+-                os.remove(full_filename)
++                try:
++                    os.remove(full_filename)
++                except OSError as e:
++                    print(e)
+ 
+         # Save the model
+         for shard_file, shard in shards.items():
 diff --git a/src/transformers/models/bert/modeling_bert.py b/src/transformers/models/bert/modeling_bert.py
 index ee7dcd5e4..5ead9c989 100755
 --- a/src/transformers/models/bert/modeling_bert.py
@@ -810,10 +921,26 @@ index ee7dcd5e4..5ead9c989 100755
              total_loss = masked_lm_loss + next_sentence_loss
  
 diff --git a/src/transformers/models/distilbert/modeling_distilbert.py b/src/transformers/models/distilbert/modeling_distilbert.py
-index 84db89e0f..3c4138e33 100755
+index 84db89e0f..cb986d7ea 100755
 --- a/src/transformers/models/distilbert/modeling_distilbert.py
 +++ b/src/transformers/models/distilbert/modeling_distilbert.py
-@@ -743,11 +743,11 @@ class DistilBertForSequenceClassification(DistilBertPreTrainedModel):
+@@ -216,12 +216,11 @@ class MultiHeadSelfAttention(nn.Module):
+         k = shape(self.k_lin(key))  # (bs, n_heads, k_length, dim_per_head)
+         v = shape(self.v_lin(value))  # (bs, n_heads, k_length, dim_per_head)
+ 
+-        q = q / math.sqrt(dim_per_head)  # (bs, n_heads, q_length, dim_per_head)
++
+         scores = torch.matmul(q, k.transpose(2, 3))  # (bs, n_heads, q_length, k_length)
++        scores = scores / math.sqrt(dim_per_head)  # (bs, n_heads, q_length, dim_per_head)
+         mask = (mask == 0).view(mask_reshp).expand_as(scores)  # (bs, n_heads, q_length, k_length)
+-        scores = scores.masked_fill(
+-            mask, torch.tensor(torch.finfo(scores.dtype).min)
+-        )  # (bs, n_heads, q_length, k_length)
++        scores = scores.masked_fill(mask, -float("inf"))  # (bs, n_heads, q_length, k_length)
+ 
+         weights = nn.functional.softmax(scores, dim=-1)  # (bs, n_heads, q_length, k_length)
+         weights = self.dropout(weights)  # (bs, n_heads, q_length, k_length)
+@@ -743,11 +742,11 @@ class DistilBertForSequenceClassification(DistilBertPreTrainedModel):
      )
      def forward(
          self,
@@ -899,7 +1026,7 @@ index 474b92f72..98aab97f0 100644
          output_hidden_states: Optional[bool] = None,
          interpolate_pos_encoding: Optional[bool] = None,
 diff --git a/src/transformers/trainer.py b/src/transformers/trainer.py
-index cf71499b0..8f61f9619 100755
+index cf71499b0..b681d82de 100755
 --- a/src/transformers/trainer.py
 +++ b/src/transformers/trainer.py
 @@ -158,6 +158,9 @@ from .utils import (
@@ -948,12 +1075,28 @@ index cf71499b0..8f61f9619 100755
              else:
                  if not is_apex_available():
                      raise ImportError(
-@@ -1329,9 +1332,38 @@ class Trainer:
+@@ -695,7 +698,7 @@ class Trainer:
+         self._memory_tracker.stop_and_update_metrics()
+ 
+         # torch.compile
+-        if args.torch_compile and not is_torch_compile_available():
++        if args.inductor and not is_torch_compile_available():
+             raise RuntimeError("Using torch.compile requires PyTorch 2.0 or higher.")
+ 
+     def add_callback(self, callback):
+@@ -1323,15 +1326,45 @@ class Trainer:
+         return model
+ 
+     def torch_jit_model_eval(self, model, dataloader, training=False):
++        print("[INFO] torch_jit_model_eval")
+         if not training:
+             if dataloader is None:
+                 logger.warning("failed to use PyTorch jit mode due to current dataloader is none.")
                  return model
              example_batch = next(iter(dataloader))
              example_batch = self._prepare_inputs(example_batch)
 +            int8_inputs=[]
-+            if self.args.int8 or self.args.do_calibration:
++            if (self.args.int8 or self.args.do_calibration) and self.args.use_ipex:
 +                import intel_extension_for_pytorch as ipex 
 +                from intel_extension_for_pytorch.quantization import prepare, convert
 +                from torch.ao.quantization import MinMaxObserver, PerChannelMinMaxObserver, QConfig
@@ -980,14 +1123,22 @@ index cf71499b0..8f61f9619 100755
 +                    model = prepared_model
                  jit_model = model.eval()
                  with ContextManagers([self.autocast_smart_context_manager(cache_enabled=False), torch.no_grad()]):
-+                    if self.args.int8:
++                    if self.args.int8 and self.args.use_ipex:
 +                        jit_model = convert(jit_model)
 +                        if self.args.smooth_quant:
 +                            jit_model(*int8_inputs)
                      if version.parse(version.parse(torch.__version__).base_version) >= version.parse("1.14.0"):
                          if isinstance(example_batch, dict):
                              jit_model = torch.jit.trace(jit_model, example_kwarg_inputs=example_batch, strict=False)
-@@ -1372,22 +1404,39 @@ class Trainer:
+@@ -1361,6 +1394,7 @@ class Trainer:
+         return model
+ 
+     def ipex_optimize_model(self, model, training=False, dtype=torch.float32):
++        print("[INFO] ipex_optimize_model")
+         if not is_ipex_available():
+             raise ImportError(
+                 "Using IPEX but IPEX is not installed or IPEX's version does not match current PyTorch, please refer"
+@@ -1372,22 +1406,39 @@ class Trainer:
          if not training:
              model.eval()
              dtype = torch.bfloat16 if not self.is_in_train and self.args.bf16_full_eval else dtype
@@ -1019,23 +1170,104 @@ index cf71499b0..8f61f9619 100755
      def _wrap_model(self, model, training=True, dataloader=None):
 -        if self.args.use_ipex:
 -            dtype = torch.bfloat16 if self.use_cpu_amp else torch.float32
--            model = self.ipex_optimize_model(model, training, dtype=dtype)
--
 +        if self.args.use_ipex and not self.args.int8 and not self.args.do_calibration:
++            dtype=torch.float32
++            if self.args.bf16:
++                dtype=torch.bfloat16
++            elif self.args.fp16_cpu:
++                dtype=torch.float16
++            elif self.args.bf32:
 +                dtype=torch.float32
-+                if self.args.bf16:
-+                    dtype=torch.bfloat16
-+                elif self.args.fp16_cpu:
-+                    dtype=torch.float16
-+                elif self.args.bf32:
-+                    dtype=torch.float32
-+                model = self.ipex_optimize_model(model, training, dtype=dtype)
-+        if self.args.int8 or self.args.do_calibration:
+             model = self.ipex_optimize_model(model, training, dtype=dtype)
+-
++        if (self.args.int8 or self.args.do_calibration) and self.args.use_ipex and not self.args.inductor:
 +            self.args.jit_mode_eval = True
          if is_sagemaker_mp_enabled():
              # Wrapping the base model twice in a DistributedModel will raise an error.
              if isinstance(self.model_wrapped, smp.model.DistributedModel):
-@@ -1993,7 +2042,11 @@ class Trainer:
+@@ -1415,6 +1466,69 @@ class Trainer:
+             model = self.torch_jit_model_eval(model, dataloader, training)
+             self.jit_compilation_time = round(time.time() - start_time, 4)
+ 
++        if self.args.inductor:
++            from torch._inductor import config as inductor_config
++            inductor_config.cpp_wrapper = True
++            example_batch = next(iter(dataloader))
++            example_batch = self._prepare_inputs(example_batch)
++            if self.args.int8:
++                from torch.ao.quantization.quantize_pt2e import prepare_pt2e, convert_pt2e
++                import torch.ao.quantization.quantizer.x86_inductor_quantizer as xiq
++                from torch.ao.quantization.quantizer.x86_inductor_quantizer import X86InductorQuantizer
++                from torch._export import capture_pre_autograd_graph, dynamic_dim
++                print('[Info] Running torch.compile() INT8 quantization')
++                with torch.no_grad():
++                    exported_model = capture_pre_autograd_graph(
++                        model,
++                        (),
++                        kwargs=example_batch,
++                    )
++                    quantizer = X86InductorQuantizer()
++                    quantizer.set_global(xiq.get_default_x86_inductor_quantization_config())
++                    prepared_model = prepare_pt2e(exported_model, quantizer)
++                    prepared_model(**example_batch)
++                    converted_model = convert_pt2e(prepared_model)
++                    torch.ao.quantization.move_exported_model_to_eval(converted_model)
++                    with torch.cpu.amp.autocast(enabled=(self.args.bf16 or self.args.int8_bf16)):
++                        if self.args.use_ipex:
++                            print('[Info] Running torch.compile() with IPEX backend')
++                            model = torch.compile(converted_model, backend="ipex")
++                        else:
++                            print('[Info] Running torch.compile() with default backend')
++                            model = torch.compile(converted_model)
++                        y = model(**example_batch)
++                        y = model(**example_batch)
++            elif self.args.bf16:
++                with torch.no_grad(), torch.cpu.amp.autocast(dtype=torch.bfloat16):
++                    if self.args.use_ipex:
++                        print('[Info] Running torch.compile() Bfloat16 with IPEX backend')
++                        model = torch.compile(model, backend="ipex")
++                    else:
++                        print('[Info] Running torch.compile() Bfloat16 with default backend')
++                        model = torch.compile(model)
++                    y = model(**example_batch)
++                    y = model(**example_batch)
++            elif self.args.fp16_cpu:
++                with torch.no_grad(), torch.cpu.amp.autocast(dtype=torch.half):
++                    if self.args.use_ipex:
++                        print('[Info] Running torch.compile() float16 with IPEX backend')
++                        model = torch.compile(model, backend="ipex")
++                    else:
++                        print('[Info] Running torch.compile() float16 with default backend')
++                        model = torch.compile(model)
++                    y = model(**example_batch)
++                    y = model(**example_batch)
++            else:
++                with torch.no_grad():
++                    if self.args.use_ipex:
++                        print('[Info] Running torch.compile() Float32 with IPEX backend')
++                        model = torch.compile(model, backend="ipex")
++                    else:
++                        print('[Info] Running torch.compile() Float32 with default backend')
++                        model = torch.compile(model)
++                    y = model(**example_batch)
++                    y = model(**example_batch)
++
+         # Note: in torch.distributed mode, there's no point in wrapping the model
+         # inside a DistributedDataParallel as we'll be under `no_grad` anyways.
+         if not training:
+@@ -1575,8 +1689,9 @@ class Trainer:
+ 
+         # torch.compile() needs to be called after wrapping the model with FSDP or DDP
+         # to ensure that it accounts for the graph breaks required by those wrappers
+-        if self.args.torch_compile:
+-            model = torch.compile(model, backend=self.args.torch_compile_backend, mode=self.args.torch_compile_mode)
++        if self.args.inductor:
++            with torch.cpu.amp.autocast(enabled=self.args.bf16 or self.args.fp16_cpu, dtype=torch.half if self.args.fp16_cpu else torch.bfloat16):
++                model = torch.compile(model, backend=self.args.torch_compile_backend, mode=self.args.torch_compile_mode)
+ 
+         return model
+ 
+@@ -1993,7 +2108,11 @@ class Trainer:
                          scale_after = self.scaler.get_scale()
                          optimizer_was_run = scale_before <= scale_after
                      else:
@@ -1048,7 +1280,7 @@ index cf71499b0..8f61f9619 100755
  
                      if optimizer_was_run and not self.deepspeed:
                          self.lr_scheduler.step()
-@@ -2714,7 +2767,10 @@ class Trainer:
+@@ -2714,7 +2833,10 @@ class Trainer:
              # loss gets scaled under gradient_accumulation_steps in deepspeed
              loss = self.deepspeed.backward(loss)
          else:
@@ -1060,7 +1292,7 @@ index cf71499b0..8f61f9619 100755
  
          return loss.detach()
  
-@@ -3085,6 +3141,47 @@ class Trainer:
+@@ -3085,6 +3207,54 @@ class Trainer:
          self._memory_tracker.stop_and_update_metrics(output.metrics)
  
          return PredictionOutput(predictions=output.predictions, label_ids=output.label_ids, metrics=output.metrics)
@@ -1098,17 +1330,38 @@ index cf71499b0..8f61f9619 100755
 +                        print("#############################")
 +                        break
 +                    with torch.no_grad():
-+                        start = time.time()
-+                        outputs = model(**batch)
-+                        #prof.step()
-+                        end = time.time()
++                        if (self.args.bf16 or self.args.int8_bf16 or self.args.fp16_cpu) and self.args.inductor:
++                            with torch.cpu.amp.autocast(dtype=torch.half if self.args.fp16_cpu else torch.bfloat16):
++                                start = time.time()
++                                outputs = model(**batch)
++                                #prof.step()
++                                end = time.time()
++                        else:
++                            start = time.time()
++                            outputs = model(**batch)
++                            #prof.step()
++                            end = time.time()
 +                        if epoch * steps_per_epoch + it > self.args.perf_begin_iter:
 +                            timeBuff.append(end-start)
 +                        pbar.update(1)
  
      def evaluation_loop(
          self,
-@@ -3158,6 +3255,20 @@ class Trainer:
+@@ -3133,7 +3303,12 @@ class Trainer:
+             logger.info("  Num examples: Unknown")
+         logger.info(f"  Batch size = {batch_size}")
+ 
+-        model.eval()
++        try:
++            model.eval()
++        except:
++            # Model after PT2 Quantization has not eval method
++            # https://github.com/pytorch/pytorch/blob/b3ad29e269aa940c6f6947627b039191856b895e/torch/ao/quantization/quantize_pt2e.py#L110
++            pass
+ 
+         self.callback_handler.eval_dataloader = dataloader
+         # Do this before wrapping.
+@@ -3158,6 +3333,20 @@ class Trainer:
          all_labels = None
          all_inputs = None
          # Will be useful when we have an iterable dataset so don't know its length.
@@ -1130,10 +1383,10 @@ index cf71499b0..8f61f9619 100755
          observed_num_examples = 0
          # Main evaluation loop
 diff --git a/src/transformers/training_args.py b/src/transformers/training_args.py
-index 088eb06b7..9051cbcb8 100644
+index 088eb06b7..2e0067671 100644
 --- a/src/transformers/training_args.py
 +++ b/src/transformers/training_args.py
-@@ -777,10 +777,109 @@ class TrainingArguments:
+@@ -777,10 +777,121 @@ class TrainingArguments:
              )
          },
      )
@@ -1163,6 +1416,14 @@ index 088eb06b7..9051cbcb8 100644
 +            )
 +        },
 +    )
++    int8_bf16: bool = field(
++        default=False,
++        metadata={
++            "help": (
++                "Whether to use int8_bf16 (mixed) precision instead of 32-bit"
++            )
++        },
++    )
 +    use_share_weight: bool = field(
 +        default=False,
 +        metadata={
@@ -1239,11 +1500,15 @@ index 088eb06b7..9051cbcb8 100644
 +    fp16_cpu: bool = field(
 +        default=False,
 +        metadata={"help": "Whether to use fp16 (mixed) precision instead of 32-bit on cpu using IPEX"},
++    )
++    inductor: bool = field(
++        default=False,
++        metadata={"help": "Whether to use torch.compile() inductor backend"},
 +    )
      fp16_opt_level: str = field(
          default="O1",
          metadata={
-@@ -1253,17 +1352,17 @@ class TrainingArguments:
+@@ -1253,17 +1364,17 @@ class TrainingArguments:
              if version.parse(version.parse(torch.__version__).base_version) == version.parse("2.0.0") and self.fp16:
                  raise ValueError("--optim adamw_torch_fused with --fp16 requires PyTorch>2.0")
  
diff --git a/models/language_modeling/tensorflow/distilbert_base/inference/requirements.txt b/models/language_modeling/tensorflow/distilbert_base/inference/requirements.txt
index c052b5606..5732cbeda 100644
--- a/models/language_modeling/tensorflow/distilbert_base/inference/requirements.txt
+++ b/models/language_modeling/tensorflow/distilbert_base/inference/requirements.txt
@@ -4,4 +4,4 @@ scikit-learn>=1.1.2
 scipy>=1.9.3
 sklearn==0.0
 tokenizers==0.13.1
-transformers==4.30.0 # fixed for Snyk scans to avoid vulnerability
+transformers==4.36.0 # fixed for Snyk scans to avoid vulnerability
diff --git a/models/language_translation/tensorflow/mlperf_gnmt/fp32/__init__.py b/models/language_translation/tensorflow/mlperf_gnmt/fp32/__init__.py
deleted file mode 100644
index e69de29bb..000000000
diff --git a/models/language_translation/tensorflow/mlperf_gnmt/fp32/bleu.py b/models/language_translation/tensorflow/mlperf_gnmt/fp32/bleu.py
deleted file mode 100644
index 25aa2f93e..000000000
--- a/models/language_translation/tensorflow/mlperf_gnmt/fp32/bleu.py
+++ /dev/null
@@ -1,172 +0,0 @@
-# Copyright 2017 Google Inc. All Rights Reserved.
-# Modifications copyright (C) 2019 MLPerf Authors.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-
-"""Python implementation of BLEU, smooth-BLEU and Running BLEU.
-
-@note The most common usage case is to invoke the function compute_bleu
-
-This module provides a Python implementation of BLEU and smooth-BLEU.
-Smooth BLEU is computed following the method outlined in the paper:
-Chin-Yew Lin, Franz Josef Och. ORANGE: a method for evaluating automatic
-evaluation metrics for machine translation. COLING 2004.
-"""
-
-import collections
-import math
-
-##
-# @brief Class to compute running BLEU scores
-# @detail BLEU scores can be computed in a non-linear way,
-# or without having access to the full translated corpus in time.
-class RunningBLEUScorer:
-
-  def __init__(self, max_order=4, smooth=False):
-    self.max_order = max_order
-    self.smooth = smooth
-    self.reset()
-
-  ##
-  # @brief Reset all variables (none of the previus sentences will be taken into account)
-  def reset(self):
-    self.matches_by_order = [0] * self.max_order
-    self.possible_matches_by_order = [0] * self.max_order
-    self.reference_length = 0
-    self.translation_length = 0
-
-  ##
-  # @brief Add a single sentence
-  # @param reference list of words for a reference sentence
-  # @param translation list of words for its corresponding translated sentence
-  # @post Updates internal structures to take this sentence's translation
-  # result into account in final BLEU score
-  def add_sentence(self, reference, translation):
-    self.add_sentence_with_multiple_refs([reference], translation)
-
-  ##
-  # @brief Add a single reference, with potentially multiple references
-  # @param reference list of list of words for a reference sentence
-  # @note That we could have multiple sentences serving as a reference
-  # @param translation (single) list of words for its corresponding translated sentence
-  # @post Updates internal structures to take this sentence's translation
-  # result into account in final BLEU score
-  def add_sentence_with_multiple_refs(self, references, translation):
-    self.reference_length += min(len(r) for r in references)
-    self.translation_length += len(translation)
-
-    merged_ref_ngram_counts = collections.Counter()
-    for reference in references:
-      merged_ref_ngram_counts |= self._get_ngrams(reference)
-
-    translation_ngram_counts = self._get_ngrams(translation)
-
-    new_matches_by_order, new_possible_matches_by_order = self._get_ngram_match_values(merged_ref_ngram_counts, translation_ngram_counts, len(translation))
-
-    for i in range(self.max_order):
-      self.matches_by_order[i] += new_matches_by_order[i]
-      self.possible_matches_by_order[i] += new_possible_matches_by_order[i]
-
-  ##
-  # @brief Calculate final BLEU score
-  def calc_BLEU_score(self):
-    precisions = [0] * self.max_order
-    for i in range(0, self.max_order):
-      if self.smooth:
-        precisions[i] = ((self.matches_by_order[i] + 1.) /
-                         (self.possible_matches_by_order[i] + 1.))
-      else:
-        if self.possible_matches_by_order[i] > 0:
-          precisions[i] = (float(self.matches_by_order[i]) /
-                           self.possible_matches_by_order[i])
-        else:
-          precisions[i] = 0.0
-
-    if min(precisions) > 0:
-      p_log_sum = sum((1. / self.max_order) * math.log(p) for p in precisions)
-      geo_mean = math.exp(p_log_sum)
-    else:
-      geo_mean = 0
-
-    ratio = float(self.translation_length) / self.reference_length
-
-    if ratio > 1.0:
-      bp = 1.
-    else:
-      bp = math.exp(1 - 1. / ratio)
-
-    bleu = geo_mean * bp
-
-    return (bleu, precisions, bp, ratio, self.translation_length, self.reference_length)
-
-  ##
-  # @brief Internal function to compute matching percentages for different order ngrams
-  def _get_ngram_match_values(self, ref_ngram_counts, translation_ngram_counts, translation_length):
-    new_matches_by_order = [0] * self.max_order
-    new_possible_matches_by_order = [0] * self.max_order
-
-    overlap = translation_ngram_counts & ref_ngram_counts
-    for ngram in overlap:
-      new_matches_by_order[len(ngram)-1] += overlap[ngram]
-    for order in range(1, self.max_order+1):
-      possible_matches = translation_length - order + 1
-      new_possible_matches_by_order[order-1] = max(0, possible_matches)
-
-    return (new_matches_by_order, new_possible_matches_by_order)
-
-  def _get_ngrams(self, segment):
-    """Internal function to extract all n-grams upto a given maximum order from an input segment.
-
-    Args:
-      segment: text segment from which n-grams will be extracted.
-
-    Returns:
-      The Counter containing all n-grams upto max_order in segment
-      with a count of how many times each n-gram occurred.
-    """
-    ngram_counts = collections.Counter()
-    for order in range(1, self.max_order + 1):
-      for i in range(0, len(segment) - order + 1):
-        ngram = tuple(segment[i:i+order])
-        ngram_counts[ngram] += 1
-    return ngram_counts
-
-def compute_bleu(reference_corpus, translation_corpus, max_order=4,
-                 smooth=False):
-  """Computes BLEU score of translated segments against one or more references.
-    This is the most common usage when calculating BLEU scores.
-
-  Args:
-    reference_corpus: list of lists of references for each translation. Each
-        reference should be tokenized into a list of tokens.
-        reference_corpus[i][j][k] represents the k'th word of the i'th sentence
-        for the j'th reference text
-    translation_corpus: list of translated sentences to score. Each sentence
-        should be tokenized into a list of tokens.
-        translation_corpus[i][j] represents the j'th word for the i'th sentence
-    max_order: Maximum n-gram order to use when computing BLEU score.
-    smooth: Whether or not to apply Lin et al. 2004 smoothing.
-
-  Returns:
-    3-Tuple with the BLEU score, n-gram precisions, geometric mean of n-gram
-    precisions and brevity penalty.
-  """
-  runningBLEU = RunningBLEUScorer(max_order=max_order, smooth=smooth)
-
-
-  for (references, translation) in zip(reference_corpus,
-                                       translation_corpus):
-    runningBLEU.add_sentence_with_multiple_refs(references, translation)
-
-  return runningBLEU.calc_BLEU_score()
\ No newline at end of file
diff --git a/models/language_translation/tensorflow/mlperf_gnmt/fp32/evaluation_utils.py b/models/language_translation/tensorflow/mlperf_gnmt/fp32/evaluation_utils.py
deleted file mode 100644
index e5eb5fd93..000000000
--- a/models/language_translation/tensorflow/mlperf_gnmt/fp32/evaluation_utils.py
+++ /dev/null
@@ -1,183 +0,0 @@
-# Copyright 2017 Google Inc. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-
-"""Utility for evaluating various tasks, e.g., translation & summarization."""
-import codecs
-import os
-import re
-import subprocess
-import shlex
-
-import tensorflow as tf
-
-import bleu
-import rouge
-
-
-__all__ = ["evaluate"]
-
-
-def evaluate(ref_file, trans_file, metric, subword_option=None):
-  """Pick a metric and evaluate depending on task."""
-  # BLEU scores for translation task
-  if metric.lower() == "bleu":
-    evaluation_score = _bleu(ref_file, trans_file,
-                             subword_option=subword_option)
-  # ROUGE scores for summarization tasks
-  elif metric.lower() == "rouge":
-    evaluation_score = _rouge(ref_file, trans_file,
-                              subword_option=subword_option)
-  elif metric.lower() == "accuracy":
-    evaluation_score = _accuracy(ref_file, trans_file)
-  elif metric.lower() == "word_accuracy":
-    evaluation_score = _word_accuracy(ref_file, trans_file)
-  else:
-    raise ValueError("Unknown metric %s" % metric)
-
-  return evaluation_score
-
-
-def _clean(sentence, subword_option):
-  """Clean and handle BPE or SPM outputs."""
-  sentence = sentence.strip()
-
-  # BPE
-  if subword_option == "bpe":
-    sentence = re.sub("@@ ", "", sentence)
-
-  # SPM
-  elif subword_option == "spm":
-    sentence = u"".join(sentence.split()).replace(u"\u2581", u" ").lstrip()
-
-  return sentence
-
-
-# Follow //transconsole/localization/machine_translation/metrics/bleu_calc.py
-def _bleu(ref_file, trans_file, subword_option=None):
-  """Compute BLEU scores and handling BPE."""
-  max_order = 4
-  smooth = False
-
-  ref_files = [ref_file]
-  reference_text = []
-  for reference_filename in ref_files:
-    with codecs.getreader("utf-8")(
-        tf.io.gfile.GFile(reference_filename, "rb")) as fh:
-      reference_text.append(fh.readlines())
-
-  per_segment_references = []
-  for references in zip(*reference_text):
-    reference_list = []
-    for reference in references:
-      reference = _clean(reference, subword_option)
-      reference_list.append(reference.split(" "))
-    per_segment_references.append(reference_list)
-
-  translations = []
-  with codecs.getreader("utf-8")(tf.io.gfile.GFile(trans_file, "rb")) as fh:
-    for line in fh:
-      line = _clean(line, subword_option=None)
-      translations.append(line.split(" "))
-
-  # bleu_score, precisions, bp, ratio, translation_length, reference_length
-  bleu_score, _, _, _, _, _ = bleu.compute_bleu(
-      per_segment_references, translations, max_order, smooth)
-  return 100 * bleu_score
-
-
-def _rouge(ref_file, summarization_file, subword_option=None):
-  """Compute ROUGE scores and handling BPE."""
-
-  references = []
-  with codecs.getreader("utf-8")(tf.io.gfile.GFile(ref_file, "rb")) as fh:
-    for line in fh:
-      references.append(_clean(line, subword_option))
-
-  hypotheses = []
-  with codecs.getreader("utf-8")(
-      tf.io.gfile.GFile(summarization_file, "rb")) as fh:
-    for line in fh:
-      hypotheses.append(_clean(line, subword_option=None))
-
-  rouge_score_map = rouge.rouge(hypotheses, references)
-  return 100 * rouge_score_map["rouge_l/f_score"]
-
-
-def _accuracy(label_file, pred_file):
-  """Compute accuracy, each line contains a label."""
-
-  with codecs.getreader("utf-8")(tf.io.gfile.GFile(label_file, "rb")) as label_fh:
-    with codecs.getreader("utf-8")(tf.io.gfile.GFile(pred_file, "rb")) as pred_fh:
-      count = 0.0
-      match = 0.0
-      for label in label_fh:
-        label = label.strip()
-        pred = pred_fh.readline().strip()
-        if label == pred:
-          match += 1
-        count += 1
-  return 100 * match / count
-
-
-def _word_accuracy(label_file, pred_file):
-  """Compute accuracy on per word basis."""
-
-  with codecs.getreader("utf-8")(tf.io.gfile.GFile(label_file, "r")) as label_fh:
-    with codecs.getreader("utf-8")(tf.io.gfile.GFile(pred_file, "r")) as pred_fh:
-      total_acc, total_count = 0., 0.
-      for sentence in label_fh:
-        labels = sentence.strip().split(" ")
-        preds = pred_fh.readline().strip().split(" ")
-        match = 0.0
-        for pos in range(min(len(labels), len(preds))):
-          label = labels[pos]
-          pred = preds[pos]
-          if label == pred:
-            match += 1
-        total_acc += 100 * match / max(len(labels), len(preds))
-        total_count += 1
-  return total_acc / total_count
-
-
-def _moses_bleu(multi_bleu_script, tgt_test, trans_file, subword_option=None):
-  """Compute BLEU scores using Moses multi-bleu.perl script."""
-
-  # TODO(thangluong): perform rewrite using python
-  # BPE
-  if subword_option == "bpe":
-    debpe_tgt_test = tgt_test + ".debpe"
-    if not os.path.exists(debpe_tgt_test):
-      subprocess.call(shlex.split("cp %s %s" % (tgt_test, debpe_tgt_test)), shell=False)
-      subprocess.call(shlex.split("sed s/@@ //g %s" % (debpe_tgt_test)),
-                      shell=False)
-    tgt_test = debpe_tgt_test
-  elif subword_option == "spm":
-    despm_tgt_test = tgt_test + ".despm"
-    if not os.path.exists(despm_tgt_test):
-      subprocess.call("cp %s %s" % (tgt_test, despm_tgt_test))
-      subprocess.call("sed s/ //g %s" % (despm_tgt_test))
-      subprocess.call(u"sed s/^\u2581/g %s" % (despm_tgt_test))
-      subprocess.call(u"sed s/\u2581/ /g %s" % (despm_tgt_test))
-    tgt_test = despm_tgt_test
-  cmd = "%s %s < %s" % (multi_bleu_script, tgt_test, trans_file)
-
-  # subprocess
-  bleu_output = subprocess.check_output(shlex.split(cmd), shell=False)
-
-  # extract BLEU score
-  m = re.search("BLEU = (.+?),", bleu_output)
-  bleu_score = float(m.group(1))
-
-  return bleu_score
diff --git a/models/language_translation/tensorflow/mlperf_gnmt/fp32/hparam.py b/models/language_translation/tensorflow/mlperf_gnmt/fp32/hparam.py
deleted file mode 100644
index e98acfa00..000000000
--- a/models/language_translation/tensorflow/mlperf_gnmt/fp32/hparam.py
+++ /dev/null
@@ -1,536 +0,0 @@
-import json
-import numbers
-import six
-
-from tensorflow.python.util import compat
-
-def _parse_fail(name, var_type, value, values):
-  """Helper function for raising a value error for bad assignment."""
-  raise ValueError(
-      'Could not parse hparam \'%s\' of type \'%s\' with value \'%s\' in %s' %
-      (name, var_type.__name__, value, values))
-
-
-def _reuse_fail(name, values):
-  """Helper function for raising a value error for reuse of name."""
-  raise ValueError('Multiple assignments to variable \'%s\' in %s' % (name,
-                                                                      values))
-
-
-def _process_scalar_value(name, parse_fn, var_type, m_dict, values,
-                          results_dictionary):
-  """Update results_dictionary with a scalar value.
-  Used to update the results_dictionary to be returned by parse_values when
-  encountering a clause with a scalar RHS (e.g.  "s=5" or "arr[0]=5".)
-  Mutates results_dictionary.
-  Args:
-    name: Name of variable in assignment ("s" or "arr").
-    parse_fn: Function for parsing the actual value.
-    var_type: Type of named variable.
-    m_dict: Dictionary constructed from regex parsing.
-      m_dict['val']: RHS value (scalar)
-      m_dict['index']: List index value (or None)
-    values: Full expression being parsed
-    results_dictionary: The dictionary being updated for return by the parsing
-      function.
-  Raises:
-    ValueError: If the name has already been used.
-  """
-  try:
-    parsed_value = parse_fn(m_dict['val'])
-  except ValueError:
-    _parse_fail(name, var_type, m_dict['val'], values)
-
-  # If no index is provided
-  if not m_dict['index']:
-    if name in results_dictionary:
-      _reuse_fail(name, values)
-    results_dictionary[name] = parsed_value
-  else:
-    if name in results_dictionary:
-      # The name has already been used as a scalar, then it
-      # will be in this dictionary and map to a non-dictionary.
-      if not isinstance(results_dictionary.get(name), dict):
-        _reuse_fail(name, values)
-    else:
-      results_dictionary[name] = {}
-
-    index = int(m_dict['index'])
-    # Make sure the index position hasn't already been assigned a value.
-    if index in results_dictionary[name]:
-      _reuse_fail('{}[{}]'.format(name, index), values)
-    results_dictionary[name][index] = parsed_value
-
-
-def _process_list_value(name, parse_fn, var_type, m_dict, values,
-                        results_dictionary):
-  """Update results_dictionary from a list of values.
-  Used to update results_dictionary to be returned by parse_values when
-  encountering a clause with a list RHS (e.g.  "arr=[1,2,3]".)
-  Mutates results_dictionary.
-  Args:
-    name: Name of variable in assignment ("arr").
-    parse_fn: Function for parsing individual values.
-    var_type: Type of named variable.
-    m_dict: Dictionary constructed from regex parsing.
-      m_dict['val']: RHS value (scalar)
-    values: Full expression being parsed
-    results_dictionary: The dictionary being updated for return by the parsing
-      function.
-  Raises:
-    ValueError: If the name has an index or the values cannot be parsed.
-  """
-  if m_dict['index'] is not None:
-    raise ValueError('Assignment of a list to a list index.')
-  elements = filter(None, re.split('[ ,]', m_dict['vals']))
-  # Make sure the name hasn't already been assigned a value
-  if name in results_dictionary:
-    raise _reuse_fail(name, values)
-  try:
-    results_dictionary[name] = [parse_fn(e) for e in elements]
-  except ValueError:
-    _parse_fail(name, var_type, m_dict['vals'], values)
-
-
-def _cast_to_type_if_compatible(name, param_type, value):
-  """Cast hparam to the provided type, if compatible.
-  Args:
-    name: Name of the hparam to be cast.
-    param_type: The type of the hparam.
-    value: The value to be cast, if compatible.
-  Returns:
-    The result of casting `value` to `param_type`.
-  Raises:
-    ValueError: If the type of `value` is not compatible with param_type.
-      * If `param_type` is a string type, but `value` is not.
-      * If `param_type` is a boolean, but `value` is not, or vice versa.
-      * If `param_type` is an integer type, but `value` is not.
-      * If `param_type` is a float type, but `value` is not a numeric type.
-  """
-  fail_msg = (
-      "Could not cast hparam '%s' of type '%s' from value %r" %
-      (name, param_type, value))
-
-  # If `value` is already of type `param_type`, return it directly.
-  # `isinstance` is too weak (e.g. isinstance(True, int) == True).
-  if type(value) == param_type:  # pylint: disable=unidiomatic-typecheck
-    return value
-
-  # Some callers use None, for which we can't do any casting/checking. :(
-  if issubclass(param_type, type(None)):
-    return value
-
-  # Avoid converting a non-string type to a string.
-  if (issubclass(param_type, (six.string_types, six.binary_type)) and
-      not isinstance(value, (six.string_types, six.binary_type))):
-    raise ValueError(fail_msg)
-
-  # Avoid converting a number or string type to a boolean or vice versa.
-  if issubclass(param_type, bool) != isinstance(value, bool):
-    raise ValueError(fail_msg)
-
-  # Avoid converting float to an integer (the reverse is fine).
-  if (issubclass(param_type, numbers.Integral) and
-      not isinstance(value, numbers.Integral)):
-    raise ValueError(fail_msg)
-
-  # Avoid converting a non-numeric type to a numeric type.
-  if (issubclass(param_type, numbers.Number) and
-      not isinstance(value, numbers.Number)):
-    raise ValueError(fail_msg)
-
-  return param_type(value)
-
-
-def parse_values(values, type_map, ignore_unknown=False):
-  """Parses hyperparameter values from a string into a python map.
-  `values` is a string containing comma-separated `name=value` pairs.
-  For each pair, the value of the hyperparameter named `name` is set to
-  `value`.
-  If a hyperparameter name appears multiple times in `values`, a ValueError
-  is raised (e.g. 'a=1,a=2', 'a[1]=1,a[1]=2').
-  If a hyperparameter name in both an index assignment and scalar assignment,
-  a ValueError is raised.  (e.g. 'a=[1,2,3],a[0] = 1').
-  The hyperparameter name may contain '.' symbols, which will result in an
-  attribute name that is only accessible through the getattr and setattr
-  functions.  (And must be first explicit added through add_hparam.)
-  WARNING: Use of '.' in your variable names is allowed, but is not well
-  supported and not recommended.
-  The `value` in `name=value` must follows the syntax according to the
-  type of the parameter:
-  *  Scalar integer: A Python-parsable integer point value.  E.g.: 1,
-     100, -12.
-  *  Scalar float: A Python-parsable floating point value.  E.g.: 1.0,
-     -.54e89.
-  *  Boolean: Either true or false.
-  *  Scalar string: A non-empty sequence of characters, excluding comma,
-     spaces, and square brackets.  E.g.: foo, bar_1.
-  *  List: A comma separated list of scalar values of the parameter type
-     enclosed in square brackets.  E.g.: [1,2,3], [1.0,1e-12], [high,low].
-  When index assignment is used, the corresponding type_map key should be the
-  list name.  E.g. for "arr[1]=0" the type_map must have the key "arr" (not
-  "arr[1]").
-  Args:
-    values: String.  Comma separated list of `name=value` pairs where
-      'value' must follow the syntax described above.
-    type_map: A dictionary mapping hyperparameter names to types.  Note every
-      parameter name in values must be a key in type_map.  The values must
-      conform to the types indicated, where a value V is said to conform to a
-      type T if either V has type T, or V is a list of elements of type T.
-      Hence, for a multidimensional parameter 'x' taking float values,
-      'x=[0.1,0.2]' will parse successfully if type_map['x'] = float.
-    ignore_unknown: Bool. Whether values that are missing a type in type_map
-      should be ignored. If set to True, a ValueError will not be raised for
-      unknown hyperparameter type.
-  Returns:
-    A python map mapping each name to either:
-    * A scalar value.
-    * A list of scalar values.
-    * A dictionary mapping index numbers to scalar values.
-    (e.g. "x=5,L=[1,2],arr[1]=3" results in {'x':5,'L':[1,2],'arr':{1:3}}")
-  Raises:
-    ValueError: If there is a problem with input.
-    * If `values` cannot be parsed.
-    * If a list is assigned to a list index (e.g. 'a[1] = [1,2,3]').
-    * If the same rvalue is assigned two different values (e.g. 'a=1,a=2',
-      'a[1]=1,a[1]=2', or 'a=1,a=[1]')
-  """
-  results_dictionary = {}
-  pos = 0
-  while pos < len(values):
-    m = PARAM_RE.match(values, pos)
-    if not m:
-      raise ValueError('Malformed hyperparameter value: %s' % values[pos:])
-    # Check that there is a comma between parameters and move past it.
-    pos = m.end()
-    # Parse the values.
-    m_dict = m.groupdict()
-    name = m_dict['name']
-    if name not in type_map:
-      if ignore_unknown:
-        continue
-      raise ValueError('Unknown hyperparameter type for %s' % name)
-    type_ = type_map[name]
-
-    # Set up correct parsing function (depending on whether type_ is a bool)
-    if type_ == bool:
-
-      def parse_bool(value):
-        if value in ['true', 'True']:
-          return True
-        elif value in ['false', 'False']:
-          return False
-        else:
-          try:
-            return bool(int(value))
-          except ValueError:
-            _parse_fail(name, type_, value, values)
-
-      parse = parse_bool
-    else:
-      parse = type_
-
-    # If a singe value is provided
-    if m_dict['val'] is not None:
-      _process_scalar_value(name, parse, type_, m_dict, values,
-                            results_dictionary)
-
-    # If the assigned value is a list:
-    elif m_dict['vals'] is not None:
-      _process_list_value(name, parse, type_, m_dict, values,
-                          results_dictionary)
-
-    else:  # Not assigned a list or value
-      _parse_fail(name, type_, '', values)
-
-  return results_dictionary
-
-
-class HParams(object):
-
-  _HAS_DYNAMIC_ATTRIBUTES = True  # Required for pytype checks.
-
-  def __init__(self, hparam_def=None, model_structure=None, **kwargs):
-    self._hparam_types = {}
-    self._model_structure = model_structure
-    if hparam_def:
-      self._init_from_proto(hparam_def)
-      if kwargs:
-        raise ValueError('hparam_def and initialization values are '
-                         'mutually exclusive')
-    else:
-      for name, value in six.iteritems(kwargs):
-        self.add_hparam(name, value)
-
-  def _init_from_proto(self, hparam_def):
-    """Creates a new HParams from `HParamDef` protocol buffer.
-    Args:
-      hparam_def: `HParamDef` protocol buffer.
-    """
-    assert isinstance(hparam_def, hparam_pb2.HParamDef)
-    for name, value in hparam_def.hparam.items():
-      kind = value.WhichOneof('kind')
-      if kind.endswith('_value'):
-        # Single value.
-        if kind.startswith('int64'):
-          # Setting attribute value to be 'int' to ensure the type is compatible
-          # with both Python2 and Python3.
-          self.add_hparam(name, int(getattr(value, kind)))
-        elif kind.startswith('bytes'):
-          # Setting attribute value to be 'str' to ensure the type is compatible
-          # with both Python2 and Python3. UTF-8 encoding is assumed.
-          self.add_hparam(name, compat.as_str(getattr(value, kind)))
-        else:
-          self.add_hparam(name, getattr(value, kind))
-      else:
-        # List of values.
-        if kind.startswith('int64'):
-          # Setting attribute value to be 'int' to ensure the type is compatible
-          # with both Python2 and Python3.
-          self.add_hparam(name, [int(v) for v in getattr(value, kind).value])
-        elif kind.startswith('bytes'):
-          # Setting attribute value to be 'str' to ensure the type is compatible
-          # with both Python2 and Python3. UTF-8 encoding is assumed.
-          self.add_hparam(
-              name, [compat.as_str(v) for v in getattr(value, kind).value])
-        else:
-          self.add_hparam(name, [v for v in getattr(value, kind).value])
-
-  def add_hparam(self, name, value):
-    """Adds {name, value} pair to hyperparameters.
-    Args:
-      name: Name of the hyperparameter.
-      value: Value of the hyperparameter. Can be one of the following types:
-        int, float, string, int list, float list, or string list.
-    Raises:
-      ValueError: if one of the arguments is invalid.
-    """
-    # Keys in kwargs are unique, but 'name' could the name of a pre-existing
-    # attribute of this object.  In that case we refuse to use it as a
-    # hyperparameter name.
-    if getattr(self, name, None) is not None:
-      raise ValueError('Hyperparameter name is reserved: %s' % name)
-    if isinstance(value, (list, tuple)):
-      if not value:
-        raise ValueError(
-            'Multi-valued hyperparameters cannot be empty: %s' % name)
-      self._hparam_types[name] = (type(value[0]), True)
-    else:
-      self._hparam_types[name] = (type(value), False)
-    setattr(self, name, value)
-
-  def set_hparam(self, name, value):
-    """Set the value of an existing hyperparameter.
-    This function verifies that the type of the value matches the type of the
-    existing hyperparameter.
-    Args:
-      name: Name of the hyperparameter.
-      value: New value of the hyperparameter.
-    Raises:
-      KeyError: If the hyperparameter doesn't exist.
-      ValueError: If there is a type mismatch.
-    """
-    param_type, is_list = self._hparam_types[name]
-    if isinstance(value, list):
-      if not is_list:
-        raise ValueError(
-            'Must not pass a list for single-valued parameter: %s' % name)
-      setattr(self, name, [
-          _cast_to_type_if_compatible(name, param_type, v) for v in value])
-    else:
-      if is_list:
-        raise ValueError(
-            'Must pass a list for multi-valued parameter: %s.' % name)
-      setattr(self, name, _cast_to_type_if_compatible(name, param_type, value))
-
-  def del_hparam(self, name):
-    """Removes the hyperparameter with key 'name'.
-    Does nothing if it isn't present.
-    Args:
-      name: Name of the hyperparameter.
-    """
-    if hasattr(self, name):
-      delattr(self, name)
-      del self._hparam_types[name]
-
-  def parse(self, values):
-    """Override existing hyperparameter values, parsing new values from a string.
-    See parse_values for more detail on the allowed format for values.
-    Args:
-      values: String.  Comma separated list of `name=value` pairs where 'value'
-        must follow the syntax described above.
-    Returns:
-      The `HParams` instance.
-    Raises:
-      ValueError: If `values` cannot be parsed or a hyperparameter in `values`
-      doesn't exist.
-    """
-    type_map = {}
-    for name, t in self._hparam_types.items():
-      param_type, _ = t
-      type_map[name] = param_type
-
-    values_map = parse_values(values, type_map)
-    return self.override_from_dict(values_map)
-
-  def override_from_dict(self, values_dict):
-    """Override existing hyperparameter values, parsing new values from a dictionary.
-    Args:
-      values_dict: Dictionary of name:value pairs.
-    Returns:
-      The `HParams` instance.
-    Raises:
-      KeyError: If a hyperparameter in `values_dict` doesn't exist.
-      ValueError: If `values_dict` cannot be parsed.
-    """
-    for name, value in values_dict.items():
-      self.set_hparam(name, value)
-    return self
-
-  def set_model_structure(self, model_structure):
-    self._model_structure = model_structure
-
-  def get_model_structure(self):
-    return self._model_structure
-
-  def to_json(self, indent=None, separators=None, sort_keys=False):
-    """Serializes the hyperparameters into JSON.
-    Args:
-      indent: If a non-negative integer, JSON array elements and object members
-        will be pretty-printed with that indent level. An indent level of 0, or
-        negative, will only insert newlines. `None` (the default) selects the
-        most compact representation.
-      separators: Optional `(item_separator, key_separator)` tuple. Default is
-        `(', ', ': ')`.
-      sort_keys: If `True`, the output dictionaries will be sorted by key.
-    Returns:
-      A JSON string.
-    """
-    return json.dumps(
-        self.values(),
-        indent=indent,
-        separators=separators,
-        sort_keys=sort_keys)
-
-  def parse_json(self, values_json):
-    """Override existing hyperparameter values, parsing new values from a json object.
-    Args:
-      values_json: String containing a json object of name:value pairs.
-    Returns:
-      The `HParams` instance.
-    Raises:
-      KeyError: If a hyperparameter in `values_json` doesn't exist.
-      ValueError: If `values_json` cannot be parsed.
-    """
-    values_map = json.loads(values_json)
-    return self.override_from_dict(values_map)
-
-  def values(self):
-    """Return the hyperparameter values as a Python dictionary.
-    Returns:
-      A dictionary with hyperparameter names as keys.  The values are the
-      hyperparameter values.
-    """
-    return {n: getattr(self, n) for n in self._hparam_types.keys()}
-
-  def get(self, key, default=None):
-    """Returns the value of `key` if it exists, else `default`."""
-    if key in self._hparam_types:
-      # Ensure that default is compatible with the parameter type.
-      if default is not None:
-        param_type, is_param_list = self._hparam_types[key]
-        type_str = 'list<%s>' % param_type if is_param_list else str(param_type)
-        fail_msg = ("Hparam '%s' of type '%s' is incompatible with "
-                    'default=%s' % (key, type_str, default))
-
-        is_default_list = isinstance(default, list)
-        if is_param_list != is_default_list:
-          raise ValueError(fail_msg)
-
-        try:
-          if is_default_list:
-            for value in default:
-              _cast_to_type_if_compatible(key, param_type, value)
-          else:
-            _cast_to_type_if_compatible(key, param_type, default)
-        except ValueError as e:
-          raise ValueError('%s. %s' % (fail_msg, e))
-
-      return getattr(self, key)
-
-    return default
-
-  def __contains__(self, key):
-    return key in self._hparam_types
-
-  def __str__(self):
-    hpdict = self.values()
-    output_list = ['{}={}'.format(key, hpdict[key]) for key in hpdict]
-    return ','.join(output_list)
-
-  def __repr__(self):
-    strval = str(sorted(self.values().items()))
-    return '%s(%s)' % (type(self).__name__, strval)
-
-  @staticmethod
-  def _get_kind_name(param_type, is_list):
-    """Returns the field name given parameter type and is_list.
-    Args:
-      param_type: Data type of the hparam.
-      is_list: Whether this is a list.
-    Returns:
-      A string representation of the field name.
-    Raises:
-      ValueError: If parameter type is not recognized.
-    """
-    if issubclass(param_type, bool):
-      # This check must happen before issubclass(param_type, six.integer_types),
-      # since Python considers bool to be a subclass of int.
-      typename = 'bool'
-    elif issubclass(param_type, six.integer_types):
-      # Setting 'int' and 'long' types to be 'int64' to ensure the type is
-      # compatible with both Python2 and Python3.
-      typename = 'int64'
-    elif issubclass(param_type, (six.string_types, six.binary_type)):
-      # Setting 'string' and 'bytes' types to be 'bytes' to ensure the type is
-      # compatible with both Python2 and Python3.
-      typename = 'bytes'
-    elif issubclass(param_type, float):
-      typename = 'float'
-    else:
-      raise ValueError('Unsupported parameter type: %s' % str(param_type))
-
-    suffix = 'list' if is_list else 'value'
-    return '_'.join([typename, suffix])
-
-  def to_proto(self, export_scope=None):  # pylint: disable=unused-argument
-    """Converts a `HParams` object to a `HParamDef` protocol buffer.
-    Args:
-      export_scope: Optional `string`. Name scope to remove.
-    Returns:
-      A `HParamDef` protocol buffer.
-    """
-    hparam_proto = hparam_pb2.HParamDef()
-    for name in self._hparam_types:
-      # Parse the values.
-      param_type, is_list = self._hparam_types.get(name, (None, None))
-      kind = HParams._get_kind_name(param_type, is_list)
-
-      if is_list:
-        if kind.startswith('bytes'):
-          v_list = [compat.as_bytes(v) for v in getattr(self, name)]
-        else:
-          v_list = [v for v in getattr(self, name)]
-        getattr(hparam_proto.hparam[name], kind).value.extend(v_list)
-      else:
-        v = getattr(self, name)
-        if kind.startswith('bytes'):
-          v = compat.as_bytes(getattr(self, name))
-        setattr(hparam_proto.hparam[name], kind, v)
-
-    return hparam_proto
-
-  @staticmethod
-  def from_proto(hparam_def, import_scope=None):  # pylint: disable=unused-argument
-    return HParams(hparam_def=hparam_def)
-
diff --git a/models/language_translation/tensorflow/mlperf_gnmt/fp32/misc_utils.py b/models/language_translation/tensorflow/mlperf_gnmt/fp32/misc_utils.py
deleted file mode 100644
index 3d33b9b29..000000000
--- a/models/language_translation/tensorflow/mlperf_gnmt/fp32/misc_utils.py
+++ /dev/null
@@ -1,182 +0,0 @@
-# Copyright 2017 Google Inc. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-
-"""Generally useful utility functions."""
-from __future__ import print_function
-
-import codecs
-import collections
-import json
-import math
-import os
-import sys
-import time
-from distutils import version
-
-import numpy as np
-import tensorflow as tf
-
-import hparam
-
-def check_tensorflow_version():
-  min_tf_version = "1.4.0-dev20171024"
-  if (version.LooseVersion(tf.__version__) <
-      version.LooseVersion(min_tf_version)):
-    raise EnvironmentError("Tensorflow version must >= %s" % min_tf_version)
-
-
-def safe_exp(value):
-  """Exponentiation with catching of overflow error."""
-  try:
-    ans = math.exp(value)
-  except OverflowError:
-    ans = float("inf")
-  return ans
-
-
-def print_time(s, start_time):
-  """Take a start time, print elapsed duration, and return a new time."""
-  print("%s, time %ds, %s." % (s, (time.time() - start_time), time.ctime()))
-  sys.stdout.flush()
-  return time.time()
-
-
-def print_out(s, f=None, new_line=True):
-  """Similar to print but with support to flush and output to a file."""
-  if isinstance(s, bytes):
-    s = s.decode("utf-8")
-
-  if f:
-    f.write(s.encode("utf-8"))
-    if new_line:
-      f.write(b"\n")
-
-  # stdout
-  out_s = s.encode("utf-8")
-  if not isinstance(out_s, str):
-    out_s = out_s.decode("utf-8")
-  print(out_s, end="", file=sys.stdout)
-
-  if new_line:
-    sys.stdout.write("\n")
-  sys.stdout.flush()
-
-
-def print_hparams(hparams, skip_patterns=None, header=None):
-  """Print hparams, can skip keys based on pattern."""
-  if header: print_out("%s" % header)
-  values = hparams.values()
-  for key in sorted(values.keys()):
-    if not skip_patterns or all(
-        [skip_pattern not in key for skip_pattern in skip_patterns]):
-      print_out("  %s=%s" % (key, str(values[key])))
-
-
-def load_hparams(model_dir):
-  """Load hparams from an existing model directory."""
-  hparams_file = os.path.join(model_dir, "hparams")
-  if tf.io.gfile.exists(hparams_file):
-    print_out("# Loading hparams from %s" % hparams_file)
-    with codecs.getreader("utf-8")(tf.io.gfile.GFile(hparams_file, "rb")) as f:
-      try:
-        hparams_values = json.load(f)
-        hparams = hparams.HParams(**hparams_values)
-      except ValueError:
-        print_out("  can't load hparams file")
-        return None
-    return hparams
-  else:
-    return None
-
-
-def maybe_parse_standard_hparams(hparams, hparams_path):
-  """Override hparams values with existing standard hparams config."""
-  if hparams_path and tf.io.gfile.exists(hparams_path):
-    print_out("# Loading standard hparams from %s" % hparams_path)
-    with codecs.getreader("utf-8")(tf.io.gfile.GFile(hparams_path, "rb")) as f:
-      hparams.parse_json(f.read())
-  return hparams
-
-
-def save_hparams(out_dir, hparams):
-  """Save hparams."""
-  hparams_file = os.path.join(out_dir, "hparams")
-  print_out("  saving hparams to %s" % hparams_file)
-  with codecs.getwriter("utf-8")(tf.io.gfile.GFile(hparams_file, "wb")) as f:
-    f.write(hparams.to_json(indent=4, sort_keys=True))
-
-
-def debug_tensor(s, msg=None, summarize=10):
-  """Print the shape and value of a tensor at test time. Return a new tensor."""
-  if not msg:
-    msg = s.name
-  return tf.compat.v1.Print(s, [tf.shape(input=s), s], msg + " ", summarize=summarize)
-
-
-def add_summary(summary_writer, global_step, tag, value):
-  """Add a new summary to the current summary_writer.
-  Useful to log things that are not part of the training graph, e.g., tag=BLEU.
-  """
-  summary = tf.compat.v1.Summary(value=[tf.compat.v1.Summary.Value(tag=tag, simple_value=value)])
-  summary_writer.add_summary(summary, global_step)
-
-
-def get_config_proto(log_device_placement=False, allow_soft_placement=True,
-                     num_intra_threads=0, num_inter_threads=0):
-  # GPU options:
-  # https://www.tensorflow.org/versions/r0.10/how_tos/using_gpu/index.html
-  config_proto = tf.compat.v1.ConfigProto(
-      log_device_placement=log_device_placement,
-      allow_soft_placement=allow_soft_placement)
-  config_proto.gpu_options.allow_growth = True
-
-  # CPU threads options
-  if num_intra_threads:
-    config_proto.intra_op_parallelism_threads = num_intra_threads
-  if num_inter_threads:
-    config_proto.inter_op_parallelism_threads = num_inter_threads
-
-  return config_proto
-
-
-def format_text(words):
-  """Convert a sequence words into sentence."""
-  if (not hasattr(words, "__len__") and  # for numpy array
-      not isinstance(words, collections.Iterable)):
-    words = [words]
-  return b" ".join(words)
-
-
-def format_bpe_text(symbols, delimiter=b"@@"):
-  """Convert a sequence of bpe words into sentence."""
-  words = []
-  word = b""
-  if isinstance(symbols, str):
-    symbols = symbols.encode()
-  delimiter_len = len(delimiter)
-  for symbol in symbols:
-    if len(symbol) >= delimiter_len and symbol[-delimiter_len:] == delimiter:
-      word += symbol[:-delimiter_len]
-    else:  # end of a word
-      word += symbol
-      words.append(word)
-      word = b""
-  return b" ".join(words)
-
-
-def format_spm_text(symbols):
-  """Decode a text in SPM (https://github.com/google/sentencepiece) format."""
-  return u"".join(format_text(symbols).decode("utf-8").split()).replace(
-      u"\u2581", u" ").strip().encode("utf-8")
diff --git a/models/language_translation/tensorflow/mlperf_gnmt/fp32/nmt_utils.py b/models/language_translation/tensorflow/mlperf_gnmt/fp32/nmt_utils.py
deleted file mode 100644
index 543110e7f..000000000
--- a/models/language_translation/tensorflow/mlperf_gnmt/fp32/nmt_utils.py
+++ /dev/null
@@ -1,111 +0,0 @@
-# Copyright 2017 Google Inc. All Rights Reserved.
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-
-"""Utility functions specifically for NMT."""
-from __future__ import print_function
-
-import codecs
-import time
-import numpy as np
-import tensorflow as tf
-import logging as log
-import evaluation_utils
-import misc_utils as utils
-
-
-__all__ = ["decode_and_evaluate", "get_translation"]
-
-
-def decode_and_evaluate(mode, sess, out_tensor, trans_file, ref_file,
-                        metric='bleu', beam_width=10,
-                        num_translations_per_input=1, iterations=1):
-  """Decode a test set and compute a score according to the evaluation task."""
-  utils.print_out("  Decoding to output %s" % trans_file)
-
-  with codecs.getwriter("utf-8")(
-      tf.io.gfile.GFile(trans_file, mode="wb")) as trans_f:
-    trans_f.write("")  # Write empty string to ensure file is created.
-    num_translations_per_input = min(num_translations_per_input, beam_width)
-
-    print("  Running inference with beam_width %g, num translations per input %d. " \
-          % (beam_width, num_translations_per_input))
-    print("  Total iterations count %d." % iterations)
-
-    # Warmup for the first batch to take out the very first runtime
-    # session overhead.
-    nmt_outputs = sess.run(out_tensor) # time x batch_size x beam_width
-    nmt_outputs = nmt_outputs.transpose() # beam_width x batch_size x time
-    batch_size = nmt_outputs.shape[1]
-    for sent_id in range(batch_size):
-      translation = get_translation(nmt_outputs[0], sent_id,
-                                    tgt_eos='</s>')
-      if mode == 'accuracy':
-        trans_f.write((translation + b"\n").decode("utf-8"))
-
-    # prediction time is the time for the model prediction only
-    # overall time is the time for data pre-processing and data post-processing
-    prediction_times = list()
-    overall_start = time.time()
-    num_sentences = 0
-    n = 0
-    while n < iterations:
-      n += 1
-      while True:
-        try:
-          start = time.time()
-          nmt_outputs = sess.run(out_tensor) # time x batch_size x beam_width
-          nmt_outputs = nmt_outputs.transpose() # beam_width x batch_size x time
-          prediction_times.append(time.time() - start)
-          batch_size = nmt_outputs.shape[1]
-          num_sentences += batch_size
-          for sent_id in range(batch_size):
-            for beam_id in range(num_translations_per_input):
-              translation = get_translation(nmt_outputs[beam_id], sent_id,
-                                            tgt_eos='</s>')
-              if mode == 'accuracy':
-                trans_f.write((translation + b"\n").decode("utf-8"))
-
-        except tf.errors.OutOfRangeError:
-          utils.print_time(
-              "  Done, num sentences %d, num translations per input %d" %
-              (num_sentences, num_translations_per_input), overall_start)
-          break
-
-  overall_time = (time.time() - overall_start)
-  print("\nAverage Prediction Latency: {:.5f} sec per batch.".format(
-    sum(prediction_times)/float(len(prediction_times))))
-  print("Overall Latency: {:.5f} sec for the entire test "
-        "dataset.".format(overall_time/float(iterations)))
-  print("Overall Throughput : {:.3f} sentences per sec.".format(
-        num_sentences/float(overall_time)))
-
-  # Evaluation
-  if mode == 'accuracy':
-    if ref_file and tf.io.gfile.exists(trans_file):
-      score = evaluation_utils.evaluate(ref_file, trans_file, metric)
-      utils.print_out("  Accuracy metric %s: %.1f" % (metric, score))
-
-
-def get_translation(nmt_outputs, sent_id, tgt_eos):
-  """Given batch decoding outputs, select a sentence and turn to text."""
-  if tgt_eos: tgt_eos = tgt_eos.encode("utf-8")
-  # Select a sentence
-  output = nmt_outputs[sent_id, :].tolist()
-
-  # If there is an eos symbol in outputs, cut them at that point.
-  if tgt_eos and tgt_eos in output:
-    output = output[:output.index(tgt_eos)]
-    translation = utils.format_text(output)
-
-  return translation
diff --git a/models/language_translation/tensorflow/mlperf_gnmt/fp32/rouge.py b/models/language_translation/tensorflow/mlperf_gnmt/fp32/rouge.py
deleted file mode 100644
index e0269b9c1..000000000
--- a/models/language_translation/tensorflow/mlperf_gnmt/fp32/rouge.py
+++ /dev/null
@@ -1,352 +0,0 @@
-"""ROUGE metric implementation.
-
-Copy from tf_seq2seq/seq2seq/metrics/rouge.py.
-This is a modified and slightly extended verison of
-https://github.com/miso-belica/sumy/blob/dev/sumy/evaluation/rouge.py.
-"""
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-from __future__ import unicode_literals
-
-import itertools
-import numpy as np
-
-#pylint: disable=C0103
-
-
-def _get_ngrams(n, text):
-  """Calcualtes n-grams.
-
-  Args:
-    n: which n-grams to calculate
-    text: An array of tokens
-
-  Returns:
-    A set of n-grams
-  """
-  ngram_set = set()
-  text_length = len(text)
-  max_index_ngram_start = text_length - n
-  for i in range(max_index_ngram_start + 1):
-    ngram_set.add(tuple(text[i:i + n]))
-  return ngram_set
-
-
-def _split_into_words(sentences):
-  """Splits multiple sentences into words and flattens the result"""
-  return list(itertools.chain(*[_.split(" ") for _ in sentences]))
-
-
-def _get_word_ngrams(n, sentences):
-  """Calculates word n-grams for multiple sentences.
-  """
-  assert len(sentences) > 0
-  assert n > 0
-
-  words = _split_into_words(sentences)
-  return _get_ngrams(n, words)
-
-
-def _len_lcs(x, y):
-  """
-  Returns the length of the Longest Common Subsequence between sequences x
-  and y.
-  Source: http://www.algorithmist.com/index.php/Longest_Common_Subsequence
-
-  Args:
-    x: sequence of words
-    y: sequence of words
-
-  Returns
-    integer: Length of LCS between x and y
-  """
-  table = _lcs(x, y)
-  n, m = len(x), len(y)
-  return table[n, m]
-
-
-def _lcs(x, y):
-  """
-  Computes the length of the longest common subsequence (lcs) between two
-  strings. The implementation below uses a DP programming algorithm and runs
-  in O(nm) time where n = len(x) and m = len(y).
-  Source: http://www.algorithmist.com/index.php/Longest_Common_Subsequence
-
-  Args:
-    x: collection of words
-    y: collection of words
-
-  Returns:
-    Table of dictionary of coord and len lcs
-  """
-  n, m = len(x), len(y)
-  table = dict()
-  for i in range(n + 1):
-    for j in range(m + 1):
-      if i == 0 or j == 0:
-        table[i, j] = 0
-      elif x[i - 1] == y[j - 1]:
-        table[i, j] = table[i - 1, j - 1] + 1
-      else:
-        table[i, j] = max(table[i - 1, j], table[i, j - 1])
-  return table
-
-
-def _recon_lcs(x, y):
-  """
-  Returns the Longest Subsequence between x and y.
-  Source: http://www.algorithmist.com/index.php/Longest_Common_Subsequence
-
-  Args:
-    x: sequence of words
-    y: sequence of words
-
-  Returns:
-    sequence: LCS of x and y
-  """
-  i, j = len(x), len(y)
-  table = _lcs(x, y)
-
-  def _recon(i, j):
-    """private recon calculation"""
-    if i == 0 or j == 0:
-      return []
-    elif x[i - 1] == y[j - 1]:
-      return _recon(i - 1, j - 1) + [(x[i - 1], i)]
-    elif table[i - 1, j] > table[i, j - 1]:
-      return _recon(i - 1, j)
-    else:
-      return _recon(i, j - 1)
-
-  recon_tuple = tuple(map(lambda x: x[0], _recon(i, j)))
-  return recon_tuple
-
-
-def rouge_n(evaluated_sentences, reference_sentences, n=2):
-  """
-  Computes ROUGE-N of two text collections of sentences.
-  Sourece: http://research.microsoft.com/en-us/um/people/cyl/download/
-  papers/rouge-working-note-v1.3.1.pdf
-
-  Args:
-    evaluated_sentences: The sentences that have been picked by the summarizer
-    reference_sentences: The sentences from the referene set
-    n: Size of ngram.  Defaults to 2.
-
-  Returns:
-    A tuple (f1, precision, recall) for ROUGE-N
-
-  Raises:
-    ValueError: raises exception if a param has len <= 0
-  """
-  if len(evaluated_sentences) <= 0 or len(reference_sentences) <= 0:
-    raise ValueError("Collections must contain at least 1 sentence.")
-
-  evaluated_ngrams = _get_word_ngrams(n, evaluated_sentences)
-  reference_ngrams = _get_word_ngrams(n, reference_sentences)
-  reference_count = len(reference_ngrams)
-  evaluated_count = len(evaluated_ngrams)
-
-  # Gets the overlapping ngrams between evaluated and reference
-  overlapping_ngrams = evaluated_ngrams.intersection(reference_ngrams)
-  overlapping_count = len(overlapping_ngrams)
-
-  # Handle edge case. This isn't mathematically correct, but it's good enough
-  if evaluated_count == 0:
-    precision = 0.0
-  else:
-    precision = overlapping_count / evaluated_count
-
-  if reference_count == 0:
-    recall = 0.0
-  else:
-    recall = overlapping_count / reference_count
-
-  f1_score = 2.0 * ((precision * recall) / (precision + recall + 1e-8))
-
-  # return overlapping_count / reference_count
-  return f1_score, precision, recall
-
-
-def _f_p_r_lcs(llcs, m, n):
-  """
-  Computes the LCS-based F-measure score
-  Source: http://research.microsoft.com/en-us/um/people/cyl/download/papers/
-  rouge-working-note-v1.3.1.pdf
-
-  Args:
-    llcs: Length of LCS
-    m: number of words in reference summary
-    n: number of words in candidate summary
-
-  Returns:
-    Float. LCS-based F-measure score
-  """
-  r_lcs = llcs / m
-  p_lcs = llcs / n
-  beta = p_lcs / (r_lcs + 1e-12)
-  num = (1 + (beta**2)) * r_lcs * p_lcs
-  denom = r_lcs + ((beta**2) * p_lcs)
-  f_lcs = num / (denom + 1e-12)
-  return f_lcs, p_lcs, r_lcs
-
-
-def rouge_l_sentence_level(evaluated_sentences, reference_sentences):
-  """
-  Computes ROUGE-L (sentence level) of two text collections of sentences.
-  http://research.microsoft.com/en-us/um/people/cyl/download/papers/
-  rouge-working-note-v1.3.1.pdf
-
-  Calculated according to:
-  R_lcs = LCS(X,Y)/m
-  P_lcs = LCS(X,Y)/n
-  F_lcs = ((1 + beta^2)*R_lcs*P_lcs) / (R_lcs + (beta^2) * P_lcs)
-
-  where:
-  X = reference summary
-  Y = Candidate summary
-  m = length of reference summary
-  n = length of candidate summary
-
-  Args:
-    evaluated_sentences: The sentences that have been picked by the summarizer
-    reference_sentences: The sentences from the referene set
-
-  Returns:
-    A float: F_lcs
-
-  Raises:
-    ValueError: raises exception if a param has len <= 0
-  """
-  if len(evaluated_sentences) <= 0 or len(reference_sentences) <= 0:
-    raise ValueError("Collections must contain at least 1 sentence.")
-  reference_words = _split_into_words(reference_sentences)
-  evaluated_words = _split_into_words(evaluated_sentences)
-  m = len(reference_words)
-  n = len(evaluated_words)
-  lcs = _len_lcs(evaluated_words, reference_words)
-  return _f_p_r_lcs(lcs, m, n)
-
-
-def _union_lcs(evaluated_sentences, reference_sentence):
-  """
-  Returns LCS_u(r_i, C) which is the LCS score of the union longest common
-  subsequence between reference sentence ri and candidate summary C. For example
-  if r_i= w1 w2 w3 w4 w5, and C contains two sentences: c1 = w1 w2 w6 w7 w8 and
-  c2 = w1 w3 w8 w9 w5, then the longest common subsequence of r_i and c1 is
-  "w1 w2" and the longest common subsequence of r_i and c2 is "w1 w3 w5". The
-  union longest common subsequence of r_i, c1, and c2 is "w1 w2 w3 w5" and
-  LCS_u(r_i, C) = 4/5.
-
-  Args:
-    evaluated_sentences: The sentences that have been picked by the summarizer
-    reference_sentence: One of the sentences in the reference summaries
-
-  Returns:
-    float: LCS_u(r_i, C)
-
-  ValueError:
-    Raises exception if a param has len <= 0
-  """
-  if len(evaluated_sentences) <= 0:
-    raise ValueError("Collections must contain at least 1 sentence.")
-
-  lcs_union = set()
-  reference_words = _split_into_words([reference_sentence])
-  combined_lcs_length = 0
-  for eval_s in evaluated_sentences:
-    evaluated_words = _split_into_words([eval_s])
-    lcs = set(_recon_lcs(reference_words, evaluated_words))
-    combined_lcs_length += len(lcs)
-    lcs_union = lcs_union.union(lcs)
-
-  union_lcs_count = len(lcs_union)
-  union_lcs_value = union_lcs_count / combined_lcs_length
-  return union_lcs_value
-
-
-def rouge_l_summary_level(evaluated_sentences, reference_sentences):
-  """
-  Computes ROUGE-L (summary level) of two text collections of sentences.
-  http://research.microsoft.com/en-us/um/people/cyl/download/papers/
-  rouge-working-note-v1.3.1.pdf
-
-  Calculated according to:
-  R_lcs = SUM(1, u)[LCS<union>(r_i,C)]/m
-  P_lcs = SUM(1, u)[LCS<union>(r_i,C)]/n
-  F_lcs = ((1 + beta^2)*R_lcs*P_lcs) / (R_lcs + (beta^2) * P_lcs)
-
-  where:
-  SUM(i,u) = SUM from i through u
-  u = number of sentences in reference summary
-  C = Candidate summary made up of v sentences
-  m = number of words in reference summary
-  n = number of words in candidate summary
-
-  Args:
-    evaluated_sentences: The sentences that have been picked by the summarizer
-    reference_sentence: One of the sentences in the reference summaries
-
-  Returns:
-    A float: F_lcs
-
-  Raises:
-    ValueError: raises exception if a param has len <= 0
-  """
-  if len(evaluated_sentences) <= 0 or len(reference_sentences) <= 0:
-    raise ValueError("Collections must contain at least 1 sentence.")
-
-  # total number of words in reference sentences
-  m = len(_split_into_words(reference_sentences))
-
-  # total number of words in evaluated sentences
-  n = len(_split_into_words(evaluated_sentences))
-
-  union_lcs_sum_across_all_references = 0
-  for ref_s in reference_sentences:
-    union_lcs_sum_across_all_references += _union_lcs(evaluated_sentences,
-                                                      ref_s)
-  return _f_p_r_lcs(union_lcs_sum_across_all_references, m, n)
-
-
-def rouge(hypotheses, references):
-  """Calculates average rouge scores for a list of hypotheses and
-  references"""
-
-  # Filter out hyps that are of 0 length
-  # hyps_and_refs = zip(hypotheses, references)
-  # hyps_and_refs = [_ for _ in hyps_and_refs if len(_[0]) > 0]
-  # hypotheses, references = zip(*hyps_and_refs)
-
-  # Calculate ROUGE-1 F1, precision, recall scores
-  rouge_1 = [
-      rouge_n([hyp], [ref], 1) for hyp, ref in zip(hypotheses, references)
-  ]
-  rouge_1_f, rouge_1_p, rouge_1_r = map(np.mean, zip(*rouge_1))
-
-  # Calculate ROUGE-2 F1, precision, recall scores
-  rouge_2 = [
-      rouge_n([hyp], [ref], 2) for hyp, ref in zip(hypotheses, references)
-  ]
-  rouge_2_f, rouge_2_p, rouge_2_r = map(np.mean, zip(*rouge_2))
-
-  # Calculate ROUGE-L F1, precision, recall scores
-  rouge_l = [
-      rouge_l_sentence_level([hyp], [ref])
-      for hyp, ref in zip(hypotheses, references)
-  ]
-  rouge_l_f, rouge_l_p, rouge_l_r = map(np.mean, zip(*rouge_l))
-
-  return {
-      "rouge_1/f_score": rouge_1_f,
-      "rouge_1/r_score": rouge_1_r,
-      "rouge_1/p_score": rouge_1_p,
-      "rouge_2/f_score": rouge_2_f,
-      "rouge_2/r_score": rouge_2_r,
-      "rouge_2/p_score": rouge_2_p,
-      "rouge_l/f_score": rouge_l_f,
-      "rouge_l/r_score": rouge_l_r,
-      "rouge_l/p_score": rouge_l_p,
-  }
diff --git a/models/language_translation/tensorflow/mlperf_gnmt/fp32/run_inference.py b/models/language_translation/tensorflow/mlperf_gnmt/fp32/run_inference.py
deleted file mode 100644
index 051d24c6b..000000000
--- a/models/language_translation/tensorflow/mlperf_gnmt/fp32/run_inference.py
+++ /dev/null
@@ -1,120 +0,0 @@
-import codecs
-import argparse
-import os
-import tensorflow as tf
-from tensorflow.core.framework import graph_pb2
-from tensorflow.python.framework import importer
-
-import misc_utils as utils
-from nmt_utils import decode_and_evaluate
-
-from tensorflow_addons import seq2seq
-
-parser = argparse.ArgumentParser()
-parser.add_argument("--in_graph", type=str, required=True,
-                    help="Specify the frozen inference graph in pb format.")
-parser.add_argument("--batch_size", type=int, default=32,
-                    help="Specify inference batch size.")
-parser.add_argument("--num_inter_threads", type=int, default=0,
-                   help="Specify number of inter-op threads.")
-parser.add_argument("--num_intra_threads", type=int, default=0,
-                   help="Specify number of intra-op threads.")
-parser.add_argument("--src_vocab_file", type=str, required=True,
-                    help="Specify source vocab file.")
-parser.add_argument("--tgt_vocab_file", type=str, required=True,
-                    help="Specify target vocabulary file.")
-parser.add_argument("--inference_input_file", type=str, required=True,
-                    help="Specify input file to be translated.")
-parser.add_argument("--inference_output_file", type=str, default=None,
-                    help="Specify output file for resulting translation.")
-parser.add_argument("--inference_ref_file", type=str, required=True,
-                    help="Specify reference output file.")
-parser.add_argument("--run", type=str, default="accuracy",
-                    help="Specify either 'accuracy' for BLEU metric or "
-                         "'performance' for latency and throughput.")
-args = parser.parse_args()
-
-out_dir = os.path.join(os.getcwd(), 'output')
-tf.io.gfile.makedirs(out_dir)
-
-if args.inference_output_file:
-  inference_output_file = args.inference_output_file
-else:
-  inference_output_file = os.path.join(out_dir, 'gnmt-out')
-
-def read_source_sentences(inference_input_file):
-  """Load inference data."""
-  with codecs.getreader("utf-8")(
-      tf.io.gfile.GFile(inference_input_file, mode="rb")) as f:
-    inference_data = f.read().splitlines()
-  return inference_data
-
-def create_new_vocab_file(vocab_file):
-  """Creates a new vocabulary file prepending three new tokens:
-  (1) <unk> for unknown tag, (2) <s> for start of sentence tag, and (3) </s> for end of
-  sentence tag."""
-  vocab = []
-  with codecs.getreader("utf-8")(tf.io.gfile.GFile(vocab_file, "rb")) as f:
-    vocab_size = 0
-    for word in f:
-      vocab_size += 1
-      vocab.append(word.strip())
-
-  if tf.io.gfile.exists(vocab_file):
-    utils.print_out("# Vocab file %s exists" % vocab_file)
-    assert len(vocab) >= 3
-    (unk, sos, eos) = ("<unk>", "<s>", "</s>")
-    if vocab[0] != unk or vocab[1] != sos or vocab[2] != eos:
-      utils.print_out("The first 3 vocab words [%s, %s, %s]"
-                      " are not [%s, %s, %s]" %
-                      (vocab[0], vocab[1], vocab[2], unk, sos, eos))
-      vocab = [unk, sos, eos] + vocab
-      vocab_size += 3
-      new_vocab_file = os.path.join(out_dir, os.path.basename(vocab_file))
-      with codecs.getwriter("utf-8")(
-          tf.io.gfile.GFile(new_vocab_file, "wb")) as f:
-        for word in vocab:
-          f.write("%s\n" % word)
-      vocab_file = new_vocab_file
-  else:
-    raise ValueError("vocab_file '%s' does not exist." % vocab_file)
-  return vocab_file
-
-if __name__ == "__main__":
-  graph_def = graph_pb2.GraphDef()
-  with tf.io.gfile.GFile(args.in_graph, "rb") as f:
-    data = f.read()
-  graph_def.ParseFromString(data)
-  graph = tf.Graph()
-  with graph.as_default():
-    importer.import_graph_def(graph_def, input_map={}, name="")
-    # Get input and output and tensors/ops for inference.
-    src_vocab_placeholder = graph.get_tensor_by_name('source_vocab_file:0')
-    tgt_vocab_placeholder = graph.get_tensor_by_name('target_vocab_file:0')
-    src_data_placeholder = graph.get_tensor_by_name('source_data:0')
-    batch_size_placeholder = graph.get_tensor_by_name('batch_size:0')
-
-    tables_initializer = graph.get_operation_by_name('init_all_tables')
-    iterator_initilizer = graph.get_operation_by_name('MakeIterator')
-    sample_words_tensor = graph.get_tensor_by_name('hash_table_Lookup_1/LookupTableFindV2:0')
-
-  # Create a session with imported graph.
-  config_proto = tf.compat.v1.ConfigProto(allow_soft_placement=True,
-      intra_op_parallelism_threads = args.num_intra_threads,
-      inter_op_parallelism_threads = args.num_inter_threads)
-  sess = tf.compat.v1.Session(graph=graph, config=config_proto)
-
-  # Read source data.
-  src_data = read_source_sentences(args.inference_input_file)
-
-  # Initialize vocabulary tables and source data iterator.
-  sess.run(tables_initializer, feed_dict={
-      src_vocab_placeholder: create_new_vocab_file(args.src_vocab_file),
-      tgt_vocab_placeholder: create_new_vocab_file(args.tgt_vocab_file)})
-  sess.run(iterator_initilizer, feed_dict={
-      src_data_placeholder: src_data,
-      batch_size_placeholder: args.batch_size})
-
-  # Decode
-  decode_and_evaluate(args.run, sess, sample_words_tensor, inference_output_file,
-                      args.inference_ref_file)
diff --git a/models/object_detection/tensorflow/faster_rcnn/__init__.py b/models/object_detection/tensorflow/faster_rcnn/__init__.py
deleted file mode 100644
index b43c21f99..000000000
--- a/models/object_detection/tensorflow/faster_rcnn/__init__.py
+++ /dev/null
@@ -1,20 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-# SPDX-License-Identifier: EPL-2.0
-#
-
diff --git a/models/object_detection/tensorflow/faster_rcnn/inference/__init__.py b/models/object_detection/tensorflow/faster_rcnn/inference/__init__.py
deleted file mode 100644
index b43c21f99..000000000
--- a/models/object_detection/tensorflow/faster_rcnn/inference/__init__.py
+++ /dev/null
@@ -1,20 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-# SPDX-License-Identifier: EPL-2.0
-#
-
diff --git a/models/object_detection/tensorflow/faster_rcnn/inference/fp32/__init__.py b/models/object_detection/tensorflow/faster_rcnn/inference/fp32/__init__.py
deleted file mode 100644
index b43c21f99..000000000
--- a/models/object_detection/tensorflow/faster_rcnn/inference/fp32/__init__.py
+++ /dev/null
@@ -1,20 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-# SPDX-License-Identifier: EPL-2.0
-#
-
diff --git a/models/object_detection/tensorflow/faster_rcnn/inference/fp32/coco_accuracy.sh b/models/object_detection/tensorflow/faster_rcnn/inference/fp32/coco_accuracy.sh
deleted file mode 100644
index 66c442189..000000000
--- a/models/object_detection/tensorflow/faster_rcnn/inference/fp32/coco_accuracy.sh
+++ /dev/null
@@ -1,59 +0,0 @@
-#!/usr/bin/env bash
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2019 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-# SPDX-License-Identifier: EPL-2.0
-#
-
-########## Variables to be defined - run it in research folder
-
-SPLIT=faster-rcnn1 #change to your favorite name
-
-FROZEN_GRAPH=$1
-TF_RECORD_FILES=$2
-TF_MODELS_ROOT=$3
-
-if [[ -z ${TF_MODELS_ROOT} ]] || [[ ! -d ${TF_MODELS_ROOT} ]]; then
-  echo "You must specify the root of the tensorflow/models source tree in the TF_MODELS_ROOT environment variable."
-  exit 1
-fi
-
-export PYTHONPATH=$PYTHONPATH:${TF_MODELS_ROOT}/research:${TF_MODELS_ROOT}/research/slim:${TF_MODELS_ROOT}/research/object_detection
-
-python object_detection/inference/infer_detections.py \
-  --input_tfrecord_paths=$TF_RECORD_FILES \
-  --output_tfrecord_path=${SPLIT}_detections.tfrecord \
-  --inference_graph=$FROZEN_GRAPH \
-  --discard_image_pixels=True
-
-
-mkdir -p ${SPLIT}_eval_metrics
-
-echo "
-label_map_path: '${TF_MODELS_ROOT}/research/object_detection/data/mscoco_label_map.pbtxt'
-tf_record_input_reader: { input_path: '${SPLIT}_detections.tfrecord' }
-" > ${SPLIT}_eval_metrics/${SPLIT}_input_config.pbtxt
-
-echo "
-metrics_set: 'coco_detection_metrics'
-" > ${SPLIT}_eval_metrics/${SPLIT}_eval_config.pbtxt
-
-
-python object_detection/metrics/offline_eval_map_corloc.py \
-  --eval_dir=${SPLIT}_eval_metrics \
-  --eval_config_path=${SPLIT}_eval_metrics/${SPLIT}_eval_config.pbtxt \
-  --input_config_path=${SPLIT}_eval_metrics/${SPLIT}_input_config.pbtxt
diff --git a/models/object_detection/tensorflow/faster_rcnn/inference/fp32/dataset_util.py b/models/object_detection/tensorflow/faster_rcnn/inference/fp32/dataset_util.py
deleted file mode 100644
index ac5fceb5e..000000000
--- a/models/object_detection/tensorflow/faster_rcnn/inference/fp32/dataset_util.py
+++ /dev/null
@@ -1,160 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-# SPDX-License-Identifier: EPL-2.0
-#
-
-# Copyright 2017 The TensorFlow Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-
-"""Utility functions for creating TFRecord data sets."""
-
-import tensorflow as tf
-
-
-def int64_feature(value):
-  return tf.train.Feature(int64_list=tf.train.Int64List(value=[value]))
-
-
-def int64_list_feature(value):
-  return tf.train.Feature(int64_list=tf.train.Int64List(value=value))
-
-
-def bytes_feature(value):
-  return tf.train.Feature(bytes_list=tf.train.BytesList(value=[value]))
-
-
-def bytes_list_feature(value):
-  return tf.train.Feature(bytes_list=tf.train.BytesList(value=value))
-
-
-def float_list_feature(value):
-  return tf.train.Feature(float_list=tf.train.FloatList(value=value))
-
-
-def read_examples_list(path):
-  """Read list of training or validation examples.
-
-  The file is assumed to contain a single example per line where the first
-  token in the line is an identifier that allows us to find the image and
-  annotation xml for that example.
-
-  For example, the line:
-  xyz 3
-  would allow us to find files xyz.jpg and xyz.xml (the 3 would be ignored).
-
-  Args:
-    path: absolute path to examples list file.
-
-  Returns:
-    list of example identifiers (strings).
-  """
-  with tf.gfile.GFile(path) as fid:
-    lines = fid.readlines()
-  return [line.strip().split(' ')[0] for line in lines]
-
-
-def recursive_parse_xml_to_dict(xml):
-  """Recursively parses XML contents to python dict.
-
-  We assume that `object` tags are the only ones that can appear
-  multiple times at the same level of a tree.
-
-  Args:
-    xml: xml tree obtained by parsing XML file contents using lxml.etree
-
-  Returns:
-    Python dictionary holding XML contents.
-  """
-  if not xml:
-    return {xml.tag: xml.text}
-  result = {}
-  for child in xml:
-    child_result = recursive_parse_xml_to_dict(child)
-    if child.tag != 'object':
-      result[child.tag] = child_result[child.tag]
-    else:
-      if child.tag not in result:
-        result[child.tag] = []
-      result[child.tag].append(child_result[child.tag])
-  return {xml.tag: result}
-
-
-def make_initializable_iterator(dataset):
-  """Creates an iterator, and initializes tables.
-
-  This is useful in cases where make_one_shot_iterator wouldn't work because
-  the graph contains a hash table that needs to be initialized.
-
-  Args:
-    dataset: A `tf.data.Dataset` object.
-
-  Returns:
-    A `tf.data.Iterator`.
-  """
-  iterator = dataset.make_initializable_iterator()
-  tf.add_to_collection(tf.GraphKeys.TABLE_INITIALIZERS, iterator.initializer)
-  return iterator
-
-
-def read_dataset(file_read_func, decode_func, input_files, config):
-  """Reads a dataset, and handles repetition and shuffling.
-
-  Args:
-    file_read_func: Function to use in tf.data.Dataset.interleave, to read
-      every individual file into a tf.data.Dataset.
-    decode_func: Function to apply to all records.
-    input_files: A list of file paths to read.
-    config: A input_reader_builder.InputReader object.
-
-  Returns:
-    A tf.data.Dataset based on config.
-  """
-  # Shard, shuffle, and read files.
-  filenames = tf.concat([tf.matching_files(pattern) for pattern in input_files],
-                        0)
-  filename_dataset = tf.data.Dataset.from_tensor_slices(filenames)
-  if config.shuffle:
-    filename_dataset = filename_dataset.shuffle(
-        config.filenames_shuffle_buffer_size)
-  elif config.num_readers > 1:
-    tf.logging.warning('`shuffle` is false, but the input data stream is '
-                       'still slightly shuffled since `num_readers` > 1.')
-
-  filename_dataset = filename_dataset.repeat(config.num_epochs or None)
-
-  records_dataset = filename_dataset.apply(
-      tf.contrib.data.parallel_interleave(
-          file_read_func, cycle_length=config.num_readers,
-          block_length=config.read_block_length, sloppy=True))
-  if config.shuffle:
-    records_dataset.shuffle(config.shuffle_buffer_size)
-  tensor_dataset = records_dataset.map(
-      decode_func, num_parallel_calls=config.num_parallel_map_calls)
-  return tensor_dataset.prefetch(config.prefetch_size)
diff --git a/models/object_detection/tensorflow/faster_rcnn/inference/fp32/eval.py b/models/object_detection/tensorflow/faster_rcnn/inference/fp32/eval.py
deleted file mode 100644
index 8aad77b0a..000000000
--- a/models/object_detection/tensorflow/faster_rcnn/inference/fp32/eval.py
+++ /dev/null
@@ -1,165 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-# SPDX-License-Identifier: EPL-2.0
-#
-
-# Copyright 2017 The TensorFlow Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-
-r"""Evaluation executable for detection models.
-
-This executable is used to evaluate DetectionModels. There are two ways of
-configuring the eval job.
-
-1) A single pipeline_pb2.TrainEvalPipelineConfig file maybe specified instead.
-In this mode, the --eval_training_data flag may be given to force the pipeline
-to evaluate on training data instead.
-
-Example usage:
-    ./eval \
-        --logtostderr \
-        --checkpoint_dir=path/to/checkpoint_dir \
-        --eval_dir=path/to/eval_dir \
-        --pipeline_config_path=pipeline_config.pbtxt
-
-2) Three configuration files may be provided: a model_pb2.DetectionModel
-configuration file to define what type of DetectionModel is being evaluated, an
-input_reader_pb2.InputReader file to specify what data the model is evaluating
-and an eval_pb2.EvalConfig file to configure evaluation parameters.
-
-Example usage:
-    ./eval \
-        --logtostderr \
-        --checkpoint_dir=path/to/checkpoint_dir \
-        --eval_dir=path/to/eval_dir \
-        --eval_config_path=eval_config.pbtxt \
-        --model_config_path=model_config.pbtxt \
-        --input_config_path=eval_input_config.pbtxt
-"""
-import functools
-import os
-import tensorflow as tf
-
-import evaluator
-import dataset_util
-from object_detection.builders import dataset_builder
-from object_detection.builders import model_builder
-from object_detection.utils import config_util
-from object_detection.utils import label_map_util
-
-
-tf.logging.set_verbosity(tf.logging.INFO)
-
-flags = tf.app.flags
-flags.DEFINE_boolean('eval_training_data', False,
-                     'If training data should be evaluated for this job.')
-flags.DEFINE_string('checkpoint_dir', '',
-                    'Directory containing checkpoints to evaluate, typically '
-                    'set to `train_dir` used in the training job.')
-flags.DEFINE_string('eval_dir', '',
-                    'Directory to write eval summaries to.')
-flags.DEFINE_string('pipeline_config_path', '',
-                    'Path to a pipeline_pb2.TrainEvalPipelineConfig config '
-                    'file. If provided, other configs are ignored')
-flags.DEFINE_string('eval_config_path', '',
-                    'Path to an eval_pb2.EvalConfig config file.')
-flags.DEFINE_string('input_config_path', '',
-                    'Path to an input_reader_pb2.InputReader config file.')
-flags.DEFINE_string('model_config_path', '',
-                    'Path to a model_pb2.DetectionModel config file.')
-flags.DEFINE_boolean('run_once', False, 'Option to only run a single pass of '
-                     'evaluation. Overrides the `max_evals` parameter in the '
-                     'provided config.')
-                     
-# add flags for mkl
-flags.DEFINE_integer('num_intra_threads', 1,
-                     'Number of threads to use for intra-op parallelism. If '
-                     'set to 0, the system will pick an appropriate number.')
-flags.DEFINE_integer('num_inter_threads', 0,
-                     'Number of threads to use for inter-op parallelism. If '
-                     'set to 0, the system will pick an appropriate number.')
-                     
-FLAGS = flags.FLAGS
-
-
-def main(unused_argv):
-  assert FLAGS.checkpoint_dir, '`checkpoint_dir` is missing.'
-  assert FLAGS.eval_dir, '`eval_dir` is missing.'
-  tf.gfile.MakeDirs(FLAGS.eval_dir)
-  if FLAGS.pipeline_config_path:
-    configs = config_util.get_configs_from_pipeline_file(
-        FLAGS.pipeline_config_path)
-    tf.gfile.Copy(FLAGS.pipeline_config_path,
-                  os.path.join(FLAGS.eval_dir, 'pipeline.config'),
-                  overwrite=True)
-  else:
-    configs = config_util.get_configs_from_multiple_files(
-        model_config_path=FLAGS.model_config_path,
-        eval_config_path=FLAGS.eval_config_path,
-        eval_input_config_path=FLAGS.input_config_path)
-    for name, config in [('model.config', FLAGS.model_config_path),
-                         ('eval.config', FLAGS.eval_config_path),
-                         ('input.config', FLAGS.input_config_path)]:
-      tf.gfile.Copy(config,
-                    os.path.join(FLAGS.eval_dir, name),
-                    overwrite=True)
-
-  model_config = configs['model']
-  eval_config = configs['eval_config']
-  input_config = configs['eval_input_config']
-  if FLAGS.eval_training_data:
-    input_config = configs['train_input_config']
-
-  model_fn = functools.partial(
-      model_builder.build,
-      model_config=model_config,
-      is_training=False)
-
-  def get_next(config):
-    return dataset_util.make_initializable_iterator(
-        dataset_builder.build(config)).get_next()
-
-  create_input_dict_fn = functools.partial(get_next, input_config)
-
-  label_map = label_map_util.load_labelmap(input_config.label_map_path)
-  max_num_classes = max([item.id for item in label_map.item])
-  categories = label_map_util.convert_label_map_to_categories(
-      label_map, max_num_classes)
-
-  if FLAGS.run_once:
-    eval_config.max_evals = 1
-
-  evaluator.evaluate(create_input_dict_fn, model_fn, eval_config, categories,
-                     FLAGS.checkpoint_dir, FLAGS.eval_dir,
-                     inter_op = FLAGS.num_inter_threads, intra_op = FLAGS.num_intra_threads)
-
-
-if __name__ == '__main__':
-  tf.app.run()
diff --git a/models/object_detection/tensorflow/faster_rcnn/inference/fp32/eval_util.py b/models/object_detection/tensorflow/faster_rcnn/inference/fp32/eval_util.py
deleted file mode 100644
index 7c15e5204..000000000
--- a/models/object_detection/tensorflow/faster_rcnn/inference/fp32/eval_util.py
+++ /dev/null
@@ -1,671 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-# SPDX-License-Identifier: EPL-2.0
-#
-
-# Copyright 2017 The TensorFlow Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-"""Common utility functions for evaluation."""
-import collections
-import logging
-import os
-import time
-
-import numpy as np
-import tensorflow as tf
-
-from object_detection.core import box_list
-from object_detection.core import box_list_ops
-from object_detection.core import keypoint_ops
-from object_detection.core import standard_fields as fields
-from object_detection.metrics import coco_evaluation
-from object_detection.utils import label_map_util
-from object_detection.utils import ops
-from object_detection.utils import visualization_utils as vis_utils
-
-slim = tf.contrib.slim
-
-
-def write_metrics(metrics, global_step, summary_dir):
-  """Write metrics to a summary directory.
-
-  Args:
-    metrics: A dictionary containing metric names and values.
-    global_step: Global step at which the metrics are computed.
-    summary_dir: Directory to write tensorflow summaries to.
-  """
-  logging.info('Writing metrics to tf summary.')
-  summary_writer = tf.summary.FileWriterCache.get(summary_dir)
-  for key in sorted(metrics):
-    summary = tf.Summary(value=[
-        tf.Summary.Value(tag=key, simple_value=metrics[key]),
-    ])
-    summary_writer.add_summary(summary, global_step)
-    logging.info('%s: %f', key, metrics[key])
-  logging.info('Metrics written to tf summary.')
-
-
-# TODO(rathodv): Add tests.
-def visualize_detection_results(result_dict,
-                                tag,
-                                global_step,
-                                categories,
-                                summary_dir='',
-                                export_dir='',
-                                agnostic_mode=False,
-                                show_groundtruth=False,
-                                groundtruth_box_visualization_color='black',
-                                min_score_thresh=.5,
-                                max_num_predictions=20,
-                                skip_scores=False,
-                                skip_labels=False,
-                                keep_image_id_for_visualization_export=False):
-  """Visualizes detection results and writes visualizations to image summaries.
-
-  This function visualizes an image with its detected bounding boxes and writes
-  to image summaries which can be viewed on tensorboard.  It optionally also
-  writes images to a directory. In the case of missing entry in the label map,
-  unknown class name in the visualization is shown as "N/A".
-
-  Args:
-    result_dict: a dictionary holding groundtruth and detection
-      data corresponding to each image being evaluated.  The following keys
-      are required:
-        'original_image': a numpy array representing the image with shape
-          [1, height, width, 3] or [1, height, width, 1]
-        'detection_boxes': a numpy array of shape [N, 4]
-        'detection_scores': a numpy array of shape [N]
-        'detection_classes': a numpy array of shape [N]
-      The following keys are optional:
-        'groundtruth_boxes': a numpy array of shape [N, 4]
-        'groundtruth_keypoints': a numpy array of shape [N, num_keypoints, 2]
-      Detections are assumed to be provided in decreasing order of score and for
-      display, and we assume that scores are probabilities between 0 and 1.
-    tag: tensorboard tag (string) to associate with image.
-    global_step: global step at which the visualization are generated.
-    categories: a list of dictionaries representing all possible categories.
-      Each dict in this list has the following keys:
-          'id': (required) an integer id uniquely identifying this category
-          'name': (required) string representing category name
-            e.g., 'cat', 'dog', 'pizza'
-          'supercategory': (optional) string representing the supercategory
-            e.g., 'animal', 'vehicle', 'food', etc
-    summary_dir: the output directory to which the image summaries are written.
-    export_dir: the output directory to which images are written.  If this is
-      empty (default), then images are not exported.
-    agnostic_mode: boolean (default: False) controlling whether to evaluate in
-      class-agnostic mode or not.
-    show_groundtruth: boolean (default: False) controlling whether to show
-      groundtruth boxes in addition to detected boxes
-    groundtruth_box_visualization_color: box color for visualizing groundtruth
-      boxes
-    min_score_thresh: minimum score threshold for a box to be visualized
-    max_num_predictions: maximum number of detections to visualize
-    skip_scores: whether to skip score when drawing a single detection
-    skip_labels: whether to skip label when drawing a single detection
-    keep_image_id_for_visualization_export: whether to keep image identifier in
-      filename when exported to export_dir
-  Raises:
-    ValueError: if result_dict does not contain the expected keys (i.e.,
-      'original_image', 'detection_boxes', 'detection_scores',
-      'detection_classes')
-  """
-  detection_fields = fields.DetectionResultFields
-  input_fields = fields.InputDataFields
-  if not set([
-      input_fields.original_image,
-      detection_fields.detection_boxes,
-      detection_fields.detection_scores,
-      detection_fields.detection_classes,
-  ]).issubset(set(result_dict.keys())):
-    raise ValueError('result_dict does not contain all expected keys.')
-  if show_groundtruth and input_fields.groundtruth_boxes not in result_dict:
-    raise ValueError('If show_groundtruth is enabled, result_dict must contain '
-                     'groundtruth_boxes.')
-  logging.info('Creating detection visualizations.')
-  category_index = label_map_util.create_category_index(categories)
-
-  image = np.squeeze(result_dict[input_fields.original_image], axis=0)
-  if image.shape[2] == 1:  # If one channel image, repeat in RGB.
-    image = np.tile(image, [1, 1, 3])
-  detection_boxes = result_dict[detection_fields.detection_boxes]
-  detection_scores = result_dict[detection_fields.detection_scores]
-  detection_classes = np.int32((result_dict[
-      detection_fields.detection_classes]))
-  detection_keypoints = result_dict.get(detection_fields.detection_keypoints)
-  detection_masks = result_dict.get(detection_fields.detection_masks)
-  detection_boundaries = result_dict.get(detection_fields.detection_boundaries)
-
-  # Plot groundtruth underneath detections
-  if show_groundtruth:
-    groundtruth_boxes = result_dict[input_fields.groundtruth_boxes]
-    groundtruth_keypoints = result_dict.get(input_fields.groundtruth_keypoints)
-    vis_utils.visualize_boxes_and_labels_on_image_array(
-        image=image,
-        boxes=groundtruth_boxes,
-        classes=None,
-        scores=None,
-        category_index=category_index,
-        keypoints=groundtruth_keypoints,
-        use_normalized_coordinates=False,
-        max_boxes_to_draw=None,
-        groundtruth_box_visualization_color=groundtruth_box_visualization_color)
-  vis_utils.visualize_boxes_and_labels_on_image_array(
-      image,
-      detection_boxes,
-      detection_classes,
-      detection_scores,
-      category_index,
-      instance_masks=detection_masks,
-      instance_boundaries=detection_boundaries,
-      keypoints=detection_keypoints,
-      use_normalized_coordinates=False,
-      max_boxes_to_draw=max_num_predictions,
-      min_score_thresh=min_score_thresh,
-      agnostic_mode=agnostic_mode,
-      skip_scores=skip_scores,
-      skip_labels=skip_labels)
-
-  if export_dir:
-    if keep_image_id_for_visualization_export and result_dict[fields.
-                                                              InputDataFields()
-                                                              .key]:
-      export_path = os.path.join(export_dir, 'export-{}-{}.png'.format(
-          tag, result_dict[fields.InputDataFields().key]))
-    else:
-      export_path = os.path.join(export_dir, 'export-{}.png'.format(tag))
-    vis_utils.save_image_array_as_png(image, export_path)
-
-  summary = tf.Summary(value=[
-      tf.Summary.Value(
-          tag=tag,
-          image=tf.Summary.Image(
-              encoded_image_string=vis_utils.encode_image_array_as_png_str(
-                  image)))
-  ])
-  summary_writer = tf.summary.FileWriterCache.get(summary_dir)
-  summary_writer.add_summary(summary, global_step)
-
-  logging.info('Detection visualizations written to summary with tag %s.', tag)
-
-
-def _run_checkpoint_once(tensor_dict,
-                         evaluators=None,
-                         batch_processor=None,
-                         checkpoint_dirs=None,
-                         variables_to_restore=None,
-                         restore_fn=None,
-                         num_batches=1,
-                         master='',
-                         save_graph=False,
-                         save_graph_dir='',
-                         losses_dict=None, inter_op = 0, intra_op = 1):
-  """Evaluates metrics defined in evaluators and returns summaries.
-
-  This function loads the latest checkpoint in checkpoint_dirs and evaluates
-  all metrics defined in evaluators. The metrics are processed in batch by the
-  batch_processor.
-
-  Args:
-    tensor_dict: a dictionary holding tensors representing a batch of detections
-      and corresponding groundtruth annotations.
-    evaluators: a list of object of type DetectionEvaluator to be used for
-      evaluation. Note that the metric names produced by different evaluators
-      must be unique.
-    batch_processor: a function taking four arguments:
-      1. tensor_dict: the same tensor_dict that is passed in as the first
-        argument to this function.
-      2. sess: a tensorflow session
-      3. batch_index: an integer representing the index of the batch amongst
-        all batches
-      By default, batch_processor is None, which defaults to running:
-        return sess.run(tensor_dict)
-      To skip an image, it suffices to return an empty dictionary in place of
-      result_dict.
-    checkpoint_dirs: list of directories to load into an EnsembleModel. If it
-      has only one directory, EnsembleModel will not be used --
-        a DetectionModel
-      will be instantiated directly. Not used if restore_fn is set.
-    variables_to_restore: None, or a dictionary mapping variable names found in
-      a checkpoint to model variables. The dictionary would normally be
-      generated by creating a tf.train.ExponentialMovingAverage object and
-      calling its variables_to_restore() method. Not used if restore_fn is set.
-    restore_fn: None, or a function that takes a tf.Session object and correctly
-      restores all necessary variables from the correct checkpoint file. If
-      None, attempts to restore from the first directory in checkpoint_dirs.
-    num_batches: the number of batches to use for evaluation.
-    master: the location of the Tensorflow session.
-    save_graph: whether or not the Tensorflow graph is stored as a pbtxt file.
-    save_graph_dir: where to store the Tensorflow graph on disk. If save_graph
-      is True this must be non-empty.
-    losses_dict: optional dictionary of scalar detection losses.
-
-  Returns:
-    global_step: the count of global steps.
-    all_evaluator_metrics: A dictionary containing metric names and values.
-
-  Raises:
-    ValueError: if restore_fn is None and checkpoint_dirs doesn't have at least
-      one element.
-    ValueError: if save_graph is True and save_graph_dir is not defined.
-  """
-  if save_graph and not save_graph_dir:
-    raise ValueError('`save_graph_dir` must be defined.')
-  config = tf.ConfigProto(inter_op_parallelism_threads=inter_op,
-                          intra_op_parallelism_threads=intra_op)  
-  sess = tf.Session(master, graph=tf.get_default_graph(), config=config)
-  sess.run(tf.global_variables_initializer())
-  sess.run(tf.local_variables_initializer())
-  sess.run(tf.tables_initializer())
-  if restore_fn:
-    restore_fn(sess)
-  else:
-    if not checkpoint_dirs:
-      raise ValueError('`checkpoint_dirs` must have at least one entry.')
-    checkpoint_file = tf.train.latest_checkpoint(checkpoint_dirs[0])
-    saver = tf.train.Saver(variables_to_restore)
-    saver.restore(sess, checkpoint_file)
-
-  if save_graph:
-    tf.train.write_graph(sess.graph_def, save_graph_dir, 'eval.pbtxt')
-
-  counters = {'skipped': 0, 'success': 0}
-  aggregate_result_losses_dict = collections.defaultdict(list)
-  with tf.contrib.slim.queues.QueueRunners(sess):
-    try:
-      loop_start_time = time.time()
-      for batch in range(int(num_batches)):
-        if (batch + 1) % 100 == 0:
-          logging.info('Running eval ops batch %d/%d', batch + 1, num_batches)
-        if not batch_processor:
-          try:
-            if not losses_dict:
-              losses_dict = {}
-            result_dict, result_losses_dict = sess.run([tensor_dict,
-                                                        losses_dict])
-            counters['success'] += 1
-          except tf.errors.InvalidArgumentError:
-            logging.info('Skipping image')
-            counters['skipped'] += 1
-            result_dict = {}
-        else:
-          result_dict, result_losses_dict = batch_processor(
-              tensor_dict, sess, batch, counters, losses_dict=losses_dict)
-        if not result_dict:
-          continue
-        for key, value in iter(result_losses_dict.items()):
-          aggregate_result_losses_dict[key].append(value)
-        for evaluator in evaluators:
-          # TODO(b/65130867): Use image_id tensor once we fix the input data
-          # decoders to return correct image_id.
-          # TODO(akuznetsa): result_dict contains batches of images, while
-          # add_single_ground_truth_image_info expects a single image. Fix
-          evaluator.add_single_ground_truth_image_info(
-              image_id=batch, groundtruth_dict=result_dict)
-          evaluator.add_single_detected_image_info(
-              image_id=batch, detections_dict=result_dict)
-      loop_end_time = time.time()
-      elapsed_time = loop_end_time - loop_start_time
-      print('Time spent : %.3f seconds.' % elapsed_time)
-      print('Time spent per BATCH: %.3f seconds.' % (elapsed_time / num_batches))
-      logging.info('Running eval batches done.')
-    except tf.errors.OutOfRangeError:
-      logging.info('Done evaluating -- epoch limit reached')
-    finally:
-      # When done, ask the threads to stop.
-      logging.info('# success: %d', counters['success'])
-      logging.info('# skipped: %d', counters['skipped'])
-      all_evaluator_metrics = {}
-      for evaluator in evaluators:
-        metrics = evaluator.evaluate()
-        evaluator.clear()
-        if any(key in all_evaluator_metrics for key in metrics):
-          raise ValueError('Metric names between evaluators must not collide.')
-        all_evaluator_metrics.update(metrics)
-      global_step = tf.train.global_step(sess, tf.train.get_global_step())
-
-      for key, value in iter(aggregate_result_losses_dict.items()):
-        all_evaluator_metrics['Losses/' + key] = np.mean(value)
-  sess.close()
-  return (global_step, all_evaluator_metrics)
-
-
-# TODO(rathodv): Add tests.
-def repeated_checkpoint_run(tensor_dict,
-                            summary_dir,
-                            evaluators,
-                            batch_processor=None,
-                            checkpoint_dirs=None,
-                            variables_to_restore=None,
-                            restore_fn=None,
-                            num_batches=1,
-                            eval_interval_secs=120,
-                            max_number_of_evaluations=None,
-                            master='',
-                            save_graph=False,
-                            save_graph_dir='',
-                            losses_dict=None,
-                            inter_op = 0,
-                            intra_op = 1):
-  """Periodically evaluates desired tensors using checkpoint_dirs or restore_fn.
-
-  This function repeatedly loads a checkpoint and evaluates a desired
-  set of tensors (provided by tensor_dict) and hands the resulting numpy
-  arrays to a function result_processor which can be used to further
-  process/save/visualize the results.
-
-  Args:
-    tensor_dict: a dictionary holding tensors representing a batch of detections
-      and corresponding groundtruth annotations.
-    summary_dir: a directory to write metrics summaries.
-    evaluators: a list of object of type DetectionEvaluator to be used for
-      evaluation. Note that the metric names produced by different evaluators
-      must be unique.
-    batch_processor: a function taking three arguments:
-      1. tensor_dict: the same tensor_dict that is passed in as the first
-        argument to this function.
-      2. sess: a tensorflow session
-      3. batch_index: an integer representing the index of the batch amongst
-        all batches
-      By default, batch_processor is None, which defaults to running:
-        return sess.run(tensor_dict)
-    checkpoint_dirs: list of directories to load into a DetectionModel or an
-      EnsembleModel if restore_fn isn't set. Also used to determine when to run
-      next evaluation. Must have at least one element.
-    variables_to_restore: None, or a dictionary mapping variable names found in
-      a checkpoint to model variables. The dictionary would normally be
-      generated by creating a tf.train.ExponentialMovingAverage object and
-      calling its variables_to_restore() method. Not used if restore_fn is set.
-    restore_fn: a function that takes a tf.Session object and correctly restores
-      all necessary variables from the correct checkpoint file.
-    num_batches: the number of batches to use for evaluation.
-    eval_interval_secs: the number of seconds between each evaluation run.
-    max_number_of_evaluations: the max number of iterations of the evaluation.
-      If the value is left as None the evaluation continues indefinitely.
-    master: the location of the Tensorflow session.
-    save_graph: whether or not the Tensorflow graph is saved as a pbtxt file.
-    save_graph_dir: where to save on disk the Tensorflow graph. If store_graph
-      is True this must be non-empty.
-    losses_dict: optional dictionary of scalar detection losses.
-
-  Returns:
-    metrics: A dictionary containing metric names and values in the latest
-      evaluation.
-
-  Raises:
-    ValueError: if max_num_of_evaluations is not None or a positive number.
-    ValueError: if checkpoint_dirs doesn't have at least one element.
-  """
-  if max_number_of_evaluations and max_number_of_evaluations <= 0:
-    raise ValueError(
-        '`number_of_steps` must be either None or a positive number.')
-
-  if not checkpoint_dirs:
-    raise ValueError('`checkpoint_dirs` must have at least one entry.')
-
-  last_evaluated_model_path = None
-  number_of_evaluations = 0
-  while True:
-    start = time.time()
-    logging.info('Starting evaluation at ' + time.strftime(
-        '%Y-%m-%d-%H:%M:%S', time.gmtime()))
-    model_path = tf.train.latest_checkpoint(checkpoint_dirs[0])
-    if not model_path:
-      logging.info('No model found in %s. Will try again in %d seconds',
-                   checkpoint_dirs[0], eval_interval_secs)
-    elif model_path == last_evaluated_model_path:
-      logging.info('Found already evaluated checkpoint. Will try again in %d '
-                   'seconds', eval_interval_secs)
-    else:
-      last_evaluated_model_path = model_path
-      global_step, metrics = _run_checkpoint_once(tensor_dict, evaluators,
-                                                  batch_processor,
-                                                  checkpoint_dirs,
-                                                  variables_to_restore,
-                                                  restore_fn, num_batches,
-                                                  master, save_graph,
-                                                  save_graph_dir,
-                                                  losses_dict=losses_dict,
-                                                  inter_op = inter_op,
-                                                  intra_op = intra_op)
-      write_metrics(metrics, global_step, summary_dir)
-    number_of_evaluations += 1
-
-    if (max_number_of_evaluations and
-        number_of_evaluations >= max_number_of_evaluations):
-      logging.info('Finished evaluation!')
-      break
-    time_to_next_eval = start + eval_interval_secs - time.time()
-    if time_to_next_eval > 0:
-      time.sleep(time_to_next_eval)
-
-  return metrics
-
-
-def result_dict_for_single_example(image,
-                                   key,
-                                   detections,
-                                   groundtruth=None,
-                                   class_agnostic=False,
-                                   scale_to_absolute=False):
-  """Merges all detection and groundtruth information for a single example.
-
-  Note that evaluation tools require classes that are 1-indexed, and so this
-  function performs the offset. If `class_agnostic` is True, all output classes
-  have label 1.
-
-  Args:
-    image: A single 4D uint8 image tensor of shape [1, H, W, C].
-    key: A single string tensor identifying the image.
-    detections: A dictionary of detections, returned from
-      DetectionModel.postprocess().
-    groundtruth: (Optional) Dictionary of groundtruth items, with fields:
-      'groundtruth_boxes': [num_boxes, 4] float32 tensor of boxes, in
-        normalized coordinates.
-      'groundtruth_classes': [num_boxes] int64 tensor of 1-indexed classes.
-      'groundtruth_area': [num_boxes] float32 tensor of bbox area. (Optional)
-      'groundtruth_is_crowd': [num_boxes] int64 tensor. (Optional)
-      'groundtruth_difficult': [num_boxes] int64 tensor. (Optional)
-      'groundtruth_group_of': [num_boxes] int64 tensor. (Optional)
-      'groundtruth_instance_masks': 3D int64 tensor of instance masks
-        (Optional).
-    class_agnostic: Boolean indicating whether the detections are class-agnostic
-      (i.e. binary). Default False.
-    scale_to_absolute: Boolean indicating whether boxes and keypoints should be
-      scaled to absolute coordinates. Note that for IoU based evaluations, it
-      does not matter whether boxes are expressed in absolute or relative
-      coordinates. Default False.
-
-  Returns:
-    A dictionary with:
-    'original_image': A [1, H, W, C] uint8 image tensor.
-    'key': A string tensor with image identifier.
-    'detection_boxes': [max_detections, 4] float32 tensor of boxes, in
-      normalized or absolute coordinates, depending on the value of
-      `scale_to_absolute`.
-    'detection_scores': [max_detections] float32 tensor of scores.
-    'detection_classes': [max_detections] int64 tensor of 1-indexed classes.
-    'detection_masks': [max_detections, H, W] float32 tensor of binarized
-      masks, reframed to full image masks.
-    'groundtruth_boxes': [num_boxes, 4] float32 tensor of boxes, in
-      normalized or absolute coordinates, depending on the value of
-      `scale_to_absolute`. (Optional)
-    'groundtruth_classes': [num_boxes] int64 tensor of 1-indexed classes.
-      (Optional)
-    'groundtruth_area': [num_boxes] float32 tensor of bbox area. (Optional)
-    'groundtruth_is_crowd': [num_boxes] int64 tensor. (Optional)
-    'groundtruth_difficult': [num_boxes] int64 tensor. (Optional)
-    'groundtruth_group_of': [num_boxes] int64 tensor. (Optional)
-    'groundtruth_instance_masks': 3D int64 tensor of instance masks
-      (Optional).
-
-  """
-  label_id_offset = 1  # Applying label id offset (b/63711816)
-
-  input_data_fields = fields.InputDataFields
-  output_dict = {
-      input_data_fields.original_image: image,
-      input_data_fields.key: key,
-  }
-
-  detection_fields = fields.DetectionResultFields
-  detection_boxes = detections[detection_fields.detection_boxes][0]
-  image_shape = tf.shape(image)
-  detection_scores = detections[detection_fields.detection_scores][0]
-
-  if class_agnostic:
-    detection_classes = tf.ones_like(detection_scores, dtype=tf.int64)
-  else:
-    detection_classes = (
-        tf.to_int64(detections[detection_fields.detection_classes][0]) +
-        label_id_offset)
-
-  num_detections = tf.to_int32(detections[detection_fields.num_detections][0])
-  detection_boxes = tf.slice(
-      detection_boxes, begin=[0, 0], size=[num_detections, -1])
-  detection_classes = tf.slice(
-      detection_classes, begin=[0], size=[num_detections])
-  detection_scores = tf.slice(
-      detection_scores, begin=[0], size=[num_detections])
-
-  if scale_to_absolute:
-    absolute_detection_boxlist = box_list_ops.to_absolute_coordinates(
-        box_list.BoxList(detection_boxes), image_shape[1], image_shape[2])
-    output_dict[detection_fields.detection_boxes] = (
-        absolute_detection_boxlist.get())
-  else:
-    output_dict[detection_fields.detection_boxes] = detection_boxes
-  output_dict[detection_fields.detection_classes] = detection_classes
-  output_dict[detection_fields.detection_scores] = detection_scores
-
-  if detection_fields.detection_masks in detections:
-    detection_masks = detections[detection_fields.detection_masks][0]
-    # TODO(rathodv): This should be done in model's postprocess
-    # function ideally.
-    detection_masks = tf.slice(
-        detection_masks, begin=[0, 0, 0], size=[num_detections, -1, -1])
-    detection_masks_reframed = ops.reframe_box_masks_to_image_masks(
-        detection_masks, detection_boxes, image_shape[1], image_shape[2])
-    detection_masks_reframed = tf.cast(
-        tf.greater(detection_masks_reframed, 0.5), tf.uint8)
-    output_dict[detection_fields.detection_masks] = detection_masks_reframed
-  if detection_fields.detection_keypoints in detections:
-    detection_keypoints = detections[detection_fields.detection_keypoints][0]
-    output_dict[detection_fields.detection_keypoints] = detection_keypoints
-    if scale_to_absolute:
-      absolute_detection_keypoints = keypoint_ops.scale(
-          detection_keypoints, image_shape[1], image_shape[2])
-      output_dict[detection_fields.detection_keypoints] = (
-          absolute_detection_keypoints)
-
-  if groundtruth:
-    if input_data_fields.groundtruth_instance_masks in groundtruth:
-      groundtruth[input_data_fields.groundtruth_instance_masks] = tf.cast(
-          groundtruth[input_data_fields.groundtruth_instance_masks], tf.uint8)
-    output_dict.update(groundtruth)
-    if scale_to_absolute:
-      groundtruth_boxes = groundtruth[input_data_fields.groundtruth_boxes]
-      absolute_gt_boxlist = box_list_ops.to_absolute_coordinates(
-          box_list.BoxList(groundtruth_boxes), image_shape[1], image_shape[2])
-      output_dict[input_data_fields.groundtruth_boxes] = (
-          absolute_gt_boxlist.get())
-    # For class-agnostic models, groundtruth classes all become 1.
-    if class_agnostic:
-      groundtruth_classes = groundtruth[input_data_fields.groundtruth_classes]
-      groundtruth_classes = tf.ones_like(groundtruth_classes, dtype=tf.int64)
-      output_dict[input_data_fields.groundtruth_classes] = groundtruth_classes
-
-  return output_dict
-
-
-def get_eval_metric_ops_for_evaluators(evaluation_metrics,
-                                       categories,
-                                       eval_dict,
-                                       include_metrics_per_category=False):
-  """Returns a dictionary of eval metric ops to use with `tf.EstimatorSpec`.
-
-  Args:
-    evaluation_metrics: List of evaluation metric names. Current options are
-      'coco_detection_metrics' and 'coco_mask_metrics'.
-    categories: A list of dicts, each of which has the following keys -
-        'id': (required) an integer id uniquely identifying this category.
-        'name': (required) string representing category name e.g., 'cat', 'dog'.
-    eval_dict: An evaluation dictionary, returned from
-      result_dict_for_single_example().
-    include_metrics_per_category: If True, include metrics for each category.
-
-  Returns:
-    A dictionary of metric names to tuple of value_op and update_op that can be
-    used as eval metric ops in tf.EstimatorSpec.
-
-  Raises:
-    ValueError: If any of the metrics in `evaluation_metric` is not
-    'coco_detection_metrics' or 'coco_mask_metrics'.
-  """
-  evaluation_metrics = list(set(evaluation_metrics))
-
-  input_data_fields = fields.InputDataFields
-  detection_fields = fields.DetectionResultFields
-  eval_metric_ops = {}
-  for metric in evaluation_metrics:
-    if metric == 'coco_detection_metrics':
-      coco_evaluator = coco_evaluation.CocoDetectionEvaluator(
-          categories, include_metrics_per_category=include_metrics_per_category)
-      eval_metric_ops.update(
-          coco_evaluator.get_estimator_eval_metric_ops(
-              image_id=eval_dict[input_data_fields.key],
-              groundtruth_boxes=eval_dict[input_data_fields.groundtruth_boxes],
-              groundtruth_classes=eval_dict[
-                  input_data_fields.groundtruth_classes],
-              detection_boxes=eval_dict[detection_fields.detection_boxes],
-              detection_scores=eval_dict[detection_fields.detection_scores],
-              detection_classes=eval_dict[detection_fields.detection_classes]))
-    elif metric == 'coco_mask_metrics':
-      coco_mask_evaluator = coco_evaluation.CocoMaskEvaluator(
-          categories, include_metrics_per_category=include_metrics_per_category)
-      eval_metric_ops.update(
-          coco_mask_evaluator.get_estimator_eval_metric_ops(
-              image_id=eval_dict[input_data_fields.key],
-              groundtruth_boxes=eval_dict[input_data_fields.groundtruth_boxes],
-              groundtruth_classes=eval_dict[
-                  input_data_fields.groundtruth_classes],
-              groundtruth_instance_masks=eval_dict[
-                  input_data_fields.groundtruth_instance_masks],
-              detection_scores=eval_dict[detection_fields.detection_scores],
-              detection_classes=eval_dict[detection_fields.detection_classes],
-              detection_masks=eval_dict[detection_fields.detection_masks]))
-    else:
-      raise ValueError('The only evaluation metrics supported are '
-                       '"coco_detection_metrics" and "coco_mask_metrics". '
-                       'Found {} in the evaluation metrics'.format(metric))
-
-  return eval_metric_ops
-
-
diff --git a/models/object_detection/tensorflow/faster_rcnn/inference/fp32/evaluator.py b/models/object_detection/tensorflow/faster_rcnn/inference/fp32/evaluator.py
deleted file mode 100644
index fcd1b45e6..000000000
--- a/models/object_detection/tensorflow/faster_rcnn/inference/fp32/evaluator.py
+++ /dev/null
@@ -1,296 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-# SPDX-License-Identifier: EPL-2.0
-#
-
-# Copyright 2017 The TensorFlow Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-"""Detection model evaluator.
-
-This file provides a generic evaluation method that can be used to evaluate a
-DetectionModel.
-"""
-
-import logging
-import tensorflow as tf
-
-import eval_util
-from object_detection.core import prefetcher
-from object_detection.core import standard_fields as fields
-from object_detection.metrics import coco_evaluation
-from object_detection.utils import object_detection_evaluation
-
-# A dictionary of metric names to classes that implement the metric. The classes
-# in the dictionary must implement
-# utils.object_detection_evaluation.DetectionEvaluator interface.
-EVAL_METRICS_CLASS_DICT = {
-    'pascal_voc_detection_metrics':
-        object_detection_evaluation.PascalDetectionEvaluator,
-    'weighted_pascal_voc_detection_metrics':
-        object_detection_evaluation.WeightedPascalDetectionEvaluator,
-    'pascal_voc_instance_segmentation_metrics':
-        object_detection_evaluation.PascalInstanceSegmentationEvaluator,
-    'weighted_pascal_voc_instance_segmentation_metrics':
-        object_detection_evaluation.WeightedPascalInstanceSegmentationEvaluator,
-    'open_images_detection_metrics':
-        object_detection_evaluation.OpenImagesDetectionEvaluator,
-    'coco_detection_metrics':
-        coco_evaluation.CocoDetectionEvaluator,
-    'coco_mask_metrics':
-        coco_evaluation.CocoMaskEvaluator,
-}
-
-EVAL_DEFAULT_METRIC = 'pascal_voc_detection_metrics'
-
-
-def _extract_predictions_and_losses(model,
-                                    create_input_dict_fn,
-                                    ignore_groundtruth=False):
-  """Constructs tensorflow detection graph and returns output tensors.
-
-  Args:
-    model: model to perform predictions with.
-    create_input_dict_fn: function to create input tensor dictionaries.
-    ignore_groundtruth: whether groundtruth should be ignored.
-
-  Returns:
-    prediction_groundtruth_dict: A dictionary with postprocessed tensors (keyed
-      by standard_fields.DetectionResultsFields) and optional groundtruth
-      tensors (keyed by standard_fields.InputDataFields).
-    losses_dict: A dictionary containing detection losses. This is empty when
-      ignore_groundtruth is true.
-  """
-  input_dict = create_input_dict_fn()
-  prefetch_queue = prefetcher.prefetch(input_dict, capacity=500)
-  input_dict = prefetch_queue.dequeue()
-  original_image = tf.expand_dims(input_dict[fields.InputDataFields.image], 0)
-  preprocessed_image, true_image_shapes = model.preprocess(
-      tf.to_float(original_image))
-  prediction_dict = model.predict(preprocessed_image, true_image_shapes)
-  detections = model.postprocess(prediction_dict, true_image_shapes)
-
-  groundtruth = None
-  losses_dict = {}
-  if not ignore_groundtruth:
-    groundtruth = {
-        fields.InputDataFields.groundtruth_boxes:
-            input_dict[fields.InputDataFields.groundtruth_boxes],
-        fields.InputDataFields.groundtruth_classes:
-            input_dict[fields.InputDataFields.groundtruth_classes],
-        fields.InputDataFields.groundtruth_area:
-            input_dict[fields.InputDataFields.groundtruth_area],
-        fields.InputDataFields.groundtruth_is_crowd:
-            input_dict[fields.InputDataFields.groundtruth_is_crowd],
-        fields.InputDataFields.groundtruth_difficult:
-            input_dict[fields.InputDataFields.groundtruth_difficult]
-    }
-    if fields.InputDataFields.groundtruth_group_of in input_dict:
-      groundtruth[fields.InputDataFields.groundtruth_group_of] = (
-          input_dict[fields.InputDataFields.groundtruth_group_of])
-    groundtruth_masks_list = None
-    if fields.DetectionResultFields.detection_masks in detections:
-      groundtruth[fields.InputDataFields.groundtruth_instance_masks] = (
-          input_dict[fields.InputDataFields.groundtruth_instance_masks])
-      groundtruth_masks_list = [
-          input_dict[fields.InputDataFields.groundtruth_instance_masks]]
-    groundtruth_keypoints_list = None
-    if fields.DetectionResultFields.detection_keypoints in detections:
-      groundtruth[fields.InputDataFields.groundtruth_keypoints] = (
-          input_dict[fields.InputDataFields.groundtruth_keypoints])
-      groundtruth_keypoints_list = [
-          input_dict[fields.InputDataFields.groundtruth_keypoints]]
-    label_id_offset = 1
-    model.provide_groundtruth(
-        [input_dict[fields.InputDataFields.groundtruth_boxes]],
-        [tf.one_hot(input_dict[fields.InputDataFields.groundtruth_classes]
-                    - label_id_offset, depth=model.num_classes)],
-        groundtruth_masks_list, groundtruth_keypoints_list)
-    losses_dict.update(model.loss(prediction_dict, true_image_shapes))
-
-  result_dict = eval_util.result_dict_for_single_example(
-      original_image,
-      input_dict[fields.InputDataFields.source_id],
-      detections,
-      groundtruth,
-      class_agnostic=(
-          fields.DetectionResultFields.detection_classes not in detections),
-      scale_to_absolute=True)
-  return result_dict, losses_dict
-
-
-def get_evaluators(eval_config, categories):
-  """Returns the evaluator class according to eval_config, valid for categories.
-
-  Args:
-    eval_config: evaluation configurations.
-    categories: a list of categories to evaluate.
-  Returns:
-    An list of instances of DetectionEvaluator.
-
-  Raises:
-    ValueError: if metric is not in the metric class dictionary.
-  """
-  eval_metric_fn_keys = eval_config.metrics_set
-  if not eval_metric_fn_keys:
-    eval_metric_fn_keys = [EVAL_DEFAULT_METRIC]
-  evaluators_list = []
-  for eval_metric_fn_key in eval_metric_fn_keys:
-    if eval_metric_fn_key not in EVAL_METRICS_CLASS_DICT:
-      raise ValueError('Metric not found: {}'.format(eval_metric_fn_key))
-    evaluators_list.append(
-        EVAL_METRICS_CLASS_DICT[eval_metric_fn_key](categories=categories))
-  return evaluators_list
-
-
-def evaluate(create_input_dict_fn, create_model_fn, eval_config, categories,
-             checkpoint_dir, eval_dir, inter_op = 0, intra_op = 1, graph_hook_fn=None, evaluator_list=None):
-  """Evaluation function for detection models.
-
-  Args:
-    create_input_dict_fn: a function to create a tensor input dictionary.
-    create_model_fn: a function that creates a DetectionModel.
-    eval_config: a eval_pb2.EvalConfig protobuf.
-    categories: a list of category dictionaries. Each dict in the list should
-                have an integer 'id' field and string 'name' field.
-    checkpoint_dir: directory to load the checkpoints to evaluate from.
-    eval_dir: directory to write evaluation metrics summary to.
-    graph_hook_fn: Optional function that is called after the training graph is
-      completely built. This is helpful to perform additional changes to the
-      training graph such as optimizing batchnorm. The function should modify
-      the default graph.
-    evaluator_list: Optional list of instances of DetectionEvaluator. If not
-      given, this list of metrics is created according to the eval_config.
-
-  Returns:
-    metrics: A dictionary containing metric names and values from the latest
-      run.
-  """
-
-  model = create_model_fn()
-
-  if eval_config.ignore_groundtruth and not eval_config.export_path:
-    logging.fatal('If ignore_groundtruth=True then an export_path is '
-                  'required. Aborting!!!')
-
-  tensor_dict, losses_dict = _extract_predictions_and_losses(
-      model=model,
-      create_input_dict_fn=create_input_dict_fn,
-      ignore_groundtruth=eval_config.ignore_groundtruth)
-
-  def _process_batch(tensor_dict, sess, batch_index, counters,
-                     losses_dict=None):
-    """Evaluates tensors in tensor_dict, losses_dict and visualizes examples.
-
-    This function calls sess.run on tensor_dict, evaluating the original_image
-    tensor only on the first K examples and visualizing detections overlaid
-    on this original_image.
-
-    Args:
-      tensor_dict: a dictionary of tensors
-      sess: tensorflow session
-      batch_index: the index of the batch amongst all batches in the run.
-      counters: a dictionary holding 'success' and 'skipped' fields which can
-        be updated to keep track of number of successful and failed runs,
-        respectively.  If these fields are not updated, then the success/skipped
-        counter values shown at the end of evaluation will be incorrect.
-      losses_dict: Optional dictonary of scalar loss tensors.
-
-    Returns:
-      result_dict: a dictionary of numpy arrays
-      result_losses_dict: a dictionary of scalar losses. This is empty if input
-        losses_dict is None.
-    """
-    try:
-      if not losses_dict:
-        losses_dict = {}
-      result_dict, result_losses_dict = sess.run([tensor_dict, losses_dict])
-      counters['success'] += 1
-    except tf.errors.InvalidArgumentError:
-      logging.info('Skipping image')
-      counters['skipped'] += 1
-      return {}, {}
-    global_step = tf.train.global_step(sess, tf.train.get_global_step())
-    if batch_index < eval_config.num_visualizations:
-      tag = 'image-{}'.format(batch_index)
-      eval_util.visualize_detection_results(
-          result_dict,
-          tag,
-          global_step,
-          categories=categories,
-          summary_dir=eval_dir,
-          export_dir=eval_config.visualization_export_dir,
-          show_groundtruth=eval_config.visualize_groundtruth_boxes,
-          groundtruth_box_visualization_color=eval_config.
-          groundtruth_box_visualization_color,
-          min_score_thresh=eval_config.min_score_threshold,
-          max_num_predictions=eval_config.max_num_boxes_to_visualize,
-          skip_scores=eval_config.skip_scores,
-          skip_labels=eval_config.skip_labels,
-          keep_image_id_for_visualization_export=eval_config.
-          keep_image_id_for_visualization_export)
-    return result_dict, result_losses_dict
-
-  variables_to_restore = tf.global_variables()
-  global_step = tf.train.get_or_create_global_step()
-  variables_to_restore.append(global_step)
-
-  if graph_hook_fn: graph_hook_fn()
-
-  if eval_config.use_moving_averages:
-    variable_averages = tf.train.ExponentialMovingAverage(0.0)
-    variables_to_restore = variable_averages.variables_to_restore()
-  saver = tf.train.Saver(variables_to_restore)
-
-  def _restore_latest_checkpoint(sess):
-    latest_checkpoint = tf.train.latest_checkpoint(checkpoint_dir)
-    saver.restore(sess, latest_checkpoint)
-
-  if not evaluator_list:
-    evaluator_list = get_evaluators(eval_config, categories)
-
-  metrics = eval_util.repeated_checkpoint_run(
-      tensor_dict=tensor_dict,
-      summary_dir=eval_dir,
-      evaluators=evaluator_list,
-      batch_processor=_process_batch,
-      checkpoint_dirs=[checkpoint_dir],
-      variables_to_restore=None,
-      restore_fn=_restore_latest_checkpoint,
-      num_batches=eval_config.num_examples,
-      eval_interval_secs=eval_config.eval_interval_secs,
-      max_number_of_evaluations=(1 if eval_config.ignore_groundtruth else
-                                 eval_config.max_evals
-                                 if eval_config.max_evals else None),
-      master=eval_config.eval_master,
-      save_graph=eval_config.save_graph,
-      save_graph_dir=(eval_dir if eval_config.save_graph else ''),
-      losses_dict=losses_dict, inter_op = inter_op, intra_op = intra_op)
-
-  return metrics
diff --git a/models/object_detection/tensorflow/faster_rcnn/inference/fp32/faster_rcnn.patch b/models/object_detection/tensorflow/faster_rcnn/inference/fp32/faster_rcnn.patch
deleted file mode 100644
index 1c93bc165..000000000
--- a/models/object_detection/tensorflow/faster_rcnn/inference/fp32/faster_rcnn.patch
+++ /dev/null
@@ -1,69 +0,0 @@
-diff --git a/research/object_detection/inference/detection_inference.py b/research/object_detection/inference/detection_inference.py
-index dc66686f..ccd8b3ae 100644
---- a/research/object_detection/inference/detection_inference.py
-+++ b/research/object_detection/inference/detection_inference.py
-@@ -65,7 +65,7 @@ def build_inference_graph(image_tensor, inference_graph_path):
-     detected_labels_tensor: Detected labels. Int64 tensor,
-         shape=[num_detections]
-   """
--  with tf.gfile.Open(inference_graph_path, 'r') as graph_def_file:
-+  with tf.gfile.Open(inference_graph_path, 'rb') as graph_def_file:
-     graph_content = graph_def_file.read()
-   graph_def = tf.GraphDef()
-   graph_def.MergeFromString(graph_content)
-diff --git a/research/object_detection/metrics/offline_eval_map_corloc.py b/research/object_detection/metrics/offline_eval_map_corloc.py
-old mode 100644
-new mode 100755
-index b5514be1..b498ddab
---- a/research/object_detection/metrics/offline_eval_map_corloc.py
-+++ b/research/object_detection/metrics/offline_eval_map_corloc.py
-@@ -88,11 +88,11 @@ def read_data_and_evaluate(input_config, eval_config):
-   Raises:
-     ValueError: if input_reader type is not supported or metric type is unknown.
-   """
--  if input_config.WhichOneof('input_reader') == 'tf_record_input_reader':
--    input_paths = input_config.tf_record_input_reader.input_path
-+  if input_config[0].WhichOneof('input_reader') == 'tf_record_input_reader':
-+    input_paths = input_config[0].tf_record_input_reader.input_path
- 
-     categories = label_map_util.create_categories_from_labelmap(
--        input_config.label_map_path)
-+        input_config[0].label_map_path)
- 
-     object_detection_evaluators = evaluator.get_evaluators(
-         eval_config, categories)
-@@ -159,7 +159,7 @@ def main(argv):
-       eval_config_path=FLAGS.eval_config_path)
- 
-   eval_config = configs['eval_config']
--  input_config = configs['eval_input_config']
-+  input_config = configs['eval_input_configs']
- 
-   metrics = read_data_and_evaluate(input_config, eval_config)
- 
-diff --git a/research/object_detection/metrics/tf_example_parser.py b/research/object_detection/metrics/tf_example_parser.py
-index 9a5f130f..fa361bf5 100644
---- a/research/object_detection/metrics/tf_example_parser.py
-+++ b/research/object_detection/metrics/tf_example_parser.py
-@@ -44,7 +44,7 @@ class StringParser(data_parser.DataToNumpyParser):
-     self.field_name = field_name
- 
-   def parse(self, tf_example):
--    return "".join(tf_example.features.feature[self.field_name]
-+    return b"".join(tf_example.features.feature[self.field_name]
-                    .bytes_list.value) if tf_example.features.feature[
-                        self.field_name].HasField("bytes_list") else None
- 
-diff --git a/research/object_detection/utils/object_detection_evaluation.py b/research/object_detection/utils/object_detection_evaluation.py
-index d65c69fb..2b36de4b 100644
---- a/research/object_detection/utils/object_detection_evaluation.py
-+++ b/research/object_detection/utils/object_detection_evaluation.py
-@@ -304,7 +304,7 @@ class ObjectDetectionEvaluator(DetectionEvaluator):
-       if idx + self._label_id_offset in category_index:
-         category_name = category_index[idx + self._label_id_offset]['name']
-         try:
--          category_name = unicode(category_name, 'utf-8')
-+          category_name = str(category_name, 'utf-8')
-         except TypeError:
-           pass
-         category_name = unicodedata.normalize(
diff --git a/models/object_detection/tensorflow/faster_rcnn/inference/int8/coco_int8.sh b/models/object_detection/tensorflow/faster_rcnn/inference/int8/coco_int8.sh
deleted file mode 100755
index 4328ab9fb..000000000
--- a/models/object_detection/tensorflow/faster_rcnn/inference/int8/coco_int8.sh
+++ /dev/null
@@ -1,55 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2019 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-# SPDX-License-Identifier: EPL-2.0
-#
-
-######## Variables to be defined - run it in research folder
-
-SPLIT=faster-rcnn1 #change to your favorite name
-FROZEN_GRAPH=$1
-########### 
-
-TF_RECORD_FILES=$2
-TF_MODEL_ROOT=$3
-
-export PYTHONPATH=$PYTHONPATH:`pwd`:`pwd`/slim
-export PYTHONPATH=$PYTHONPATH:`pwd`:`pwd`/object_detection
-
-python object_detection/inference/infer_detections.py \
-  --input_tfrecord_paths=$TF_RECORD_FILES \
-  --output_tfrecord_path=${SPLIT}_detections.tfrecord \
-  --inference_graph=$FROZEN_GRAPH \
-  --discard_image_pixels=True
-
-
-mkdir -p ${SPLIT}_eval_metrics
-
-echo "
-label_map_path: '${TF_MODEL_ROOT}/research/object_detection/data/mscoco_label_map.pbtxt'
-tf_record_input_reader: { input_path: '${SPLIT}_detections.tfrecord' }
-" > ${SPLIT}_eval_metrics/${SPLIT}_input_config.pbtxt
-
-echo "
-metrics_set: 'coco_detection_metrics'
-" > ${SPLIT}_eval_metrics/${SPLIT}_eval_config.pbtxt
-
-
-python object_detection/metrics/offline_eval_map_corloc.py \
-  --eval_dir=${SPLIT}_eval_metrics \
-  --eval_config_path=${SPLIT}_eval_metrics/${SPLIT}_eval_config.pbtxt \
-  --input_config_path=${SPLIT}_eval_metrics/${SPLIT}_input_config.pbtxt
diff --git a/models/object_detection/tensorflow/faster_rcnn/inference/int8/faster_rcnn.patch b/models/object_detection/tensorflow/faster_rcnn/inference/int8/faster_rcnn.patch
deleted file mode 100644
index 1c93bc165..000000000
--- a/models/object_detection/tensorflow/faster_rcnn/inference/int8/faster_rcnn.patch
+++ /dev/null
@@ -1,69 +0,0 @@
-diff --git a/research/object_detection/inference/detection_inference.py b/research/object_detection/inference/detection_inference.py
-index dc66686f..ccd8b3ae 100644
---- a/research/object_detection/inference/detection_inference.py
-+++ b/research/object_detection/inference/detection_inference.py
-@@ -65,7 +65,7 @@ def build_inference_graph(image_tensor, inference_graph_path):
-     detected_labels_tensor: Detected labels. Int64 tensor,
-         shape=[num_detections]
-   """
--  with tf.gfile.Open(inference_graph_path, 'r') as graph_def_file:
-+  with tf.gfile.Open(inference_graph_path, 'rb') as graph_def_file:
-     graph_content = graph_def_file.read()
-   graph_def = tf.GraphDef()
-   graph_def.MergeFromString(graph_content)
-diff --git a/research/object_detection/metrics/offline_eval_map_corloc.py b/research/object_detection/metrics/offline_eval_map_corloc.py
-old mode 100644
-new mode 100755
-index b5514be1..b498ddab
---- a/research/object_detection/metrics/offline_eval_map_corloc.py
-+++ b/research/object_detection/metrics/offline_eval_map_corloc.py
-@@ -88,11 +88,11 @@ def read_data_and_evaluate(input_config, eval_config):
-   Raises:
-     ValueError: if input_reader type is not supported or metric type is unknown.
-   """
--  if input_config.WhichOneof('input_reader') == 'tf_record_input_reader':
--    input_paths = input_config.tf_record_input_reader.input_path
-+  if input_config[0].WhichOneof('input_reader') == 'tf_record_input_reader':
-+    input_paths = input_config[0].tf_record_input_reader.input_path
- 
-     categories = label_map_util.create_categories_from_labelmap(
--        input_config.label_map_path)
-+        input_config[0].label_map_path)
- 
-     object_detection_evaluators = evaluator.get_evaluators(
-         eval_config, categories)
-@@ -159,7 +159,7 @@ def main(argv):
-       eval_config_path=FLAGS.eval_config_path)
- 
-   eval_config = configs['eval_config']
--  input_config = configs['eval_input_config']
-+  input_config = configs['eval_input_configs']
- 
-   metrics = read_data_and_evaluate(input_config, eval_config)
- 
-diff --git a/research/object_detection/metrics/tf_example_parser.py b/research/object_detection/metrics/tf_example_parser.py
-index 9a5f130f..fa361bf5 100644
---- a/research/object_detection/metrics/tf_example_parser.py
-+++ b/research/object_detection/metrics/tf_example_parser.py
-@@ -44,7 +44,7 @@ class StringParser(data_parser.DataToNumpyParser):
-     self.field_name = field_name
- 
-   def parse(self, tf_example):
--    return "".join(tf_example.features.feature[self.field_name]
-+    return b"".join(tf_example.features.feature[self.field_name]
-                    .bytes_list.value) if tf_example.features.feature[
-                        self.field_name].HasField("bytes_list") else None
- 
-diff --git a/research/object_detection/utils/object_detection_evaluation.py b/research/object_detection/utils/object_detection_evaluation.py
-index d65c69fb..2b36de4b 100644
---- a/research/object_detection/utils/object_detection_evaluation.py
-+++ b/research/object_detection/utils/object_detection_evaluation.py
-@@ -304,7 +304,7 @@ class ObjectDetectionEvaluator(DetectionEvaluator):
-       if idx + self._label_id_offset in category_index:
-         category_name = category_index[idx + self._label_id_offset]['name']
-         try:
--          category_name = unicode(category_name, 'utf-8')
-+          category_name = str(category_name, 'utf-8')
-         except TypeError:
-           pass
-         category_name = unicodedata.normalize(
diff --git a/models/object_detection/tensorflow/faster_rcnn/inference/int8/run_frozen_graph_rcnn.py b/models/object_detection/tensorflow/faster_rcnn/inference/int8/run_frozen_graph_rcnn.py
deleted file mode 100644
index 1e4a03d9c..000000000
--- a/models/object_detection/tensorflow/faster_rcnn/inference/int8/run_frozen_graph_rcnn.py
+++ /dev/null
@@ -1,227 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2019 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-# SPDX-License-Identifier: EPL-2.0
-#
-
-import numpy as np
-import os
-import six.moves.urllib as urllib
-import sys
-import tarfile
-import tensorflow as tf
-import zipfile
-
-from collections import defaultdict
-from io import StringIO
-import matplotlib
-matplotlib.use('Agg')
-from matplotlib import pyplot as plt
-from PIL import Image
-import time
-import argparse
-from tensorflow.python.client import timeline
-
-
-parser = argparse.ArgumentParser()
-parser.add_argument('-g', '--graph', help='Path to input graph to run', type=str, required=True)
-parser.add_argument('-d', '--dataset', help='Full Path to input dataset to run', type=str, required=True)
-parser.add_argument('-s', '--single_image', help='Run for single image onle, if false, run for the whole dataset', action='store_true')
-parser.add_argument('-x', '--single_socket', help='Run for single socket, if false, run both sockets', action='store_true')
-parser.add_argument('-v', '--visualize', help='Whether to visulize the output image', action='store_true' )
-parser.add_argument('-t', '--timeline', help='Output file name for TF timeline', type=str, default=None)
-parser.add_argument('-e', '--evaluate_tensor', help='Full tensor name to evaluate', type=str, default=None)
-parser.add_argument('-p', '--print_accuracy', help='Print accuracy results', action='store_true')
-parser.add_argument('-n', '--number_of_steps', help='Run for n number of steps', type=int, default=None)
-parser.add_argument('--num-inter-threads', help='Num inter threads', type=int, default=None, dest="num_inter_threads")
-parser.add_argument('--num-intra-threads', help='Num intra threads', type=int, default=None, dest="num_intra_threads")
-
-args = parser.parse_args()
-
-
-# This is needed since the notebook is stored in the object_detection folder.
-sys.path.append("..")
-
-# This is needed to display the images.
-if (args.visualize and args.single_image and args.evaluate_tensor is None):
-  from IPython import get_ipython
-  get_ipython().run_line_magic('matplotlib', 'tk')
-
-import importlib
-label_map_util = importlib.import_module('..label_map_util', package='object_detection.utils.label_map_util')
-vis_util = importlib.import_module('..visualization_utils', package='object_detection.utils.visualization_utils')
-
-# List of the strings that is used to add correct label for each box.
-PATH_TO_LABELS = os.path.join('object_detection/data', 'mscoco_label_map.pbtxt')
-
-NUM_CLASSES = 90
-
-detection_graph = tf.Graph()
-with detection_graph.as_default():
-  od_graph_def = tf.GraphDef()
-  with tf.gfile.GFile(args.graph, 'rb') as fid:
-    serialized_graph = fid.read()
-    od_graph_def.ParseFromString(serialized_graph)
-    tf.import_graph_def(od_graph_def, name='')
-
-label_map = label_map_util.load_labelmap(PATH_TO_LABELS)
-categories = label_map_util.convert_label_map_to_categories(
-    label_map, max_num_classes=NUM_CLASSES, use_display_name=True)
-category_index = label_map_util.create_category_index(categories)
-
-def load_image_into_numpy_array(image):
-  (im_width, im_height) = image.size
-  if image.mode == 'L':
-    np_image = np.array(image.getdata()).reshape(
-        (im_height, im_width)).astype(np.uint8)
-    return np.stack((np_image,)*3, -1)
-  else:
-    return np.array(image.getdata()).reshape(
-        (im_height, im_width, 3)).astype(np.uint8)
-
-if (args.single_image):
-  TEST_IMAGE_PATHS = [args.dataset + "/000000578871.jpg"]  
-else:
-  PATH_TO_TEST_IMAGES_DIR = args.dataset 
-  print(PATH_TO_TEST_IMAGES_DIR)
-  TEST_IMAGE_PATHS = []
-  for root, dirs, files in os.walk(PATH_TO_TEST_IMAGES_DIR):
-    for file in files:
-      TEST_IMAGE_PATHS.append(os.path.join(PATH_TO_TEST_IMAGES_DIR, file))
-
-# Size, in inches, of the output images.
-IMAGE_SIZE = (12, 8)
-
-
-def run_inference_for_single_image(graph):
-  sess_config = tf.ConfigProto()
-  sess_config.intra_op_parallelism_threads = args.num_intra_threads
-  sess_config.inter_op_parallelism_threads = args.num_inter_threads
-  if not os.environ.get("OMP_NUM_THREADS"):
-    os.environ["OMP_NUM_THREADS"] = args.num_intra_threads
-
-  with graph.as_default():
-    with tf.Session(config=sess_config) as sess:
-      # Get handles to input and output tensors
-      tensor_dict = {}
-      if (args.evaluate_tensor is None):
-        ops = tf.get_default_graph().get_operations()
-        all_tensor_names = {output.name for op in ops for output in op.outputs}
-        for key in [
-            'num_detections', 'detection_boxes', 'detection_scores',
-            'detection_classes' 
-        ]:
-          tensor_name = key + ':0'
-          if tensor_name in all_tensor_names:
-            tensor_dict[key] = tf.get_default_graph().get_tensor_by_name(
-                tensor_name)
-      else:
-        our_op = tf.get_default_graph().get_operation_by_name(args.evaluate_tensor)
-        tensor_names = our_op.outputs
-        list_ops = []
-        for i, tensor in enumerate(tensor_names):
-          list_ops.append(tensor.name)
-        tensor_dict[args.evaluate_tensor] = list_ops
-
-      if (args.timeline is not None):
-        run_options = tf.RunOptions(trace_level=tf.RunOptions.FULL_TRACE)
-        run_metadata = tf.RunMetadata()
-      total_duration = 0
-      for index, image_path in enumerate(TEST_IMAGE_PATHS):
-        image = Image.open(image_path)
-        # the array based representation of the image will be used later in order to prepare the
-        # result image with boxes and labels on it.
-        image_np = load_image_into_numpy_array(image)
-        image_tensor = tf.get_default_graph().get_tensor_by_name('image_tensor:0')
-
-        # Run inference
-        start_time = time.time()
-        if (args.timeline is not None):
-          output_dict = sess.run(tensor_dict,
-                                feed_dict={image_tensor: np.expand_dims(image_np, 0)}, options=run_options, run_metadata=run_metadata)
-        else:
-          output_dict = sess.run(tensor_dict,
-                                 feed_dict={image_tensor: np.expand_dims(image_np, 0)})
-        step_duration = time.time() - start_time
-        if(index > 20):
-          total_duration = total_duration + step_duration
-
-        if (args.single_image):
-          if index == 0:
-            print('Avg. Duration per Step:' + str(total_duration / 1))
-        else:
-          if (index % 10 == 0):
-            print('Step ' + str(index) + ': ' + str(step_duration) + ' seconds')
-          if index == 4999:
-            print('Avg. Duration per Step:' + str(total_duration / 5000))
-
-        # Flush print messages
-        sys.stdout.flush()
-
-        if (args.number_of_steps is not None):
-          if (args.single_image):
-            sys.exit("single_iamge and number_of_steps cannot be both enabled!")
-          elif (index == (args.number_of_steps - 1)):
-            print('Avg. Duration per Step:' +
-                  str(total_duration / (args.number_of_steps - 20)))
-            break
-
-        if (args.timeline is not None):
-          trace = timeline.Timeline(step_stats=run_metadata.step_stats)
-          with open('tl-' + time.strftime("%Y%m%d-%H%M%S") + '-' + args.timeline, 'w') as file:
-            file.write(trace.generate_chrome_trace_format(show_memory=False))
-
-
-        if (args.evaluate_tensor is not None):
-          for tensor in output_dict[args.evaluate_tensor]:
-            print(tensor.shape)
-          return None, None
-
-        # all outputs are float32 numpy arrays, so convert types as appropriate
-        output_dict['num_detections'] = int(output_dict['num_detections'][0])
-        output_dict['detection_classes'] = output_dict[
-            'detection_classes'][0].astype(np.uint8)
-        output_dict['detection_boxes'] = output_dict['detection_boxes'][0]
-        output_dict['detection_scores'] = output_dict['detection_scores'][0]
-
-        if (args.print_accuracy):
-          print('num_detections:\n' + str(output_dict['num_detections']))
-          print('detection_classes:\n' + str(output_dict['detection_classes']))
-          print('detection_boxes:\n' + str(output_dict['detection_boxes']))
-          print('detection_scores:\n' + str(output_dict['detection_scores']))
-
-        if 'detection_masks' in output_dict:
-          output_dict['detection_masks'] = output_dict['detection_masks'][0]
-  return output_dict, image_np
-
-
-# Actual detection.
-output_dict, image_np = run_inference_for_single_image(detection_graph)
-
-# Visualization of the results of a detection.
-if (args.visualize and args.single_image and args.evaluate_tensor is None):
-  vis_util.visualize_boxes_and_labels_on_image_array(
-      image_np,
-      output_dict['detection_boxes'],
-      output_dict['detection_classes'],
-      output_dict['detection_scores'],
-      category_index,
-      instance_masks=output_dict.get('detection_masks'),
-      use_normalized_coordinates=True,
-      line_thickness=8)
-  plt.figure(figsize=IMAGE_SIZE)
-  plt.imshow(image_np)
diff --git a/models/object_detection/tensorflow/rfcn/__init__.py b/models/object_detection/tensorflow/rfcn/__init__.py
deleted file mode 100644
index f730c0c1c..000000000
--- a/models/object_detection/tensorflow/rfcn/__init__.py
+++ /dev/null
@@ -1,20 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
-
diff --git a/models/object_detection/tensorflow/rfcn/inference/__init__.py b/models/object_detection/tensorflow/rfcn/inference/__init__.py
deleted file mode 100644
index f730c0c1c..000000000
--- a/models/object_detection/tensorflow/rfcn/inference/__init__.py
+++ /dev/null
@@ -1,20 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
-
diff --git a/models/object_detection/tensorflow/rfcn/inference/fp32/__init__.py b/models/object_detection/tensorflow/rfcn/inference/fp32/__init__.py
deleted file mode 100644
index f730c0c1c..000000000
--- a/models/object_detection/tensorflow/rfcn/inference/fp32/__init__.py
+++ /dev/null
@@ -1,20 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
-
diff --git a/models/object_detection/tensorflow/rfcn/inference/fp32/coco_mAP.sh b/models/object_detection/tensorflow/rfcn/inference/fp32/coco_mAP.sh
deleted file mode 100755
index d52f04398..000000000
--- a/models/object_detection/tensorflow/rfcn/inference/fp32/coco_mAP.sh
+++ /dev/null
@@ -1,64 +0,0 @@
-#!/usr/bin/env bash
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
-
-########## Variables to be defined
-
-SPLIT=${SPLIT:-"RFCN_final_graph"} #change to your favorite room
-FROZEN_GRAPH=${FROZEN_GRAPH:-"/in_graph/frozen_inference_graph.pb"}
-TF_RECORD_FILES=${TF_RECORD_FILES:-"/dataset/coco_val.record"}
-if [[ -z ${TF_MODELS_ROOT} ]] || [[ ! -d ${TF_MODELS_ROOT} ]]; then
-  echo "You must specify the root of the tensorflow/models source tree in the TF_MODELS_ROOT environment variable."
-  exit 1
-fi
-
-export PYTHONPATH=$PYTHONPATH:${TF_MODELS_ROOT}/research:${TF_MODELS_ROOT}/research/slim:${TF_MODELS_ROOT}/research/object_detection
-
-echo "PWD=$PWD"
-echo "SPLIT=${SPLIT}"
-echo "FROZEN_GRAPH=${FROZEN_GRAPH}"
-echo "TF_RECORD_FILES=${TF_RECORD_FILES}"
-echo "PYTHONPATH=${PYTHONPATH}"
-echo "TF_MODELS_ROOT=$TF_MODELS_ROOT"
-
-python -m object_detection.inference.infer_detections \
-  --input_tfrecord_paths=$TF_RECORD_FILES \
-  --output_tfrecord_path=${SPLIT}_detections.tfrecord \
-  --inference_graph=$FROZEN_GRAPH \
-  --discard_image_pixels
-
-
-mkdir -p ${SPLIT}_eval_metrics
-
-echo "
-label_map_path: '${TF_MODELS_ROOT}/research/object_detection/data/mscoco_label_map.pbtxt'
-tf_record_input_reader: { input_path: '${SPLIT}_detections.tfrecord' }
-" > ${SPLIT}_eval_metrics/$(basename ${SPLIT})_input_config.pbtxt
-
-echo "
-metrics_set: 'coco_detection_metrics'
-" > ${SPLIT}_eval_metrics/$(basename ${SPLIT})_eval_config.pbtxt
-
-
-python -m object_detection.metrics.offline_eval_map_corloc \
-  --eval_dir=${SPLIT}_eval_metrics \
-  --eval_config_path=${SPLIT}_eval_metrics/$(basename ${SPLIT})_eval_config.pbtxt \
-  --input_config_path=${SPLIT}_eval_metrics/$(basename ${SPLIT})_input_config.pbtxt
-
diff --git a/models/object_detection/tensorflow/rfcn/inference/fp32/dataset_util.py b/models/object_detection/tensorflow/rfcn/inference/fp32/dataset_util.py
deleted file mode 100644
index d607ba372..000000000
--- a/models/object_detection/tensorflow/rfcn/inference/fp32/dataset_util.py
+++ /dev/null
@@ -1,160 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
-
-# Copyright 2017 The TensorFlow Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-
-"""Utility functions for creating TFRecord data sets."""
-
-import tensorflow as tf
-
-
-def int64_feature(value):
-  return tf.train.Feature(int64_list=tf.train.Int64List(value=[value]))
-
-
-def int64_list_feature(value):
-  return tf.train.Feature(int64_list=tf.train.Int64List(value=value))
-
-
-def bytes_feature(value):
-  return tf.train.Feature(bytes_list=tf.train.BytesList(value=[value]))
-
-
-def bytes_list_feature(value):
-  return tf.train.Feature(bytes_list=tf.train.BytesList(value=value))
-
-
-def float_list_feature(value):
-  return tf.train.Feature(float_list=tf.train.FloatList(value=value))
-
-
-def read_examples_list(path):
-  """Read list of training or validation examples.
-
-  The file is assumed to contain a single example per line where the first
-  token in the line is an identifier that allows us to find the image and
-  annotation xml for that example.
-
-  For example, the line:
-  xyz 3
-  would allow us to find files xyz.jpg and xyz.xml (the 3 would be ignored).
-
-  Args:
-    path: absolute path to examples list file.
-
-  Returns:
-    list of example identifiers (strings).
-  """
-  with tf.io.gfile.GFile(path) as fid:
-    lines = fid.readlines()
-  return [line.strip().split(' ')[0] for line in lines]
-
-
-def recursive_parse_xml_to_dict(xml):
-  """Recursively parses XML contents to python dict.
-
-  We assume that `object` tags are the only ones that can appear
-  multiple times at the same level of a tree.
-
-  Args:
-    xml: xml tree obtained by parsing XML file contents using lxml.etree
-
-  Returns:
-    Python dictionary holding XML contents.
-  """
-  if not xml:
-    return {xml.tag: xml.text}
-  result = {}
-  for child in xml:
-    child_result = recursive_parse_xml_to_dict(child)
-    if child.tag != 'object':
-      result[child.tag] = child_result[child.tag]
-    else:
-      if child.tag not in result:
-        result[child.tag] = []
-      result[child.tag].append(child_result[child.tag])
-  return {xml.tag: result}
-
-
-def make_initializable_iterator(dataset):
-  """Creates an iterator, and initializes tables.
-
-  This is useful in cases where make_one_shot_iterator wouldn't work because
-  the graph contains a hash table that needs to be initialized.
-
-  Args:
-    dataset: A `tf.data.Dataset` object.
-
-  Returns:
-    A `tf.data.Iterator`.
-  """
-  iterator = tf.compat.v1.data.make_initializable_iterator(dataset)
-  tf.compat.v1.add_to_collection(tf.compat.v1.GraphKeys.TABLE_INITIALIZERS, iterator.initializer)
-  return iterator
-
-
-def read_dataset(file_read_func, decode_func, input_files, config):
-  """Reads a dataset, and handles repetition and shuffling.
-
-  Args:
-    file_read_func: Function to use in tf.data.Dataset.interleave, to read
-      every individual file into a tf.data.Dataset.
-    decode_func: Function to apply to all records.
-    input_files: A list of file paths to read.
-    config: A input_reader_builder.InputReader object.
-
-  Returns:
-    A tf.data.Dataset based on config.
-  """
-  # Shard, shuffle, and read files.
-  filenames = tf.concat([tf.io.matching_files(pattern) for pattern in input_files],
-                        0)
-  filename_dataset = tf.data.Dataset.from_tensor_slices(filenames)
-  if config.shuffle:
-    filename_dataset = filename_dataset.shuffle(
-        config.filenames_shuffle_buffer_size)
-  elif config.num_readers > 1:
-    tf.compat.v1.logging.warning('`shuffle` is false, but the input data stream is '
-                       'still slightly shuffled since `num_readers` > 1.')
-
-  filename_dataset = filename_dataset.repeat(config.num_epochs or None)
-
-  records_dataset = filename_dataset.apply(
-      tf.data.experimental.parallel_interleave(
-          file_read_func, cycle_length=config.num_readers,
-          block_length=config.read_block_length, sloppy=True))
-  if config.shuffle:
-    records_dataset.shuffle(config.shuffle_buffer_size)
-  tensor_dataset = records_dataset.map(
-      decode_func, num_parallel_calls=config.num_parallel_map_calls)
-  return tensor_dataset.prefetch(config.prefetch_size)
diff --git a/models/object_detection/tensorflow/rfcn/inference/fp32/eval.py b/models/object_detection/tensorflow/rfcn/inference/fp32/eval.py
deleted file mode 100644
index edc3ddfc9..000000000
--- a/models/object_detection/tensorflow/rfcn/inference/fp32/eval.py
+++ /dev/null
@@ -1,173 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
-
-# Copyright 2017 The TensorFlow Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-
-r"""Evaluation executable for detection models.
-
-This executable is used to evaluate DetectionModels. There are two ways of
-configuring the eval job.
-
-1) A single pipeline_pb2.TrainEvalPipelineConfig file maybe specified instead.
-In this mode, the --eval_training_data flag may be given to force the pipeline
-to evaluate on training data instead.
-
-Example usage:
-    ./eval \
-        --logtostderr \
-        --checkpoint_dir=path/to/checkpoint_dir \
-        --eval_dir=path/to/eval_dir \
-        --pipeline_config_path=pipeline_config.pbtxt
-
-2) Three configuration files may be provided: a model_pb2.DetectionModel
-configuration file to define what type of DetectionModel is being evaluated, an
-input_reader_pb2.InputReader file to specify what data the model is evaluating
-and an eval_pb2.EvalConfig file to configure evaluation parameters.
-
-Example usage:
-    ./eval \
-        --logtostderr \
-        --checkpoint_dir=path/to/checkpoint_dir \
-        --eval_dir=path/to/eval_dir \
-        --eval_config_path=eval_config.pbtxt \
-        --model_config_path=model_config.pbtxt \
-        --input_config_path=eval_input_config.pbtxt
-"""
-import functools
-import os
-import tensorflow as tf
-
-import evaluator
-import dataset_util
-from object_detection.builders import dataset_builder
-from object_detection.builders import model_builder
-from object_detection.utils import config_util
-from object_detection.utils import label_map_util
-
-
-tf.compat.v1.logging.set_verbosity(tf.compat.v1.logging.INFO)
-import logging
-# logging.basicConfig(level=logging.INFO)
-
-flags = tf.app.flags
-flags.DEFINE_boolean('eval_training_data', False,
-                     'If training data should be evaluated for this job.')
-flags.DEFINE_string('checkpoint_dir', '',
-                    'Directory containing checkpoints to evaluate, typically '
-                    'set to `train_dir` used in the training job.')
-flags.DEFINE_string('eval_dir', '',
-                    'Directory to write eval summaries to.')
-flags.DEFINE_string('pipeline_config_path', '',
-                    'Path to a pipeline_pb2.TrainEvalPipelineConfig config '
-                    'file. If provided, other configs are ignored')
-flags.DEFINE_string('eval_config_path', '',
-                    'Path to an eval_pb2.EvalConfig config file.')
-flags.DEFINE_string('input_config_path', '',
-                    'Path to an input_reader_pb2.InputReader config file.')
-flags.DEFINE_string('model_config_path', '',
-                    'Path to a model_pb2.DetectionModel config file.')
-flags.DEFINE_boolean('run_once', False, 'Option to only run a single pass of '
-                     'evaluation. Overrides the `max_evals` parameter in the '
-                     'provided config.')
-flags.DEFINE_integer('omp', 0, 'number of OMP threads')
-flags.DEFINE_integer('intra_op', 0, 'number of intra_op threads')
-flags.DEFINE_integer('inter_op', 0, 'number of inter_op threads')
-flags.DEFINE_integer('blocktime', 1, 'Blocktime value')
-flags.DEFINE_integer('batch_size', 1, 'batch size')
-
-FLAGS = flags.FLAGS
-
-
-def main(unused_argv):
-  if (FLAGS.omp > 0):
-    if not os.environ.get("OMP_NUM_THREADS"):
-      logging.info('OMP_NUM_THREADS value= %d', FLAGS.omp)
-      os.environ["OMP_NUM_THREADS"] = str(FLAGS.omp)
-    if not os.environ.get("KMP_BLOCKTIME"):
-      logging.info('KMP_BLOCKTIME value= %d', FLAGS.blocktime)
-      os.environ["KMP_BLOCKTIME"] = str(FLAGS.blocktime)
-    if not os.environ.get("KMP_SETTINGS"):
-      os.environ["KMP_SETTINGS"] = "1"
-    # os.environ["KMP_AFFINITY"]= "granularity=fine,verbose,compact,1,0"
-  assert FLAGS.checkpoint_dir, '`checkpoint_dir` is missing.'
-  assert FLAGS.eval_dir, '`eval_dir` is missing.'
-  tf.io.gfile.makedirs(FLAGS.eval_dir)
-  if FLAGS.pipeline_config_path:
-    configs = config_util.get_configs_from_pipeline_file(
-        FLAGS.pipeline_config_path)
-    tf.io.gfile.copy(FLAGS.pipeline_config_path,
-                  os.path.join(FLAGS.eval_dir, 'pipeline.config'),
-                  overwrite=True)
-  else:
-    configs = config_util.get_configs_from_multiple_files(
-        model_config_path=FLAGS.model_config_path,
-        eval_config_path=FLAGS.eval_config_path,
-        eval_input_config_path=FLAGS.input_config_path)
-    for name, config in [('model.config', FLAGS.model_config_path),
-                         ('eval.config', FLAGS.eval_config_path),
-                         ('input.config', FLAGS.input_config_path)]:
-      tf.io.gfile.copy(config,
-                    os.path.join(FLAGS.eval_dir, name),
-                    overwrite=True)
-
-  model_config = configs['model']
-  eval_config = configs['eval_config']
-  input_config = configs['eval_input_config']
-  if FLAGS.eval_training_data:
-    input_config = configs['train_input_config']
-
-  model_fn = functools.partial(
-      model_builder.build,
-      model_config=model_config,
-      is_training=False)
-
-  def get_next(config):
-    return tf.compat.v1.data.make_initializable_iterator(
-        dataset_util, dataset_builder.build(config)).get_next()
-
-  create_input_dict_fn = functools.partial(get_next, input_config)
-
-  label_map = label_map_util.load_labelmap(input_config.label_map_path)
-  max_num_classes = max([item.id for item in label_map.item])
-  categories = label_map_util.convert_label_map_to_categories(
-      label_map, max_num_classes)
-
-  if FLAGS.run_once:
-    eval_config.max_evals = 1
-
-  evaluator.evaluate(create_input_dict_fn, model_fn, eval_config, categories,
-                     FLAGS.checkpoint_dir, FLAGS.eval_dir, intra_op=FLAGS.intra_op, inter_op=FLAGS.inter_op)
-
-
-if __name__ == '__main__':
-  tf.compat.v1.app.run()
diff --git a/models/object_detection/tensorflow/rfcn/inference/fp32/eval_util.py b/models/object_detection/tensorflow/rfcn/inference/fp32/eval_util.py
deleted file mode 100644
index 5f3741e3a..000000000
--- a/models/object_detection/tensorflow/rfcn/inference/fp32/eval_util.py
+++ /dev/null
@@ -1,673 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
-
-# Copyright 2017 The TensorFlow Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-"""Common utility functions for evaluation."""
-import collections
-import logging
-import os
-import time
-
-import numpy as np
-import tensorflow as tf
-
-from object_detection.core import box_list
-from object_detection.core import box_list_ops
-from object_detection.core import keypoint_ops
-from object_detection.core import standard_fields as fields
-from object_detection.metrics import coco_evaluation
-from object_detection.utils import label_map_util
-from object_detection.utils import ops
-from object_detection.utils import visualization_utils as vis_utils
-
-def write_metrics(metrics, global_step, summary_dir):
-  """Write metrics to a summary directory.
-
-  Args:
-    metrics: A dictionary containing metric names and values.
-    global_step: Global step at which the metrics are computed.
-    summary_dir: Directory to write tensorflow summaries to.
-  """
-  logging.info('Writing metrics to tf summary.')
-  summary_writer = tf.compat.v1.summary.FileWriterCache.get(summary_dir)
-  for key in sorted(metrics):
-    summary = tf.compat.v1.Summary(value=[
-        tf.compat.v1.Summary.Value(tag=key, simple_value=metrics[key]),
-    ])
-    summary_writer.add_summary(summary, global_step)
-    logging.info('%s: %f', key, metrics[key])
-  logging.info('Metrics written to tf summary.')
-
-
-# TODO(rathodv): Add tests.
-def visualize_detection_results(result_dict,
-                                tag,
-                                global_step,
-                                categories,
-                                summary_dir='',
-                                export_dir='',
-                                agnostic_mode=False,
-                                show_groundtruth=False,
-                                groundtruth_box_visualization_color='black',
-                                min_score_thresh=.5,
-                                max_num_predictions=20,
-                                skip_scores=False,
-                                skip_labels=False,
-                                keep_image_id_for_visualization_export=False):
-  """Visualizes detection results and writes visualizations to image summaries.
-
-  This function visualizes an image with its detected bounding boxes and writes
-  to image summaries which can be viewed on tensorboard.  It optionally also
-  writes images to a directory. In the case of missing entry in the label map,
-  unknown class name in the visualization is shown as "N/A".
-
-  Args:
-    result_dict: a dictionary holding groundtruth and detection
-      data corresponding to each image being evaluated.  The following keys
-      are required:
-        'original_image': a numpy array representing the image with shape
-          [1, height, width, 3] or [1, height, width, 1]
-        'detection_boxes': a numpy array of shape [N, 4]
-        'detection_scores': a numpy array of shape [N]
-        'detection_classes': a numpy array of shape [N]
-      The following keys are optional:
-        'groundtruth_boxes': a numpy array of shape [N, 4]
-        'groundtruth_keypoints': a numpy array of shape [N, num_keypoints, 2]
-      Detections are assumed to be provided in decreasing order of score and for
-      display, and we assume that scores are probabilities between 0 and 1.
-    tag: tensorboard tag (string) to associate with image.
-    global_step: global step at which the visualization are generated.
-    categories: a list of dictionaries representing all possible categories.
-      Each dict in this list has the following keys:
-          'id': (required) an integer id uniquely identifying this category
-          'name': (required) string representing category name
-            e.g., 'cat', 'dog', 'pizza'
-          'supercategory': (optional) string representing the supercategory
-            e.g., 'animal', 'vehicle', 'food', etc
-    summary_dir: the output directory to which the image summaries are written.
-    export_dir: the output directory to which images are written.  If this is
-      empty (default), then images are not exported.
-    agnostic_mode: boolean (default: False) controlling whether to evaluate in
-      class-agnostic mode or not.
-    show_groundtruth: boolean (default: False) controlling whether to show
-      groundtruth boxes in addition to detected boxes
-    groundtruth_box_visualization_color: box color for visualizing groundtruth
-      boxes
-    min_score_thresh: minimum score threshold for a box to be visualized
-    max_num_predictions: maximum number of detections to visualize
-    skip_scores: whether to skip score when drawing a single detection
-    skip_labels: whether to skip label when drawing a single detection
-    keep_image_id_for_visualization_export: whether to keep image identifier in
-      filename when exported to export_dir
-  Raises:
-    ValueError: if result_dict does not contain the expected keys (i.e.,
-      'original_image', 'detection_boxes', 'detection_scores',
-      'detection_classes')
-  """
-  detection_fields = fields.DetectionResultFields
-  input_fields = fields.InputDataFields
-  if not set([
-      input_fields.original_image,
-      detection_fields.detection_boxes,
-      detection_fields.detection_scores,
-      detection_fields.detection_classes,
-  ]).issubset(set(result_dict.keys())):
-    raise ValueError('result_dict does not contain all expected keys.')
-  if show_groundtruth and input_fields.groundtruth_boxes not in result_dict:
-    raise ValueError('If show_groundtruth is enabled, result_dict must contain '
-                     'groundtruth_boxes.')
-  logging.info('Creating detection visualizations.')
-  category_index = label_map_util.create_category_index(categories)
-
-  image = np.squeeze(result_dict[input_fields.original_image], axis=0)
-  if image.shape[2] == 1:  # If one channel image, repeat in RGB.
-    image = np.tile(image, [1, 1, 3])
-  detection_boxes = result_dict[detection_fields.detection_boxes]
-  detection_scores = result_dict[detection_fields.detection_scores]
-  detection_classes = np.int32((result_dict[
-      detection_fields.detection_classes]))
-  detection_keypoints = result_dict.get(detection_fields.detection_keypoints)
-  detection_masks = result_dict.get(detection_fields.detection_masks)
-  detection_boundaries = result_dict.get(detection_fields.detection_boundaries)
-
-  # Plot groundtruth underneath detections
-  if show_groundtruth:
-    groundtruth_boxes = result_dict[input_fields.groundtruth_boxes]
-    groundtruth_keypoints = result_dict.get(input_fields.groundtruth_keypoints)
-    vis_utils.visualize_boxes_and_labels_on_image_array(
-        image=image,
-        boxes=groundtruth_boxes,
-        classes=None,
-        scores=None,
-        category_index=category_index,
-        keypoints=groundtruth_keypoints,
-        use_normalized_coordinates=False,
-        max_boxes_to_draw=None,
-        groundtruth_box_visualization_color=groundtruth_box_visualization_color)
-  vis_utils.visualize_boxes_and_labels_on_image_array(
-      image,
-      detection_boxes,
-      detection_classes,
-      detection_scores,
-      category_index,
-      instance_masks=detection_masks,
-      instance_boundaries=detection_boundaries,
-      keypoints=detection_keypoints,
-      use_normalized_coordinates=False,
-      max_boxes_to_draw=max_num_predictions,
-      min_score_thresh=min_score_thresh,
-      agnostic_mode=agnostic_mode,
-      skip_scores=skip_scores,
-      skip_labels=skip_labels)
-
-  if export_dir:
-    if keep_image_id_for_visualization_export and result_dict[fields.
-                                                              InputDataFields()
-                                                              .key]:
-      export_path = os.path.join(export_dir, 'export-{}-{}.png'.format(
-          tag, result_dict[fields.InputDataFields().key]))
-    else:
-      export_path = os.path.join(export_dir, 'export-{}.png'.format(tag))
-    vis_utils.save_image_array_as_png(image, export_path)
-
-  summary = tf.compat.v1.Summary(value=[
-      tf.compat.v1.Summary.Value(
-          tag=tag,
-          image=tf.compat.v1.Summary.Image(
-              encoded_image_string=vis_utils.encode_image_array_as_png_str(
-                  image)))
-  ])
-  summary_writer = tf.compat.v1.summary.FileWriterCache.get(summary_dir)
-  summary_writer.add_summary(summary, global_step)
-
-  logging.info('Detection visualizations written to summary with tag %s.', tag)
-
-
-def _run_checkpoint_once(tensor_dict,
-                         evaluators=None,
-                         batch_processor=None,
-                         checkpoint_dirs=None,
-                         variables_to_restore=None,
-                         restore_fn=None,
-                         num_batches=1,
-                         master='',
-                         save_graph=False,
-                         save_graph_dir='',
-                         losses_dict=None,
-                         intra_op=0,
-                         inter_op=0):
-
-  """Evaluates metrics defined in evaluators and returns summaries.
-
-  This function loads the latest checkpoint in checkpoint_dirs and evaluates
-  all metrics defined in evaluators. The metrics are processed in batch by the
-  batch_processor.
-
-  Args:
-    tensor_dict: a dictionary holding tensors representing a batch of detections
-      and corresponding groundtruth annotations.
-    evaluators: a list of object of type DetectionEvaluator to be used for
-      evaluation. Note that the metric names produced by different evaluators
-      must be unique.
-    batch_processor: a function taking four arguments:
-      1. tensor_dict: the same tensor_dict that is passed in as the first
-        argument to this function.
-      2. sess: a tensorflow session
-      3. batch_index: an integer representing the index of the batch amongst
-        all batches
-      By default, batch_processor is None, which defaults to running:
-        return sess.run(tensor_dict)
-      To skip an image, it suffices to return an empty dictionary in place of
-      result_dict.
-    checkpoint_dirs: list of directories to load into an EnsembleModel. If it
-      has only one directory, EnsembleModel will not be used --
-        a DetectionModel
-      will be instantiated directly. Not used if restore_fn is set.
-    variables_to_restore: None, or a dictionary mapping variable names found in
-      a checkpoint to model variables. The dictionary would normally be
-      generated by creating a tf.train.ExponentialMovingAverage object and
-      calling its variables_to_restore() method. Not used if restore_fn is set.
-    restore_fn: None, or a function that takes a tf.Session object and correctly
-      restores all necessary variables from the correct checkpoint file. If
-      None, attempts to restore from the first directory in checkpoint_dirs.
-    num_batches: the number of batches to use for evaluation.
-    master: the location of the Tensorflow session.
-    save_graph: whether or not the Tensorflow graph is stored as a pbtxt file.
-    save_graph_dir: where to store the Tensorflow graph on disk. If save_graph
-      is True this must be non-empty.
-    losses_dict: optional dictionary of scalar detection losses.
-
-  Returns:
-    global_step: the count of global steps.
-    all_evaluator_metrics: A dictionary containing metric names and values.
-
-  Raises:
-    ValueError: if restore_fn is None and checkpoint_dirs doesn't have at least
-      one element.
-    ValueError: if save_graph is True and save_graph_dir is not defined.
-  """
-  if save_graph and not save_graph_dir:
-    raise ValueError('`save_graph_dir` must be defined.')
-  if (inter_op > 0 or intra_op > 0):
-    config = tf.compat.v1.ConfigProto(inter_op_parallelism_threads=inter_op,
-                            intra_op_parallelism_threads=intra_op)
-    logging.info('inter_op value= %d', inter_op)
-    logging.info('intra_op value= %d', intra_op)
-  else:
-    config = None
-  sess = tf.compat.v1.Session(master, graph=tf.compat.v1.get_default_graph(), config=config)
-  sess.run(tf.compat.v1.global_variables_initializer())
-  sess.run(tf.compat.v1.local_variables_initializer())
-  sess.run(tf.compat.v1.tables_initializer())
-  if restore_fn:
-    restore_fn(sess)
-  else:
-    if not checkpoint_dirs:
-      raise ValueError('`checkpoint_dirs` must have at least one entry.')
-    checkpoint_file = tf.train.latest_checkpoint(checkpoint_dirs[0])
-    saver = tf.compat.v1.train.Saver(variables_to_restore)
-    saver.restore(sess, checkpoint_file)
-
-  if save_graph:
-    tf.io.write_graph(sess.graph_def, save_graph_dir, 'eval.pbtxt')
-
-  counters = {'skipped': 0, 'success': 0}
-  aggregate_result_losses_dict = collections.defaultdict(list)
-  with tf.compat.v1.train.QueueRunner(sess):
-    try:
-      loop_start_time = time.time()
-      for batch in range(int(num_batches)):
-        if (batch + 1) % 100 == 0:
-          logging.info('Running eval ops batch %d/%d', batch + 1, num_batches)
-        start_time = time.time()
-        if not batch_processor:
-          try:
-            if not losses_dict:
-              losses_dict = {}
-            result_dict, result_losses_dict = sess.run([tensor_dict,
-                                                        losses_dict])
-            counters['success'] += 1
-          except tf.errors.InvalidArgumentError:
-            logging.info('Skipping image')
-            counters['skipped'] += 1
-            result_dict = {}
-        else:
-          result_dict, result_losses_dict = batch_processor(
-              tensor_dict, sess, batch, counters, losses_dict=losses_dict)
-        if ((batch % 100) == 0):
-          logging.info('Iteration %d: %.3f sec', batch, time.time() - start_time)
-        if not result_dict:
-          continue
-        for key, value in iter(result_losses_dict.items()):
-          aggregate_result_losses_dict[key].append(value)
-        for evaluator in evaluators:
-          # TODO(b/65130867): Use image_id tensor once we fix the input data
-          # decoders to return correct image_id.
-          # TODO(akuznetsa): result_dict contains batches of images, while
-          # add_single_ground_truth_image_info expects a single image. Fix
-          evaluator.add_single_ground_truth_image_info(
-              image_id=batch, groundtruth_dict=result_dict)
-          evaluator.add_single_detected_image_info(
-              image_id=batch, detections_dict=result_dict)
-      loop_end_time = time.time()
-      total_run_time = loop_end_time - loop_start_time
-      avg_time_per_batch = total_run_time / num_batches
-      print('Average time per step: %.3f sec' % avg_time_per_batch)
-      logging.info('Running eval batches done.')
-    except tf.errors.OutOfRangeError:
-      logging.info('Done evaluating -- epoch limit reached')
-    finally:
-      # When done, ask the threads to stop.
-      logging.info('# success: %d', counters['success'])
-      logging.info('# skipped: %d', counters['skipped'])
-      all_evaluator_metrics = {}
-      for evaluator in evaluators:
-        metrics = evaluator.evaluate()
-        evaluator.clear()
-        if any(key in all_evaluator_metrics for key in metrics):
-          raise ValueError('Metric names between evaluators must not collide.')
-        all_evaluator_metrics.update(metrics)
-      global_step = tf.compat.v1.train.global_step(sess, tf.compat.v1.train.get_global_step())
-
-      for key, value in iter(aggregate_result_losses_dict.items()):
-        all_evaluator_metrics['Losses/' + key] = np.mean(value)
-  sess.close()
-  return (global_step, all_evaluator_metrics)
-
-
-# TODO(rathodv): Add tests.
-def repeated_checkpoint_run(tensor_dict,
-                            summary_dir,
-                            evaluators,
-                            batch_processor=None,
-                            checkpoint_dirs=None,
-                            variables_to_restore=None,
-                            restore_fn=None,
-                            num_batches=1,
-                            eval_interval_secs=120,
-                            max_number_of_evaluations=None,
-                            master='',
-                            save_graph=False,
-                            save_graph_dir='',
-                            losses_dict=None):
-  """Periodically evaluates desired tensors using checkpoint_dirs or restore_fn.
-
-  This function repeatedly loads a checkpoint and evaluates a desired
-  set of tensors (provided by tensor_dict) and hands the resulting numpy
-  arrays to a function result_processor which can be used to further
-  process/save/visualize the results.
-
-  Args:
-    tensor_dict: a dictionary holding tensors representing a batch of detections
-      and corresponding groundtruth annotations.
-    summary_dir: a directory to write metrics summaries.
-    evaluators: a list of object of type DetectionEvaluator to be used for
-      evaluation. Note that the metric names produced by different evaluators
-      must be unique.
-    batch_processor: a function taking three arguments:
-      1. tensor_dict: the same tensor_dict that is passed in as the first
-        argument to this function.
-      2. sess: a tensorflow session
-      3. batch_index: an integer representing the index of the batch amongst
-        all batches
-      By default, batch_processor is None, which defaults to running:
-        return sess.run(tensor_dict)
-    checkpoint_dirs: list of directories to load into a DetectionModel or an
-      EnsembleModel if restore_fn isn't set. Also used to determine when to run
-      next evaluation. Must have at least one element.
-    variables_to_restore: None, or a dictionary mapping variable names found in
-      a checkpoint to model variables. The dictionary would normally be
-      generated by creating a tf.train.ExponentialMovingAverage object and
-      calling its variables_to_restore() method. Not used if restore_fn is set.
-    restore_fn: a function that takes a tf.Session object and correctly restores
-      all necessary variables from the correct checkpoint file.
-    num_batches: the number of batches to use for evaluation.
-    eval_interval_secs: the number of seconds between each evaluation run.
-    max_number_of_evaluations: the max number of iterations of the evaluation.
-      If the value is left as None the evaluation continues indefinitely.
-    master: the location of the Tensorflow session.
-    save_graph: whether or not the Tensorflow graph is saved as a pbtxt file.
-    save_graph_dir: where to save on disk the Tensorflow graph. If store_graph
-      is True this must be non-empty.
-    losses_dict: optional dictionary of scalar detection losses.
-
-  Returns:
-    metrics: A dictionary containing metric names and values in the latest
-      evaluation.
-
-  Raises:
-    ValueError: if max_num_of_evaluations is not None or a positive number.
-    ValueError: if checkpoint_dirs doesn't have at least one element.
-  """
-  if max_number_of_evaluations and max_number_of_evaluations <= 0:
-    raise ValueError(
-        '`number_of_steps` must be either None or a positive number.')
-
-  if not checkpoint_dirs:
-    raise ValueError('`checkpoint_dirs` must have at least one entry.')
-
-  last_evaluated_model_path = None
-  number_of_evaluations = 0
-  while True:
-    start = time.time()
-    logging.info('Starting evaluation at ' + time.strftime(
-        '%Y-%m-%d-%H:%M:%S', time.gmtime()))
-    model_path = tf.train.latest_checkpoint(checkpoint_dirs[0])
-    if not model_path:
-      logging.info('No model found in %s. Will try again in %d seconds',
-                   checkpoint_dirs[0], eval_interval_secs)
-    elif model_path == last_evaluated_model_path:
-      logging.info('Found already evaluated checkpoint. Will try again in %d '
-                   'seconds', eval_interval_secs)
-    else:
-      last_evaluated_model_path = model_path
-      global_step, metrics = _run_checkpoint_once(tensor_dict, evaluators,
-                                                  batch_processor,
-                                                  checkpoint_dirs,
-                                                  variables_to_restore,
-                                                  restore_fn, num_batches,
-                                                  master, save_graph,
-                                                  save_graph_dir,
-                                                  losses_dict=losses_dict)
-      write_metrics(metrics, global_step, summary_dir)
-    number_of_evaluations += 1
-
-    if (max_number_of_evaluations and
-        number_of_evaluations >= max_number_of_evaluations):
-      logging.info('Finished evaluation!')
-      break
-    time_to_next_eval = start + eval_interval_secs - time.time()
-    if time_to_next_eval > 0:
-      time.sleep(time_to_next_eval)
-
-  return metrics
-
-
-def result_dict_for_single_example(image,
-                                   key,
-                                   detections,
-                                   groundtruth=None,
-                                   class_agnostic=False,
-                                   scale_to_absolute=False):
-  """Merges all detection and groundtruth information for a single example.
-
-  Note that evaluation tools require classes that are 1-indexed, and so this
-  function performs the offset. If `class_agnostic` is True, all output classes
-  have label 1.
-
-  Args:
-    image: A single 4D uint8 image tensor of shape [1, H, W, C].
-    key: A single string tensor identifying the image.
-    detections: A dictionary of detections, returned from
-      DetectionModel.postprocess().
-    groundtruth: (Optional) Dictionary of groundtruth items, with fields:
-      'groundtruth_boxes': [num_boxes, 4] float32 tensor of boxes, in
-        normalized coordinates.
-      'groundtruth_classes': [num_boxes] int64 tensor of 1-indexed classes.
-      'groundtruth_area': [num_boxes] float32 tensor of bbox area. (Optional)
-      'groundtruth_is_crowd': [num_boxes] int64 tensor. (Optional)
-      'groundtruth_difficult': [num_boxes] int64 tensor. (Optional)
-      'groundtruth_group_of': [num_boxes] int64 tensor. (Optional)
-      'groundtruth_instance_masks': 3D int64 tensor of instance masks
-        (Optional).
-    class_agnostic: Boolean indicating whether the detections are class-agnostic
-      (i.e. binary). Default False.
-    scale_to_absolute: Boolean indicating whether boxes and keypoints should be
-      scaled to absolute coordinates. Note that for IoU based evaluations, it
-      does not matter whether boxes are expressed in absolute or relative
-      coordinates. Default False.
-
-  Returns:
-    A dictionary with:
-    'original_image': A [1, H, W, C] uint8 image tensor.
-    'key': A string tensor with image identifier.
-    'detection_boxes': [max_detections, 4] float32 tensor of boxes, in
-      normalized or absolute coordinates, depending on the value of
-      `scale_to_absolute`.
-    'detection_scores': [max_detections] float32 tensor of scores.
-    'detection_classes': [max_detections] int64 tensor of 1-indexed classes.
-    'detection_masks': [max_detections, H, W] float32 tensor of binarized
-      masks, reframed to full image masks.
-    'groundtruth_boxes': [num_boxes, 4] float32 tensor of boxes, in
-      normalized or absolute coordinates, depending on the value of
-      `scale_to_absolute`. (Optional)
-    'groundtruth_classes': [num_boxes] int64 tensor of 1-indexed classes.
-      (Optional)
-    'groundtruth_area': [num_boxes] float32 tensor of bbox area. (Optional)
-    'groundtruth_is_crowd': [num_boxes] int64 tensor. (Optional)
-    'groundtruth_difficult': [num_boxes] int64 tensor. (Optional)
-    'groundtruth_group_of': [num_boxes] int64 tensor. (Optional)
-    'groundtruth_instance_masks': 3D int64 tensor of instance masks
-      (Optional).
-
-  """
-  label_id_offset = 1  # Applying label id offset (b/63711816)
-
-  input_data_fields = fields.InputDataFields
-  output_dict = {
-      input_data_fields.original_image: image,
-      input_data_fields.key: key,
-  }
-
-  detection_fields = fields.DetectionResultFields
-  detection_boxes = detections[detection_fields.detection_boxes][0]
-  image_shape = tf.shape(input=image)
-  detection_scores = detections[detection_fields.detection_scores][0]
-
-  if class_agnostic:
-    detection_classes = tf.ones_like(detection_scores, dtype=tf.int64)
-  else:
-    detection_classes = (
-        tf.cast(detections[detection_fields.detection_classes][0], dtype=tf.int64) +
-        label_id_offset)
-
-  num_detections = tf.cast(detections[detection_fields.num_detections][0], dtype=tf.int32)
-  detection_boxes = tf.slice(
-      detection_boxes, begin=[0, 0], size=[num_detections, -1])
-  detection_classes = tf.slice(
-      detection_classes, begin=[0], size=[num_detections])
-  detection_scores = tf.slice(
-      detection_scores, begin=[0], size=[num_detections])
-
-  if scale_to_absolute:
-    absolute_detection_boxlist = box_list_ops.to_absolute_coordinates(
-        box_list.BoxList(detection_boxes), image_shape[1], image_shape[2])
-    output_dict[detection_fields.detection_boxes] = (
-        absolute_detection_boxlist.get())
-  else:
-    output_dict[detection_fields.detection_boxes] = detection_boxes
-  output_dict[detection_fields.detection_classes] = detection_classes
-  output_dict[detection_fields.detection_scores] = detection_scores
-
-  if detection_fields.detection_masks in detections:
-    detection_masks = detections[detection_fields.detection_masks][0]
-    # TODO(rathodv): This should be done in model's postprocess
-    # function ideally.
-    detection_masks = tf.slice(
-        detection_masks, begin=[0, 0, 0], size=[num_detections, -1, -1])
-    detection_masks_reframed = ops.reframe_box_masks_to_image_masks(
-        detection_masks, detection_boxes, image_shape[1], image_shape[2])
-    detection_masks_reframed = tf.cast(
-        tf.greater(detection_masks_reframed, 0.5), tf.uint8)
-    output_dict[detection_fields.detection_masks] = detection_masks_reframed
-  if detection_fields.detection_keypoints in detections:
-    detection_keypoints = detections[detection_fields.detection_keypoints][0]
-    output_dict[detection_fields.detection_keypoints] = detection_keypoints
-    if scale_to_absolute:
-      absolute_detection_keypoints = keypoint_ops.scale(
-          detection_keypoints, image_shape[1], image_shape[2])
-      output_dict[detection_fields.detection_keypoints] = (
-          absolute_detection_keypoints)
-
-  if groundtruth:
-    if input_data_fields.groundtruth_instance_masks in groundtruth:
-      groundtruth[input_data_fields.groundtruth_instance_masks] = tf.cast(
-          groundtruth[input_data_fields.groundtruth_instance_masks], tf.uint8)
-    output_dict.update(groundtruth)
-    if scale_to_absolute:
-      groundtruth_boxes = groundtruth[input_data_fields.groundtruth_boxes]
-      absolute_gt_boxlist = box_list_ops.to_absolute_coordinates(
-          box_list.BoxList(groundtruth_boxes), image_shape[1], image_shape[2])
-      output_dict[input_data_fields.groundtruth_boxes] = (
-          absolute_gt_boxlist.get())
-    # For class-agnostic models, groundtruth classes all become 1.
-    if class_agnostic:
-      groundtruth_classes = groundtruth[input_data_fields.groundtruth_classes]
-      groundtruth_classes = tf.ones_like(groundtruth_classes, dtype=tf.int64)
-      output_dict[input_data_fields.groundtruth_classes] = groundtruth_classes
-
-  return output_dict
-
-
-def get_eval_metric_ops_for_evaluators(evaluation_metrics,
-                                       categories,
-                                       eval_dict,
-                                       include_metrics_per_category=False):
-  """Returns a dictionary of eval metric ops to use with `tf.EstimatorSpec`.
-
-  Args:
-    evaluation_metrics: List of evaluation metric names. Current options are
-      'coco_detection_metrics' and 'coco_mask_metrics'.
-    categories: A list of dicts, each of which has the following keys -
-        'id': (required) an integer id uniquely identifying this category.
-        'name': (required) string representing category name e.g., 'cat', 'dog'.
-    eval_dict: An evaluation dictionary, returned from
-      result_dict_for_single_example().
-    include_metrics_per_category: If True, include metrics for each category.
-
-  Returns:
-    A dictionary of metric names to tuple of value_op and update_op that can be
-    used as eval metric ops in tf.EstimatorSpec.
-
-  Raises:
-    ValueError: If any of the metrics in `evaluation_metric` is not
-    'coco_detection_metrics' or 'coco_mask_metrics'.
-  """
-  evaluation_metrics = list(set(evaluation_metrics))
-
-  input_data_fields = fields.InputDataFields
-  detection_fields = fields.DetectionResultFields
-  eval_metric_ops = {}
-  for metric in evaluation_metrics:
-    if metric == 'coco_detection_metrics':
-      coco_evaluator = coco_evaluation.CocoDetectionEvaluator(
-          categories, include_metrics_per_category=include_metrics_per_category)
-      eval_metric_ops.update(
-          coco_evaluator.get_estimator_eval_metric_ops(
-              image_id=eval_dict[input_data_fields.key],
-              groundtruth_boxes=eval_dict[input_data_fields.groundtruth_boxes],
-              groundtruth_classes=eval_dict[
-                  input_data_fields.groundtruth_classes],
-              detection_boxes=eval_dict[detection_fields.detection_boxes],
-              detection_scores=eval_dict[detection_fields.detection_scores],
-              detection_classes=eval_dict[detection_fields.detection_classes]))
-    elif metric == 'coco_mask_metrics':
-      coco_mask_evaluator = coco_evaluation.CocoMaskEvaluator(
-          categories, include_metrics_per_category=include_metrics_per_category)
-      eval_metric_ops.update(
-          coco_mask_evaluator.get_estimator_eval_metric_ops(
-              image_id=eval_dict[input_data_fields.key],
-              groundtruth_boxes=eval_dict[input_data_fields.groundtruth_boxes],
-              groundtruth_classes=eval_dict[
-                  input_data_fields.groundtruth_classes],
-              groundtruth_instance_masks=eval_dict[
-                  input_data_fields.groundtruth_instance_masks],
-              detection_scores=eval_dict[detection_fields.detection_scores],
-              detection_classes=eval_dict[detection_fields.detection_classes],
-              detection_masks=eval_dict[detection_fields.detection_masks]))
-    else:
-      raise ValueError('The only evaluation metrics supported are '
-                       '"coco_detection_metrics" and "coco_mask_metrics". '
-                       'Found {} in the evaluation metrics'.format(metric))
-
-  return eval_metric_ops
diff --git a/models/object_detection/tensorflow/rfcn/inference/fp32/evaluator.py b/models/object_detection/tensorflow/rfcn/inference/fp32/evaluator.py
deleted file mode 100644
index 61a2136ea..000000000
--- a/models/object_detection/tensorflow/rfcn/inference/fp32/evaluator.py
+++ /dev/null
@@ -1,317 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
-
-# Copyright 2017 The TensorFlow Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-"""Detection model evaluator.
-
-This file provides a generic evaluation method that can be used to evaluate a
-DetectionModel.
-"""
-
-import logging
-import tensorflow as tf
-import os
-import time
-
-import eval_util
-from object_detection.core import prefetcher
-from object_detection.core import standard_fields as fields
-from object_detection.metrics import coco_evaluation
-from object_detection.utils import object_detection_evaluation
-from tensorflow.python.client import timeline
-
-# A dictionary of metric names to classes that implement the metric. The classes
-# in the dictionary must implement
-# utils.object_detection_evaluation.DetectionEvaluator interface.
-EVAL_METRICS_CLASS_DICT = {
-    'pascal_voc_detection_metrics':
-        object_detection_evaluation.PascalDetectionEvaluator,
-    'weighted_pascal_voc_detection_metrics':
-        object_detection_evaluation.WeightedPascalDetectionEvaluator,
-    'pascal_voc_instance_segmentation_metrics':
-        object_detection_evaluation.PascalInstanceSegmentationEvaluator,
-    'weighted_pascal_voc_instance_segmentation_metrics':
-        object_detection_evaluation.WeightedPascalInstanceSegmentationEvaluator,
-    'open_images_detection_metrics':
-        object_detection_evaluation.OpenImagesDetectionEvaluator,
-    'coco_detection_metrics':
-        coco_evaluation.CocoDetectionEvaluator,
-    'coco_mask_metrics':
-        coco_evaluation.CocoMaskEvaluator,
-}
-
-EVAL_DEFAULT_METRIC = 'pascal_voc_detection_metrics'
-
-
-def _extract_predictions_and_losses(model,
-                                    create_input_dict_fn,
-                                    ignore_groundtruth=False):
-  """Constructs tensorflow detection graph and returns output tensors.
-
-  Args:
-    model: model to perform predictions with.
-    create_input_dict_fn: function to create input tensor dictionaries.
-    ignore_groundtruth: whether groundtruth should be ignored.
-
-  Returns:
-    prediction_groundtruth_dict: A dictionary with postprocessed tensors (keyed
-      by standard_fields.DetectionResultsFields) and optional groundtruth
-      tensors (keyed by standard_fields.InputDataFields).
-    losses_dict: A dictionary containing detection losses. This is empty when
-      ignore_groundtruth is true.
-  """
-  input_dict = create_input_dict_fn()
-  prefetch_queue = prefetcher.prefetch(input_dict, capacity=500)
-  input_dict = prefetch_queue.dequeue()
-  original_image = tf.expand_dims(input_dict[fields.InputDataFields.image], 0)
-  preprocessed_image, true_image_shapes = model.preprocess(
-      tf.cast(original_image, dtype=tf.float32))
-  prediction_dict = model.predict(preprocessed_image, true_image_shapes)
-  detections = model.postprocess(prediction_dict, true_image_shapes)
-
-  groundtruth = None
-  losses_dict = {}
-  if not ignore_groundtruth:
-    groundtruth = {
-        fields.InputDataFields.groundtruth_boxes:
-            input_dict[fields.InputDataFields.groundtruth_boxes],
-        fields.InputDataFields.groundtruth_classes:
-            input_dict[fields.InputDataFields.groundtruth_classes],
-        fields.InputDataFields.groundtruth_area:
-            input_dict[fields.InputDataFields.groundtruth_area],
-        fields.InputDataFields.groundtruth_is_crowd:
-            input_dict[fields.InputDataFields.groundtruth_is_crowd],
-        fields.InputDataFields.groundtruth_difficult:
-            input_dict[fields.InputDataFields.groundtruth_difficult]
-    }
-    if fields.InputDataFields.groundtruth_group_of in input_dict:
-      groundtruth[fields.InputDataFields.groundtruth_group_of] = (
-          input_dict[fields.InputDataFields.groundtruth_group_of])
-    groundtruth_masks_list = None
-    if fields.DetectionResultFields.detection_masks in detections:
-      groundtruth[fields.InputDataFields.groundtruth_instance_masks] = (
-          input_dict[fields.InputDataFields.groundtruth_instance_masks])
-      groundtruth_masks_list = [
-          input_dict[fields.InputDataFields.groundtruth_instance_masks]]
-    groundtruth_keypoints_list = None
-    if fields.DetectionResultFields.detection_keypoints in detections:
-      groundtruth[fields.InputDataFields.groundtruth_keypoints] = (
-          input_dict[fields.InputDataFields.groundtruth_keypoints])
-      groundtruth_keypoints_list = [
-          input_dict[fields.InputDataFields.groundtruth_keypoints]]
-    label_id_offset = 1
-    model.provide_groundtruth(
-        [input_dict[fields.InputDataFields.groundtruth_boxes]],
-        [tf.one_hot(input_dict[fields.InputDataFields.groundtruth_classes]
-                    - label_id_offset, depth=model.num_classes)],
-        groundtruth_masks_list, groundtruth_keypoints_list)
-    losses_dict.update(model.loss(prediction_dict, true_image_shapes))
-
-  result_dict = eval_util.result_dict_for_single_example(
-      original_image,
-      input_dict[fields.InputDataFields.source_id],
-      detections,
-      groundtruth,
-      class_agnostic=(
-          fields.DetectionResultFields.detection_classes not in detections),
-      scale_to_absolute=True)
-  return result_dict, losses_dict
-
-
-def get_evaluators(eval_config, categories):
-  """Returns the evaluator class according to eval_config, valid for categories.
-
-  Args:
-    eval_config: evaluation configurations.
-    categories: a list of categories to evaluate.
-  Returns:
-    An list of instances of DetectionEvaluator.
-
-  Raises:
-    ValueError: if metric is not in the metric class dictionary.
-  """
-  eval_metric_fn_keys = eval_config.metrics_set
-  if not eval_metric_fn_keys:
-    eval_metric_fn_keys = [EVAL_DEFAULT_METRIC]
-  evaluators_list = []
-  for eval_metric_fn_key in eval_metric_fn_keys:
-    if eval_metric_fn_key not in EVAL_METRICS_CLASS_DICT:
-      raise ValueError('Metric not found: {}'.format(eval_metric_fn_key))
-    evaluators_list.append(
-        EVAL_METRICS_CLASS_DICT[eval_metric_fn_key](categories=categories))
-  return evaluators_list
-
-
-def evaluate(create_input_dict_fn, create_model_fn, eval_config, categories,
-             checkpoint_dir, eval_dir, graph_hook_fn=None, evaluator_list=None, intra_op=0, inter_op=0):
-  """Evaluation function for detection models.
-
-  Args:
-    create_input_dict_fn: a function to create a tensor input dictionary.
-    create_model_fn: a function that creates a DetectionModel.
-    eval_config: a eval_pb2.EvalConfig protobuf.
-    categories: a list of category dictionaries. Each dict in the list should
-                have an integer 'id' field and string 'name' field.
-    checkpoint_dir: directory to load the checkpoints to evaluate from.
-    eval_dir: directory to write evaluation metrics summary to.
-    graph_hook_fn: Optional function that is called after the training graph is
-      completely built. This is helpful to perform additional changes to the
-      training graph such as optimizing batchnorm. The function should modify
-      the default graph.
-    evaluator_list: Optional list of instances of DetectionEvaluator. If not
-      given, this list of metrics is created according to the eval_config.
-
-  Returns:
-    metrics: A dictionary containing metric names and values from the latest
-      run.
-  """
-
-  model = create_model_fn()
-
-  if eval_config.ignore_groundtruth and not eval_config.export_path:
-    logging.fatal('If ignore_groundtruth=True then an export_path is '
-                  'required. Aborting!!!')
-
-  tensor_dict, losses_dict = _extract_predictions_and_losses(
-      model=model,
-      create_input_dict_fn=create_input_dict_fn,
-      ignore_groundtruth=eval_config.ignore_groundtruth)
-
-  def _process_batch(tensor_dict, sess, batch_index, counters,
-                     losses_dict=None):
-    """Evaluates tensors in tensor_dict, losses_dict and visualizes examples.
-
-    This function calls sess.run on tensor_dict, evaluating the original_image
-    tensor only on the first K examples and visualizing detections overlaid
-    on this original_image.
-
-    Args:
-      tensor_dict: a dictionary of tensors
-      sess: tensorflow session
-      batch_index: the index of the batch amongst all batches in the run.
-      counters: a dictionary holding 'success' and 'skipped' fields which can
-        be updated to keep track of number of successful and failed runs,
-        respectively.  If these fields are not updated, then the success/skipped
-        counter values shown at the end of evaluation will be incorrect.
-      losses_dict: Optional dictonary of scalar loss tensors.
-
-    Returns:
-      result_dict: a dictionary of numpy arrays
-      result_losses_dict: a dictionary of scalar losses. This is empty if input
-        losses_dict is None.
-    """
-    try:
-      if not losses_dict:
-        losses_dict = {}
-      trace = False
-      if batch_index == 0 and trace:
-        run_options = tf.compat.v1.RunOptions(trace_level=tf.compat.v1.RunOptions.FULL_TRACE)
-        run_metadata = tf.compat.v1.RunMetadata()
-      else:
-        run_options = None
-        run_metadata = None
-      start_time = time.time()
-      result_dict, result_losses_dict = sess.run(
-          [tensor_dict, losses_dict], options=run_options, run_metadata=run_metadata)
-      if (batch_index % 100 == 0):
-        logging.info('Step %d: %.3f sec', batch_index, time.time() - start_time)
-      if batch_index == 0 and trace:
-        trace = timeline.Timeline(step_stats=run_metadata.step_stats)
-        dir = 'logs'
-        if not os.path.exists(dir):
-          os.makedirs(dir)
-        with open(dir + '/rfcn-timeline-' + time.strftime("%Y%m%d-%H%M%S") + '.json', 'w') as file:
-          file.write(trace.generate_chrome_trace_format(show_memory=False))
-      counters['success'] += 1
-    except tf.errors.InvalidArgumentError:
-      logging.info('Skipping image')
-      counters['skipped'] += 1
-      return {}, {}
-    global_step = tf.compat.v1.train.global_step(sess, tf.compat.v1.train.get_global_step())
-    if batch_index < eval_config.num_visualizations:
-      tag = 'image-{}'.format(batch_index)
-      eval_util.visualize_detection_results(
-          result_dict,
-          tag,
-          global_step,
-          categories=categories,
-          summary_dir=eval_dir,
-          export_dir=eval_config.visualization_export_dir,
-          show_groundtruth=eval_config.visualize_groundtruth_boxes,
-          groundtruth_box_visualization_color=eval_config.
-          groundtruth_box_visualization_color,
-          min_score_thresh=eval_config.min_score_threshold,
-          max_num_predictions=eval_config.max_num_boxes_to_visualize,
-          skip_scores=eval_config.skip_scores,
-          skip_labels=eval_config.skip_labels,
-          keep_image_id_for_visualization_export=eval_config.
-          keep_image_id_for_visualization_export)
-    return result_dict, result_losses_dict
-
-  variables_to_restore = tf.compat.v1.global_variables()
-  global_step = tf.compat.v1.train.get_or_create_global_step()
-  variables_to_restore.append(global_step)
-
-  if graph_hook_fn: graph_hook_fn()
-
-  if eval_config.use_moving_averages:
-    variable_averages = tf.train.ExponentialMovingAverage(0.0)
-    variables_to_restore = variable_averages.variables_to_restore()
-  saver = tf.compat.v1.train.Saver(variables_to_restore)
-
-  def _restore_latest_checkpoint(sess):
-    latest_checkpoint = tf.train.latest_checkpoint(checkpoint_dir)
-    saver.restore(sess, latest_checkpoint)
-
-  if not evaluator_list:
-    evaluator_list = get_evaluators(eval_config, categories)
-
-  metrics = eval_util.repeated_checkpoint_run(
-      tensor_dict=tensor_dict,
-      summary_dir=eval_dir,
-      evaluators=evaluator_list,
-      batch_processor=_process_batch,
-      checkpoint_dirs=[checkpoint_dir],
-      variables_to_restore=None,
-      restore_fn=_restore_latest_checkpoint,
-      num_batches=eval_config.num_examples,
-      eval_interval_secs=eval_config.eval_interval_secs,
-      max_number_of_evaluations=(1 if eval_config.ignore_groundtruth else
-                                 eval_config.max_evals
-                                 if eval_config.max_evals else None),
-      master=eval_config.eval_master,
-      save_graph=eval_config.save_graph,
-      save_graph_dir=(eval_dir if eval_config.save_graph else ''),
-      losses_dict=losses_dict)
-
-  return metrics
diff --git a/models/object_detection/tensorflow/rfcn/inference/fp32/run_rfcn_inference.py b/models/object_detection/tensorflow/rfcn/inference/fp32/run_rfcn_inference.py
deleted file mode 100644
index 36e73d5cf..000000000
--- a/models/object_detection/tensorflow/rfcn/inference/fp32/run_rfcn_inference.py
+++ /dev/null
@@ -1,284 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-#
-
-import numpy as np
-import os
-import six.moves.urllib as urllib
-import sys
-import tarfile
-import tensorflow as tf
-import zipfile
-import subprocess
-
-from collections import defaultdict
-from io import StringIO
-import matplotlib
-matplotlib.use('Agg')
-from matplotlib import pyplot as plt
-from PIL import Image
-import time
-import argparse
-from tensorflow.python.client import timeline
-import importlib
-
-class RFCNRunner:
-  '''Add code here to detect the environment and set necessary variables before launching the model'''
-  args=None
-  custom_args=None
-  RESEARCH_DIR = 'research'
-  OBJ_DETECTION_DIR = 'object_detection'
-  DATA_DIR = 'data'
-  label_map_file = 'mscoco_label_map.pbtxt'
-  research_dir = ''
-  NUM_CLASSES = 90
-  detection_graph = None
-  test_image_paths = []
-  TEST_IMG_FILE = "000000578871.jpg"
-  DEFAULT_INTEROP_THREADS = 2
-  RFCN_OUTPUTS = ['num_detections', 'detection_boxes', 'detection_scores',
-              'detection_classes', 'detection_masks']
-  STEP_SIZE = 10
-  MAX_STEPS = 5000
-  label_map_util = None
-  vis_util = None
-  # Size, in inches, of the output images.
-  IMAGE_SIZE = (12, 8)
-
-  def __init__(self, args):
-    self.args = args
-    self.parse_args()
-    self.log('Received these standard args: {}'.format(self.args))
-
-  def parse_args(self):
-    parser = argparse.ArgumentParser()
-    mutex_group = parser.add_mutually_exclusive_group()
-    mutex_group.add_argument('-x', '--number_of_steps', help='Run for n number of steps', type=int, default=None)
-    mutex_group.add_argument('-z', '--visualize', help='Whether to visulize the output image', action='store_true' )
-    parser.add_argument('-v', '--verbose', help='Print some useful info.', action='store_true' )
-    parser.add_argument('-t', '--timeline', help='Output file name for TF timeline', type=str, default=None)
-    parser.add_argument('-e', '--evaluate_tensor', help='Full tensor name to evaluate', type=str, default=None)
-    parser.add_argument('-p', '--print_accuracy', help='Print accuracy results', action='store_true')
-    parser.add_argument('-g', '--input_graph', help='The input frozen graph pb file', dest='input_graph', required=True, default=None)
-    parser.add_argument('-d', '--data_location', help='The location of the image data to be analyzed.', dest='data_location', default=None, required=True)
-    parser.add_argument('-m', '--tensorflow-models-path',
-        help='Path to the tensorflow-models directory (or clone of github.com/tensorflow/models',
-        dest='tf_models_path', default=None, required=True)
-    parser.add_argument(
-        '--num-inter-threads', dest='num_inter_threads',
-        help='number threads across operators',
-        type=int, default=2)
-    parser.add_argument(
-        '--num-intra-threads', dest='num_intra_threads',
-        help='number threads for an operator',
-        type=int, default=56)
-    self.args = parser.parse_args()
-    self.validate_args()
-    self.finish_import()
-
-  def log(self, msg):
-    if self.args.verbose: print(msg)
-
-  def validate_args(self):
-    self.log('Validating Args...')
-    self.research_dir = os.path.join(self.args.tf_models_path, self.RESEARCH_DIR)
-    if not ( self.args.data_location and
-        os.path.exists(os.path.join(self.args.data_location, self.TEST_IMG_FILE))):
-      raise ValueError ("Unable to locate images for evaluation at {}".format(self.args.data_location))
-    if os.path.isdir(self.research_dir):
-      # List of the strings that is used to add correct label for each box.
-      self.label_map_file = os.path.join(self.research_dir,
-                                          self.OBJ_DETECTION_DIR,
-                                          self.DATA_DIR,
-                                          self.label_map_file)
-      if not os.path.exists(self.label_map_file):
-        raise ValueError ("Unable to locate label map file at {}".format(self.label_map_file))
-    else:
-      raise ValueError ("{} is not a valid path to the TensorFlow models.".format(self.args.tf_models_path))
-
-    if not os.path.exists(self.args.input_graph):
-      raise ValueError("Unable to find the input graph protobuf file: {}".format(self.args.input_graph))
-
-  def finish_import(self):
-    # This is needed since the notebook is stored in the object_detection folder.
-    sys.path.append(self.research_dir)
-    # This is needed to display the images.
-    if (self.args.visualize and self.args.evaluate_tensor is None):
-      from IPython import get_ipython
-      get_ipython().run_line_magic('matplotlib', 'tk')
-    self.label_map_util = importlib.import_module('..label_map_util', package='object_detection.utils.label_map_util')
-    self.vis_util = importlib.import_module('..visualization_utils', package='object_detection.utils.visualization_utils')
-
-  def run(self):
-      self.log("Running performance test")
-      self.read_graph()
-      self.get_image_paths()
-      #self.load_label_map()
-      # Actual detection.
-      output_dict, image_np = self.run_inference(self.detection_graph)
-      self.visualize(output_dict, image_np)
-
-  def visualize(self, output_dict, image_np):
-    # Visualization of the results of a detection.
-    if (self.args.visualize and
-        self.args.evaluate_tensor is None and
-        self.category_index and
-        output_dict and
-        image_np ):
-      self.vis_util.visualize_boxes_and_labels_on_image_array(
-          image_np,
-          output_dict['detection_boxes'],
-          output_dict['detection_classes'],
-          output_dict['detection_scores'],
-          self.category_index,
-          instance_masks=output_dict.get('detection_masks'),
-          use_normalized_coordinates=True,
-          line_thickness=8)
-      plt.figure(figsize=self.IMAGE_SIZE)
-      plt.imshow(image_np)
-
-  def read_graph(self):
-    self.detection_graph = tf.Graph()
-    with self.detection_graph.as_default():
-      od_graph_def = tf.compat.v1.GraphDef()
-      with tf.io.gfile.GFile(self.args.input_graph, 'rb') as fid:
-        serialized_graph = fid.read()
-        od_graph_def.ParseFromString(serialized_graph)
-        tf.import_graph_def(od_graph_def, name='')
-
-
-  def get_image_paths(self):
-    if (self.args.visualize):
-      self.test_image_paths = [os.path.join(self.args.data_location, self.TEST_IMG_FILE)]
-    else:
-      self.test_image_paths = []
-      for root, dirs, files in os.walk(self.args.data_location):
-        for file in files:
-          self.test_image_paths.append(os.path.join(self.args.data_location, file))
-
-  def load_label_map(self):
-    label_map = self.label_map_util.load_labelmap(self.label_map_file)
-    categories = self.label_map_util.convert_label_map_to_categories(
-        label_map, max_num_classes=self.NUM_CLASSES, use_display_name=True)
-    if (self.args.visualize and self.args.evaluate_tensor is None):
-      self.category_index = self.label_map_util.create_category_index(categories)
-
-  def load_image_into_numpy_array(self, image):
-    (im_width, im_height) = image.size
-    if image.mode == 'L':
-      np_image = np.array(image.getdata()).reshape(
-          (im_height, im_width)).astype(np.uint8)
-      return np.stack((np_image,)*3, -1)
-    else:
-      return np.array(image.getdata()).reshape(
-          (im_height, im_width, 3)).astype(np.uint8)
-
-  def run_inference(self,graph):
-    sess_config = tf.compat.v1.ConfigProto()
-    sess_config.intra_op_parallelism_threads = self.args.num_intra_threads
-    sess_config.inter_op_parallelism_threads = self.args.num_inter_threads
-    with self.detection_graph.as_default():
-      with tf.compat.v1.Session(config=sess_config) as sess:
-        # Get handles to input and output tensors
-        tensor_dict = {}
-        if not self.args.evaluate_tensor:
-          ops = tf.compat.v1.get_default_graph().get_operations()
-          all_tensor_names = {output.name for op in ops for output in op.outputs}
-          for key in self.RFCN_OUTPUTS:
-            tensor_name = key + ':0'
-            if tensor_name in all_tensor_names:
-              tensor_dict[key] = tf.compat.v1.get_default_graph().get_tensor_by_name(
-                  tensor_name)
-        else:
-          our_op = tf.compat.v1.get_default_graph().get_operation_by_name(self.args.evaluate_tensor)
-          tensor_names = our_op.outputs
-          list_ops = []
-          for i, tensor in enumerate(tensor_names):
-            list_ops.append(tensor.name)
-          tensor_dict[self.args.evaluate_tensor] = list_ops
-
-        run_options = None
-        run_metadata = None
-        if self.args.timeline:
-          run_options = tf.compat.v1.RunOptions(trace_level=tf.compat.v1.RunOptions.FULL_TRACE)
-          run_metadata = tf.compat.v1.RunMetadata()
-
-        total_duration = 0
-        for index, image_path in enumerate(self.test_image_paths):
-          image = Image.open(image_path)
-          # the array based representation of the image will be used later in order to prepare the
-          # result image with boxes and labels on it.
-          image_np = self.load_image_into_numpy_array(image)
-          image_tensor = tf.compat.v1.get_default_graph().get_tensor_by_name('image_tensor:0')
-
-          # Run inference
-          start_time = time.time()
-          #if self.args.timeline:
-          output_dict = sess.run(tensor_dict,
-                       feed_dict={image_tensor: np.expand_dims(image_np, 0)},
-                       options=run_options, run_metadata=run_metadata)
-          # else:
-          #   output_dict = sess.run(tensor_dict,
-          #                         feed_dict={image_tensor: np.expand_dims(image_np, 0)})
-          step_duration = time.time() - start_time
-          total_duration = total_duration + step_duration
-
-          if (self.args.visualize):
-            if index == 0:
-              print ('Avg. Duration per Step:' + str(total_duration / 1))
-          else:
-            if (index % self.STEP_SIZE == 0):
-              print ('Step ' + str(index) + ': ' + str(step_duration) + ' seconds', flush=True)
-            if index == self.MAX_STEPS - 1:
-              print ('Avg. Duration per Step:' + str(total_duration / self.MAX_STEPS))
-
-          if self.args.number_of_steps and index == (self.args.number_of_steps - 1):
-              print ('Avg. Duration per Step:' +
-                    str(total_duration / self.args.number_of_steps))
-              break
-
-          if self.args.timeline:
-            trace = timeline.Timeline(step_stats=run_metadata.step_stats)
-            with open('tl-' + time.strftime("%Y%m%d-%H%M%S") + '-' + self.args.timeline, 'w') as file:
-              file.write(trace.generate_chrome_trace_format(show_memory=False))
-
-          if self.args.evaluate_tensor:
-            for tensor in output_dict[self.args.evaluate_tensor]:
-              print (tensor.shape)
-            return None, None
-
-          # all outputs are float32 numpy arrays, so convert types as appropriate
-          output_dict['num_detections'] = int(output_dict['num_detections'][0])
-          output_dict['detection_classes'] = output_dict[
-              'detection_classes'][0].astype(np.uint8)
-          output_dict['detection_boxes'] = output_dict['detection_boxes'][0]
-          output_dict['detection_scores'] = output_dict['detection_scores'][0]
-
-          if (self.args.print_accuracy):
-            print ('num_detections:\n' + str(output_dict['num_detections']))
-            print ('detection_classes:\n' + str(output_dict['detection_classes']))
-            print ('detection_boxes:\n' + str(output_dict['detection_boxes']))
-            print ('detection_scores:\n' + str(output_dict['detection_scores']))
-
-          if 'detection_masks' in output_dict:
-            output_dict['detection_masks'] = output_dict['detection_masks'][0]
-    return output_dict, image_np
-
-if __name__ == "__main__":
-  rr = RFCNRunner(sys.argv)
-  rr.run()
diff --git a/models/object_detection/tensorflow/rfcn/inference/int8/__init__.py b/models/object_detection/tensorflow/rfcn/inference/int8/__init__.py
deleted file mode 100644
index c4fdb7d61..000000000
--- a/models/object_detection/tensorflow/rfcn/inference/int8/__init__.py
+++ /dev/null
@@ -1,19 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
diff --git a/models/object_detection/tensorflow/rfcn/inference/int8/coco_mAP.sh b/models/object_detection/tensorflow/rfcn/inference/int8/coco_mAP.sh
deleted file mode 100755
index 94b4d8b05..000000000
--- a/models/object_detection/tensorflow/rfcn/inference/int8/coco_mAP.sh
+++ /dev/null
@@ -1,63 +0,0 @@
-#!/usr/bin/env bash
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
-
-########## Variables to be defined
-
-SPLIT=${SPLIT:-"RFCN_final_graph"} #change to your favorite room
-FROZEN_GRAPH=${FROZEN_GRAPH:-"/in_graph/rfcn_resnet101_int8_coco_pretrained_model.pb"}
-TF_RECORD_FILE=${TF_RECORD_FILE:-"/dataset/coco_val.record"}
-if [[ -z ${TF_MODELS_ROOT} ]] || [[ ! -d ${TF_MODELS_ROOT} ]]; then
-  echo "You must specify the root of the tensorflow/models source tree in the TF_MODELS_ROOT environment variable."
-  exit 1
-fi
-
-export PYTHONPATH=$PYTHONPATH:${TF_MODELS_ROOT}/research:${TF_MODELS_ROOT}/research/slim:${TF_MODELS_ROOT}/research/object_detection
-
-echo "SPLIT=${SPLIT}"
-echo "FROZEN_GRAPH=${FROZEN_GRAPH}"
-echo "TF_RECORD_FILE=${TF_RECORD_FILE}"
-echo "PYTHONPATH=${PYTHONPATH}"
-echo "TF_MODELS_ROOT=$TF_MODELS_ROOT"
-
-python -m object_detection.inference.infer_detections \
-  --input_tfrecord_paths=$TF_RECORD_FILE \
-  --output_tfrecord_path=${SPLIT}_detections.tfrecord \
-  --inference_graph=$FROZEN_GRAPH \
-  --discard_image_pixels
-
-
-mkdir -p ${SPLIT}_eval_metrics
-
-echo "
-label_map_path: '${TF_MODELS_ROOT}/research/object_detection/data/mscoco_label_map.pbtxt'
-tf_record_input_reader: { input_path: '${SPLIT}_detections.tfrecord' }
-" > ${SPLIT}_eval_metrics/${SPLIT}_input_config.pbtxt
-
-echo "
-metrics_set: 'coco_detection_metrics'
-" > ${SPLIT}_eval_metrics/${SPLIT}_eval_config.pbtxt
-
-
-python -m object_detection.metrics.offline_eval_map_corloc \
-  --eval_dir=${SPLIT}_eval_metrics \
-  --eval_config_path=${SPLIT}_eval_metrics/${SPLIT}_eval_config.pbtxt \
-  --input_config_path=${SPLIT}_eval_metrics/${SPLIT}_input_config.pbtxt
-
diff --git a/models/object_detection/tensorflow/rfcn/inference/int8/run_rfcn_inference.py b/models/object_detection/tensorflow/rfcn/inference/int8/run_rfcn_inference.py
deleted file mode 100644
index e2ff30c20..000000000
--- a/models/object_detection/tensorflow/rfcn/inference/int8/run_rfcn_inference.py
+++ /dev/null
@@ -1,284 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-#
-
-import numpy as np
-import os
-import six.moves.urllib as urllib
-import sys
-import tarfile
-import tensorflow as tf
-import zipfile
-import subprocess
-
-from collections import defaultdict
-from io import StringIO
-import matplotlib
-matplotlib.use('Agg')
-from matplotlib import pyplot as plt
-from PIL import Image
-import time
-import argparse
-from tensorflow.python.client import timeline
-import importlib
-
-class RFCNRunner:
-  '''Add code here to detect the environment and set necessary variables before launching the model'''
-  args=None
-  custom_args=None
-  RESEARCH_DIR = 'research'
-  OBJ_DETECTION_DIR = 'object_detection'
-  DATA_DIR = 'data'
-  label_map_file = 'mscoco_label_map.pbtxt'
-  research_dir = ''
-  NUM_CLASSES = 90
-  detection_graph = None
-  test_image_paths = []
-  TEST_IMG_FILE = "000000578871.jpg"
-  DEFAULT_INTEROP_THREADS = 2
-  RFCN_OUTPUTS = ['num_detections', 'detection_boxes', 'detection_scores',
-              'detection_classes', 'detection_masks']
-  STEP_SIZE = 10
-  MAX_STEPS = 5000
-  label_map_util = None
-  vis_util = None
-  # Size, in inches, of the output images.
-  IMAGE_SIZE = (12, 8)
-
-  def __init__(self, args):
-    self.args = args
-    self.parse_args()
-    self.log('Received these standard args: {}'.format(self.args))
-
-  def parse_args(self):
-    parser = argparse.ArgumentParser()
-    mutex_group = parser.add_mutually_exclusive_group()
-    mutex_group.add_argument('-x', '--number_of_steps', help='Run for n number of steps', type=int, default=None)
-    mutex_group.add_argument('-z', '--visualize', help='Whether to visulize the output image', action='store_true' )
-    parser.add_argument('-v', '--verbose', help='Print some useful info.', action='store_true' )
-    parser.add_argument('-t', '--timeline', help='Output file name for TF timeline', type=str, default=None)
-    parser.add_argument('-e', '--evaluate_tensor', help='Full tensor name to evaluate', type=str, default=None)
-    parser.add_argument('-p', '--print_accuracy', help='Print accuracy results', action='store_true')
-    parser.add_argument('-g', '--input_graph', help='The input frozen graph pb file', dest='input_graph', required=True, default=None)
-    parser.add_argument('-d', '--data_location', help='The location of the image data to be analyzed.', dest='data_location', default=None, required=True)
-    parser.add_argument('-m', '--tensorflow-models-path', 
-        help='Path to the tensorflow-models directory (or clone of github.com/tensorflow/models', 
-        dest='tf_models_path', default=None, required=True)
-    parser.add_argument(
-        '--num-inter-threads', dest='num_inter_threads',
-        help='number threads across operators',
-        type=int, default=2)
-    parser.add_argument(
-        '--num-intra-threads', dest='num_intra_threads',
-        help='number threads for an operator',
-        type=int, default=56)
-    self.args = parser.parse_args()
-    self.validate_args()
-    self.finish_import()
-
-  def log(self, msg):
-    if self.args.verbose: print(msg)
-
-  def validate_args(self):
-    self.log('Validating Args...')
-    self.research_dir = os.path.join(self.args.tf_models_path, self.RESEARCH_DIR)
-    if not ( self.args.data_location and 
-        os.path.exists(os.path.join(self.args.data_location, self.TEST_IMG_FILE))):
-      raise ValueError ("Unable to locate images for evaluation at {}".format(self.args.data_location))
-    if os.path.isdir(self.research_dir):
-      # List of the strings that is used to add correct label for each box.
-      self.label_map_file = os.path.join(self.research_dir, 
-                                          self.OBJ_DETECTION_DIR, 
-                                          self.DATA_DIR, 
-                                          self.label_map_file)
-      if not os.path.exists(self.label_map_file):
-        raise ValueError ("Unable to locate label map file at {}".format(self.label_map_file))
-    else:
-      raise ValueError ("{} is not a valid path to the TensorFlow models.".format(self.args.tf_models_path))
-    
-    if not os.path.exists(self.args.input_graph):
-      raise ValueError("Unable to find the input graph protobuf file: {}".format(self.args.input_graph))
-
-  def finish_import(self):
-    # This is needed since the notebook is stored in the object_detection folder.
-    sys.path.append(self.research_dir)
-    # This is needed to display the images.
-    if (self.args.visualize and self.args.evaluate_tensor is None):
-      from IPython import get_ipython
-      get_ipython().run_line_magic('matplotlib', 'tk')
-    self.label_map_util = importlib.import_module('..label_map_util', package='object_detection.utils.label_map_util')
-    self.vis_util = importlib.import_module('..visualization_utils', package='object_detection.utils.visualization_utils')
-
-  def run(self):
-      self.log("Running performance test")
-      self.read_graph()
-      self.get_image_paths()
-      #self.load_label_map()
-      # Actual detection.
-      output_dict, image_np = self.run_inference(self.detection_graph)
-      self.visualize(output_dict, image_np)
-
-  def visualize(self, output_dict, image_np):
-    # Visualization of the results of a detection.
-    if (self.args.visualize and 
-        self.args.evaluate_tensor is None and 
-        self.category_index and
-        output_dict and
-        image_np ):
-      self.vis_util.visualize_boxes_and_labels_on_image_array(
-          image_np,
-          output_dict['detection_boxes'],
-          output_dict['detection_classes'],
-          output_dict['detection_scores'],
-          self.category_index,
-          instance_masks=output_dict.get('detection_masks'),
-          use_normalized_coordinates=True,
-          line_thickness=8)
-      plt.figure(figsize=self.IMAGE_SIZE)
-      plt.imshow(image_np)
-
-  def read_graph(self):
-    self.detection_graph = tf.Graph()
-    with self.detection_graph.as_default():
-      od_graph_def = tf.compat.v1.GraphDef()
-      with tf.io.gfile.GFile(self.args.input_graph, 'rb') as fid:
-        serialized_graph = fid.read()
-        od_graph_def.ParseFromString(serialized_graph)
-        tf.import_graph_def(od_graph_def, name='')
-
-
-  def get_image_paths(self):
-    if (self.args.visualize):
-      self.test_image_paths = [os.path.join(self.args.data_location, self.TEST_IMG_FILE)]
-    else:
-      self.test_image_paths = []
-      for root, dirs, files in os.walk(self.args.data_location):
-        for file in files:
-          self.test_image_paths.append(os.path.join(self.args.data_location, file))
-
-  def load_label_map(self):
-    label_map = self.label_map_util.load_labelmap(self.label_map_file)
-    categories = self.label_map_util.convert_label_map_to_categories(
-        label_map, max_num_classes=self.NUM_CLASSES, use_display_name=True)
-    if (self.args.visualize and self.args.evaluate_tensor is None):
-      self.category_index = self.label_map_util.create_category_index(categories)
-
-  def load_image_into_numpy_array(self, image):
-    (im_width, im_height) = image.size
-    if image.mode == 'L':
-      np_image = np.array(image.getdata()).reshape(
-          (im_height, im_width)).astype(np.uint8)
-      return np.stack((np_image,)*3, -1)
-    else:
-      return np.array(image.getdata()).reshape(
-          (im_height, im_width, 3)).astype(np.uint8)
-
-  def run_inference(self,graph):
-    sess_config = tf.compat.v1.ConfigProto()
-    sess_config.intra_op_parallelism_threads = self.args.num_intra_threads
-    sess_config.inter_op_parallelism_threads = self.args.num_inter_threads
-    with self.detection_graph.as_default():
-      with tf.compat.v1.Session(config=sess_config) as sess:
-        # Get handles to input and output tensors
-        tensor_dict = {}
-        if not self.args.evaluate_tensor:
-          ops = tf.compat.v1.get_default_graph().get_operations()
-          all_tensor_names = {output.name for op in ops for output in op.outputs}
-          for key in self.RFCN_OUTPUTS:
-            tensor_name = key + ':0'
-            if tensor_name in all_tensor_names:
-              tensor_dict[key] = tf.compat.v1.get_default_graph().get_tensor_by_name(
-                  tensor_name)
-        else:
-          our_op = tf.compat.v1.get_default_graph().get_operation_by_name(self.args.evaluate_tensor)
-          tensor_names = our_op.outputs
-          list_ops = []
-          for i, tensor in enumerate(tensor_names):
-            list_ops.append(tensor.name)
-          tensor_dict[self.args.evaluate_tensor] = list_ops
-
-        run_options = None
-        run_metadata = None
-        if self.args.timeline:
-          run_options = tf.compat.v1.RunOptions(trace_level=tf.compat.v1.RunOptions.FULL_TRACE)
-          run_metadata = tf.compat.v1.RunMetadata()
-
-        total_duration = 0
-        for index, image_path in enumerate(self.test_image_paths):
-          image = Image.open(image_path)
-          # the array based representation of the image will be used later in order to prepare the
-          # result image with boxes and labels on it.
-          image_np = self.load_image_into_numpy_array(image)
-          image_tensor = tf.compat.v1.get_default_graph().get_tensor_by_name('image_tensor:0')
-
-          # Run inference
-          start_time = time.time()
-          #if self.args.timeline:
-          output_dict = sess.run(tensor_dict,
-                       feed_dict={image_tensor: np.expand_dims(image_np, 0)},
-                       options=run_options, run_metadata=run_metadata)
-          # else:
-          #   output_dict = sess.run(tensor_dict,
-          #                         feed_dict={image_tensor: np.expand_dims(image_np, 0)})
-          step_duration = time.time() - start_time
-          total_duration = total_duration + step_duration
-
-          if (self.args.visualize):
-            if index == 0:
-              print ('Avg. Duration per Step:' + str(total_duration / 1))
-          else:
-            if (index % self.STEP_SIZE == 0):
-              print ('Step ' + str(index) + ': ' + str(step_duration) + ' seconds')
-            if index == self.MAX_STEPS - 1:
-              print ('Avg. Duration per Step:' + str(total_duration / self.MAX_STEPS))
-
-          if self.args.number_of_steps and index == (self.args.number_of_steps - 1):
-              print ('Avg. Duration per Step:' +
-                    str(total_duration / self.args.number_of_steps))
-              break
-
-          if self.args.timeline:
-            trace = timeline.Timeline(step_stats=run_metadata.step_stats)
-            with open('tl-' + time.strftime("%Y%m%d-%H%M%S") + '-' + self.args.timeline, 'w') as file:
-              file.write(trace.generate_chrome_trace_format(show_memory=False))
-
-          if self.args.evaluate_tensor:
-            for tensor in output_dict[self.args.evaluate_tensor]:
-              print (tensor.shape)
-            return None, None
-
-          # all outputs are float32 numpy arrays, so convert types as appropriate
-          output_dict['num_detections'] = int(output_dict['num_detections'][0])
-          output_dict['detection_classes'] = output_dict[
-              'detection_classes'][0].astype(np.uint8)
-          output_dict['detection_boxes'] = output_dict['detection_boxes'][0]
-          output_dict['detection_scores'] = output_dict['detection_scores'][0]
-
-          if (self.args.print_accuracy):
-            print ('num_detections:\n' + str(output_dict['num_detections']))
-            print ('detection_classes:\n' + str(output_dict['detection_classes']))
-            print ('detection_boxes:\n' + str(output_dict['detection_boxes']))
-            print ('detection_scores:\n' + str(output_dict['detection_scores']))
-
-          if 'detection_masks' in output_dict:
-            output_dict['detection_masks'] = output_dict['detection_masks'][0]
-    return output_dict, image_np
-
-if __name__ == "__main__":
-  rr = RFCNRunner(sys.argv)
-  rr.run()
diff --git a/models/object_detection/tensorflow/rfcn/inference/tf-2.0.patch b/models/object_detection/tensorflow/rfcn/inference/tf-2.0.patch
deleted file mode 100644
index adb5c9566..000000000
--- a/models/object_detection/tensorflow/rfcn/inference/tf-2.0.patch
+++ /dev/null
@@ -1,617 +0,0 @@
-diff --git a/research/object_detection/eval_util.py b/research/object_detection/eval_util.py
-index 51c29455..e3970e4f 100644
---- a/research/object_detection/eval_util.py
-+++ b/research/object_detection/eval_util.py
-@@ -30,8 +30,6 @@ from object_detection.utils import label_map_util
- from object_detection.utils import ops
- from object_detection.utils import visualization_utils as vis_utils
- 
--slim = tf.contrib.slim
--
- 
- def write_metrics(metrics, global_step, summary_dir):
-   """Write metrics to a summary directory.
-@@ -42,10 +40,10 @@ def write_metrics(metrics, global_step, summary_dir):
-     summary_dir: Directory to write tensorflow summaries to.
-   """
-   logging.info('Writing metrics to tf summary.')
--  summary_writer = tf.summary.FileWriterCache.get(summary_dir)
-+  summary_writer = tf.compat.v1.summary.FileWriterCache.get(summary_dir)
-   for key in sorted(metrics):
--    summary = tf.Summary(value=[
--        tf.Summary.Value(tag=key, simple_value=metrics[key]),
-+    summary = tf.compat.v1.Summary(value=[
-+        tf.compat.v1.Summary.Value(tag=key, simple_value=metrics[key]),
-     ])
-     summary_writer.add_summary(summary, global_step)
-     logging.info('%s: %f', key, metrics[key])
-@@ -183,14 +181,14 @@ def visualize_detection_results(result_dict,
-       export_path = os.path.join(export_dir, 'export-{}.png'.format(tag))
-     vis_utils.save_image_array_as_png(image, export_path)
- 
--  summary = tf.Summary(value=[
--      tf.Summary.Value(
-+  summary = tf.compat.v1.Summary(value=[
-+      tf.compat.v1.Summary.Value(
-           tag=tag,
--          image=tf.Summary.Image(
-+          image=tf.compat.v1.Summary.Image(
-               encoded_image_string=vis_utils.encode_image_array_as_png_str(
-                   image)))
-   ])
--  summary_writer = tf.summary.FileWriterCache.get(summary_dir)
-+  summary_writer = tf.compat.v1.summary.FileWriterCache.get(summary_dir)
-   summary_writer.add_summary(summary, global_step)
- 
-   logging.info('Detection visualizations written to summary with tag %s.', tag)
-@@ -258,25 +256,25 @@ def _run_checkpoint_once(tensor_dict,
-   """
-   if save_graph and not save_graph_dir:
-     raise ValueError('`save_graph_dir` must be defined.')
--  sess = tf.Session(master, graph=tf.get_default_graph())
--  sess.run(tf.global_variables_initializer())
--  sess.run(tf.local_variables_initializer())
--  sess.run(tf.tables_initializer())
-+  sess = tf.compat.v1.Session(master, graph=tf.compat.v1.get_default_graph())
-+  sess.run(tf.compat.v1.global_variables_initializer())
-+  sess.run(tf.compat.v1.local_variables_initializer())
-+  sess.run(tf.compat.v1.tables_initializer())
-   if restore_fn:
-     restore_fn(sess)
-   else:
-     if not checkpoint_dirs:
-       raise ValueError('`checkpoint_dirs` must have at least one entry.')
-     checkpoint_file = tf.train.latest_checkpoint(checkpoint_dirs[0])
--    saver = tf.train.Saver(variables_to_restore)
-+    saver = tf.compat.v1.train.Saver(variables_to_restore)
-     saver.restore(sess, checkpoint_file)
- 
-   if save_graph:
--    tf.train.write_graph(sess.graph_def, save_graph_dir, 'eval.pbtxt')
-+    tf.io.write_graph(sess.graph_def, save_graph_dir, 'eval.pbtxt')
- 
-   counters = {'skipped': 0, 'success': 0}
-   aggregate_result_losses_dict = collections.defaultdict(list)
--  with tf.contrib.slim.queues.QueueRunners(sess):
-+  with queues.QueueRunners(sess):
-     try:
-       for batch in range(int(num_batches)):
-         if (batch + 1) % 100 == 0:
-@@ -322,7 +320,7 @@ def _run_checkpoint_once(tensor_dict,
-         if any(key in all_evaluator_metrics for key in metrics):
-           raise ValueError('Metric names between evaluators must not collide.')
-         all_evaluator_metrics.update(metrics)
--      global_step = tf.train.global_step(sess, tf.train.get_global_step())
-+      global_step = tf.compat.v1.train.global_step(sess, tf.compat.v1.train.get_global_step())
- 
-       for key, value in iter(aggregate_result_losses_dict.items()):
-         all_evaluator_metrics['Losses/' + key] = np.mean(value)
-@@ -506,17 +504,17 @@ def result_dict_for_single_example(image,
- 
-   detection_fields = fields.DetectionResultFields
-   detection_boxes = detections[detection_fields.detection_boxes][0]
--  image_shape = tf.shape(image)
-+  image_shape = tf.shape(input=image)
-   detection_scores = detections[detection_fields.detection_scores][0]
- 
-   if class_agnostic:
-     detection_classes = tf.ones_like(detection_scores, dtype=tf.int64)
-   else:
-     detection_classes = (
--        tf.to_int64(detections[detection_fields.detection_classes][0]) +
-+        tf.cast(detections[detection_fields.detection_classes][0], dtype=tf.int64) +
-         label_id_offset)
- 
--  num_detections = tf.to_int32(detections[detection_fields.num_detections][0])
-+  num_detections = tf.cast(detections[detection_fields.num_detections][0], dtype=tf.int32)
-   detection_boxes = tf.slice(
-       detection_boxes, begin=[0, 0], size=[num_detections, -1])
-   detection_classes = tf.slice(
-@@ -558,11 +556,10 @@ def result_dict_for_single_example(image,
-     if input_data_fields.groundtruth_instance_masks in groundtruth:
-       masks = groundtruth[input_data_fields.groundtruth_instance_masks]
-       masks = tf.expand_dims(masks, 3)
--      masks = tf.image.resize_images(
-+      masks = tf.image.resize(
-           masks,
-           image_shape[1:3],
--          method=tf.image.ResizeMethod.NEAREST_NEIGHBOR,
--          align_corners=True)
-+          method=tf.image.ResizeMethod.NEAREST_NEIGHBOR)
-       masks = tf.squeeze(masks, 3)
-       groundtruth[input_data_fields.groundtruth_instance_masks] = tf.cast(
-           masks, tf.uint8)
-diff --git a/research/object_detection/inference/detection_inference.py b/research/object_detection/inference/detection_inference.py
-index dc66686f..cbdf106b 100644
---- a/research/object_detection/inference/detection_inference.py
-+++ b/research/object_detection/inference/detection_inference.py
-@@ -31,16 +31,16 @@ def build_input(tfrecord_paths):
-     image_tensor: The decoded image of the example. Uint8 tensor,
-         shape=[1, None, None,3]
-   """
--  filename_queue = tf.train.string_input_producer(
-+  filename_queue = tf.compat.v1.train.string_input_producer(
-       tfrecord_paths, shuffle=False, num_epochs=1)
- 
--  tf_record_reader = tf.TFRecordReader()
-+  tf_record_reader = tf.compat.v1.TFRecordReader()
-   _, serialized_example_tensor = tf_record_reader.read(filename_queue)
--  features = tf.parse_single_example(
--      serialized_example_tensor,
-+  features = tf.io.parse_single_example(
-+      serialized=serialized_example_tensor,
-       features={
-           standard_fields.TfExampleFields.image_encoded:
--              tf.FixedLenFeature([], tf.string),
-+              tf.io.FixedLenFeature([], tf.string),
-       })
-   encoded_image = features[standard_fields.TfExampleFields.image_encoded]
-   image_tensor = tf.image.decode_image(encoded_image, channels=3)
-@@ -65,15 +65,15 @@ def build_inference_graph(image_tensor, inference_graph_path):
-     detected_labels_tensor: Detected labels. Int64 tensor,
-         shape=[num_detections]
-   """
--  with tf.gfile.Open(inference_graph_path, 'r') as graph_def_file:
-+  with tf.io.gfile.GFile(inference_graph_path, 'rb') as graph_def_file:
-     graph_content = graph_def_file.read()
--  graph_def = tf.GraphDef()
-+  graph_def = tf.compat.v1.GraphDef()
-   graph_def.MergeFromString(graph_content)
- 
-   tf.import_graph_def(
-       graph_def, name='', input_map={'image_tensor': image_tensor})
- 
--  g = tf.get_default_graph()
-+  g = tf.compat.v1.get_default_graph()
- 
-   num_detections_tensor = tf.squeeze(
-       g.get_tensor_by_name('num_detections:0'), 0)
-@@ -114,7 +114,7 @@ def infer_detections_and_add_to_example(
-   """
-   tf_example = tf.train.Example()
-   (serialized_example, detected_boxes, detected_scores,
--   detected_classes) = tf.get_default_session().run([
-+   detected_classes) = tf.compat.v1.get_default_session().run([
-        serialized_example_tensor, detected_boxes_tensor, detected_scores_tensor,
-        detected_labels_tensor
-    ])
-diff --git a/research/object_detection/inference/infer_detections.py b/research/object_detection/inference/infer_detections.py
-index a251009e..23a61f20 100644
---- a/research/object_detection/inference/infer_detections.py
-+++ b/research/object_detection/inference/infer_detections.py
-@@ -37,24 +37,27 @@ metrics).
- import itertools
- import tensorflow as tf
- from object_detection.inference import detection_inference
-+import argparse
- 
--tf.flags.DEFINE_string('input_tfrecord_paths', None,
--                       'A comma separated list of paths to input TFRecords.')
--tf.flags.DEFINE_string('output_tfrecord_path', None,
--                       'Path to the output TFRecord.')
--tf.flags.DEFINE_string('inference_graph', None,
--                       'Path to the inference graph with embedded weights.')
--tf.flags.DEFINE_boolean('discard_image_pixels', False,
--                        'Discards the images in the output TFExamples. This'
--                        ' significantly reduces the output size and is useful'
--                        ' if the subsequent tools don\'t need access to the'
--                        ' images (e.g. when computing evaluation measures).')
--
--FLAGS = tf.flags.FLAGS
--
-+parser = argparse.ArgumentParser()
-+parser.add_argument('-i', '--input_tfrecord_paths',
-+                    help='A comma separated list of paths to input TFRecords.',
-+                    default=None)
-+parser.add_argument('-o', '--output_tfrecord_path',
-+                    help='Path to the output TFRecord.', default=None)
-+parser.add_argument('-g', '--inference_graph',
-+                    help='Path to the inference graph with embedded weights.',
-+                    default=None)
-+parser.add_argument('-d', '--discard_image_pixels',
-+                    help='Discards the images in the output TFExamples. This'
-+                    ' significantly reduces the output size and is useful'
-+                    ' if the subsequent tools don\'t need access to the'
-+                    ' images (e.g. when computing evaluation measures).',
-+                    action='store_true', default=False)
-+FLAGS = parser.parse_args()
- 
- def main(_):
--  tf.logging.set_verbosity(tf.logging.INFO)
-+  tf.compat.v1.logging.set_verbosity(tf.compat.v1.logging.INFO)
- 
-   required_flags = ['input_tfrecord_paths', 'output_tfrecord_path',
-                     'inference_graph']
-@@ -62,26 +65,26 @@ def main(_):
-     if not getattr(FLAGS, flag_name):
-       raise ValueError('Flag --{} is required'.format(flag_name))
- 
--  with tf.Session() as sess:
-+  with tf.compat.v1.Session() as sess:
-     input_tfrecord_paths = [
-         v for v in FLAGS.input_tfrecord_paths.split(',') if v]
--    tf.logging.info('Reading input from %d files', len(input_tfrecord_paths))
-+    tf.compat.v1.logging.info('Reading input from %d files', len(input_tfrecord_paths))
-     serialized_example_tensor, image_tensor = detection_inference.build_input(
-         input_tfrecord_paths)
--    tf.logging.info('Reading graph and building model...')
-+    tf.compat.v1.logging.info('Reading graph and building model...')
-     (detected_boxes_tensor, detected_scores_tensor,
-      detected_labels_tensor) = detection_inference.build_inference_graph(
-          image_tensor, FLAGS.inference_graph)
- 
--    tf.logging.info('Running inference and writing output to {}'.format(
-+    tf.compat.v1.logging.info('Running inference and writing output to {}'.format(
-         FLAGS.output_tfrecord_path))
--    sess.run(tf.local_variables_initializer())
--    tf.train.start_queue_runners()
--    with tf.python_io.TFRecordWriter(
-+    sess.run(tf.compat.v1.local_variables_initializer())
-+    tf.compat.v1.train.start_queue_runners()
-+    with tf.io.TFRecordWriter(
-         FLAGS.output_tfrecord_path) as tf_record_writer:
-       try:
-         for counter in itertools.count():
--          tf.logging.log_every_n(tf.logging.INFO, 'Processed %d images...', 10,
-+          tf.compat.v1.logging.log_every_n(tf.compat.v1.logging.INFO, 'Processed %d images...', 10,
-                                  counter)
-           tf_example = detection_inference.infer_detections_and_add_to_example(
-               serialized_example_tensor, detected_boxes_tensor,
-@@ -89,8 +92,8 @@ def main(_):
-               FLAGS.discard_image_pixels)
-           tf_record_writer.write(tf_example.SerializeToString())
-       except tf.errors.OutOfRangeError:
--        tf.logging.info('Finished processing records')
-+        tf.compat.v1.logging.info('Finished processing records')
- 
- 
- if __name__ == '__main__':
--  tf.app.run()
-+  tf.compat.v1.app.run()
-diff --git a/research/object_detection/metrics/coco_tools.py b/research/object_detection/metrics/coco_tools.py
-index 71b747bc..145e93ba 100644
---- a/research/object_detection/metrics/coco_tools.py
-+++ b/research/object_detection/metrics/coco_tools.py
-@@ -106,7 +106,7 @@ class COCOWrapper(coco.COCO):
-     results = coco.COCO()
-     results.dataset['images'] = [img for img in self.dataset['images']]
- 
--    tf.logging.info('Loading and preparing annotation results...')
-+    tf.compat.v1.logging.info('Loading and preparing annotation results...')
-     tic = time.time()
- 
-     if not isinstance(annotations, list):
-@@ -128,7 +128,7 @@ class COCOWrapper(coco.COCO):
-         ann['bbox'] = mask.toBbox(ann['segmentation'])
-         ann['id'] = idx + 1
-         ann['iscrowd'] = 0
--    tf.logging.info('DONE (t=%0.2fs)', (time.time() - tic))
-+    tf.compat.v1.logging.info('DONE (t=%0.2fs)', (time.time() - tic))
- 
-     results.dataset['annotations'] = annotations
-     results.createIndex()
-@@ -479,7 +479,7 @@ def ExportGroundtruthToCOCO(image_ids,
-       'categories': categories
-   }
-   if output_path:
--    with tf.gfile.GFile(output_path, 'w') as fid:
-+    with tf.io.gfile.GFile(output_path, 'w') as fid:
-       json_utils.Dump(groundtruth_dict, fid, float_digits=4, indent=2)
-   return groundtruth_dict
- 
-@@ -666,7 +666,7 @@ def ExportDetectionsToCOCO(image_ids,
-         scores,
-         classes))
-   if output_path:
--    with tf.gfile.GFile(output_path, 'w') as fid:
-+    with tf.io.gfile.GFile(output_path, 'w') as fid:
-       json_utils.Dump(detections_export_list, fid, float_digits=4, indent=2)
-   return detections_export_list
- 
-@@ -746,7 +746,7 @@ def ExportSegmentsToCOCO(image_ids,
-         image_id, category_id_set, np.squeeze(masks, axis=3), scores, classes))
- 
-   if output_path:
--    with tf.gfile.GFile(output_path, 'w') as fid:
-+    with tf.io.gfile.GFile(output_path, 'w') as fid:
-       json_utils.Dump(segment_export_list, fid, float_digits=4, indent=2)
-   return segment_export_list
- 
-@@ -845,6 +845,6 @@ def ExportKeypointsToCOCO(image_ids,
-         })
- 
-   if output_path:
--    with tf.gfile.GFile(output_path, 'w') as fid:
-+    with tf.io.gfile.GFile(output_path, 'w') as fid:
-       json_utils.Dump(keypoints_export_list, fid, float_digits=4, indent=2)
-   return keypoints_export_list
-diff --git a/research/object_detection/metrics/offline_eval_map_corloc.py b/research/object_detection/metrics/offline_eval_map_corloc.py
-index ff2efbaf..dbf6a82d 100644
---- a/research/object_detection/metrics/offline_eval_map_corloc.py
-+++ b/research/object_detection/metrics/offline_eval_map_corloc.py
-@@ -35,6 +35,7 @@ import csv
- import os
- import re
- import tensorflow as tf
-+import argparse
- 
- from object_detection.core import standard_fields
- from object_detection.legacy import evaluator
-@@ -42,16 +43,16 @@ from object_detection.metrics import tf_example_parser
- from object_detection.utils import config_util
- from object_detection.utils import label_map_util
- 
--flags = tf.app.flags
--tf.logging.set_verbosity(tf.logging.INFO)
-+tf.compat.v1.logging.set_verbosity(tf.compat.v1.logging.INFO)
- 
--flags.DEFINE_string('eval_dir', None, 'Directory to write eval summaries to.')
--flags.DEFINE_string('eval_config_path', None,
--                    'Path to an eval_pb2.EvalConfig config file.')
--flags.DEFINE_string('input_config_path', None,
--                    'Path to an eval_pb2.InputConfig config file.')
--
--FLAGS = flags.FLAGS
-+parser = argparse.ArgumentParser()
-+parser.add_argument('-d', '--eval_dir',
-+                    help='Directory to write eval summaries to.', default=None)
-+parser.add_argument('-e', '--eval_config_path',
-+                    help='Path to an eval_pb2.EvalConfig config file.', default=None)
-+parser.add_argument('-p', '--input_config_path',
-+                    help='Path to an eval_pb2.InputConfig config file.', default=None)
-+FLAGS = parser.parse_args()
- 
- 
- def _generate_sharded_filenames(filename):
-@@ -104,13 +105,13 @@ def read_data_and_evaluate(input_config, eval_config):
-     skipped_images = 0
-     processed_images = 0
-     for input_path in _generate_filenames(input_paths):
--      tf.logging.info('Processing file: {0}'.format(input_path))
-+      tf.compat.v1.logging.info('Processing file: {0}'.format(input_path))
- 
--      record_iterator = tf.python_io.tf_record_iterator(path=input_path)
-+      record_iterator = tf.compat.v1.python_io.tf_record_iterator(path=input_path)
-       data_parser = tf_example_parser.TfExampleDetectionAndGTParser()
- 
-       for string_record in record_iterator:
--        tf.logging.log_every_n(tf.logging.INFO, 'Processed %d images...', 1000,
-+        tf.compat.v1.logging.log_every_n(tf.compat.v1.logging.INFO, 'Processed %d images...', 1000,
-                                processed_images)
-         processed_images += 1
- 
-@@ -127,7 +128,7 @@ def read_data_and_evaluate(input_config, eval_config):
-               decoded_dict)
-         else:
-           skipped_images += 1
--          tf.logging.info('Skipped images: {0}'.format(skipped_images))
-+          tf.compat.v1.logging.info('Skipped images: {0}'.format(skipped_images))
- 
-     return object_detection_evaluator.evaluate()
- 
-@@ -141,7 +142,7 @@ def write_metrics(metrics, output_dir):
-     metrics: A dictionary containing metric names and values.
-     output_dir: Directory to write metrics to.
-   """
--  tf.logging.info('Writing metrics.')
-+  tf.compat.v1.logging.info('Writing metrics.')
- 
-   with open(os.path.join(output_dir, 'metrics.csv'), 'w') as csvfile:
-     metrics_writer = csv.writer(csvfile, delimiter=',')
-@@ -170,4 +171,4 @@ def main(argv):
- 
- 
- if __name__ == '__main__':
--  tf.app.run(main)
-+  tf.compat.v1.app.run(main)
-diff --git a/research/object_detection/metrics/tf_example_parser.py b/research/object_detection/metrics/tf_example_parser.py
-index 9a5f130f..fa361bf5 100644
---- a/research/object_detection/metrics/tf_example_parser.py
-+++ b/research/object_detection/metrics/tf_example_parser.py
-@@ -44,7 +44,7 @@ class StringParser(data_parser.DataToNumpyParser):
-     self.field_name = field_name
- 
-   def parse(self, tf_example):
--    return "".join(tf_example.features.feature[self.field_name]
-+    return b"".join(tf_example.features.feature[self.field_name]
-                    .bytes_list.value) if tf_example.features.feature[
-                        self.field_name].HasField("bytes_list") else None
- 
-diff --git a/research/object_detection/queues.py b/research/object_detection/queues.py
-new file mode 100644
-index 00000000..1bc8d2c8
---- /dev/null
-+++ b/research/object_detection/queues.py
-@@ -0,0 +1,74 @@
-+# Copyright 2015 The TensorFlow Authors. All Rights Reserved.
-+#
-+# Licensed under the Apache License, Version 2.0 (the "License");
-+# you may not use this file except in compliance with the License.
-+# You may obtain a copy of the License at
-+#
-+#     http://www.apache.org/licenses/LICENSE-2.0
-+#
-+# Unless required by applicable law or agreed to in writing, software
-+# distributed under the License is distributed on an "AS IS" BASIS,
-+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-+# See the License for the specific language governing permissions and
-+# limitations under the License.
-+# ==============================================================================
-+"""Contains a helper context for running queue runners.
-+
-+@@NestedQueueRunnerError
-+@@QueueRunners
-+"""
-+
-+from __future__ import absolute_import
-+from __future__ import division
-+from __future__ import print_function
-+
-+from contextlib import contextmanager
-+import threading
-+
-+from tensorflow.python.framework import ops
-+from tensorflow.python.training import coordinator
-+
-+__all__ = [
-+    'NestedQueueRunnerError',
-+    'QueueRunners',
-+]
-+
-+_queue_runner_lock = threading.Lock()
-+
-+
-+class NestedQueueRunnerError(Exception):
-+  pass
-+
-+
-+@contextmanager
-+def QueueRunners(session):
-+  """Creates a context manager that handles starting and stopping queue runners.
-+
-+  Args:
-+    session: the currently running session.
-+
-+  Yields:
-+    a context in which queues are run.
-+
-+  Raises:
-+    NestedQueueRunnerError: if a QueueRunners context is nested within another.
-+  """
-+  if not _queue_runner_lock.acquire(False):
-+    raise NestedQueueRunnerError('QueueRunners cannot be nested')
-+
-+  coord = coordinator.Coordinator()
-+  threads = []
-+  for qr in ops.get_collection(ops.GraphKeys.QUEUE_RUNNERS):
-+    threads.extend(
-+        qr.create_threads(
-+            session, coord=coord, daemon=True, start=True))
-+  try:
-+    yield
-+  finally:
-+    coord.request_stop()
-+    try:
-+      coord.join(threads, stop_grace_period_secs=120)
-+    except RuntimeError:
-+      session.close()
-+
-+    _queue_runner_lock.release()
-diff --git a/research/object_detection/utils/config_util.py b/research/object_detection/utils/config_util.py
-index e7835223..4e8125ab 100644
---- a/research/object_detection/utils/config_util.py
-+++ b/research/object_detection/utils/config_util.py
-@@ -89,7 +89,7 @@ def get_configs_from_pipeline_file(pipeline_config_path):
-       corresponding config objects.
-   """
-   pipeline_config = pipeline_pb2.TrainEvalPipelineConfig()
--  with tf.gfile.GFile(pipeline_config_path, "r") as f:
-+  with tf.io.gfile.GFile(pipeline_config_path, "r") as f:
-     proto_str = f.read()
-     text_format.Merge(proto_str, pipeline_config)
-   return create_configs_from_pipeline_proto(pipeline_config)
-@@ -128,7 +128,7 @@ def get_graph_rewriter_config_from_file(graph_rewriter_config_file):
-     graph_rewriter_pb2.GraphRewriter proto
-   """
-   graph_rewriter_config = graph_rewriter_pb2.GraphRewriter()
--  with tf.gfile.GFile(graph_rewriter_config_file, "r") as f:
-+  with tf.io.gfile.GFile(graph_rewriter_config_file, "r") as f:
-     text_format.Merge(f.read(), graph_rewriter_config)
-   return graph_rewriter_config
- 
-@@ -168,8 +168,8 @@ def save_pipeline_config(pipeline_config, directory):
-     file_io.recursive_create_dir(directory)
-   pipeline_config_path = os.path.join(directory, "pipeline.config")
-   config_text = text_format.MessageToString(pipeline_config)
--  with tf.gfile.Open(pipeline_config_path, "wb") as f:
--    tf.logging.info("Writing pipeline config file to %s",
-+  with tf.io.gfile.GFile(pipeline_config_path, "wb") as f:
-+    tf.compat.v1.logging.info("Writing pipeline config file to %s",
-                     pipeline_config_path)
-     f.write(config_text)
- 
-@@ -198,31 +198,31 @@ def get_configs_from_multiple_files(model_config_path="",
-   configs = {}
-   if model_config_path:
-     model_config = model_pb2.DetectionModel()
--    with tf.gfile.GFile(model_config_path, "r") as f:
-+    with tf.io.gfile.GFile(model_config_path, "r") as f:
-       text_format.Merge(f.read(), model_config)
-       configs["model"] = model_config
- 
-   if train_config_path:
-     train_config = train_pb2.TrainConfig()
--    with tf.gfile.GFile(train_config_path, "r") as f:
-+    with tf.io.gfile.GFile(train_config_path, "r") as f:
-       text_format.Merge(f.read(), train_config)
-       configs["train_config"] = train_config
- 
-   if train_input_config_path:
-     train_input_config = input_reader_pb2.InputReader()
--    with tf.gfile.GFile(train_input_config_path, "r") as f:
-+    with tf.io.gfile.GFile(train_input_config_path, "r") as f:
-       text_format.Merge(f.read(), train_input_config)
-       configs["train_input_config"] = train_input_config
- 
-   if eval_config_path:
-     eval_config = eval_pb2.EvalConfig()
--    with tf.gfile.GFile(eval_config_path, "r") as f:
-+    with tf.io.gfile.GFile(eval_config_path, "r") as f:
-       text_format.Merge(f.read(), eval_config)
-       configs["eval_config"] = eval_config
- 
-   if eval_input_config_path:
-     eval_input_config = input_reader_pb2.InputReader()
--    with tf.gfile.GFile(eval_input_config_path, "r") as f:
-+    with tf.io.gfile.GFile(eval_input_config_path, "r") as f:
-       text_format.Merge(f.read(), eval_input_config)
-       configs["eval_input_config"] = eval_input_config
- 
-@@ -325,7 +325,7 @@ def merge_external_params_with_configs(configs, hparams=None, **kwargs):
-   if hparams:
-     kwargs.update(hparams.values())
-   for key, value in kwargs.items():
--    tf.logging.info("Maybe overwriting %s: %s", key, value)
-+    tf.compat.v1.logging.info("Maybe overwriting %s: %s", key, value)
-     # pylint: disable=g-explicit-bool-comparison
-     if value == "" or value is None:
-       continue
-@@ -366,7 +366,7 @@ def merge_external_params_with_configs(configs, hparams=None, **kwargs):
-     elif _is_generic_key(key):
-       _update_generic(configs, key, value)
-     else:
--      tf.logging.info("Ignoring config override key: %s", key)
-+      tf.compat.v1.logging.info("Ignoring config override key: %s", key)
-   return configs
- 
- 
-diff --git a/research/object_detection/utils/label_map_util.py b/research/object_detection/utils/label_map_util.py
-index aef46c1d..8ccad0cb 100644
---- a/research/object_detection/utils/label_map_util.py
-+++ b/research/object_detection/utils/label_map_util.py
-@@ -128,7 +128,7 @@ def load_labelmap(path):
-   Returns:
-     a StringIntLabelMapProto
-   """
--  with tf.gfile.GFile(path, 'r') as fid:
-+  with tf.io.gfile.GFile(path, 'r') as fid:
-     label_map_string = fid.read()
-     label_map = string_int_label_map_pb2.StringIntLabelMap()
-     try:
-diff --git a/research/object_detection/utils/object_detection_evaluation.py b/research/object_detection/utils/object_detection_evaluation.py
-index 8a38d8c2..99af2324 100644
---- a/research/object_detection/utils/object_detection_evaluation.py
-+++ b/research/object_detection/utils/object_detection_evaluation.py
-@@ -839,9 +839,9 @@ class ObjectDetectionEvaluation(object):
-       if self.use_weighted_mean_ap:
-         all_scores = np.append(all_scores, scores)
-         all_tp_fp_labels = np.append(all_tp_fp_labels, tp_fp_labels)
--      print 'Scores and tpfp per class label: {}'.format(class_index)
--      print tp_fp_labels
--      print scores
-+      print('Scores and tpfp per class label: {}'.format(class_index))
-+      print(tp_fp_labels)
-+      print(scores)
-       precision, recall = metrics.compute_precision_recall(
-           scores, tp_fp_labels, self.num_gt_instances_per_class[class_index])
-       self.precisions_per_class.append(precision)
diff --git a/models/object_detection/tensorflow/ssd-mobilenet/inference/int8/infer_detections.py b/models/object_detection/tensorflow/ssd-mobilenet/inference/int8/infer_detections.py
index 4b84c0709..72fea7b7a 100644
--- a/models/object_detection/tensorflow/ssd-mobilenet/inference/int8/infer_detections.py
+++ b/models/object_detection/tensorflow/ssd-mobilenet/inference/int8/infer_detections.py
@@ -119,9 +119,13 @@ def __init__(self):
     arg_parser.add_argument('-w', "--warmup_iter",
                             help='For accuracy measurement only.',
                             dest='warmup_iter', default=200, type=int)
+                            
     arg_parser.add_argument('--onednn-graph', dest='onednn_graph',
                             help='enable OneDNN Graph', action='store_true')
 
+    arg_parser.add_argument('-sg', '--input-subgraph', dest='input_subgraph',
+                            help='int8 subgraph')
+
     # parse the arguments
     self.args = arg_parser.parse_args()
 
@@ -150,7 +154,7 @@ def build_data_sess(self):
       self.input_images, self.bbox, self.label, self.image_id = self.get_input()
     self.data_sess = tf.compat.v1.Session(graph=data_graph, config=self.config)
 
-    dir_path = os.path.dirname(os.path.realpath(__file__))
+    dir_path = self.args.input_subgraph
     preprocess_graph = tf.Graph()
     with preprocess_graph.as_default():
       graph_def = tf.compat.v1.GraphDef()
diff --git a/models/object_detection/tensorflow/ssd-mobilenet/inference/ssdmobilenet_preprocess.pb b/models/object_detection/tensorflow/ssd-mobilenet/inference/ssdmobilenet_preprocess.pb
deleted file mode 100644
index e8b51b171..000000000
Binary files a/models/object_detection/tensorflow/ssd-mobilenet/inference/ssdmobilenet_preprocess.pb and /dev/null differ
diff --git a/models/recommendation/pytorch/dlrm/product/dlrm_s_pytorch.py b/models/recommendation/pytorch/dlrm/product/dlrm_s_pytorch.py
index 9b6c4bc90..e88afa159 100644
--- a/models/recommendation/pytorch/dlrm/product/dlrm_s_pytorch.py
+++ b/models/recommendation/pytorch/dlrm/product/dlrm_s_pytorch.py
@@ -95,13 +95,10 @@
 from torch.optim.lr_scheduler import _LRScheduler
 from torch.profiler import profile, record_function, ProfilerActivity
 
-# intel
-import intel_extension_for_pytorch as ipex
 from torch.utils import ThroughputBenchmark
 
 # int8
 from torch.ao.quantization import MinMaxObserver, PerChannelMinMaxObserver, QConfig
-from intel_extension_for_pytorch.quantization import prepare, convert
 # For distributed run
 import extend_distributed as ext_dist
 
@@ -269,7 +266,11 @@ def create_emb(self, m, ln, local_ln_emb_sparse, ln_emb_dense, np_init_emb_weigh
             embs = local_ln_emb_sparse + ln_emb_dense
         for i in embs:
             n = ln[i]
-            sparse_grad = n >= self.sparse_dense_boundary and ext_dist.my_size > 1
+            
+            # TODO: the previous implementation uses sparse_grad = n >= self.sparse_dense_boundary and ext_dist.my_size > 1,
+            # making sparse_grad a numpy.bool_ instead of a python bool, which is unsupported in TorchInductor.
+            # Update the implementation to make sparse_grad a python bool.
+            sparse_grad = True if n >= self.sparse_dense_boundary and ext_dist.my_size > 1 else False
             if np_init_emb_weight:
                 W = np.random.uniform(
                         low=-np.sqrt(1 / n), high=np.sqrt(1 / n), size=(n, m)
@@ -353,21 +354,13 @@ def apply_emb(self, emb_l, lS_o, lS_i):
         # 2. for each embedding the lookups are further organized into a batch
         # 3. for a list of embedding tables there is a list of batched lookups
         is_ddp = isinstance(emb_l,  ext_dist.DDP)
-        
-        if (not is_ddp and isinstance(emb_l,ipex.nn.modules.MergedEmbeddingBag)) or (is_ddp and isinstance(emb_l.module,ipex.nn.modules.MergedEmbeddingBag)): 
-             start = time.time()
-             n_tables = len(lS_i)
-             #prepare the input for sparse mergedembedding
-             idx = [lS_i[i] for i in range(n_tables)]
-             offset = [lS_o[i] for i in range(n_tables)]
-             include_last = [False for i in range(n_tables)]
-             if is_ddp:
-                 indices, offsets, indices_with_row_offsets = emb_l.module.linearize_indices_and_offsets(idx, offset, include_last)
-             else:
-                 indices, offsets, indices_with_row_offsets = emb_l.linearize_indices_and_offsets(idx, offset, include_last)
-             emb_args = (indices, offsets, indices_with_row_offsets)
-             self.emb_args_preprocessing += time.time() - start
-             return emb_l(emb_args, self.need_linearize_indices_and_offsets)
+        module_to_check = emb_l if not is_ddp else emb_l.module
+        if args.ipex_interaction and isinstance(module_to_check, ipex.nn.modules.MergedEmbeddingBag):
+            n_tables = module_to_check.n_tables
+            indices = [lS_i[i] for i in range(n_tables)]
+            offsets = [lS_o[i] for i in range(n_tables)]
+            return emb_l(indices, offsets)
+
         ly = []
         for k, sparse_index_group_batch in enumerate(lS_i):
             sparse_offset_group_batch = lS_o[k]
@@ -849,10 +842,14 @@ def trace_model(args, dlrm, test_ld):
             # but new thread generate from throughputbench mark will 
             # init this flag to true, so we temporal cast embedding's
             # weight to bfloat16 for now
-            if args.inference_only:
+            if args.inference_only and args.ipex_interaction:
                 dlrm.emb_l.bfloat16()
-            dlrm = ipex.optimize(dlrm, dtype=torch.bfloat16, inplace=True)
-        elif args.int8:
+            if args.ipex_interaction:
+                dlrm = ipex.optimize(dlrm, dtype=torch.bfloat16, inplace=True)
+            else:
+                torch._C._jit_set_autocast_mode(False)
+        elif args.int8 and args.ipex_interaction:
+            from intel_extension_for_pytorch.quantization import prepare, convert
             if args.num_cpu_cores != 0:
                 torch.set_num_threads(args.num_cpu_cores)
             qconfig = QConfig(activation=MinMaxObserver.with_args(qscheme=torch.per_tensor_symmetric, dtype=torch.qint8),
@@ -860,24 +857,59 @@ def trace_model(args, dlrm, test_ld):
             prepare(dlrm, qconfig, example_inputs=(X, lS_o, lS_i), inplace=True)
             dlrm.load_qconf_summary(qconf_summary = args.int8_configure)
             convert(dlrm, inplace=True)
-        else:
+        elif args.ipex_interaction:
             if args.bf32:
                 ipex.set_fp32_math_mode(mode=ipex.FP32MathMode.BF32, device="cpu")
             dlrm = ipex.optimize(dlrm, dtype=torch.float, inplace=True, auto_kernel_selection=True)
         if args.int8:
             print("Start to trace/freeze for int8, may need {} to save int8 weight".format(dlrm.numel / 1024 / 1024 / 1024))
             print("Current mem usage: {} G".format(psutil.Process(os.getpid()).memory_info().rss / 1024 / 1024 / 1024))
-            dlrm = torch.jit.trace(dlrm, [X, lS_o, lS_i])
-            dlrm = torch.jit.freeze(dlrm)
+            if args.ipex_interaction:
+                dlrm = torch.jit.trace(dlrm, [X, lS_o, lS_i])
+                dlrm = torch.jit.freeze(dlrm)
             print("After trace/freeze, current mem usage: {} G".format(psutil.Process(os.getpid()).memory_info().rss / 1024 / 1024 / 1024))
             dlrm(X, lS_o, lS_i)
             dlrm(X, lS_o, lS_i)
         else:
-            with torch.cpu.amp.autocast(enabled=args.bf16):
-                dlrm = torch.jit.trace(dlrm, (X, lS_o, lS_i), check_trace=True)
-                dlrm = torch.jit.freeze(dlrm)
-        dlrm(X, lS_o, lS_i)
-        dlrm(X, lS_o, lS_i)
+            if args.ipex_interaction:
+                with torch.cpu.amp.autocast(enabled=args.bf16):
+                    dlrm = torch.jit.trace(dlrm, (X, lS_o, lS_i), check_trace=True)
+                    dlrm = torch.jit.freeze(dlrm)
+        # torch.compile() path
+        if args.inductor:
+            from torch._inductor import config as inductor_config
+            inductor_config.cpp_wrapper = True
+            if args.int8:
+                from torch.ao.quantization.quantize_pt2e import prepare_pt2e, convert_pt2e
+                import torch.ao.quantization.quantizer.x86_inductor_quantizer as xiq
+                from torch.ao.quantization.quantizer.x86_inductor_quantizer import X86InductorQuantizer
+                from torch._export import capture_pre_autograd_graph, dynamic_dim
+                print('[Info] Running torch.compile() INT8 quantization')
+                with torch.no_grad():
+                    example_inputs = (X, lS_o, lS_i)
+                    exported_model = capture_pre_autograd_graph(
+                        dlrm,
+                        example_inputs
+                    )
+                    quantizer = X86InductorQuantizer()
+                    quantizer.set_global(xiq.get_default_x86_inductor_quantization_config())
+                    prepared_model = prepare_pt2e(exported_model, quantizer)
+                    prepared_model(example_inputs)
+                    converted_model = convert_pt2e(prepared_model)
+                    torch.ao.quantization.move_exported_model_to_eval(converted_model)
+                    print('[Info] Running torch.compile() with default backend')
+                    dlrm = torch.compile(dlrm)
+            elif args.bf16:
+                with torch.no_grad(), torch.cpu.amp.autocast(dtype=torch.bfloat16):
+                    print('[Info] Running torch.compile() with default backend')
+                    dlrm = torch.compile(dlrm)
+            else:
+                with torch.no_grad():
+                    print('[Info] Running torch.compile() with default backend')
+                    dlrm = torch.compile(dlrm)
+        with torch.no_grad(), torch.cpu.amp.autocast(enabled=args.bf16):
+            dlrm(X, lS_o, lS_i)
+            dlrm(X, lS_o, lS_i)
         return dlrm
 
 
@@ -1170,7 +1202,7 @@ def run():
     )
     # training
     parser.add_argument("--mini-batch-size", type=int, default=-1)
-    parser.add_argument("--local-batch-size", type=int, default=1)
+    parser.add_argument("--local-batch-size", type=int, default=-1)
     parser.add_argument("--nepochs", type=int, default=1)
     parser.add_argument("--learning-rate", type=float, default=0.01)
     parser.add_argument("--print-precision", type=int, default=5)
@@ -1213,6 +1245,8 @@ def run():
      # embedding table is sparse table only if sparse_dense_boundary >= 2048
     parser.add_argument("--sparse-dense-boundary", type=int, default=2048)
     parser.add_argument("--hybrid-gradient-emb", action="store_true", default=False)
+    parser.add_argument('--inductor', action='store_true', default=False,
+                        help='using torch.compile()')
 
     global args
     global nbatches
@@ -1220,9 +1254,18 @@ def run():
     args = parser.parse_args()
     
     print(args)
+    if args.ipex_interaction:
+        print('Using ipex')
+        import intel_extension_for_pytorch as ipex
+        global ipex
     ext_dist.init_distributed(backend=args.dist_backend)
     if args.mini_batch_size < 0:
-        args.mini_batch_size = args.local_batch_size * ext_dist.my_size
+        if args.local_batch_size > 0:
+            assert ext_dist.my_size > 1
+            args.mini_batch_size = args.local_batch_size * ext_dist.my_size
+        else:
+            assert args.test_mini_batch_size > 0
+            args.mini_batch_size = args.test_mini_batch_size
 
     ### some basic setup ###
     np.random.seed(args.numpy_rand_seed)
@@ -1441,9 +1484,9 @@ def run():
         if args.bf16:
             print("Start to split weight to bf16 and trail part, or saving whole fp32 master weight, create bf16 weight copy")
             print("Maximum will use ~ {} G memory, may use less memory if useless fp32 weight (for split path) will be released in time ".format(dlrm.numel * 4 / 1024 / 1024 /1024))
-            dlrm, optimizer = ipex.optimize(dlrm, dtype=torch.bfloat16, optimizer=optimizer, inplace=True, sample_input=sample_input)
-            print("args.ipex_merged_emb:", args.ipex_merged_emb)
             if args.ipex_merged_emb:
+                dlrm, optimizer = ipex.optimize(dlrm, dtype=torch.bfloat16, optimizer=optimizer, inplace=True, sample_input=sample_input)
+                print("args.ipex_merged_emb:", args.ipex_merged_emb)
                 print("###############Current mem usage: {} G".format(psutil.Process(os.getpid()).memory_info().rss / 1024 / 1024 / 1024))
                 if ext_dist.my_size > 1:
                     dlrm.emb_sparse.to_bfloat16_train()
@@ -1451,29 +1494,37 @@ def run():
                     dlrm.emb_l.to_bfloat16_train()
             print("Weight cast done, current mem usage: {} G".format(psutil.Process(os.getpid()).memory_info().rss / 1024 / 1024 / 1024))
         else:
-            dlrm, optimizer = ipex.optimize(dlrm, dtype=torch.float, optimizer=optimizer, inplace=True, sample_input=sample_input, auto_kernel_selection=True)
-            if args.bf32:
-                ipex.set_fp32_math_mode(mode=ipex.FP32MathMode.BF32, device="cpu")
+            if args.ipex_merged_emb:
+                dlrm, optimizer = ipex.optimize(dlrm, dtype=torch.float, optimizer=optimizer, inplace=True, sample_input=sample_input, auto_kernel_selection=True)
+                if args.bf32:
+                    ipex.set_fp32_math_mode(mode=ipex.FP32MathMode.BF32, device="cpu")
 
         for i in range(len(dlrm.top_l)):
-            if isinstance(dlrm.top_l[i], ipex.nn.utils._weight_prepack._IPEXLinear):
-                if isinstance(dlrm.top_l[i+1], torch.nn.ReLU):
-                    dlrm.top_l[i] = ipex.nn.modules.IPEXLinearEltwise(dlrm.top_l[i], 'relu')
-                else:
-                    dlrm.top_l[i] = ipex.nn.modules.IPEXLinearEltwise(dlrm.top_l[i], 'sigmoid')
-                dlrm.top_l[i + 1] = torch.nn.Identity()
+            if args.ipex_merged_emb:
+                if isinstance(dlrm.top_l[i], ipex.nn.utils._weight_prepack._IPEXLinear):
+                    if isinstance(dlrm.top_l[i+1], torch.nn.ReLU):
+                        dlrm.top_l[i] = ipex.nn.modules.IPEXLinearEltwise(dlrm.top_l[i], 'relu')
+                    else:
+                        dlrm.top_l[i] = ipex.nn.modules.IPEXLinearEltwise(dlrm.top_l[i], 'sigmoid')
+                    dlrm.top_l[i + 1] = torch.nn.Identity()
         for i in range(len(dlrm.bot_l)):
-            if isinstance(dlrm.bot_l[i], ipex.nn.utils._weight_prepack._IPEXLinear):
-                if isinstance(dlrm.bot_l[i+1], torch.nn.ReLU):
-                    dlrm.bot_l[i] = ipex.nn.modules.IPEXLinearEltwise(dlrm.bot_l[i], 'relu')
-                else:
-                    dlrm.bot_l[i] = ipex.nn.modules.IPEXLinearEltwise(dlrm.bot_l[i], 'sigmoid')
-                dlrm.bot_l[i + 1] = torch.nn.Identity()
+            if args.ipex_merged_emb:
+                if isinstance(dlrm.bot_l[i], ipex.nn.utils._weight_prepack._IPEXLinear):
+                    if isinstance(dlrm.bot_l[i+1], torch.nn.ReLU):
+                        dlrm.bot_l[i] = ipex.nn.modules.IPEXLinearEltwise(dlrm.bot_l[i], 'relu')
+                    else:
+                        dlrm.bot_l[i] = ipex.nn.modules.IPEXLinearEltwise(dlrm.bot_l[i], 'sigmoid')
+                    dlrm.bot_l[i + 1] = torch.nn.Identity()
 
         if ext_dist.my_size > 1:
             dlrm.bot_l = ext_dist.DDP(dlrm.bot_l, gradient_as_bucket_view=True, broadcast_buffers=False, find_unused_parameters=True)
             dlrm.top_l = ext_dist.DDP(dlrm.top_l, gradient_as_bucket_view=True, broadcast_buffers=False, find_unused_parameters=True)
             dlrm.emb_dense = ext_dist.DDP(dlrm.emb_dense, gradient_as_bucket_view=True, broadcast_buffers=False)
+
+        if args.inductor:
+            with torch.cpu.amp.autocast(enabled=args.bf16):
+                print('[Info] Running training steps torch.compile() with default backend')
+                dlrm = torch.compile(dlrm)
     training_record = [0, 0]
     def update_training_performance(time, iters, training_record=training_record):
         if iters > args.num_warmup_iters:
@@ -1548,10 +1599,10 @@ def trace_handler(prof):
 
                         # forward pass
 
-                        if hasattr(dlrm, 'emb_l') and isinstance(dlrm.emb_l, ipex.nn.modules.MergedEmbeddingBagWithSGD):
+                        if args.ipex_merged_emb and hasattr(dlrm, 'emb_l') and isinstance(dlrm.emb_l, ipex.nn.modules.MergedEmbeddingBagWithSGD):
                             dlrm.emb_l.sgd_args = dlrm.emb_l.sgd_args._replace(lr=lr_scheduler.get_last_lr()[0])
 
-                        if hasattr(dlrm, 'emb_sparse') and isinstance(dlrm.emb_sparse, ipex.nn.modules.MergedEmbeddingBagWithSGD):
+                        if args.ipex_merged_emb and hasattr(dlrm, 'emb_sparse') and isinstance(dlrm.emb_sparse, ipex.nn.modules.MergedEmbeddingBagWithSGD):
                             dlrm.emb_sparse.sgd_args = dlrm.emb_sparse.sgd_args._replace(lr=lr_scheduler.get_last_lr()[0]/ext_dist.my_size)
 
                         with torch.cpu.amp.autocast(enabled=args.bf16):
diff --git a/models/recommendation/pytorch/torchrec_dlrm/data_process/multi_hot_criteo.py b/models/recommendation/pytorch/torchrec_dlrm/data_process/multi_hot_criteo.py
index 9cf15bc51..72f383c87 100644
--- a/models/recommendation/pytorch/torchrec_dlrm/data_process/multi_hot_criteo.py
+++ b/models/recommendation/pytorch/torchrec_dlrm/data_process/multi_hot_criteo.py
@@ -331,6 +331,6 @@ def load_batch(self, sample_list=None) -> Batch:
         if sample_list is None:
             sample_list = list(range(self.batch_size))
         dense = self.dense_arrs[0][sample_list, :]
-        sparse = [arr[sample_list, :] for arr in self.sparse_arrs[0]]
+        sparse = [arr[sample_list, :] % self.hashes[i] for i, arr in enumerate(self.sparse_arrs[0])]
         labels = self.labels_arrs[0][sample_list, :]
         return self._np_arrays_to_batch(dense, sparse, labels)
diff --git a/models/recommendation/pytorch/torchrec_dlrm/dlrm_main.py b/models/recommendation/pytorch/torchrec_dlrm/dlrm_main.py
index 6de291bf7..e381cc28c 100644
--- a/models/recommendation/pytorch/torchrec_dlrm/dlrm_main.py
+++ b/models/recommendation/pytorch/torchrec_dlrm/dlrm_main.py
@@ -34,6 +34,7 @@
 import time
 
 import torch
+import torch.distributed as dist
 print(f"torch version {torch.__version__}")
 import torchmetrics as metrics
 from pyre_extensions import none_throws
@@ -48,8 +49,6 @@
 from torchrec.optim.optimizers import in_backward_optimizer_filter
 from tqdm import tqdm
 from jit_trace_able_utils import unpack, SparseArchTraceAbleWrapper
-import intel_extension_for_pytorch as ipex
-print(f"IPEX version {ipex.__version__}")
 
 # OSS import
 try:
@@ -100,8 +99,23 @@ def load_snapshot(model, snapshot_dir):
     snapshot.restore(app_state={"model": model})
 
 def trace_handler(prof):
+    import os
+    try:
+        profile_dir = os.environ["PROFLIE_DIR"]
+    except:
+        profile_dir = "./"
     print(prof.key_averages().table(sort_by="self_cpu_time_total"))
-    prof.export_chrome_trace(f"{prof._mode}-{prof._dtype}-dlrm_trace_step{prof.step_num}.json")
+    try:
+        my_size = dist.get_world_size()
+    except:
+        my_size = 1
+    if my_size > 1:
+        my_rank = dist.get_rank()
+        trace_dir = f"{profile_dir}/rank_{my_rank}_{prof._mode}-{prof._dtype}-dlrm_trace_step{prof.step_num}.json"
+    else:
+        trace_dir = f"{profile_dir}/{prof._mode}-{prof._dtype}-dlrm_trace_step{prof.step_num}.json"
+    print(trace_dir)
+    prof.export_chrome_trace(trace_dir)
 
 prof_schedule=torch.profiler.schedule(
     wait=10,
@@ -119,7 +133,25 @@ def fetch_batch(dataloader):
     try:
         batch = dataloader.dataset.load_batch()
     except:
-        batch = dataloader.source.dataset.batch_generator._generate_batch()
+        import torchrec
+        dataset = dataloader.source.dataset
+        if isinstance(dataset, torchrec.datasets.criteo.InMemoryBinaryCriteoIterDataPipe):
+            sample_list = list(range(dataset.batch_size))
+            dense = dataset.dense_arrs[0][sample_list, :]
+            sparse = [arr[sample_list, :] for arr in dataset.sparse_arrs][0] % dataset.hashes
+            labels = dataset.labels_arrs[0][sample_list, :]
+            return dataloader.func(dataset._np_arrays_to_batch(dense, sparse, labels))
+        batch = dataloader.func(dataloader.source.dataset.batch_generator._generate_batch())
+    return batch
+
+def split_dense_input_and_label_for_ranks(batch):
+    my_rank = dist.get_rank()
+    my_size = dist.get_world_size()
+    local_bs = int(batch.dense_features.shape[0] / my_size)
+    start = local_bs * my_rank
+    end = start + local_bs
+    batch.dense_features = batch.dense_features[start:end]
+    batch.labels = batch.labels[start:end]
     return batch
 
 def parse_autocast(dtype: str):
@@ -177,15 +209,26 @@ def convert_int8(args, model, dataloader):
 
 def ipex_optimize(args, model, optimizer, dataloader):
     example_batch = fetch_batch(dataloader)
+    if args.distributed_training and dist.get_world_size() > 1:
+        example_batch = split_dense_input_and_label_for_ranks(example_batch)
     example_batch.sparse_features = unpack(example_batch.sparse_features)
     dense, sparse = example_batch.dense_features, example_batch.sparse_features
     autocast, dtype = parse_autocast(args.dtype)
-    auto_kernel_selection = True if args.dtype == 'bf32' else False
+    auto_kernel_selection = True
     import intel_extension_for_pytorch as ipex
     if dtype == torch.int8:
         assert args.inference_only
         with torch.no_grad():
-            model = convert_int8(args, model, dataloader)
+            if args.int8_prepare or args.calibration:
+                    model = convert_int8(args, model, dataloader)
+                    torch.jit.save(model, args.int8_model_dir)
+                    print(f"save int8 model to {args.int8_model_dir}")
+                    exit()
+            else:
+                # just run JIT, since we load optimized INT8 model
+                print_memory("int8 jit optimize")
+                model(dense, sparse)
+                model(dense, sparse)            
     elif args.inference_only:
         with torch.no_grad():
             model = ipex.optimize(
@@ -217,11 +260,70 @@ def ipex_optimize(args, model, optimizer, dataloader):
             linear_bn_folding=False,
             conv_bn_folding=False
         )
-        pass
+        if dtype == torch.bfloat16:
+            model.sparse_arch.embedding_bag_collection.to_bfloat16_train()
+        # if dtype == torch.float16:
+        #     model.sparse_arch.embedding_bag_collection.to_float16_train()
     if args.dtype == 'bf32':
         ipex.set_fp32_math_mode(mode=ipex.FP32MathMode.BF32, device="cpu")
     return model, optimizer
 
+def stock_pt_optimize(args, model, optimizer, dataloader):
+    example_batch = fetch_batch(dataloader)
+    example_batch.sparse_features = unpack(example_batch.sparse_features)
+    dense, sparse = example_batch.dense_features, example_batch.sparse_features
+    autocast, dtype = parse_autocast(args.dtype)
+    if args.inductor:
+        from torch._inductor import config as inductor_config
+        from torch._dynamo import config
+        config.error_on_recompile = True
+        inductor_config.cpp_wrapper = True
+        inductor_config.cpp.enable_kernel_profile = True
+        if args.inference_only:
+            inductor_config.freezing = True
+            if not dtype == torch.int8:
+                model.eval()
+        if dtype == torch.int8:
+            assert args.inference_only
+            from torch.ao.quantization.quantize_pt2e import prepare_pt2e, convert_pt2e
+            import torch.ao.quantization.quantizer.x86_inductor_quantizer as xiq
+            from torch.ao.quantization.quantizer.x86_inductor_quantizer import X86InductorQuantizer
+            from torch._export import capture_pre_autograd_graph, dynamic_dim
+            print('[Info] Running torch.compile() INT8 quantization')
+            with torch.no_grad():
+                example_inputs = (dense, sparse)
+                exported_model = capture_pre_autograd_graph(
+                    model,
+                    example_inputs
+                )
+                quantizer = X86InductorQuantizer()
+                quantizer.set_global(xiq.get_default_x86_inductor_quantization_config())
+                prepared_model = prepare_pt2e(exported_model, quantizer)
+                prepared_model(dense, sparse)
+                converted_model = convert_pt2e(prepared_model)
+                torch.ao.quantization.move_exported_model_to_eval(converted_model)
+                print(converted_model.graph)
+                print("===========================")
+                converted_model.graph.print_tabular()
+                if args.ipex:
+                    print('[Info] Running torch.compile() with IPEX backend')
+                    model(dense, sparse)
+                    model = torch.compile(converted_model, backend="ipex")
+                else:
+                    print('[Info] Running torch.compile() with default backend')
+                    model(dense, sparse)
+                    model = torch.compile(converted_model)
+                model(dense, sparse)
+                model(dense, sparse)
+        else:
+            with torch.no_grad(), torch.cpu.amp.autocast(enabled=autocast, dtype=dtype):
+                print('[Info] Running torch.compile() with default backend')
+                model(dense, sparse)
+                model = torch.compile(model)
+                model(dense, sparse)
+                model(dense, sparse)
+    return model, optimizer
+
 def parse_args(argv: List[str]) -> argparse.Namespace:
     parser = argparse.ArgumentParser(description="torchrec dlrm example trainer")
     parser.add_argument(
@@ -266,6 +368,12 @@ def parse_args(argv: List[str]) -> argparse.Namespace:
         default=None,
         help="number of test batches",
     )
+    parser.add_argument(
+        "--warmup_batches",
+        type=int,
+        default=100,
+        help="number of test batches",
+    )
     parser.add_argument(
         "--dataset_name",
         type=str,
@@ -331,7 +439,7 @@ def parse_args(argv: List[str]) -> argparse.Namespace:
     parser.add_argument(
         "--seed",
         type=int,
-        default=None,
+        default=1696543516,
         help="Random seed for reproducibility.",
     )
     parser.add_argument(
@@ -475,6 +583,11 @@ def parse_args(argv: List[str]) -> argparse.Namespace:
         action="store_true",
         help="Whether optimize model by ipex.optimize",
     )
+    parser.add_argument(
+        "--ipex",
+        action="store_true",
+        help="Whether optimize model with dynamo + ipex backend",
+    )
     parser.add_argument(
         "--dtype",
         type=str,
@@ -509,6 +622,12 @@ def parse_args(argv: List[str]) -> argparse.Namespace:
         default=0,
         help="log-freq to print performance statistics.",
     )
+    parser.add_argument(
+        "--log-freq-eval",
+        type=int,
+        default=0,
+        help="log-freq to print performance/auroc statistics for eval.",
+    )
     parser.add_argument(
         "--snapshot-dir",
         type=str,
@@ -539,6 +658,26 @@ def parse_args(argv: List[str]) -> argparse.Namespace:
         default=0,
         help="if value > 0, will use pytorch throughputbenchmark to share weight with `value` instance",
     )
+    parser.add_argument(
+        "--inductor",
+        action="store_true",
+        help="whether use torch.compile()",
+    )
+    parser.add_argument(
+        "--distributed-training",
+        action="store_true",
+        help="whether train dlrm distributed",
+    )
+    parser.add_argument(
+        "--ipex-dist-merged-emb-adagrad",
+        action="store_true",
+        help="whether use ipex customer op for distributed merged embedding",
+    )
+    parser.add_argument(
+        "--ipex-merged-emb-adagrad",
+        action="store_true",
+        help="whether use ipex customer op for merged embedding with adagrad",
+    )
     return parser.parse_args(argv)
 
 
@@ -566,11 +705,13 @@ def _evaluate(
     limit_batches = args.limit_val_batches
     print_progress = args.print_progress
     autocast_enabled, autocast_dtype = parse_autocast(args.dtype)
-    log_freq = args.log_freq
+    log_freq = args.log_freq_eval
     enable_torch_profile = args.profile
 
     benckmark_batch = fetch_batch(eval_dataloader)
     benckmark_batch.sparse_features = unpack(benckmark_batch.sparse_features)
+    if args.distributed_training and dist.get_world_size() > 1:
+        benckmark_batch = split_dense_input_and_label_for_ranks(benckmark_batch)
 
     def fetch_next(iterator, current_it):
         if args.benchmark:
@@ -582,6 +723,8 @@ def fetch_next(iterator, current_it):
                 next_batch = next(iterator)
             with record_function("unpack KeyJaggedTensor"):
                 next_batch.sparse_features = unpack(next_batch.sparse_features)
+            if args.distributed_training and dist.get_world_size() > 1:
+                next_batch = split_dense_input_and_label_for_ranks(next_batch)
             return next_batch
 
     def eval_step(model, iterator, current_it):
@@ -592,7 +735,23 @@ def eval_step(model, iterator, current_it):
             t2 = time.time()
         return logits, batch.labels, t2 - t1
 
-        
+    def gather_output(label, pred):
+        my_rank = dist.get_rank()
+        my_size = dist.get_world_size()
+        if my_rank == 0:
+            label_list = [torch.empty_like(label) for _ in range(my_size)]
+            pred_list = [torch.empty_like(pred) for _ in range(my_size)]
+        else:
+            label_list = None
+            pred_list = None
+        dist.barrier()
+        dist.gather(label, label_list, dst=0)
+        dist.gather(pred, pred_list, dst=0)
+        if my_rank == 0:
+            return torch.cat(label_list), torch.cat(pred_list)
+        else:
+            return None, None
+
     pbar = tqdm(
         iter(int, 1),
         desc=f"Evaluating {stage} set",
@@ -602,7 +761,9 @@ def eval_step(model, iterator, current_it):
 
     logger.info(f"EVAL_START, EPOCH_NUM: {epoch_num}")
 
-    eval_model.eval()
+    if not (args.inductor and args.dtype == 'int8'):
+        eval_model.eval()
+
     device = torch.device('cpu')
 
     iterator = itertools.islice(iter(eval_dataloader), limit_batches)
@@ -615,9 +776,14 @@ def eval_step(model, iterator, current_it):
 
     preds = []
     labels = []
-    auroc = metrics.AUROC(compute_on_step=False, task='binary').to(device)
+
+    if not args.ipex_optimize:
+        auroc_computer = metrics.AUROC(task='binary').to(device)
+    else:
+        import intel_extension_for_pytorch as ipex
+
     total_t = 0
-    it = 1
+    it = 0
     ctx1 = torch.no_grad()
     ctx2 = torch.cpu.amp.autocast(enabled=autocast_enabled, dtype=autocast_dtype)
     ctx3 = torch.profiler.profile(activities=[ProfilerActivity.CPU], schedule=prof_schedule, on_trace_ready=trace_handler)
@@ -629,20 +795,26 @@ def eval_step(model, iterator, current_it):
         while True:
             try:
                 logits, label, fw_t = eval_step(eval_model, iterator, it)
-                total_t += fw_t
-                if log_freq != 0 and it % log_freq == 0:
-                    preds = [torch.cat(preds)]
-                    labels = [torch.cat(labels)]
-                    num_samples = labels[0].shape[0]
-                    auroc = ipex._C.roc_auc_score(labels[0].float(), preds[0].squeeze().float())
-                    logger.info(f"avg eval time per iter at ITER: {it}, {total_t/it} s, num_samples: {num_samples}, AUROC: {auroc}")
+                if it > args.warmup_batches:
+                    if enable_torch_profile:
+                        p.step()
+                    total_t += fw_t
+                    if log_freq != 0 and it % log_freq == 0 and it > args.warmup_batches:
+                        assert not args.distributed_training
+                        preds = [torch.cat(preds)]
+                        labels = [torch.cat(labels)]
+                        num_samples = labels[0].shape[0] - args.warmup_batches * args.batch_size
+                        if not args.ipex_optimize:
+                            auroc = auroc_computer(preds[0].squeeze().float(), labels[0].float())
+                        else:
+                            auroc = ipex._C.roc_auc_score(labels[0].float(), preds[0].squeeze().float())
+
+                        logger.info(f"avg eval time per iter at ITER: {it}, {total_t/it} s, num_samples: {num_samples}, AUROC: {auroc}")
                 pred = torch.sigmoid(logits)
                 preds.append(pred)
                 labels.append(label)
                 pbar.update(1)
                 it += 1
-                if enable_torch_profile:
-                    p.step()
             except StopIteration:
                 # Dataset traversal complete
                 break
@@ -651,14 +823,29 @@ def eval_step(model, iterator, current_it):
 
     preds = torch.cat(preds)
     labels = torch.cat(labels)
-    num_samples = labels.shape[0]
-    auroc = ipex._C.roc_auc_score(labels.float(), preds.squeeze().float())
-
-    logger.info(f"AUROC over {stage} set: {auroc}.")
-    logger.info(f"Number of {stage} samples: {num_samples}")
-    logger.info(f"Throughput: {num_samples/total_t} fps")
-    logger.info(f"Final AUROC: {auroc} ")
-
+    if args.distributed_training and dist.get_world_size() > 1:
+        labels, preds = gather_output(labels, preds)
+
+    is_rank_zero = args.distributed_training and dist.get_world_size() > 1 and dist.get_rank() == 0
+    if is_rank_zero or not args.distributed_training:
+        num_samples = labels.shape[0] - args.warmup_batches * args.batch_size
+        if not args.ipex_optimize:
+            auroc = auroc = auroc_computer(preds.squeeze().float(), labels.float())
+        else:
+            auroc = ipex._C.roc_auc_score(labels.float(), preds.squeeze().float())
+        logger.info(f"AUROC over {stage} set: {auroc}.")
+        logger.info(f"Number of {stage} samples: {num_samples}")
+        logger.info(f"Throughput: {num_samples/total_t} fps")
+        logger.info(f"Final AUROC: {auroc} ")
+
+    if args.distributed_training and dist.get_world_size() > 1:
+        dist.barrier()
+        if is_rank_zero:
+            bd_list = [auroc]
+        else:
+            bd_list = [None]
+        dist.broadcast_object_list(bd_list, src=0)
+        auroc = bd_list[0]
     return auroc
 
 
@@ -706,6 +893,8 @@ def _train(
 
     benckmark_batch = fetch_batch(train_dataloader)
     benckmark_batch.sparse_features = unpack(benckmark_batch.sparse_features)
+    if args.distributed_training and dist.get_world_size() > 1:
+        benckmark_batch = split_dense_input_and_label_for_ranks(benckmark_batch)
 
     def fetch_next(iterator, current_it):
         if args.benchmark:
@@ -717,6 +906,8 @@ def fetch_next(iterator, current_it):
                 next_batch = next(iterator)
             with record_function("unpack KeyJaggedTensor"):
                 next_batch.sparse_features = unpack(next_batch.sparse_features)
+            if args.distributed_training and dist.get_world_size() > 1:
+                next_batch = split_dense_input_and_label_for_ranks(next_batch)
             return next_batch
 
     def train_step(model, opt, iterator, current_it):
@@ -777,16 +968,17 @@ def train_step(model, opt, iterator, current_it):
                 for i, g in enumerate(train_optimizer.param_groups):
                     logger.info(f"lr: {it} {i} {g['lr']:.6f}")
             samples, train_t = train_step(train_model, train_optimizer, iterator, it)
-            if enable_torch_profile:
-                p.step()
-            num_samples += samples
-            total_t += train_t
+            if it >= args.warmup_batches:
+                num_samples += samples
+                total_t += train_t
+                if enable_torch_profile:
+                    p.step()
 
-            if log_freq != 0 and it % log_freq == 0:
-                logger.info(f"avg training time per iter at ITER: {it}, {total_t/it} s")
+            if log_freq != 0 and it % log_freq == 0 and it > args.warmup_batches: 
+                logger.info(f"avg training time per iter at ITER: {it}, {total_t/ (it - args.warmup_batches)} s")
                 print_memory(f"memory usage at iter {it}")
 
-            lr_scheduler.step()
+            # lr_scheduler.step()
             pbar.update(1)
             if validation_freq and it % validation_freq == 0:
                 epoch_num = epoch + it / len(train_dataloader)
@@ -797,6 +989,8 @@ def train_step(model, opt, iterator, current_it):
                     epoch_num,
                     args
                 )
+                if isinstance(auroc_result, list):
+                    auroc_result = auroc_result[0]
                 if validation_auroc is not None and auroc_result >= validation_auroc:
                     logger.info("auc = {auroc_result} >= validation_auroc {validation_auroc}, stopped since success")
                     is_success = True
@@ -818,6 +1012,7 @@ def _share_weight_benchmark(
     data_loader,
     args,
 ):
+    import contextlib
     from torch.utils import ThroughputBenchmark
     print_memory("start to init throughput benchmark")
     bench = ThroughputBenchmark(model)
@@ -826,12 +1021,18 @@ def _share_weight_benchmark(
     print_memory("start to add input to throughput benchmark")
     bench.add_input(batch.dense_features, batch.sparse_features)
     print_memory("start to run throughput benchmark")
-    stats = bench.benchmark(
-        num_calling_threads=args.share_weight_instance,
-        num_warmup_iters=100,
-        num_iters=args.limit_val_batches * args.share_weight_instance,
-    )
-    print(stats)
+    ctx = contextlib.suppress()
+    if args.dtype == 'bf16':
+        ctx = torch.cpu.amp.autocast(enabled=True, dtype=torch.bfloat16)
+    if args.dtype == 'fp16':
+        ctx = torch.cpu.amp.autocast(enabled=True, dtype=torch.float16)
+    with ctx:
+        stats = bench.benchmark(
+            num_calling_threads=args.share_weight_instance,
+            num_warmup_iters=200,
+            num_iters=args.limit_val_batches * args.share_weight_instance,
+        )
+        print(stats)
     latency = stats.latency_avg_ms
     batch_size = batch.dense_features.shape[0]
     throughput = (1 / latency) * 1000 * batch_size * args.share_weight_instance
@@ -872,13 +1073,14 @@ def train_val_test(
     results = TrainValTestResults()
 
     if args.inference_only:
-        if args.share_weight_instance > 0:
-            _share_weight_benchmark(
-                model.model,
-                val_dataloader,
-                args
-            )
-            exit()
+        with torch.no_grad():
+            if args.share_weight_instance > 0:
+                _share_weight_benchmark(
+                    model.model,
+                    val_dataloader,
+                    args
+                )
+                exit()
         # Mlperf is using val set to test auroc
         # https://github.com/mlcommons/inference/blob/master/recommendation/dlrm_v2/pytorch/python/multihot_criteo.py#L99-L107
         test_auroc = _evaluate(
@@ -943,49 +1145,13 @@ def train_val_test(
 
     return results
 
-
-def main(argv: List[str]) -> None:
-    """
-    Trains, validates, and tests a Deep Learning Recommendation Model (DLRM)
-    (https://arxiv.org/abs/1906.00091). The DLRM model contains both data parallel
-    components (e.g. multi-layer perceptrons & interaction arch) and model parallel
-    components (e.g. embedding tables). The DLRM model is pipelined so that dataloading,
-    data-parallel to model-parallel comms, and forward/backward are overlapped. Can be
-    run with either a random dataloader or an in-memory Criteo 1 TB click logs dataset
-    (https://ailab.criteo.com/download-criteo-1tb-click-logs-dataset/).
-
-    Args:
-        argv (List[str]): command line args.
-
-    Returns:
-        None.
-    """
-
-    args = parse_args(argv)
-    for name, val in vars(args).items():
-        try:
-            vars(args)[name] = list(map(int, val.split(",")))
-        except (ValueError, AttributeError):
-            pass
-
-    if args.multi_hot_sizes is not None:
-        assert (
-            args.num_embeddings_per_feature is not None
-            and len(args.multi_hot_sizes) == len(args.num_embeddings_per_feature)
-            or args.num_embeddings_per_feature is None
-            and len(args.multi_hot_sizes) == len(DEFAULT_CAT_NAMES)
-        ), "--multi_hot_sizes must be a comma delimited list the same size as the number of embedding tables."
-    assert (
-        args.in_memory_binary_criteo_path is None
-        or args.synthetic_multi_hot_criteo_path is None
-    ), "--in_memory_binary_criteo_path and --synthetic_multi_hot_criteo_path are mutually exclusive CLI arguments."
-    assert (
-        args.multi_hot_sizes is None or args.synthetic_multi_hot_criteo_path is None
-    ), "--multi_hot_sizes is used to convert 1-hot to multi-hot. It's inapplicable with --synthetic_multi_hot_criteo_path."
-    assert (
-        args.multi_hot_distribution_type is None
-        or args.synthetic_multi_hot_criteo_path is None
-    ), "--multi_hot_distribution_type is used to convert 1-hot to multi-hot. It's inapplicable with --synthetic_multi_hot_criteo_path."
+def construct_model(args):
+    if args.dtype == "int8" and not args.int8_prepare and not args.calibration and args.jit:
+        assert args.inference_only
+        print(f"loading int8 model from {args.int8_model_dir}")
+        model = torch.jit.load(args.int8_model_dir)
+        print_memory("int8 loading finished")
+        return DLRMTrain(model), None, None
 
     device: torch.device = torch.device("cpu")
     backend = "gloo"
@@ -1028,54 +1194,24 @@ def main(argv: List[str]) -> None:
         )
         for feature_idx, feature_name in enumerate(DEFAULT_CAT_NAMES)
     ]
-    sharded_module_kwargs = {}
-    if args.over_arch_layer_sizes is not None:
-        sharded_module_kwargs["over_arch_layer_sizes"] = args.over_arch_layer_sizes
 
-    print_memory("start init dlrm ")
-    if args.interaction_type == InteractionType.ORIGINAL:
-        dlrm_model = DLRM(
-            embedding_bag_collection=EmbeddingBagCollection(
-                tables=eb_configs, device=torch.device("meta")
-            ),
-            dense_in_features=len(DEFAULT_INT_NAMES),
-            dense_arch_layer_sizes=args.dense_arch_layer_sizes,
-            over_arch_layer_sizes=args.over_arch_layer_sizes,
-            dense_device=device,
-        )
-    elif args.interaction_type == InteractionType.DCN:
-        if args.ipex_optimize:
-            from ipex_optimized_model.model import IPEX_DLRM_DCN
-            dcn_init_fn = IPEX_DLRM_DCN
-        else:
-            dcn_init_fn = DLRM_DCN
-        dlrm_model = dcn_init_fn(
-            embedding_bag_collection=EmbeddingBagCollection(
-                tables=eb_configs, device=torch.device("cpu")
-            ),
-            dense_in_features=len(DEFAULT_INT_NAMES),
-            dense_arch_layer_sizes=args.dense_arch_layer_sizes,
-            over_arch_layer_sizes=args.over_arch_layer_sizes,
-            dcn_num_layers=args.dcn_num_layers,
-            dcn_low_rank_dim=args.dcn_low_rank_dim,
-            dense_device=device,
-        )
-    elif args.interaction_type == InteractionType.PROJECTION:
-        dlrm_model = DLRM_Projection(
-            embedding_bag_collection=EmbeddingBagCollection(
-                tables=eb_configs, device=torch.device("meta")
-            ),
-            dense_in_features=len(DEFAULT_INT_NAMES),
-            dense_arch_layer_sizes=args.dense_arch_layer_sizes,
-            over_arch_layer_sizes=args.over_arch_layer_sizes,
-            interaction_branch1_layer_sizes=args.interaction_branch1_layer_sizes,
-            interaction_branch2_layer_sizes=args.interaction_branch2_layer_sizes,
-            dense_device=device,
-        )
-    else:
-        raise ValueError(
-            "Unknown interaction option set. Should be original, dcn, or projection."
-        )
+    assert args.interaction_type == InteractionType.DCN
+    from ipex_optimized_model.model import IPEX_DLRM_DCN, init_weight
+    dcn_init_fn = IPEX_DLRM_DCN
+    dlrm_model = dcn_init_fn(
+        embedding_bag_collection=EmbeddingBagCollection(
+            tables=eb_configs, device=torch.device("cpu")
+        ),
+        dense_in_features=len(DEFAULT_INT_NAMES),
+        dense_arch_layer_sizes=args.dense_arch_layer_sizes,
+        over_arch_layer_sizes=args.over_arch_layer_sizes,
+        dcn_num_layers=args.dcn_num_layers,
+        dcn_low_rank_dim=args.dcn_low_rank_dim,
+        dense_device=device,
+    )
+    if args.validation_auroc:
+        # init weight to test convergence
+        init_weight(dlrm_model)
 
     train_model = DLRMTrain(dlrm_model)
     if args.test_auroc or args.calibration:
@@ -1084,12 +1220,13 @@ def main(argv: List[str]) -> None:
         load_snapshot(train_model, args.snapshot_dir)
     
     from ipex_optimized_model.model import replace_crossnet, replace_embeddingbag_collection
-    replace_crossnet(train_model.model)
-    if args.inference_only and args.ipex_merged_emb_cat:
-        replace_embeddingbag_collection(train_model.model)
-    else:
-        for emb in train_model.model.sparse_arch.embedding_bag_collection.embedding_bags.values():
-            emb.sparse = True
+    # change the crossnet and embeddingbag-collection for ipex
+    # we do not integrate this part in IPEX_DLRM_DCN because this will change the paramter names
+    # and will impact the loading for checkpoint, so we do this after loading snapshot
+    if args.ipex_optimize:
+        # re-write crossnet with using nn.linear instead of only using torch.linear
+        replace_crossnet(train_model.model)
+    replace_embeddingbag_collection(train_model.model, args)
     # embedding_optimizer = torch.optim.Adagrad if args.adagrad else torch.optim.SGD
     # This will apply the Adagrad optimizer in the backward pass for the embeddings (sparse_arch). This means that
     # the optimizer update will be applied in the backward pass, in this case through a fused op.
@@ -1134,26 +1271,73 @@ def main(argv: List[str]) -> None:
     optimizer,  lr_scheduler = None, None
     if not args.inference_only:
         print_memory("start create optimizer ")
-        if args.adagrad:
-            optimizer = torch.optim.Adagrad(
-                model.parameters(),
-                lr=args.learning_rate,
-                lr_decay=ADAGRAD_LR_DECAY,
-                weight_decay=WEIGHT_DECAY,
-                initial_accumulator_value=ADAGRAD_INIT_ACC,
-                eps=ADAGRAD_EPS,
-            )
-        else:
-            optimizer = torch.optim.SGD(
-                model.parameters(),
-                lr=args.learning_rate,
-                weight_decay=WEIGHT_DECAY,
-            )
+        assert args.adagrad
+        optimizer = torch.optim.Adagrad(
+            model.parameters(),
+            lr=args.learning_rate,
+            lr_decay=ADAGRAD_LR_DECAY,
+            weight_decay=WEIGHT_DECAY,
+            initial_accumulator_value=ADAGRAD_INIT_ACC,
+            eps=ADAGRAD_EPS,
+        )
 
         lr_scheduler = LRPolicyScheduler(
             optimizer, args.lr_warmup_steps, args.lr_decay_start, args.lr_decay_steps
         )
 
+def main(argv: List[str]) -> None:
+    """
+    Trains, validates, and tests a Deep Learning Recommendation Model (DLRM)
+    (https://arxiv.org/abs/1906.00091). The DLRM model contains both data parallel
+    components (e.g. multi-layer perceptrons & interaction arch) and model parallel
+    components (e.g. embedding tables). The DLRM model is pipelined so that dataloading,
+    data-parallel to model-parallel comms, and forward/backward are overlapped. Can be
+    run with either a random dataloader or an in-memory Criteo 1 TB click logs dataset
+    (https://ailab.criteo.com/download-criteo-1tb-click-logs-dataset/).
+
+    Args:
+        argv (List[str]): command line args.
+
+    Returns:
+        None.
+    """
+
+    args = parse_args(argv)
+    if args.ipex_optimize:
+        import intel_extension_for_pytorch as ipex
+        print(f"IPEX version {ipex.__version__}")
+    for name, val in vars(args).items():
+        try:
+            vars(args)[name] = list(map(int, val.split(",")))
+        except (ValueError, AttributeError):
+            pass
+
+    if args.multi_hot_sizes is not None:
+        assert (
+            args.num_embeddings_per_feature is not None
+            and len(args.multi_hot_sizes) == len(args.num_embeddings_per_feature)
+            or args.num_embeddings_per_feature is None
+            and len(args.multi_hot_sizes) == len(DEFAULT_CAT_NAMES)
+        ), "--multi_hot_sizes must be a comma delimited list the same size as the number of embedding tables."
+    assert (
+        args.in_memory_binary_criteo_path is None
+        or args.synthetic_multi_hot_criteo_path is None
+    ), "--in_memory_binary_criteo_path and --synthetic_multi_hot_criteo_path are mutually exclusive CLI arguments."
+    assert (
+        args.multi_hot_sizes is None or args.synthetic_multi_hot_criteo_path is None
+    ), "--multi_hot_sizes is used to convert 1-hot to multi-hot. It's inapplicable with --synthetic_multi_hot_criteo_path."
+    assert (
+        args.multi_hot_distribution_type is None
+        or args.synthetic_multi_hot_criteo_path is None
+    ), "--multi_hot_distribution_type is used to convert 1-hot to multi-hot. It's inapplicable with --synthetic_multi_hot_criteo_path."
+
+    backend = "gloo"
+    device = torch.device('cpu')
+
+    pprint(vars(args))
+    import numpy as np
+    np.random.seed(args.seed)
+    torch.manual_seed(args.seed)
     # logger.event(
     #     key=mllog_constants.OPT_NAME,
     #     value=mllog_constants.ADAGRAD if args.adagrad else mllog_constants.SGD,
@@ -1191,6 +1375,40 @@ def main(argv: List[str]) -> None:
     #     value=args.lr_decay_steps,
     # )
 
+    if args.num_embeddings_per_feature is not None:
+        args.num_embeddings = None
+
+    # Sets default limits for random dataloader iterations when left unspecified.
+    if (
+        args.in_memory_binary_criteo_path is None
+        and args.synthetic_multi_hot_criteo_path is None
+    ):
+        for split in ["train", "val", "test"]:
+            attr = f"limit_{split}_batches"
+            if getattr(args, attr) is None:
+                setattr(args, attr, 10)
+
+    sharded_module_kwargs = {}
+    if args.over_arch_layer_sizes is not None:
+        sharded_module_kwargs["over_arch_layer_sizes"] = args.over_arch_layer_sizes
+
+    if args.distributed_training:
+        import oneccl_bindings_for_pytorch
+        def env2int(env_list, default=-1):
+            for e in env_list:
+                val = int(os.environ.get(e, -1))
+                if val >= 0:
+                    return val
+            return default
+        rank = env2int(
+            ["PMI_RANK", "OMPI_COMM_WORLD_RANK", "MV2_COMM_WORLD_RANK", "RANK"], 0
+        )
+        world_size = env2int(os.environ["W_SIZE"])
+        os.environ["WORLD_SIZE"] = os.environ["W_SIZE"]
+        dist.init_process_group("ccl", world_size=world_size, rank=rank)
+
+    print_memory("start init dlrm ")
+    model, optimizer, lr_scheduler = construct_model(args)
     print_memory("start create dataloader ")
 
     # only create 1 multi-hot sample to save memory
@@ -1238,6 +1456,50 @@ def main(argv: List[str]) -> None:
         print_memory("start ipex_optimize ")
         model.model, optimizer = ipex_optimize(args, model.model, optimizer, test_dataloader)
 
+    if args.distributed_training and dist.get_world_size() > 1:
+        from torch.nn.parallel import DistributedDataParallel as DDP
+        model.model.dense_arch = DDP(model.model.dense_arch, gradient_as_bucket_view=True, broadcast_buffers=False, find_unused_parameters=True)
+        model.model.inter_arch = DDP(model.model.inter_arch, gradient_as_bucket_view=True, broadcast_buffers=False, find_unused_parameters=True)
+        model.model.over_arch = DDP(model.model.over_arch, gradient_as_bucket_view=True, broadcast_buffers=False, find_unused_parameters=True)
+
+    if args.inductor:
+        def randomrize_crossnet_bias(bias):
+            r"""
+            the bias is initialized as all zeros and in inductor will create 1 bias for all 3 bias since they are same:
+            crossnet init:
+                self.bias: torch.nn.ParameterList = torch.nn.ParameterList(
+                    [
+                        torch.nn.Parameter(torch.nn.init.zeros_(torch.empty(in_features)))
+                        for i in range(self._num_layers)
+                    ]
+                )
+            inductor process:
+                def allocate(name):
+                    for constant_name, value in self.constants.items():
+                        if (
+                            not data.is_mkldnn
+                            and data.size() == value.size()
+                            and data.stride() == value.stride()
+                            and data.dtype == value.dtype
+                            and data.device == value.device
+                            and torch.eq(data, value).all()
+                        ):
+                            return constant_name
+            But in real world, they should be different (load from pre-trained weight), so we randomrize the bias here
+            """
+            with torch.no_grad():
+                for b in bias:
+                    b.data = torch.randn_like(b)
+        randomrize_crossnet_bias(model.model.inter_arch.crossnet.bias)
+        print_memory("start StockPT ")
+        if args.dtype == 'bf16':
+            model.model.sparse_arch = model.model.sparse_arch.bfloat16()
+            # model.model.inter_arch.crossnet.bias = model.model.inter_arch.crossnet.bias.bfloat16()
+        if args.dtype == 'fp16':
+            model.model.sparse_arch = model.model.sparse_arch.half()
+            # model.model.inter_arch.crossnet.bias = model.model.inter_arch.crossnet.bias.half()
+        model.model, optimizer = stock_pt_optimize(args, model.model, optimizer, test_dataloader)
+
     print_memory("start running model")
     train_val_test(
         args,
diff --git a/models/recommendation/pytorch/torchrec_dlrm/int8_configure.json b/models/recommendation/pytorch/torchrec_dlrm/int8_configure.json
index 01d7d3abb..c3c0de8bc 100644
--- a/models/recommendation/pytorch/torchrec_dlrm/int8_configure.json
+++ b/models/recommendation/pytorch/torchrec_dlrm/int8_configure.json
@@ -19,9 +19,9 @@
                 "input_tensor_infos": [
                     {
                         "id": 7,
-                        "orig_dtype": "torch.int64",
-                        "inf_dtype": "torch.int64",
-                        "force_dtype": "torch.int64",
+                        "orig_dtype": "torch.int32",
+                        "inf_dtype": "torch.int32",
+                        "force_dtype": "torch.int32",
                         "scale": [
                             313710.8125
                         ],
@@ -31,9 +31,9 @@
                     },
                     {
                         "id": 8,
-                        "orig_dtype": "torch.int64",
-                        "inf_dtype": "torch.int64",
-                        "force_dtype": "torch.int64",
+                        "orig_dtype": "torch.int32",
+                        "inf_dtype": "torch.int32",
+                        "force_dtype": "torch.int32",
                         "scale": [
                             306.2494812011719
                         ],
@@ -43,9 +43,9 @@
                     },
                     {
                         "id": 9,
-                        "orig_dtype": "torch.int64",
-                        "inf_dtype": "torch.int64",
-                        "force_dtype": "torch.int64",
+                        "orig_dtype": "torch.int32",
+                        "inf_dtype": "torch.int32",
+                        "force_dtype": "torch.int32",
                         "scale": [
                             135.24298095703125
                         ],
@@ -55,9 +55,9 @@
                     },
                     {
                         "id": 10,
-                        "orig_dtype": "torch.int64",
-                        "inf_dtype": "torch.int64",
-                        "force_dtype": "torch.int64",
+                        "orig_dtype": "torch.int32",
+                        "inf_dtype": "torch.int32",
+                        "force_dtype": "torch.int32",
                         "scale": [
                             58.20223617553711
                         ],
@@ -67,9 +67,9 @@
                     },
                     {
                         "id": 11,
-                        "orig_dtype": "torch.int64",
-                        "inf_dtype": "torch.int64",
-                        "force_dtype": "torch.int64",
+                        "orig_dtype": "torch.int32",
+                        "inf_dtype": "torch.int32",
+                        "force_dtype": "torch.int32",
                         "scale": [
                             158.9283905029297
                         ],
@@ -79,9 +79,9 @@
                     },
                     {
                         "id": 12,
-                        "orig_dtype": "torch.int64",
-                        "inf_dtype": "torch.int64",
-                        "force_dtype": "torch.int64",
+                        "orig_dtype": "torch.int32",
+                        "inf_dtype": "torch.int32",
+                        "force_dtype": "torch.int32",
                         "scale": [
                             0.015685785561800003
                         ],
@@ -91,9 +91,9 @@
                     },
                     {
                         "id": 13,
-                        "orig_dtype": "torch.int64",
-                        "inf_dtype": "torch.int64",
-                        "force_dtype": "torch.int64",
+                        "orig_dtype": "torch.int32",
+                        "inf_dtype": "torch.int32",
+                        "force_dtype": "torch.int32",
                         "scale": [
                             55.723880767822266
                         ],
@@ -103,9 +103,9 @@
                     },
                     {
                         "id": 14,
-                        "orig_dtype": "torch.int64",
-                        "inf_dtype": "torch.int64",
-                        "force_dtype": "torch.int64",
+                        "orig_dtype": "torch.int32",
+                        "inf_dtype": "torch.int32",
+                        "force_dtype": "torch.int32",
                         "scale": [
                             12.078180313110352
                         ],
@@ -115,9 +115,9 @@
                     },
                     {
                         "id": 15,
-                        "orig_dtype": "torch.int64",
-                        "inf_dtype": "torch.int64",
-                        "force_dtype": "torch.int64",
+                        "orig_dtype": "torch.int32",
+                        "inf_dtype": "torch.int32",
+                        "force_dtype": "torch.int32",
                         "scale": [
                             0.31440040469169617
                         ],
@@ -127,9 +127,9 @@
                     },
                     {
                         "id": 16,
-                        "orig_dtype": "torch.int64",
-                        "inf_dtype": "torch.int64",
-                        "force_dtype": "torch.int64",
+                        "orig_dtype": "torch.int32",
+                        "inf_dtype": "torch.int32",
+                        "force_dtype": "torch.int32",
                         "scale": [
                             313715.65625
                         ],
@@ -139,9 +139,9 @@
                     },
                     {
                         "id": 17,
-                        "orig_dtype": "torch.int64",
-                        "inf_dtype": "torch.int64",
-                        "force_dtype": "torch.int64",
+                        "orig_dtype": "torch.int32",
+                        "inf_dtype": "torch.int32",
+                        "force_dtype": "torch.int32",
                         "scale": [
                             24061.390625
                         ],
@@ -151,9 +151,9 @@
                     },
                     {
                         "id": 18,
-                        "orig_dtype": "torch.int64",
-                        "inf_dtype": "torch.int64",
-                        "force_dtype": "torch.int64",
+                        "orig_dtype": "torch.int32",
+                        "inf_dtype": "torch.int32",
+                        "force_dtype": "torch.int32",
                         "scale": [
                             3178.567626953125
                         ],
@@ -163,9 +163,9 @@
                     },
                     {
                         "id": 19,
-                        "orig_dtype": "torch.int64",
-                        "inf_dtype": "torch.int64",
-                        "force_dtype": "torch.int64",
+                        "orig_dtype": "torch.int32",
+                        "inf_dtype": "torch.int32",
+                        "force_dtype": "torch.int32",
                         "scale": [
                             0.07058604061603546
                         ],
@@ -175,9 +175,9 @@
                     },
                     {
                         "id": 20,
-                        "orig_dtype": "torch.int64",
-                        "inf_dtype": "torch.int64",
-                        "force_dtype": "torch.int64",
+                        "orig_dtype": "torch.int32",
+                        "inf_dtype": "torch.int32",
+                        "force_dtype": "torch.int32",
                         "scale": [
                             17.317106246948242
                         ],
@@ -187,9 +187,9 @@
                     },
                     {
                         "id": 21,
-                        "orig_dtype": "torch.int64",
-                        "inf_dtype": "torch.int64",
-                        "force_dtype": "torch.int64",
+                        "orig_dtype": "torch.int32",
+                        "inf_dtype": "torch.int32",
+                        "force_dtype": "torch.int32",
                         "scale": [
                             93.62061309814453
                         ],
@@ -199,9 +199,9 @@
                     },
                     {
                         "id": 22,
-                        "orig_dtype": "torch.int64",
-                        "inf_dtype": "torch.int64",
-                        "force_dtype": "torch.int64",
+                        "orig_dtype": "torch.int32",
+                        "inf_dtype": "torch.int32",
+                        "force_dtype": "torch.int32",
                         "scale": [
                             1.2078055143356323
                         ],
@@ -211,9 +211,9 @@
                     },
                     {
                         "id": 23,
-                        "orig_dtype": "torch.int64",
-                        "inf_dtype": "torch.int64",
-                        "force_dtype": "torch.int64",
+                        "orig_dtype": "torch.int32",
+                        "inf_dtype": "torch.int32",
+                        "force_dtype": "torch.int32",
                         "scale": [
                             0.023528680205345154
                         ],
@@ -223,9 +223,9 @@
                     },
                     {
                         "id": 24,
-                        "orig_dtype": "torch.int64",
-                        "inf_dtype": "torch.int64",
-                        "force_dtype": "torch.int64",
+                        "orig_dtype": "torch.int32",
+                        "inf_dtype": "torch.int32",
+                        "force_dtype": "torch.int32",
                         "scale": [
                             7.113068580627441
                         ],
@@ -235,9 +235,9 @@
                     },
                     {
                         "id": 25,
-                        "orig_dtype": "torch.int64",
-                        "inf_dtype": "torch.int64",
-                        "force_dtype": "torch.int64",
+                        "orig_dtype": "torch.int32",
+                        "inf_dtype": "torch.int32",
+                        "force_dtype": "torch.int32",
                         "scale": [
                             0.10195761173963547
                         ],
@@ -247,9 +247,9 @@
                     },
                     {
                         "id": 26,
-                        "orig_dtype": "torch.int64",
-                        "inf_dtype": "torch.int64",
-                        "force_dtype": "torch.int64",
+                        "orig_dtype": "torch.int32",
+                        "inf_dtype": "torch.int32",
+                        "force_dtype": "torch.int32",
                         "scale": [
                             313715.28125
                         ],
@@ -259,9 +259,9 @@
                     },
                     {
                         "id": 27,
-                        "orig_dtype": "torch.int64",
-                        "inf_dtype": "torch.int64",
-                        "force_dtype": "torch.int64",
+                        "orig_dtype": "torch.int32",
+                        "inf_dtype": "torch.int32",
+                        "force_dtype": "torch.int32",
                         "scale": [
                             313715.65625
                         ],
@@ -271,9 +271,9 @@
                     },
                     {
                         "id": 28,
-                        "orig_dtype": "torch.int64",
-                        "inf_dtype": "torch.int64",
-                        "force_dtype": "torch.int64",
+                        "orig_dtype": "torch.int32",
+                        "inf_dtype": "torch.int32",
+                        "force_dtype": "torch.int32",
                         "scale": [
                             313715.03125
                         ],
@@ -283,9 +283,9 @@
                     },
                     {
                         "id": 29,
-                        "orig_dtype": "torch.int64",
-                        "inf_dtype": "torch.int64",
-                        "force_dtype": "torch.int64",
+                        "orig_dtype": "torch.int32",
+                        "inf_dtype": "torch.int32",
+                        "force_dtype": "torch.int32",
                         "scale": [
                             4628.4443359375
                         ],
@@ -295,9 +295,9 @@
                     },
                     {
                         "id": 30,
-                        "orig_dtype": "torch.int64",
-                        "inf_dtype": "torch.int64",
-                        "force_dtype": "torch.int64",
+                        "orig_dtype": "torch.int32",
+                        "inf_dtype": "torch.int32",
+                        "force_dtype": "torch.int32",
                         "scale": [
                             101.72232055664062
                         ],
@@ -307,9 +307,9 @@
                     },
                     {
                         "id": 31,
-                        "orig_dtype": "torch.int64",
-                        "inf_dtype": "torch.int64",
-                        "force_dtype": "torch.int64",
+                        "orig_dtype": "torch.int32",
+                        "inf_dtype": "torch.int32",
+                        "force_dtype": "torch.int32",
                         "scale": [
                             0.45003268122673035
                         ],
@@ -319,9 +319,9 @@
                     },
                     {
                         "id": 32,
-                        "orig_dtype": "torch.int64",
-                        "inf_dtype": "torch.int64",
-                        "force_dtype": "torch.int64",
+                        "orig_dtype": "torch.int32",
+                        "inf_dtype": "torch.int32",
+                        "force_dtype": "torch.int32",
                         "scale": [
                             0.26678285002708435
                         ],
@@ -331,9 +331,9 @@
                     },
                     {
                         "id": 33,
-                        "orig_dtype": "torch.int64",
-                        "inf_dtype": "torch.int64",
-                        "force_dtype": "torch.int64",
+                        "orig_dtype": "torch.int32",
+                        "inf_dtype": "torch.int32",
+                        "force_dtype": "torch.int32",
                         "scale": [
                             3011.6708984375
                         ],
@@ -343,9 +343,9 @@
                     },
                     {
                         "id": 34,
-                        "orig_dtype": "torch.int64",
-                        "inf_dtype": "torch.int64",
-                        "force_dtype": "torch.int64",
+                        "orig_dtype": "torch.int32",
+                        "inf_dtype": "torch.int32",
+                        "force_dtype": "torch.int32",
                         "scale": [
                             2007.7806396484375
                         ],
@@ -355,9 +355,9 @@
                     },
                     {
                         "id": 35,
-                        "orig_dtype": "torch.int64",
-                        "inf_dtype": "torch.int64",
-                        "force_dtype": "torch.int64",
+                        "orig_dtype": "torch.int32",
+                        "inf_dtype": "torch.int32",
+                        "force_dtype": "torch.int32",
                         "scale": [
                             1003.8903198242188
                         ],
@@ -367,9 +367,9 @@
                     },
                     {
                         "id": 36,
-                        "orig_dtype": "torch.int64",
-                        "inf_dtype": "torch.int64",
-                        "force_dtype": "torch.int64",
+                        "orig_dtype": "torch.int32",
+                        "inf_dtype": "torch.int32",
+                        "force_dtype": "torch.int32",
                         "scale": [
                             2007.7806396484375
                         ],
@@ -379,9 +379,9 @@
                     },
                     {
                         "id": 37,
-                        "orig_dtype": "torch.int64",
-                        "inf_dtype": "torch.int64",
-                        "force_dtype": "torch.int64",
+                        "orig_dtype": "torch.int32",
+                        "inf_dtype": "torch.int32",
+                        "force_dtype": "torch.int32",
                         "scale": [
                             6023.341796875
                         ],
@@ -391,9 +391,9 @@
                     },
                     {
                         "id": 38,
-                        "orig_dtype": "torch.int64",
-                        "inf_dtype": "torch.int64",
-                        "force_dtype": "torch.int64",
+                        "orig_dtype": "torch.int32",
+                        "inf_dtype": "torch.int32",
+                        "force_dtype": "torch.int32",
                         "scale": [
                             1003.8903198242188
                         ],
@@ -403,9 +403,9 @@
                     },
                     {
                         "id": 39,
-                        "orig_dtype": "torch.int64",
-                        "inf_dtype": "torch.int64",
-                        "force_dtype": "torch.int64",
+                        "orig_dtype": "torch.int32",
+                        "inf_dtype": "torch.int32",
+                        "force_dtype": "torch.int32",
                         "scale": [
                             1003.8903198242188
                         ],
@@ -415,9 +415,9 @@
                     },
                     {
                         "id": 40,
-                        "orig_dtype": "torch.int64",
-                        "inf_dtype": "torch.int64",
-                        "force_dtype": "torch.int64",
+                        "orig_dtype": "torch.int32",
+                        "inf_dtype": "torch.int32",
+                        "force_dtype": "torch.int32",
                         "scale": [
                             1003.8903198242188
                         ],
@@ -427,9 +427,9 @@
                     },
                     {
                         "id": 41,
-                        "orig_dtype": "torch.int64",
-                        "inf_dtype": "torch.int64",
-                        "force_dtype": "torch.int64",
+                        "orig_dtype": "torch.int32",
+                        "inf_dtype": "torch.int32",
+                        "force_dtype": "torch.int32",
                         "scale": [
                             1003.8903198242188
                         ],
@@ -439,9 +439,9 @@
                     },
                     {
                         "id": 42,
-                        "orig_dtype": "torch.int64",
-                        "inf_dtype": "torch.int64",
-                        "force_dtype": "torch.int64",
+                        "orig_dtype": "torch.int32",
+                        "inf_dtype": "torch.int32",
+                        "force_dtype": "torch.int32",
                         "scale": [
                             7027.232421875
                         ],
@@ -451,9 +451,9 @@
                     },
                     {
                         "id": 43,
-                        "orig_dtype": "torch.int64",
-                        "inf_dtype": "torch.int64",
-                        "force_dtype": "torch.int64",
+                        "orig_dtype": "torch.int32",
+                        "inf_dtype": "torch.int32",
+                        "force_dtype": "torch.int32",
                         "scale": [
                             3011.6708984375
                         ],
@@ -463,9 +463,9 @@
                     },
                     {
                         "id": 44,
-                        "orig_dtype": "torch.int64",
-                        "inf_dtype": "torch.int64",
-                        "force_dtype": "torch.int64",
+                        "orig_dtype": "torch.int32",
+                        "inf_dtype": "torch.int32",
+                        "force_dtype": "torch.int32",
                         "scale": [
                             8031.12255859375
                         ],
@@ -475,9 +475,9 @@
                     },
                     {
                         "id": 45,
-                        "orig_dtype": "torch.int64",
-                        "inf_dtype": "torch.int64",
-                        "force_dtype": "torch.int64",
+                        "orig_dtype": "torch.int32",
+                        "inf_dtype": "torch.int32",
+                        "force_dtype": "torch.int32",
                         "scale": [
                             1003.8903198242188
                         ],
@@ -487,9 +487,9 @@
                     },
                     {
                         "id": 46,
-                        "orig_dtype": "torch.int64",
-                        "inf_dtype": "torch.int64",
-                        "force_dtype": "torch.int64",
+                        "orig_dtype": "torch.int32",
+                        "inf_dtype": "torch.int32",
+                        "force_dtype": "torch.int32",
                         "scale": [
                             6023.341796875
                         ],
@@ -499,9 +499,9 @@
                     },
                     {
                         "id": 47,
-                        "orig_dtype": "torch.int64",
-                        "inf_dtype": "torch.int64",
-                        "force_dtype": "torch.int64",
+                        "orig_dtype": "torch.int32",
+                        "inf_dtype": "torch.int32",
+                        "force_dtype": "torch.int32",
                         "scale": [
                             9035.0126953125
                         ],
@@ -511,9 +511,9 @@
                     },
                     {
                         "id": 48,
-                        "orig_dtype": "torch.int64",
-                        "inf_dtype": "torch.int64",
-                        "force_dtype": "torch.int64",
+                        "orig_dtype": "torch.int32",
+                        "inf_dtype": "torch.int32",
+                        "force_dtype": "torch.int32",
                         "scale": [
                             5019.45166015625
                         ],
@@ -523,9 +523,9 @@
                     },
                     {
                         "id": 49,
-                        "orig_dtype": "torch.int64",
-                        "inf_dtype": "torch.int64",
-                        "force_dtype": "torch.int64",
+                        "orig_dtype": "torch.int32",
+                        "inf_dtype": "torch.int32",
+                        "force_dtype": "torch.int32",
                         "scale": [
                             1003.8903198242188
                         ],
@@ -535,9 +535,9 @@
                     },
                     {
                         "id": 50,
-                        "orig_dtype": "torch.int64",
-                        "inf_dtype": "torch.int64",
-                        "force_dtype": "torch.int64",
+                        "orig_dtype": "torch.int32",
+                        "inf_dtype": "torch.int32",
+                        "force_dtype": "torch.int32",
                         "scale": [
                             1003.8903198242188
                         ],
@@ -547,9 +547,9 @@
                     },
                     {
                         "id": 51,
-                        "orig_dtype": "torch.int64",
-                        "inf_dtype": "torch.int64",
-                        "force_dtype": "torch.int64",
+                        "orig_dtype": "torch.int32",
+                        "inf_dtype": "torch.int32",
+                        "force_dtype": "torch.int32",
                         "scale": [
                             1003.8903198242188
                         ],
@@ -559,9 +559,9 @@
                     },
                     {
                         "id": 52,
-                        "orig_dtype": "torch.int64",
-                        "inf_dtype": "torch.int64",
-                        "force_dtype": "torch.int64",
+                        "orig_dtype": "torch.int32",
+                        "inf_dtype": "torch.int32",
+                        "force_dtype": "torch.int32",
                         "scale": [
                             12046.68359375
                         ],
@@ -571,9 +571,9 @@
                     },
                     {
                         "id": 53,
-                        "orig_dtype": "torch.int64",
-                        "inf_dtype": "torch.int64",
-                        "force_dtype": "torch.int64",
+                        "orig_dtype": "torch.int32",
+                        "inf_dtype": "torch.int32",
+                        "force_dtype": "torch.int32",
                         "scale": [
                             100389.03125
                         ],
@@ -583,9 +583,9 @@
                     },
                     {
                         "id": 54,
-                        "orig_dtype": "torch.int64",
-                        "inf_dtype": "torch.int64",
-                        "force_dtype": "torch.int64",
+                        "orig_dtype": "torch.int32",
+                        "inf_dtype": "torch.int32",
+                        "force_dtype": "torch.int32",
                         "scale": [
                             27105.037109375
                         ],
@@ -595,9 +595,9 @@
                     },
                     {
                         "id": 55,
-                        "orig_dtype": "torch.int64",
-                        "inf_dtype": "torch.int64",
-                        "force_dtype": "torch.int64",
+                        "orig_dtype": "torch.int32",
+                        "inf_dtype": "torch.int32",
+                        "force_dtype": "torch.int32",
                         "scale": [
                             10038.9033203125
                         ],
@@ -607,9 +607,9 @@
                     },
                     {
                         "id": 56,
-                        "orig_dtype": "torch.int64",
-                        "inf_dtype": "torch.int64",
-                        "force_dtype": "torch.int64",
+                        "orig_dtype": "torch.int32",
+                        "inf_dtype": "torch.int32",
+                        "force_dtype": "torch.int32",
                         "scale": [
                             3011.6708984375
                         ],
@@ -619,9 +619,9 @@
                     },
                     {
                         "id": 57,
-                        "orig_dtype": "torch.int64",
-                        "inf_dtype": "torch.int64",
-                        "force_dtype": "torch.int64",
+                        "orig_dtype": "torch.int32",
+                        "inf_dtype": "torch.int32",
+                        "force_dtype": "torch.int32",
                         "scale": [
                             1003.8903198242188
                         ],
@@ -631,9 +631,9 @@
                     },
                     {
                         "id": 58,
-                        "orig_dtype": "torch.int64",
-                        "inf_dtype": "torch.int64",
-                        "force_dtype": "torch.int64",
+                        "orig_dtype": "torch.int32",
+                        "inf_dtype": "torch.int32",
+                        "force_dtype": "torch.int32",
                         "scale": [
                             1003.8903198242188
                         ],
@@ -659,7 +659,7 @@
                         "orig_dtype": "torch.float32",
                         "inf_dtype": "torch.qint8",
                         "scale": [
-                            0.0007458091131411493
+                            0.0008381635998375714
                         ],
                         "zero_point": [
                             0
@@ -849,7 +849,7 @@
                         "orig_dtype": "torch.float32",
                         "inf_dtype": "torch.qint8",
                         "scale": [
-                            0.0007534330943599343
+                            0.0008294559083878994
                         ],
                         "zero_point": [
                             0
@@ -2091,7 +2091,7 @@
                 }
             },
             "1": {
-                "op_type": "<built-in method relu of type object at 0x1553761a7640>",
+                "op_type": "<built-in method relu of type object at 0x7fe360e70da0>",
                 "op_type_is_module": false,
                 "fqn": "dense_arch.model._mlp.0",
                 "input_tensor_infos": [
@@ -2734,7 +2734,7 @@
                 }
             },
             "1": {
-                "op_type": "<built-in method relu of type object at 0x1553761a7640>",
+                "op_type": "<built-in method relu of type object at 0x7fe360e70da0>",
                 "op_type_is_module": false,
                 "fqn": "dense_arch.model._mlp.1",
                 "input_tensor_infos": [
@@ -3121,7 +3121,7 @@
                 }
             },
             "1": {
-                "op_type": "<built-in method relu of type object at 0x1553761a7640>",
+                "op_type": "<built-in method relu of type object at 0x7fe360e70da0>",
                 "op_type_is_module": false,
                 "fqn": "dense_arch.model._mlp.2",
                 "input_tensor_infos": [
@@ -3189,11 +3189,22 @@
         ]
     },
     "inter_arch": {
+        "q_op_infos": {},
+        "nonq_op_infos": {},
+        "layer_output_infos": [
+            {
+                "id": 71,
+                "orig_dtype": "torch.float32",
+                "inf_dtype": "torch.qint8"
+            }
+        ]
+    },
+    "inter_arch:crossnet": {
         "q_op_infos": {
             "0": {
                 "op_type": "<class 'torch.nn.modules.linear.Linear'>",
                 "op_type_is_module": true,
-                "fqn": "inter_arch.V_linears.0",
+                "fqn": "inter_arch.crossnet.V_linears.0",
                 "input_tensor_infos": [
                     {
                         "id": 59,
@@ -4278,7 +4289,7 @@
             "1": {
                 "op_type": "<class 'torch.nn.modules.linear.Linear'>",
                 "op_type_is_module": true,
-                "fqn": "inter_arch.W_linears.0",
+                "fqn": "inter_arch.crossnet.W_linears.0",
                 "input_tensor_infos": [
                     {
                         "id": 60,
@@ -11243,9 +11254,9 @@
                 }
             },
             "2": {
-                "op_type": "<method 'add' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'add' of 'torch._C.TensorBase' objects>",
                 "op_type_is_module": false,
-                "fqn": "inter_arch",
+                "fqn": "inter_arch.crossnet",
                 "input_tensor_infos": [
                     {
                         "id": 62,
@@ -11309,7 +11320,7 @@
             "3": {
                 "op_type": "<class 'torch.nn.modules.linear.Linear'>",
                 "op_type_is_module": true,
-                "fqn": "inter_arch.V_linears.1",
+                "fqn": "inter_arch.crossnet.V_linears.1",
                 "input_tensor_infos": [
                     {
                         "id": 63,
@@ -12394,7 +12405,7 @@
             "4": {
                 "op_type": "<class 'torch.nn.modules.linear.Linear'>",
                 "op_type_is_module": true,
-                "fqn": "inter_arch.W_linears.1",
+                "fqn": "inter_arch.crossnet.W_linears.1",
                 "input_tensor_infos": [
                     {
                         "id": 64,
@@ -19359,9 +19370,9 @@
                 }
             },
             "5": {
-                "op_type": "<method 'add' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'add' of 'torch._C.TensorBase' objects>",
                 "op_type_is_module": false,
-                "fqn": "inter_arch",
+                "fqn": "inter_arch.crossnet",
                 "input_tensor_infos": [
                     {
                         "id": 66,
@@ -19425,7 +19436,7 @@
             "6": {
                 "op_type": "<class 'torch.nn.modules.linear.Linear'>",
                 "op_type_is_module": true,
-                "fqn": "inter_arch.V_linears.2",
+                "fqn": "inter_arch.crossnet.V_linears.2",
                 "input_tensor_infos": [
                     {
                         "id": 67,
@@ -20510,7 +20521,7 @@
             "7": {
                 "op_type": "<class 'torch.nn.modules.linear.Linear'>",
                 "op_type_is_module": true,
-                "fqn": "inter_arch.W_linears.2",
+                "fqn": "inter_arch.crossnet.W_linears.2",
                 "input_tensor_infos": [
                     {
                         "id": 68,
@@ -27475,9 +27486,9 @@
                 }
             },
             "8": {
-                "op_type": "<method 'add' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'add' of 'torch._C.TensorBase' objects>",
                 "op_type_is_module": false,
-                "fqn": "inter_arch",
+                "fqn": "inter_arch.crossnet",
                 "input_tensor_infos": [
                     {
                         "id": 70,
@@ -27535,8 +27546,8 @@
         },
         "nonq_op_infos": {
             "0": {
-                "op_type": "<method 'mul' of 'torch._C._TensorBase' objects>",
-                "fqn": "inter_arch",
+                "op_type": "<method 'mul' of 'torch._C.TensorBase' objects>",
+                "fqn": "inter_arch.crossnet",
                 "input_tensor_infos": [
                     {
                         "id": 59,
@@ -27558,8 +27569,8 @@
                 ]
             },
             "1": {
-                "op_type": "<method 'mul' of 'torch._C._TensorBase' objects>",
-                "fqn": "inter_arch",
+                "op_type": "<method 'mul' of 'torch._C.TensorBase' objects>",
+                "fqn": "inter_arch.crossnet",
                 "input_tensor_infos": [
                     {
                         "id": 59,
@@ -27581,8 +27592,8 @@
                 ]
             },
             "2": {
-                "op_type": "<method 'mul' of 'torch._C._TensorBase' objects>",
-                "fqn": "inter_arch",
+                "op_type": "<method 'mul' of 'torch._C.TensorBase' objects>",
+                "fqn": "inter_arch.crossnet",
                 "input_tensor_infos": [
                     {
                         "id": 59,
@@ -27612,12 +27623,12 @@
             }
         ]
     },
-    "inter_arch:V_linears": {
+    "inter_arch:crossnet:V_linears": {
         "q_op_infos": {},
         "nonq_op_infos": {},
         "layer_output_infos": []
     },
-    "inter_arch:W_linears": {
+    "inter_arch:crossnet:W_linears": {
         "q_op_infos": {},
         "nonq_op_infos": {},
         "layer_output_infos": []
@@ -29836,7 +29847,7 @@
                 }
             },
             "1": {
-                "op_type": "<built-in method relu of type object at 0x1553761a7640>",
+                "op_type": "<built-in method relu of type object at 0x7fe360e70da0>",
                 "op_type_is_module": false,
                 "fqn": "over_arch.model.0._mlp.0",
                 "input_tensor_infos": [
@@ -32015,7 +32026,7 @@
                 }
             },
             "1": {
-                "op_type": "<built-in method relu of type object at 0x1553761a7640>",
+                "op_type": "<built-in method relu of type object at 0x7fe360e70da0>",
                 "op_type_is_module": false,
                 "fqn": "over_arch.model.0._mlp.1",
                 "input_tensor_infos": [
@@ -33170,7 +33181,7 @@
                 }
             },
             "1": {
-                "op_type": "<built-in method relu of type object at 0x1553761a7640>",
+                "op_type": "<built-in method relu of type object at 0x7fe360e70da0>",
                 "op_type_is_module": false,
                 "fqn": "over_arch.model.0._mlp.2",
                 "input_tensor_infos": [
@@ -33813,7 +33824,7 @@
                 }
             },
             "1": {
-                "op_type": "<built-in method relu of type object at 0x1553761a7640>",
+                "op_type": "<built-in method relu of type object at 0x7fe360e70da0>",
                 "op_type_is_module": false,
                 "fqn": "over_arch.model.0._mlp.3",
                 "input_tensor_infos": [
diff --git a/models/recommendation/pytorch/torchrec_dlrm/ipex_optimized_model/model.py b/models/recommendation/pytorch/torchrec_dlrm/ipex_optimized_model/model.py
index 698432290..ffc74d335 100644
--- a/models/recommendation/pytorch/torchrec_dlrm/ipex_optimized_model/model.py
+++ b/models/recommendation/pytorch/torchrec_dlrm/ipex_optimized_model/model.py
@@ -24,7 +24,124 @@
 
 from typing import Dict, List, Optional, Tuple
 import intel_extension_for_pytorch as ipex
-from intel_extension_for_pytorch.nn.modules import MergedEmbeddingBagWithCat
+from intel_extension_for_pytorch.nn.modules import MergedEmbeddingBagWithCat, MergedEmbeddingBagWithAdaGrad, DistMergeEmbeddingBagWithAdaGrad
+import numpy as np
+
+def _calculate_fan_in_and_fan_out(shape):
+    # numpy array version
+    dimensions = len(shape)
+    assert dimensions >= 2, "Fan in and fan out can not be computed for tensor with fewer than 2 dimensions"
+
+    num_input_fmaps = shape[1]
+    num_output_fmaps = shape[0]
+    receptive_field_size = 1
+    if len(shape) > 2:
+        # math.prod is not always available, accumulate the product manually
+        # we could use functools.reduce but that is not supported by TorchScript
+        for s in shape[2:]:
+            receptive_field_size *= s
+    fan_in = num_input_fmaps * receptive_field_size
+    fan_out = num_output_fmaps * receptive_field_size
+
+    return fan_in, fan_out
+
+def _calculate_correct_fan(shape, mode):
+    mode = mode.lower()
+    valid_modes = ['fan_in', 'fan_out']
+    if mode not in valid_modes:
+        raise ValueError("Mode {} not supported, please use one of {}".format(mode, valid_modes))
+
+    fan_in, fan_out = _calculate_fan_in_and_fan_out(shape)
+    return fan_in if mode == 'fan_in' else fan_out
+
+def calculate_gain(nonlinearity, param=None):
+    linear_fns = ['linear', 'conv1d', 'conv2d', 'conv3d', 'conv_transpose1d', 'conv_transpose2d', 'conv_transpose3d']
+    if nonlinearity in linear_fns or nonlinearity == 'sigmoid':
+        return 1
+    elif nonlinearity == 'tanh':
+        return 5.0 / 3
+    elif nonlinearity == 'relu':
+        return np.sqrt(2.0)
+    elif nonlinearity == 'leaky_relu':
+        if param is None:
+            negative_slope = 0.01
+        elif not isinstance(param, bool) and isinstance(param, int) or isinstance(param, float):
+            # True/False are instances of int, hence check above
+            negative_slope = param
+        else:
+            raise ValueError("negative_slope {} not a valid number".format(param))
+        return np.sqrt(2.0 / (1 + negative_slope ** 2))
+    elif nonlinearity == 'selu':
+        return 3.0 / 4  # Value found empirically (https://github.com/pytorch/pytorch/pull/50664)
+    else:
+        raise ValueError("Unsupported nonlinearity {}".format(nonlinearity))
+
+def xavier_norm_(shape: tuple, gain: float = 1.):
+    fan_in, fan_out = _calculate_fan_in_and_fan_out(shape)
+    std = gain * np.sqrt(2.0 / float(fan_in + fan_out))
+    mean = 0.0
+    d = np.random.normal(mean, std, size=shape).astype(np.float32)
+    return d
+
+def kaiming_uniform_(shape: tuple, a: float = 0,
+                     mode: str = 'fan_in',
+                     nonlinearity: str = 'leaky_relu'
+):
+    assert (0 not in shape), "Initializing zero-element tensors is a no-op"
+    fan = _calculate_correct_fan(shape, mode)
+    gain = calculate_gain(nonlinearity, a)
+    std = gain / np.sqrt(fan)
+    bound = np.sqrt(3.0) * std  # Calculate uniform bounds from standard deviation
+    return np.random.uniform(-bound, bound, size=shape)
+
+def init_weight(dlrm):
+    # Sparse Arch
+    for k, m in dlrm.sparse_arch.embedding_bag_collection.embedding_bags.items():
+        num_embeddings, embedding_dim = m.weight.shape
+        W = np.random.uniform(
+            low=-np.sqrt(1 / num_embeddings),
+            high=np.sqrt(1 / num_embeddings),
+            size=(num_embeddings, embedding_dim)).astype(np.float32)
+        with torch.no_grad():
+            m.weight.data = torch.tensor(W, requires_grad=True)
+    # Dense Arch
+    dense_mlp = dlrm.dense_arch.model._mlp
+    for layer in dense_mlp:
+        n, m = layer._linear.weight.shape
+        W = kaiming_uniform_((n, m), a=np.sqrt(5)).astype(np.float32)
+        with torch.no_grad():
+            layer._linear.weight.data = torch.tensor(W, requires_grad=True)
+            layer._linear.bias.data.zero_()
+    # LowRankCrossNet
+    crossnet = dlrm.inter_arch.crossnet
+    for w in crossnet.W_kernels:
+        m, n = w.shape
+        W = xavier_norm_((m, n))
+        with torch.no_grad():
+            w.data = torch.tensor(W, requires_grad=True)
+    for w in crossnet.V_kernels:
+        m, n = w.shape
+        W = xavier_norm_((m, n))
+        with torch.no_grad():
+            w.data = torch.tensor(W, requires_grad=True)
+    for b in crossnet.bias:
+        with torch.no_grad():
+            b.data.zero_()
+
+    # Over Arch
+    over_mlp = dlrm.over_arch.model[0]._mlp
+    for layer in over_mlp:
+        n, m = layer._linear.weight.shape
+        W = kaiming_uniform_((n, m), a=np.sqrt(5)).astype(np.float32)
+        with torch.no_grad():
+            layer._linear.weight.data = torch.tensor(W, requires_grad=True)
+            layer._linear.bias.data.zero_()
+    last_layer = dlrm.over_arch.model[1]
+    n, m = last_layer.weight.shape
+    W = kaiming_uniform_((n, m), a=np.sqrt(5)).astype(np.float32)
+    with torch.no_grad():
+        last_layer.weight.data = torch.tensor(W, requires_grad=True)
+        last_layer.bias.data.zero_()
 
 class _LowRankCrossNet(torch.nn.Module):
     def __init__(
@@ -56,7 +173,7 @@ def forward(self, input: torch.Tensor) -> torch.Tensor:
 def replace_crossnet(dlrm):
     crossnet = dlrm.inter_arch.crossnet
     new_crossnet = _LowRankCrossNet(crossnet)
-    dlrm.inter_arch = new_crossnet
+    dlrm.inter_arch.crossnet = new_crossnet
     del(crossnet)
 
 class SparseArchCatDense(SparseArch):
@@ -74,11 +191,17 @@ def forward(
         Returns:
             torch.Tensor: tensor of shape B X (F + 1) X D.
         """
+        (B, _) = embedded_dense_features.shape
         embedding_bag_collection = self.embedding_bag_collection
+        indices = tuple([sf['values'] for _, sf in sparse_features.items()])
+        offsets = tuple([sf['offsets'] for _, sf in sparse_features.items()])
         if isinstance(embedding_bag_collection, MergedEmbeddingBagWithCat):
-            indices = tuple([sf['values'] for _, sf in sparse_features.items()])
-            offsets = tuple([sf['offsets'] for _, sf in sparse_features.items()])
             return embedding_bag_collection(indices, offsets, embedded_dense_features)
+        elif isinstance(embedding_bag_collection, DistMergeEmbeddingBagWithAdaGrad):
+            embedded_sparse_features = embedding_bag_collection(indices, offsets)
+            return torch.cat((embedded_dense_features.unsqueeze(1), embedded_sparse_features), dim=1).reshape(B, -1)
+        elif isinstance(embedding_bag_collection, MergedEmbeddingBagWithAdaGrad):
+            embedded_sparse_features = embedding_bag_collection(indices, offsets)
         else:
             embedded_sparse_features: List[torch.Tensor] = []
             for i, embedding_bag in enumerate(embedding_bag_collection.embedding_bags.values()):
@@ -90,14 +213,28 @@ def forward(
                         per_sample_weights=None,
                     )
                     embedded_sparse_features.append(res)
-            to_cat = [embedded_dense_features] + embedded_sparse_features
-            return torch.cat(to_cat, dim=1)
-
-def replace_embeddingbag_collection(dlrm):
-    collection = list(dlrm.sparse_arch.embedding_bag_collection.embedding_bags.values())
-    new_collection = MergedEmbeddingBagWithCat.from_embeddingbag_list(collection)
-    dlrm.sparse_arch.embedding_bag_collection = new_collection
-    del(collection)
+        to_cat = [embedded_dense_features] + list(embedded_sparse_features)
+        out = torch.cat(to_cat, dim=1)
+        return out
+
+def replace_embeddingbag_collection(dlrm, args):
+    if args.ipex_optimize:
+        new_collection = None
+        collection = list(dlrm.sparse_arch.embedding_bag_collection.embedding_bags.values())
+        if args.inference_only and args.ipex_merged_emb_cat:
+            new_collection = MergedEmbeddingBagWithCat.from_embeddingbag_list(collection)
+        if not args.inference_only:
+            if args.ipex_dist_merged_emb_adagrad:
+                new_collection = DistMergeEmbeddingBagWithAdaGrad.from_embeddingbag_list(collection, lr=args.learning_rate, eps=1e-8)
+            elif args.ipex_merged_emb_adagrad:
+                new_collection = MergedEmbeddingBagWithAdaGrad.from_embeddingbag_list(collection, lr=args.learning_rate, eps=1e-8)
+            else:
+                for emb in collection:
+                    emb.sparse = True
+
+        if new_collection:
+            dlrm.sparse_arch.embedding_bag_collection = new_collection
+            del(collection)
 
 class InteractionDCNArchWithoutCat(InteractionDCNArch):
 
diff --git a/models/recommendation/tensorflow/ncf/__init__.py b/models/recommendation/tensorflow/ncf/__init__.py
deleted file mode 100644
index c4fdb7d61..000000000
--- a/models/recommendation/tensorflow/ncf/__init__.py
+++ /dev/null
@@ -1,19 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
diff --git a/models/recommendation/tensorflow/ncf/inference/__init__.py b/models/recommendation/tensorflow/ncf/inference/__init__.py
deleted file mode 100644
index c4fdb7d61..000000000
--- a/models/recommendation/tensorflow/ncf/inference/__init__.py
+++ /dev/null
@@ -1,19 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
diff --git a/models/recommendation/tensorflow/ncf/inference/fp32/__init__.py b/models/recommendation/tensorflow/ncf/inference/fp32/__init__.py
deleted file mode 100644
index c4fdb7d61..000000000
--- a/models/recommendation/tensorflow/ncf/inference/fp32/__init__.py
+++ /dev/null
@@ -1,19 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
diff --git a/models/recommendation/tensorflow/ncf/inference/fp32/ncf_main.py b/models/recommendation/tensorflow/ncf/inference/fp32/ncf_main.py
deleted file mode 100644
index 338a2b00f..000000000
--- a/models/recommendation/tensorflow/ncf/inference/fp32/ncf_main.py
+++ /dev/null
@@ -1,594 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
-
-# Copyright 2018 The TensorFlow Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-"""NCF framework to train and evaluate the NeuMF model.
-The NeuMF model assembles both MF and MLP models under the NCF framework. Check
-`neumf_model.py` for more details about the models.
-"""
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-
-import contextlib
-import gc
-import heapq
-import math
-import multiprocessing
-import os
-import signal
-import typing
-import time
-from datetime import datetime
-
-# pylint: disable=g-bad-import-order
-import numpy as np
-from absl import app as absl_app
-from absl import flags
-import tensorflow as tf
-# pylint: enable=g-bad-import-order
-
-from official.datasets import movielens
-from official.recommendation import constants as rconst
-from official.recommendation import data_preprocessing
-from official.recommendation import neumf_model
-from official.utils.flags import core as flags_core
-from official.utils.logs import hooks_helper
-from official.utils.logs import logger
-from official.utils.misc import distribution_utils
-from official.utils.misc import model_helpers
-
-_TOP_K = 10  # Top-k list for evaluation
-# keys for evaluation metrics
-_HR_KEY = "HR"
-_NDCG_KEY = "NDCG"
-
-
-class LoggerHook(tf.train.SessionRunHook):
-    """ Logs runtime. """
-
-    def begin(self):
-        self._step = -1
-        self._displayed_steps = 0
-        self._total_recommendations_per_sec = 0
-        self._total_duration = 0
-
-    def before_run(self, run_context):
-        self._step += 1
-        self._start_time = time.time()
-
-    def after_run(self, run_context, run_values):
-        duration = time.time() - self._start_time
-        # Display benchmarking metrics
-        display_every = 100 if FLAGS.dataset == 'ml-1m' else 50000
-        if self._step != 0 and self._step % display_every == 0:
-            recommendations_per_sec = FLAGS.batch_size / duration
-            self._displayed_steps += 1
-            self._total_recommendations_per_sec += recommendations_per_sec
-            self._total_duration += duration
-            format_str = ('%s: step %d, %.1f recommendations/sec, %.5f msec/batch')
-            print(format_str % (datetime.now(), self._step, recommendations_per_sec, duration * 1000))
-
-    def end(self, run_context):
-        print('Average recommendations/sec across %d steps: %.1f (%.5f msec/batch)' %
-              (self._step, self._total_recommendations_per_sec / self._displayed_steps,
-               (self._total_duration * 1000) / self._displayed_steps))
-
-
-def get_hit_rate_and_ndcg(predicted_scores_by_user, items_by_user, top_k=_TOP_K,
-                          match_mlperf=False):
-    """Returns the hit rate and the normalized DCG for evaluation.
-    `predicted_scores_by_user` and `items_by_user` are parallel NumPy arrays with
-    shape (num_users, num_items) such that `predicted_scores_by_user[i, j]` is the
-    predicted score that user `i` would rate item `items_by_user[i][j]`.
-    `items_by_user[i, 0]` is the item that user `i` interacted with, while
-    `items_by_user[i, 1:] are items that user `i` did not interact with. The goal
-    of the NCF model to give a high score for `predicted_scores_by_user[i, 0]`
-    compared to `predicted_scores_by_user[i, 1:]`, and the returned HR and NDCG
-    will be higher the more successful the model is at this goal.
-    If `match_mlperf` is True, then the HR and NDCG computations are done in a
-    slightly unusual way to match the MLPerf reference implementation.
-    Specifically, if `items_by_user[i, :]` contains duplicate items, it will be
-    treated as if the item only appeared once. Effectively, for duplicate items in
-    a row, the predicted score for all but one of the items will be set to
-    -infinity
-    For example, suppose we have that following inputs:
-    predicted_scores_by_user: [[ 2,  3,  3],
-                               [ 5,  4,  4]]
-    items_by_user:            [[10, 20, 20],
-                               [30, 40, 40]]
-    top_k: 2
-    Then with match_mlperf=True, the HR would be 2/2 = 1.0. With
-    match_mlperf=False, the HR would be 1/2 = 0.5. This is because each user has
-    predicted scores for only 2 unique items: 10 and 20 for the first user, and 30
-    and 40 for the second. Therefore, with match_mlperf=True, it's guarenteed the
-    first item's score is in the top 2. With match_mlperf=False, this function
-    would compute the first user's first item is not in the top 2, because item 20
-    has a higher score, and item 20 occurs twice.
-    Args:
-      predicted_scores_by_user: 2D Numpy array of the predicted scores.
-        `predicted_scores_by_user[i, j]` is the predicted score that user `i`
-        would rate item `items_by_user[i][j]`.
-      items_by_user: 2d numpy array of the item IDs. For user `i`,
-        `items_by_user[i][0]` is the itme that user `i` interacted with, while
-        `predicted_scores_by_user[i, 1:] are items that user `i` did not interact
-        with.
-      top_k: Only consider the highest rated `top_k` items per user. The HR and
-        NDCG for that user will only be nonzero if the predicted score for that
-        user's first item is in the `top_k` top scores.
-      match_mlperf: If True, compute HR and NDCG slightly differently to match the
-        MLPerf reference implementation.
-    Returns:
-      (hr, ndcg) tuple of floats, averaged across all users.
-    """
-    num_users = predicted_scores_by_user.shape[0]
-    zero_indices = np.zeros((num_users, 1), dtype=np.int32)
-
-    if match_mlperf:
-        predicted_scores_by_user = predicted_scores_by_user.copy()
-        items_by_user = items_by_user.copy()
-
-        # For each user, sort the items and predictions by increasing item number.
-        # We use mergesort since it's the only stable sort, which we need to be
-        # equivalent to the MLPerf reference implementation.
-        sorted_items_indices = items_by_user.argsort(kind="mergesort")
-        sorted_items = items_by_user[
-            np.arange(num_users)[:, np.newaxis], sorted_items_indices]
-        sorted_predictions = predicted_scores_by_user[
-            np.arange(num_users)[:, np.newaxis], sorted_items_indices]
-
-        # For items that occur more than once in a user's row, set the predicted
-        # score of the subsequent occurrences to -infinity, which effectively
-        # removes them from the array.
-        diffs = sorted_items[:, :-1] - sorted_items[:, 1:]
-        diffs = np.concatenate(
-            [np.ones((diffs.shape[0], 1), dtype=diffs.dtype), diffs], axis=1)
-        predicted_scores_by_user = np.where(diffs, sorted_predictions, -np.inf)
-
-        # After this block, `zero_indices` will be a (num_users, 1) shaped array
-        # indicating, for each user, the index of item of value 0 in
-        # `sorted_items_indices`. This item is the one we want to check if it is in
-        # the top_k items.
-        zero_indices = np.array(np.where(sorted_items_indices == 0))
-        assert np.array_equal(zero_indices[0, :], np.arange(num_users))
-        zero_indices = zero_indices[1, :, np.newaxis]
-
-    # NumPy has an np.argparition() method, however log(1000) is so small that
-    # sorting the whole array is simpler and fast enough.
-    top_indicies = np.argsort(predicted_scores_by_user, axis=1)[:, -top_k:]
-    top_indicies = np.flip(top_indicies, axis=1)
-
-    # Both HR and NDCG vectorized computation takes advantage of the fact that if
-    # the positive example for a user is not in the top k, that index does not
-    # appear. That is to say:   hit_ind.shape[0] <= num_users
-    hit_ind = np.argwhere(np.equal(top_indicies, zero_indices))
-    hr = hit_ind.shape[0] / num_users
-    ndcg = np.sum(np.log(2) / np.log(hit_ind[:, 1] + 2)) / num_users
-    return hr, ndcg
-
-
-def evaluate_model(estimator, ncf_dataset, pred_input_fn):
-    # type: (tf.estimator.Estimator, prepare.NCFDataset, typing.Callable) -> dict
-    """Model evaluation with HR and NDCG metrics.
-    The evaluation protocol is to rank the test interacted item (truth items)
-    among the randomly chosen 999 items that are not interacted by the user.
-    The performance of the ranked list is judged by Hit Ratio (HR) and Normalized
-    Discounted Cumulative Gain (NDCG).
-    For evaluation, the ranked list is truncated at 10 for both metrics. As such,
-    the HR intuitively measures whether the test item is present on the top-10
-    list, and the NDCG accounts for the position of the hit by assigning higher
-    scores to hits at top ranks. Both metrics are calculated for each test user,
-    and the average scores are reported.
-    Args:
-      estimator: The Estimator.
-      ncf_dataset: An NCFDataSet object, which contains the information about
-        test/eval dataset, such as:
-          num_users: How many unique users are in the eval set.
-          test_data: The points which are used for consistent evaluation. These
-            are already included in the pred_input_fn.
-      pred_input_fn: The input function for the test data.
-    Returns:
-      eval_results: A dict of evaluation results for benchmark logging.
-        eval_results = {
-          _HR_KEY: hr,
-          _NDCG_KEY: ndcg,
-          tf.GraphKeys.GLOBAL_STEP: global_step
-        }
-        where hr is an integer indicating the average HR scores across all users,
-        ndcg is an integer representing the average NDCG scores across all users,
-        and global_step is the global step
-    """
-
-    tf.logging.info("Computing predictions for eval set...")
-
-    # OpenMP settings
-    # os.environ["OMP_NUM_THREADS"] = "11"
-    if not os.environ.get("KMP_BLOCKTIME"):
-        os.environ["KMP_BLOCKTIME"] = "1"
-    if not os.environ.get("KMP_SETTINGS"):
-        os.environ["KMP_SETTINGS"] = "1"
-    if not os.environ.get("KMP_AFFINITY"):
-        os.environ["KMP_AFFINITY"] = "granularity=fine,noverbose,compact,1,0"
-
-    # Get predictions
-    prediction_hooks = None if FLAGS.accuracy_only else [LoggerHook()]
-    predictions = estimator.predict(input_fn=pred_input_fn,
-                                    yield_single_examples=False,
-                                    hooks=prediction_hooks)
-    predictions = list(predictions)
-
-    prediction_batches = [p[movielens.RATING_COLUMN] for p in predictions]
-    item_batches = [p[movielens.ITEM_COLUMN] for p in predictions]
-
-    # Reshape the predicted scores and items. Each user takes one row.
-    prediction_with_padding = np.concatenate(prediction_batches, axis=0)
-    predicted_scores_by_user = prediction_with_padding[
-        :ncf_dataset.num_users * (1 + rconst.NUM_EVAL_NEGATIVES)]\
-        .reshape(ncf_dataset.num_users, -1)
-    item_with_padding = np.concatenate(item_batches, axis=0)
-    items_by_user = item_with_padding[
-        :ncf_dataset.num_users * (1 + rconst.NUM_EVAL_NEGATIVES)]\
-        .reshape(ncf_dataset.num_users, -1)
-
-    tf.logging.info("Computing metrics...")
-
-    hr, ndcg = get_hit_rate_and_ndcg(predicted_scores_by_user, items_by_user,
-                                     match_mlperf=FLAGS.ml_perf)
-
-    global_step = estimator.get_variable_value(tf.GraphKeys.GLOBAL_STEP)
-    eval_results = {
-        _HR_KEY: hr,
-        _NDCG_KEY: ndcg,
-        tf.GraphKeys.GLOBAL_STEP: global_step
-    }
-
-    return eval_results
-
-
-def construct_estimator(num_gpus, model_dir, params, batch_size,
-                        eval_batch_size):
-    """Construct either an Estimator or TPUEstimator for NCF.
-    Args:
-      num_gpus: The number of gpus (Used to select distribution strategy)
-      model_dir: The model directory for the estimator
-      params: The params dict for the estimator
-      batch_size: The mini-batch size for training.
-      eval_batch_size: The batch size used during evaluation.
-    Returns:
-      An Estimator or TPUEstimator.
-    """
-
-    if params["use_tpu"]:
-        tpu_cluster_resolver = tf.contrib.cluster_resolver.TPUClusterResolver(
-            tpu=params["tpu"],
-            zone=params["tpu_zone"],
-            project=params["tpu_gcp_project"],
-        )
-
-        tpu_config = tf.contrib.tpu.TPUConfig(
-            iterations_per_loop=100,
-            num_shards=8)
-
-        run_config = tf.contrib.tpu.RunConfig(
-            cluster=tpu_cluster_resolver,
-            model_dir=model_dir,
-            session_config=tf.ConfigProto(
-                allow_soft_placement=True, log_device_placement=False),
-            tpu_config=tpu_config)
-
-        tpu_params = {k: v for k, v in params.items() if k != "batch_size"}
-
-        train_estimator = tf.contrib.tpu.TPUEstimator(
-            model_fn=neumf_model.neumf_model_fn,
-            use_tpu=True,
-            train_batch_size=batch_size,
-            params=tpu_params,
-            config=run_config)
-
-        eval_estimator = tf.contrib.tpu.TPUEstimator(
-            model_fn=neumf_model.neumf_model_fn,
-            use_tpu=False,
-            train_batch_size=1,
-            predict_batch_size=eval_batch_size,
-            params=tpu_params,
-            config=run_config)
-
-        return train_estimator, eval_estimator
-
-    distribution = distribution_utils.get_distribution_strategy(num_gpus=num_gpus)
-    cpu_session_config = tf.ConfigProto(allow_soft_placement=True, log_device_placement=False,
-                                        inter_op_parallelism_threads=FLAGS.inter_op_parallelism_threads,
-                                        intra_op_parallelism_threads=FLAGS.intra_op_parallelism_threads)
-
-    run_config = tf.estimator.RunConfig(train_distribute=distribution,
-                                        session_config=cpu_session_config)
-    params["eval_batch_size"] = eval_batch_size
-    estimator = tf.estimator.Estimator(model_fn=neumf_model.neumf_model_fn,
-                                       model_dir=model_dir, config=run_config,
-                                       params=params)
-    return estimator, estimator
-
-
-def main(_):
-    with logger.benchmark_context(FLAGS):
-        run_ncf(FLAGS)
-
-
-def serving_input_receiver_fn():
-    inputs = {
-        'user_id': tf.placeholder(dtype=tf.int32, shape=[None], name='user_id'),
-        'item_id': tf.placeholder(dtype=tf.uint16, shape=[None], name='item_id'),
-    }
-    return tf.estimator.export.ServingInputReceiver(inputs, inputs)
-
-
-def run_ncf(_):
-    """Run NCF training and eval loop."""
-    if FLAGS.download_if_missing:
-        movielens.download(FLAGS.dataset, FLAGS.data_dir)
-
-    num_gpus = flags_core.get_num_gpus(FLAGS)
-    batch_size = distribution_utils.per_device_batch_size(
-        int(FLAGS.batch_size), num_gpus)
-    eval_batch_size = int(FLAGS.eval_batch_size or FLAGS.batch_size)
-    ncf_dataset = data_preprocessing.instantiate_pipeline(
-        dataset=FLAGS.dataset, data_dir=FLAGS.data_dir,
-        batch_size=batch_size,
-        eval_batch_size=eval_batch_size,
-        num_neg=FLAGS.num_neg,
-        epochs_per_cycle=FLAGS.epochs_between_evals,
-        match_mlperf=FLAGS.ml_perf)
-
-    model_helpers.apply_clean(flags.FLAGS)
-
-    train_estimator, eval_estimator = construct_estimator(
-        num_gpus=num_gpus, model_dir=FLAGS.model_dir, params={
-            "batch_size": batch_size,
-            "learning_rate": FLAGS.learning_rate,
-            "num_users": ncf_dataset.num_users,
-            "num_items": ncf_dataset.num_items,
-            "mf_dim": FLAGS.num_factors,
-            "model_layers": [int(layer) for layer in FLAGS.layers],
-            "mf_regularization": FLAGS.mf_regularization,
-            "mlp_reg_layers": [float(reg) for reg in FLAGS.mlp_regularization],
-            "use_tpu": FLAGS.tpu is not None,
-            "tpu": FLAGS.tpu,
-            "tpu_zone": FLAGS.tpu_zone,
-            "tpu_gcp_project": FLAGS.tpu_gcp_project,
-        }, batch_size=flags.FLAGS.batch_size, eval_batch_size=eval_batch_size)
-
-    # Create hooks that log information about the training and metric values
-    train_hooks = hooks_helper.get_train_hooks(
-        FLAGS.hooks,
-        model_dir=FLAGS.model_dir,
-        batch_size=FLAGS.batch_size  # for ExamplesPerSecondHook
-    )
-    run_params = {
-        "batch_size": FLAGS.batch_size,
-        "eval_batch_size": eval_batch_size,
-        "number_factors": FLAGS.num_factors,
-        "hr_threshold": FLAGS.hr_threshold,
-        "train_epochs": FLAGS.train_epochs,
-    }
-    benchmark_logger = logger.get_benchmark_logger()
-    benchmark_logger.log_run_info(
-        model_name="recommendation",
-        dataset_name=FLAGS.dataset,
-        run_params=run_params,
-        test_id=FLAGS.benchmark_test_id)
-
-    approx_train_steps = int(ncf_dataset.num_train_positives
-                             * (1 + FLAGS.num_neg) // FLAGS.batch_size)
-    pred_input_fn = data_preprocessing.make_pred_input_fn(ncf_dataset=ncf_dataset)
-
-    total_training_cycle = 1 if FLAGS.inference_only else FLAGS.train_epochs // FLAGS.epochs_between_evals
-    for cycle_index in range(total_training_cycle):
-        tf.logging.info("Starting a training cycle: {}/{}".format(
-            cycle_index + 1, total_training_cycle))
-
-        if not FLAGS.inference_only:
-            # Train the model
-            train_input_fn, train_record_dir, batch_count = \
-                data_preprocessing.make_train_input_fn(ncf_dataset=ncf_dataset)
-
-            if np.abs(approx_train_steps - batch_count) > 1:
-                tf.logging.warning(
-                    "Estimated ({}) and reported ({}) number of batches differ by more "
-                    "than one".format(approx_train_steps, batch_count))
-            train_estimator.train(input_fn=train_input_fn, hooks=train_hooks,
-                                  steps=batch_count)
-            tf.gfile.DeleteRecursively(train_record_dir)
-
-        # Evaluate the model
-        eval_results = evaluate_model(
-            eval_estimator, ncf_dataset, pred_input_fn)
-
-        # Benchmark the evaluation results
-        benchmark_logger.log_evaluation_result(eval_results)
-        # Log the HR and NDCG results.
-        hr = eval_results[_HR_KEY]
-        ndcg = eval_results[_NDCG_KEY]
-        tf.logging.fatal(
-            "Iteration {}: HR = {:.4f}, NDCG = {:.4f}".format(
-                cycle_index + 1, hr, ndcg))
-
-        # Export SavedModel
-        if FLAGS.export_savedmodel:
-            eval_estimator.export_savedmodel(FLAGS.model_dir, serving_input_receiver_fn)
-            print("SavedModel successfully exported to: {}/<timestamp>".format(
-                FLAGS.model_dir))
-
-        # Some of the NumPy vector math can be quite large and likes to stay in
-        # memory for a while.
-        gc.collect()
-
-        # If some evaluation threshold is met
-        if model_helpers.past_stop_threshold(FLAGS.hr_threshold, hr):
-            break
-
-    # Clear the session explicitly to avoid session delete error
-    tf.keras.backend.clear_session()
-
-
-def define_ncf_flags():
-    """Add flags for running ncf_main."""
-    # Add common flags
-    flags_core.define_base(export_dir=False)
-    flags_core.define_performance(
-        num_parallel_calls=False,
-        inter_op=True,
-        intra_op=True,
-        synthetic_data=False,
-        max_train_steps=False,
-        dtype=False,
-        all_reduce_alg=False
-    )
-    flags_core.define_device(tpu=True)
-    flags_core.define_benchmark()
-
-    flags.adopt_module_key_flags(flags_core)
-
-    flags_core.set_defaults(
-        model_dir="/tmp/ncf/",
-        data_dir="/tmp/movielens-data/",
-        train_epochs=2,
-        batch_size=256,
-        hooks=None,
-        tpu=None
-    )
-
-    # Add ncf-specific flags
-    flags.DEFINE_enum(
-        name="dataset", default="ml-1m",
-        enum_values=["ml-1m", "ml-20m"], case_sensitive=False,
-        help=flags_core.help_wrap(
-            "Dataset to be trained and evaluated."))
-
-    flags.DEFINE_boolean(
-        name="download_if_missing", default=True, help=flags_core.help_wrap(
-            "Download data to data_dir if it is not already present."))
-
-    flags.DEFINE_string(
-        name="eval_batch_size", default=None, help=flags_core.help_wrap(
-            "The batch size used for evaluation. This should generally be larger"
-            "than the training batch size as the lack of back propagation during"
-            "evaluation can allow for larger batch sizes to fit in memory. If not"
-            "specified, the training batch size (--batch_size) will be used."))
-
-    flags.DEFINE_integer(
-        name="num_factors", default=8,
-        help=flags_core.help_wrap("The Embedding size of MF model."))
-
-    # Set the default as a list of strings to be consistent with input arguments
-    flags.DEFINE_list(
-        name="layers", default=["64", "32", "16", "8"],
-        help=flags_core.help_wrap(
-            "The sizes of hidden layers for MLP. Example "
-            "to specify different sizes of MLP layers: --layers=32,16,8,4"))
-
-    flags.DEFINE_float(
-        name="mf_regularization", default=0.,
-        help=flags_core.help_wrap(
-            "The regularization factor for MF embeddings. The factor is used by "
-            "regularizer which allows to apply penalties on layer parameters or "
-            "layer activity during optimization."))
-
-    flags.DEFINE_list(
-        name="mlp_regularization", default=["0.", "0.", "0.", "0."],
-        help=flags_core.help_wrap(
-            "The regularization factor for each MLP layer. See mf_regularization "
-            "help for more info about regularization factor."))
-
-    flags.DEFINE_integer(
-        name="num_neg", default=4,
-        help=flags_core.help_wrap(
-            "The Number of negative instances to pair with a positive instance."))
-
-    flags.DEFINE_float(
-        name="learning_rate", default=0.001,
-        help=flags_core.help_wrap("The learning rate."))
-
-    flags.DEFINE_float(
-        name="hr_threshold", default=None,
-        help=flags_core.help_wrap(
-            "If passed, training will stop when the evaluation metric HR is "
-            "greater than or equal to hr_threshold. For dataset ml-1m, the "
-            "desired hr_threshold is 0.68 which is the result from the paper; "
-            "For dataset ml-20m, the threshold can be set as 0.95 which is "
-            "achieved by MLPerf implementation."))
-
-    flags.DEFINE_bool(
-        name="ml_perf", default=None,
-        help=flags_core.help_wrap(
-            "If set, changes the behavior of the model slightly to match the "
-            "MLPerf reference implementations here: \n"
-            "https://github.com/mlperf/reference/tree/master/recommendation/"
-            "pytorch\n"
-            "The two changes are:\n"
-            "1. When computing the HR and NDCG during evaluation, remove "
-            "duplicate user-item pairs before the computation. This results in "
-            "better HRs and NDCGs.\n"
-            "2. Use a different soring algorithm when sorting the input data, "
-            "which performs better due to the fact the sorting algorithms are "
-            "not stable."))
-
-    flags.DEFINE_bool(
-        name="inference_only", default=False,
-        help=flags_core.help_wrap(
-            "If set, runs only the forward pass."))
-
-    flags.DEFINE_bool(
-        name="accuracy_only", default=False,
-        help=flags_core.help_wrap(
-            "If set, only accuracy (i.e. no performance benchmarking) "
-            "metrics are computed."))
-
-    flags.DEFINE_bool(
-        name="benchmark_only", default=True,
-        help=flags_core.help_wrap(
-            "If set, only performance benchmarking (i.e. no accuracy) "
-            "metrics are computed."))
-
-    flags.DEFINE_bool(
-        name="export_savedmodel", default=False,
-        help=flags_core.help_wrap(
-            "If set, the model is exported in serving-compatible format to "
-            "the model_dir."))
-
-
-if __name__ == "__main__":
-    tf.logging.set_verbosity(tf.logging.FATAL)
-    define_ncf_flags()
-    FLAGS = flags.FLAGS
-    absl_app.run(main)
diff --git a/models/recommendation/tensorflow/ncf/training/__init__.py b/models/recommendation/tensorflow/ncf/training/__init__.py
deleted file mode 100644
index d9c4123de..000000000
--- a/models/recommendation/tensorflow/ncf/training/__init__.py
+++ /dev/null
@@ -1,19 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2019 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-# SPDX-License-Identifier: EPL-2.0
-#
diff --git a/models/recommendation/tensorflow/ncf/training/ncf_common.py b/models/recommendation/tensorflow/ncf/training/ncf_common.py
deleted file mode 100644
index c30fe3f10..000000000
--- a/models/recommendation/tensorflow/ncf/training/ncf_common.py
+++ /dev/null
@@ -1,356 +0,0 @@
-# Copyright 2018 The TensorFlow Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-"""Common functionalities used by both Keras and Estimator implementations.
-"""
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-
-import json
-import os
-
-# pylint: disable=g-bad-import-order
-import numpy as np
-from absl import flags
-from absl import logging
-import tensorflow as tf
-# pylint: enable=g-bad-import-order
-
-from official.recommendation import constants as rconst
-from official.recommendation import data_pipeline
-from official.recommendation import data_preprocessing
-from official.recommendation import movielens
-from official.utils.flags import core as flags_core
-from official.utils.misc import distribution_utils
-from official.utils.misc import keras_utils
-
-FLAGS = flags.FLAGS
-
-
-def get_inputs(params):
-  """Returns some parameters used by the model."""
-  if FLAGS.download_if_missing and not FLAGS.use_synthetic_data:
-    movielens.download(FLAGS.dataset, FLAGS.data_dir)
-
-  if FLAGS.seed is not None:
-    np.random.seed(FLAGS.seed)
-
-  if FLAGS.use_synthetic_data:
-    producer = data_pipeline.DummyConstructor()
-    num_users, num_items = data_preprocessing.DATASET_TO_NUM_USERS_AND_ITEMS[
-        FLAGS.dataset]
-    num_train_steps = rconst.SYNTHETIC_BATCHES_PER_EPOCH
-    num_eval_steps = rconst.SYNTHETIC_BATCHES_PER_EPOCH
-  else:
-    num_users, num_items, producer = data_preprocessing.instantiate_pipeline(
-        dataset=FLAGS.dataset, data_dir=FLAGS.data_dir, params=params,
-        constructor_type=FLAGS.constructor_type,
-        deterministic=FLAGS.seed is not None)
-    num_train_steps = producer.train_batches_per_epoch
-    num_eval_steps = producer.eval_batches_per_epoch
-
-  return num_users, num_items, num_train_steps, num_eval_steps, producer
-
-
-def parse_flags(flags_obj):
-  """Convenience function to turn flags into params."""
-  num_gpus = flags_core.get_num_gpus(flags_obj)
-
-  batch_size = flags_obj.batch_size
-  eval_batch_size = flags_obj.eval_batch_size or flags_obj.batch_size
-
-  return {
-      "train_epochs": flags_obj.train_epochs,
-      "batches_per_step": 1,
-      "use_seed": flags_obj.seed is not None,
-      "batch_size": batch_size,
-      "eval_batch_size": eval_batch_size,
-      "learning_rate": flags_obj.learning_rate,
-      "mf_dim": flags_obj.num_factors,
-      "model_layers": [int(layer) for layer in flags_obj.layers],
-      "mf_regularization": flags_obj.mf_regularization,
-      "mlp_reg_layers": [float(reg) for reg in flags_obj.mlp_regularization],
-      "num_neg": flags_obj.num_neg,
-      "distribution_strategy": flags_obj.distribution_strategy,
-      "num_gpus": num_gpus,
-      "use_tpu": flags_obj.tpu is not None,
-      "tpu": flags_obj.tpu,
-      "tpu_zone": flags_obj.tpu_zone,
-      "tpu_gcp_project": flags_obj.tpu_gcp_project,
-      "beta1": flags_obj.beta1,
-      "beta2": flags_obj.beta2,
-      "epsilon": flags_obj.epsilon,
-      "match_mlperf": flags_obj.ml_perf,
-      "use_xla_for_gpu": flags_obj.use_xla_for_gpu,
-      "epochs_between_evals": FLAGS.epochs_between_evals,
-      "keras_use_ctl": flags_obj.keras_use_ctl,
-      "hr_threshold": flags_obj.hr_threshold,
-      "stream_files": flags_obj.tpu is not None,
-      "train_dataset_path": flags_obj.train_dataset_path,
-      "eval_dataset_path": flags_obj.eval_dataset_path,
-      "input_meta_data_path": flags_obj.input_meta_data_path,
-      "use_bfloat16" : flags_obj.use_bfloat16,
-  }
-
-
-def get_v1_distribution_strategy(params):
-  """Returns the distribution strategy to use."""
-  if params["use_tpu"]:
-    # Some of the networking libraries are quite chatty.
-    for name in ["googleapiclient.discovery", "googleapiclient.discovery_cache",
-                 "oauth2client.transport"]:
-      logging.getLogger(name).setLevel(logging.ERROR)
-
-    tpu_cluster_resolver = tf.distribute.cluster_resolver.TPUClusterResolver(
-        tpu=params["tpu"],
-        zone=params["tpu_zone"],
-        project=params["tpu_gcp_project"],
-        coordinator_name="coordinator"
-    )
-
-    logging.info("Issuing reset command to TPU to ensure a clean state.")
-    tf.Session.reset(tpu_cluster_resolver.get_master())
-
-    # Estimator looks at the master it connects to for MonitoredTrainingSession
-    # by reading the `TF_CONFIG` environment variable, and the coordinator
-    # is used by StreamingFilesDataset.
-    tf_config_env = {
-        "session_master": tpu_cluster_resolver.get_master(),
-        "eval_session_master": tpu_cluster_resolver.get_master(),
-        "coordinator": tpu_cluster_resolver.cluster_spec()
-                       .as_dict()["coordinator"]
-    }
-    os.environ["TF_CONFIG"] = json.dumps(tf_config_env)
-
-    distribution = tf.distribute.experimental.TPUStrategy(
-        tpu_cluster_resolver, steps_per_run=100)
-
-  else:
-    distribution = distribution_utils.get_distribution_strategy(
-        num_gpus=params["num_gpus"])
-
-  return distribution
-
-
-def define_ncf_flags():
-  """Add flags for running ncf_main."""
-  # Add common flags
-  flags_core.define_base(clean=True, train_epochs=True,
-                         epochs_between_evals=True, export_dir=False,
-                         run_eagerly=True, stop_threshold=True, num_gpu=True,
-                         hooks=True, distribution_strategy=True)
-  flags_core.define_performance(
-      synthetic_data=True,
-      dtype=True,
-      fp16_implementation=True,
-      loss_scale=True,
-      dynamic_loss_scale=True,
-      enable_xla=False,
-      force_v2_in_keras_compile=True
-  )
-  flags_core.define_device(tpu=True)
-  flags_core.define_benchmark()
-
-  flags.adopt_module_key_flags(flags_core)
-
-  flags_core.set_defaults(
-      model_dir="/tmp/ncf/",
-      data_dir="/tmp/movielens-data/",
-      train_epochs=2,
-      batch_size=256,
-      hooks="ProfilerHook",
-      tpu=None
-  )
-
-  # Add ncf-specific flags
-  flags.DEFINE_enum(
-      name="dataset", default="ml-1m",
-      enum_values=["ml-1m", "ml-20m"], case_sensitive=False,
-      help=flags_core.help_wrap(
-          "Dataset to be trained and evaluated."))
-
-  flags.DEFINE_boolean(
-      name="download_if_missing", default=True, help=flags_core.help_wrap(
-          "Download data to data_dir if it is not already present."))
-
-  flags.DEFINE_integer(
-      name="eval_batch_size", default=None, help=flags_core.help_wrap(
-          "The batch size used for evaluation. This should generally be larger"
-          "than the training batch size as the lack of back propagation during"
-          "evaluation can allow for larger batch sizes to fit in memory. If not"
-          "specified, the training batch size (--batch_size) will be used."))
-
-  flags.DEFINE_integer(
-      name="num_factors", default=8,
-      help=flags_core.help_wrap("The Embedding size of MF model."))
-
-  # Set the default as a list of strings to be consistent with input arguments
-  flags.DEFINE_list(
-      name="layers", default=["64", "32", "16", "8"],
-      help=flags_core.help_wrap(
-          "The sizes of hidden layers for MLP. Example "
-          "to specify different sizes of MLP layers: --layers=32,16,8,4"))
-
-  flags.DEFINE_float(
-      name="mf_regularization", default=0.,
-      help=flags_core.help_wrap(
-          "The regularization factor for MF embeddings. The factor is used by "
-          "regularizer which allows to apply penalties on layer parameters or "
-          "layer activity during optimization."))
-
-  flags.DEFINE_list(
-      name="mlp_regularization", default=["0.", "0.", "0.", "0."],
-      help=flags_core.help_wrap(
-          "The regularization factor for each MLP layer. See mf_regularization "
-          "help for more info about regularization factor."))
-
-  flags.DEFINE_integer(
-      name="num_neg", default=4,
-      help=flags_core.help_wrap(
-          "The Number of negative instances to pair with a positive instance."))
-
-  flags.DEFINE_float(
-      name="learning_rate", default=0.001,
-      help=flags_core.help_wrap("The learning rate."))
-
-  flags.DEFINE_float(
-      name="beta1", default=0.9,
-      help=flags_core.help_wrap("beta1 hyperparameter for the Adam optimizer."))
-
-  flags.DEFINE_float(
-      name="beta2", default=0.999,
-      help=flags_core.help_wrap("beta2 hyperparameter for the Adam optimizer."))
-
-  flags.DEFINE_float(
-      name="epsilon", default=1e-8,
-      help=flags_core.help_wrap("epsilon hyperparameter for the Adam "
-                                "optimizer."))
-
-  flags.DEFINE_float(
-      name="hr_threshold", default=1.0,
-      help=flags_core.help_wrap(
-          "If passed, training will stop when the evaluation metric HR is "
-          "greater than or equal to hr_threshold. For dataset ml-1m, the "
-          "desired hr_threshold is 0.68 which is the result from the paper; "
-          "For dataset ml-20m, the threshold can be set as 0.95 which is "
-          "achieved by MLPerf implementation."))
-
-  flags.DEFINE_enum(
-      name="constructor_type", default="bisection",
-      enum_values=["bisection", "materialized"], case_sensitive=False,
-      help=flags_core.help_wrap(
-          "Strategy to use for generating false negatives. materialized has a"
-          "precompute that scales badly, but a faster per-epoch construction"
-          "time and can be faster on very large systems."))
-
-  flags.DEFINE_string(
-      name="train_dataset_path",
-      default=None,
-      help=flags_core.help_wrap("Path to training data."))
-
-  flags.DEFINE_string(
-      name="eval_dataset_path",
-      default=None,
-      help=flags_core.help_wrap("Path to evaluation data."))
-
-  flags.DEFINE_string(
-      name="input_meta_data_path",
-      default=None,
-      help=flags_core.help_wrap("Path to input meta data file."))
-
-  flags.DEFINE_bool(
-      name="ml_perf", default=False,
-      help=flags_core.help_wrap(
-          "If set, changes the behavior of the model slightly to match the "
-          "MLPerf reference implementations here: \n"
-          "https://github.com/mlperf/reference/tree/master/recommendation/"
-          "pytorch\n"
-          "The two changes are:\n"
-          "1. When computing the HR and NDCG during evaluation, remove "
-          "duplicate user-item pairs before the computation. This results in "
-          "better HRs and NDCGs.\n"
-          "2. Use a different soring algorithm when sorting the input data, "
-          "which performs better due to the fact the sorting algorithms are "
-          "not stable."))
-
-  flags.DEFINE_bool(
-      name="output_ml_perf_compliance_logging", default=False,
-      help=flags_core.help_wrap(
-          "If set, output the MLPerf compliance logging. This is only useful "
-          "if one is running the model for MLPerf. See "
-          "https://github.com/mlperf/policies/blob/master/training_rules.adoc"
-          "#submission-compliance-logs for details. This uses sudo and so may "
-          "ask for your password, as root access is needed to clear the system "
-          "caches, which is required for MLPerf compliance."
-      )
-  )
-
-  flags.DEFINE_integer(
-      name="seed", default=None, help=flags_core.help_wrap(
-          "This value will be used to seed both NumPy and TensorFlow."))
-
-  @flags.validator("eval_batch_size", "eval_batch_size must be at least {}"
-                   .format(rconst.NUM_EVAL_NEGATIVES + 1))
-  def eval_size_check(eval_batch_size):
-    return (eval_batch_size is None or
-            int(eval_batch_size) > rconst.NUM_EVAL_NEGATIVES)
-
-  flags.DEFINE_bool(
-      name="use_xla_for_gpu", default=False, help=flags_core.help_wrap(
-          "If True, use XLA for the model function. Only works when using a "
-          "GPU. On TPUs, XLA is always used"))
-
-  xla_message = "--use_xla_for_gpu is incompatible with --tpu"
-  @flags.multi_flags_validator(["use_xla_for_gpu", "tpu"], message=xla_message)
-  def xla_validator(flag_dict):
-    return not flag_dict["use_xla_for_gpu"] or not flag_dict["tpu"]
-
-  flags.DEFINE_bool(
-      name="early_stopping",
-      default=False,
-      help=flags_core.help_wrap(
-          "If True, we stop the training when it reaches hr_threshold"))
-
-  flags.DEFINE_bool(
-      name="keras_use_ctl",
-      default=False,
-      help=flags_core.help_wrap(
-          "If True, we use a custom training loop for keras."))
-
-  flags.DEFINE_bool(
-      name="profiling",
-      default=False,
-      help=flags_core.help_wrap(
-          "If True, enable timeline profiling."))
-
-  flags.DEFINE_bool(
-      name="use_bfloat16",
-      default=False,
-      help=flags_core.help_wrap(
-          "If True, use bfloat16 as data type for model."))
-
-
-def convert_to_softmax_logits(logits):
-  """Convert the logits returned by the base model to softmax logits.
-
-  Args:
-    logits: used to create softmax.
-
-  Returns:
-    Softmax with the first column of zeros is equivalent to sigmoid.
-  """
-  softmax_logits = tf.concat([logits * 0, logits], axis=1)
-  return softmax_logits
diff --git a/models/recommendation/tensorflow/ncf/training/ncf_estimator_main.py b/models/recommendation/tensorflow/ncf/training/ncf_estimator_main.py
deleted file mode 100644
index 676cfa8da..000000000
--- a/models/recommendation/tensorflow/ncf/training/ncf_estimator_main.py
+++ /dev/null
@@ -1,199 +0,0 @@
-# Copyright 2018 The TensorFlow Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-"""NCF framework to train and evaluate the NeuMF model.
-
-The NeuMF model assembles both MF and MLP models under the NCF framework. Check
-`neumf_model.py` for more details about the models.
-"""
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-
-import contextlib
-import heapq
-import json
-import math
-import multiprocessing
-import os
-import signal
-import typing
-
-# pylint: disable=g-bad-import-order
-import numpy as np
-from absl import app as absl_app
-from absl import flags
-from absl import logging
-import tensorflow as tf
-# pylint: enable=g-bad-import-order
-
-from official.recommendation import constants as rconst
-from official.recommendation import data_pipeline
-from official.recommendation import data_preprocessing
-from official.recommendation import movielens
-from official.utils.flags import core as flags_core
-from official.utils.logs import hooks_helper
-from official.utils.logs import logger
-from official.utils.logs import mlperf_helper
-from official.utils.misc import distribution_utils
-from official.utils.misc import model_helpers
-from training import ncf_common
-from training import neumf_model
-
-
-FLAGS = flags.FLAGS
-
-
-def construct_estimator(model_dir, params):
-  """Construct either an Estimator or TPUEstimator for NCF.
-
-  Args:
-    model_dir: The model directory for the estimator
-    params: The params dict for the estimator
-
-  Returns:
-    An Estimator or TPUEstimator.
-  """
-  distribution = ncf_common.get_v1_distribution_strategy(params)
-  run_config = tf.estimator.RunConfig(train_distribute=distribution,
-                                      eval_distribute=distribution)
-
-  model_fn = neumf_model.neumf_model_fn
-  if params["use_xla_for_gpu"]:
-    # TODO(seemuch): remove the contrib imput
-    from tensorflow.contrib.compiler import xla
-    logging.info("Using XLA for GPU for training and evaluation.")
-    model_fn = xla.estimator_model_fn(model_fn)
-  estimator = tf.estimator.Estimator(model_fn=model_fn, model_dir=model_dir,
-                                     config=run_config, params=params)
-  return estimator
-
-
-def log_and_get_hooks(eval_batch_size):
-  """Convenience function for hook and logger creation."""
-  # Create hooks that log information about the training and metric values
-  train_hooks = hooks_helper.get_train_hooks(
-      FLAGS.hooks,
-      model_dir=FLAGS.model_dir,
-      batch_size=FLAGS.batch_size,  # for ExamplesPerSecondHook
-      tensors_to_log={"cross_entropy": "cross_entropy"}
-  )
-  run_params = {
-      "batch_size": FLAGS.batch_size,
-      "eval_batch_size": eval_batch_size,
-      "number_factors": FLAGS.num_factors,
-      "hr_threshold": FLAGS.hr_threshold,
-      "train_epochs": FLAGS.train_epochs,
-  }
-  benchmark_logger = logger.get_benchmark_logger()
-  benchmark_logger.log_run_info(
-      model_name="recommendation",
-      dataset_name=FLAGS.dataset,
-      run_params=run_params,
-      test_id=FLAGS.benchmark_test_id)
-
-  return benchmark_logger, train_hooks
-
-
-def main(_):
-  with logger.benchmark_context(FLAGS), \
-       mlperf_helper.LOGGER(FLAGS.output_ml_perf_compliance_logging):
-    mlperf_helper.set_ncf_root(os.path.split(os.path.abspath(__file__))[0])
-    run_ncf(FLAGS)
-
-
-def run_ncf(_):
-  """Run NCF training and eval loop."""
-  params = ncf_common.parse_flags(FLAGS)
-
-  num_users, num_items, num_train_steps, num_eval_steps, producer = (
-      ncf_common.get_inputs(params))
-
-  params["num_users"], params["num_items"] = num_users, num_items
-  producer.start()
-  model_helpers.apply_clean(flags.FLAGS)
-
-  estimator = construct_estimator(model_dir=FLAGS.model_dir, params=params)
-
-  benchmark_logger, train_hooks = log_and_get_hooks(params["eval_batch_size"])
-  total_training_cycle = FLAGS.train_epochs // FLAGS.epochs_between_evals
-
-  target_reached = False
-  mlperf_helper.ncf_print(key=mlperf_helper.TAGS.TRAIN_LOOP)
-  for cycle_index in range(total_training_cycle):
-    assert FLAGS.epochs_between_evals == 1 or not mlperf_helper.LOGGER.enabled
-    logging.info("Starting a training cycle: {}/{}".format(
-        cycle_index + 1, total_training_cycle))
-
-    mlperf_helper.ncf_print(key=mlperf_helper.TAGS.TRAIN_EPOCH,
-                            value=cycle_index)
-
-    if FLAGS.profiling:
-      profile_hook = tf.estimator.ProfilerHook(save_steps=500, output_dir="./timeline")
-      train_hooks.append(profile_hook)
-
-    train_input_fn = producer.make_input_fn(is_training=True)
-    estimator.train(input_fn=train_input_fn, hooks=train_hooks,
-                    steps=num_train_steps)
-
-    logging.info("Beginning evaluation.")
-    eval_input_fn = producer.make_input_fn(is_training=False)
-
-    mlperf_helper.ncf_print(key=mlperf_helper.TAGS.EVAL_START,
-                            value=cycle_index)
-    eval_results = estimator.evaluate(eval_input_fn, steps=num_eval_steps)
-    logging.info("Evaluation complete.")
-
-    hr = float(eval_results[rconst.HR_KEY])
-    ndcg = float(eval_results[rconst.NDCG_KEY])
-    loss = float(eval_results["loss"])
-
-    mlperf_helper.ncf_print(
-        key=mlperf_helper.TAGS.EVAL_TARGET,
-        value={"epoch": cycle_index, "value": FLAGS.hr_threshold})
-    mlperf_helper.ncf_print(key=mlperf_helper.TAGS.EVAL_ACCURACY,
-                            value={"epoch": cycle_index, "value": hr})
-    mlperf_helper.ncf_print(
-        key=mlperf_helper.TAGS.EVAL_HP_NUM_NEG,
-        value={"epoch": cycle_index, "value": rconst.NUM_EVAL_NEGATIVES})
-
-    mlperf_helper.ncf_print(key=mlperf_helper.TAGS.EVAL_STOP, value=cycle_index)
-
-    # Benchmark the evaluation results
-    benchmark_logger.log_evaluation_result(eval_results)
-    # Log the HR and NDCG results.
-    logging.info(
-        "Iteration {}: HR = {:.4f}, NDCG = {:.4f}, Loss = {:.4f}".format(
-            cycle_index + 1, hr, ndcg, loss))
-
-    # If some evaluation threshold is met
-    if model_helpers.past_stop_threshold(FLAGS.hr_threshold, hr):
-      target_reached = True
-      break
-
-  mlperf_helper.ncf_print(key=mlperf_helper.TAGS.RUN_STOP,
-                          value={"success": target_reached})
-  producer.stop_loop()
-  producer.join()
-
-  # Clear the session explicitly to avoid session delete error
-  tf.keras.backend.clear_session()
-  mlperf_helper.ncf_print(key=mlperf_helper.TAGS.RUN_FINAL)
-
-
-if __name__ == "__main__":
-  logging.set_verbosity(logging.INFO)
-  ncf_common.define_ncf_flags()
-  absl_app.run(main)
diff --git a/models/recommendation/tensorflow/ncf/training/neumf_model.py b/models/recommendation/tensorflow/ncf/training/neumf_model.py
deleted file mode 100644
index b8ed4b6ba..000000000
--- a/models/recommendation/tensorflow/ncf/training/neumf_model.py
+++ /dev/null
@@ -1,513 +0,0 @@
-# Copyright 2018 The TensorFlow Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-"""Defines NeuMF model for NCF framework.
-
-Some abbreviations used in the code base:
-NeuMF: Neural Matrix Factorization
-NCF: Neural Collaborative Filtering
-GMF: Generalized Matrix Factorization
-MLP: Multi-Layer Perceptron
-
-GMF applies a linear kernel to model the latent feature interactions, and MLP
-uses a nonlinear kernel to learn the interaction function from data. NeuMF model
-is a fused model of GMF and MLP to better model the complex user-item
-interactions, and unifies the strengths of linearity of MF and non-linearity of
-MLP for modeling the user-item latent structures.
-
-In NeuMF model, it allows GMF and MLP to learn separate embeddings, and combine
-the two models by concatenating their last hidden layer.
-"""
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-
-import sys
-
-from six.moves import xrange  # pylint: disable=redefined-builtin
-import tensorflow as tf
-
-from official.recommendation import constants as rconst
-from official.recommendation import movielens
-from official.recommendation import stat_utils
-from official.utils.logs import mlperf_helper
-from training import ncf_common
-
-
-def sparse_to_dense_grads(grads_and_vars):
-  """Convert sparse gradients to dense gradients.
-
-  All sparse gradients, which are represented as instances of tf.IndexedSlices,
-  are converted to dense Tensors. Dense gradients, which are represents as
-  Tensors, are unchanged.
-
-  The purpose of this conversion is that for small embeddings, which are used by
-  this model, applying dense gradients with the AdamOptimizer is faster than
-  applying sparse gradients.
-
-  Args
-    grads_and_vars: A list of (gradient, variable) tuples. Each gradient can
-      be a Tensor or an IndexedSlices. Tensors are unchanged, and IndexedSlices
-      are converted to dense Tensors.
-  Returns:
-    The same list of (gradient, variable) as `grads_and_vars`, except each
-    IndexedSlices gradient is converted to a Tensor.
-  """
-
-  # Calling convert_to_tensor changes IndexedSlices into Tensors, and leaves
-  # Tensors unchanged.
-  return [(tf.convert_to_tensor(g), v) for g, v in grads_and_vars]
-
-
-def neumf_model_fn(features, labels, mode, params):
-  """Model Function for NeuMF estimator."""
-  if params.get("use_seed"):
-    tf.set_random_seed(stat_utils.random_int32())
-
-  users = features[movielens.USER_COLUMN]
-  items = features[movielens.ITEM_COLUMN]
-
-  user_input = tf.keras.layers.Input(tensor=users)
-  item_input = tf.keras.layers.Input(tensor=items)
-  logits = construct_model(user_input, item_input, params).output
-
-  # Softmax with the first column of zeros is equivalent to sigmoid.
-  softmax_logits = ncf_common.convert_to_softmax_logits(logits)
-
-  if mode == tf.estimator.ModeKeys.EVAL:
-    duplicate_mask = tf.cast(features[rconst.DUPLICATE_MASK], tf.float32)
-    return _get_estimator_spec_with_metrics(
-        logits,
-        softmax_logits,
-        duplicate_mask,
-        params["num_neg"],
-        params["match_mlperf"],
-        use_tpu_spec=params["use_xla_for_gpu"])
-
-  elif mode == tf.estimator.ModeKeys.TRAIN:
-    labels = tf.cast(labels, tf.int32)
-    valid_pt_mask = features[rconst.VALID_POINT_MASK]
-
-    mlperf_helper.ncf_print(key=mlperf_helper.TAGS.OPT_NAME, value="adam")
-    mlperf_helper.ncf_print(key=mlperf_helper.TAGS.OPT_LR,
-                            value=params["learning_rate"])
-    mlperf_helper.ncf_print(key=mlperf_helper.TAGS.OPT_HP_ADAM_BETA1,
-                            value=params["beta1"])
-    mlperf_helper.ncf_print(key=mlperf_helper.TAGS.OPT_HP_ADAM_BETA2,
-                            value=params["beta2"])
-    mlperf_helper.ncf_print(key=mlperf_helper.TAGS.OPT_HP_ADAM_EPSILON,
-                            value=params["epsilon"])
-
-    optimizer = tf.compat.v1.train.AdamOptimizer(
-        learning_rate=params["learning_rate"],
-        beta1=params["beta1"],
-        beta2=params["beta2"],
-        epsilon=params["epsilon"])
-    if params["use_tpu"]:
-      # TODO(seemuch): remove this contrib import
-      optimizer = tf.contrib.tpu.CrossShardOptimizer(optimizer)
-
-    mlperf_helper.ncf_print(key=mlperf_helper.TAGS.MODEL_HP_LOSS_FN,
-                            value=mlperf_helper.TAGS.BCE)
-
-    loss = tf.compat.v1.losses.sparse_softmax_cross_entropy(
-        labels=labels,
-        logits=softmax_logits,
-        weights=tf.cast(valid_pt_mask, tf.float32)
-    )
-
-    # This tensor is used by logging hooks.
-    tf.identity(loss, name="cross_entropy")
-
-    global_step = tf.compat.v1.train.get_global_step()
-    tvars = tf.compat.v1.trainable_variables()
-    gradients = optimizer.compute_gradients(
-        loss, tvars, colocate_gradients_with_ops=True)
-    gradients = sparse_to_dense_grads(gradients)
-    minimize_op = optimizer.apply_gradients(
-        gradients, global_step=global_step, name="train")
-    update_ops = tf.compat.v1.get_collection(tf.compat.v1.GraphKeys.UPDATE_OPS)
-    train_op = tf.group(minimize_op, update_ops)
-
-    return tf.estimator.EstimatorSpec(mode=mode, loss=loss, train_op=train_op)
-
-  else:
-    raise NotImplementedError
-
-
-def _strip_first_and_last_dimension(x, batch_size):
-  return tf.reshape(x[0, :], (batch_size,))
-
-
-def construct_model(user_input, item_input, params):
-  # type: (tf.Tensor, tf.Tensor, dict) -> tf.keras.Model
-  """Initialize NeuMF model.
-
-  Args:
-    user_input: keras input layer for users
-    item_input: keras input layer for items
-    params: Dict of hyperparameters.
-  Raises:
-    ValueError: if the first model layer is not even.
-  Returns:
-    model:  a keras Model for computing the logits
-  """
-  num_users = params["num_users"]
-  num_items = params["num_items"]
-
-  model_layers = params["model_layers"]
-
-  mf_regularization = params["mf_regularization"]
-  mlp_reg_layers = params["mlp_reg_layers"]
-
-  mf_dim = params["mf_dim"]
-
-  mlperf_helper.ncf_print(key=mlperf_helper.TAGS.MODEL_HP_MF_DIM, value=mf_dim)
-  mlperf_helper.ncf_print(key=mlperf_helper.TAGS.MODEL_HP_MLP_LAYER_SIZES,
-                          value=model_layers)
-
-  if model_layers[0] % 2 != 0:
-    raise ValueError("The first layer size should be multiple of 2!")
-
-  # Initializer for embedding layers
-  embedding_initializer = "glorot_uniform"
-
-  def mf_slice_fn(x):
-    x = tf.squeeze(x, [1])
-    return x[:, :mf_dim]
-
-  def mlp_slice_fn(x):
-    x = tf.squeeze(x, [1])
-    return x[:, mf_dim:]
-
-  if params["use_tpu"]:
-    # It turns out to be significantly more effecient to store the MF and MLP
-    # embedding portions in the same table, and then slice as needed.
-    embedding_user = tf.keras.layers.Embedding(
-        num_users,
-        mf_dim + model_layers[0] // 2,
-        embeddings_initializer=embedding_initializer,
-        embeddings_regularizer=tf.keras.regularizers.l2(mf_regularization),
-        input_length=1,
-        name="embedding_user")(
-            user_input)
-
-    embedding_item = tf.keras.layers.Embedding(
-        num_items,
-        mf_dim + model_layers[0] // 2,
-        embeddings_initializer=embedding_initializer,
-        embeddings_regularizer=tf.keras.regularizers.l2(mf_regularization),
-        input_length=1,
-        name="embedding_item")(
-            item_input)
-
-    # GMF part
-    mf_user_latent = tf.keras.layers.Lambda(
-        mf_slice_fn, name="embedding_user_mf")(embedding_user)
-    mf_item_latent = tf.keras.layers.Lambda(
-        mf_slice_fn, name="embedding_item_mf")(embedding_item)
-
-    # MLP part
-    mlp_user_latent = tf.keras.layers.Lambda(
-        mlp_slice_fn, name="embedding_user_mlp")(embedding_user)
-    mlp_item_latent = tf.keras.layers.Lambda(
-        mlp_slice_fn, name="embedding_item_mlp")(embedding_item)
-  else:
-    # Embedding layers of GMF and MLP
-    mf_embedding_user = tf.keras.layers.Embedding(
-        num_users,
-        mf_dim,
-        embeddings_initializer=embedding_initializer,
-        embeddings_regularizer=tf.keras.regularizers.l2(mf_regularization),
-        input_length=1)(user_input)
-    mf_embedding_item = tf.keras.layers.Embedding(
-        num_items,
-        mf_dim,
-        embeddings_initializer=embedding_initializer,
-        embeddings_regularizer=tf.keras.regularizers.l2(mf_regularization),
-        input_length=1)(item_input)
-
-    mlp_embedding_user = tf.keras.layers.Embedding(
-        num_users,
-        model_layers[0]//2,
-        embeddings_initializer=embedding_initializer,
-        embeddings_regularizer=tf.keras.regularizers.l2(mlp_reg_layers[0]),
-        input_length=1)(user_input)
-    mlp_embedding_item = tf.keras.layers.Embedding(
-        num_items,
-        model_layers[0]//2,
-        embeddings_initializer=embedding_initializer,
-        embeddings_regularizer=tf.keras.regularizers.l2(mlp_reg_layers[0]),
-        input_length=1)(item_input)
-
-    # GMF part
-    mf_user_latent = tf.keras.layers.Lambda(
-        mf_slice_fn, name="embedding_user_mf")(mf_embedding_user)
-    mf_item_latent = tf.keras.layers.Lambda(
-        mf_slice_fn, name="embedding_item_mf")(mf_embedding_item)
-
-    # MLP part
-    mlp_user_latent = tf.keras.layers.Lambda(
-        mlp_slice_fn, name="embedding_user_mlp")(mlp_embedding_user)
-    mlp_item_latent = tf.keras.layers.Lambda(
-        mlp_slice_fn, name="embedding_item_mlp")(mlp_embedding_item)
-
-  policy = None
-  if params["use_bfloat16"]:
-    policy = tf.keras.mixed_precision.experimental.Policy('mixed_bfloat16')
-    mf_user_latent = tf.cast(mf_user_latent, tf.bfloat16)
-    mf_item_latent = tf.cast(mf_item_latent, tf.bfloat16)
-    mlp_user_latent = tf.cast(mlp_user_latent, tf.bfloat16)
-    mlp_item_latent = tf.cast(mlp_item_latent, tf.bfloat16)
-
-  # Element-wise multiply
-  mf_vector = tf.keras.layers.multiply([mf_user_latent, mf_item_latent])
-
-  # Concatenation of two latent features
-  mlp_vector = tf.keras.layers.concatenate([mlp_user_latent, mlp_item_latent])
-
-  num_layer = len(model_layers)  # Number of layers in the MLP
-  for layer in xrange(1, num_layer):
-    model_layer = tf.keras.layers.Dense(
-        model_layers[layer],
-        kernel_regularizer=tf.keras.regularizers.l2(mlp_reg_layers[layer]),
-        dtype=policy,
-        activation="relu")
-    mlp_vector = model_layer(mlp_vector)
-
-  # Concatenate GMF and MLP parts
-  predict_vector = tf.keras.layers.concatenate([mf_vector, mlp_vector])
-
-  # Final prediction layer
-  logits = tf.keras.layers.Dense(
-      1, activation=None, kernel_initializer="lecun_uniform",
-      dtype="float",
-      name=movielens.RATING_COLUMN)(predict_vector)
-
-  # Print model topology.
-  model = tf.keras.models.Model([user_input, item_input], logits)
-  model.summary()
-  sys.stdout.flush()
-
-  return model
-
-
-def _get_estimator_spec_with_metrics(logits,              # type: tf.Tensor
-                                     softmax_logits,      # type: tf.Tensor
-                                     duplicate_mask,      # type: tf.Tensor
-                                     num_training_neg,    # type: int
-                                     match_mlperf=False,  # type: bool
-                                     use_tpu_spec=False   # type: bool
-                                    ):
-  """Returns a EstimatorSpec that includes the metrics."""
-  cross_entropy, \
-  metric_fn, \
-  in_top_k, \
-  ndcg, \
-  metric_weights = compute_eval_loss_and_metrics_helper(
-      logits,
-      softmax_logits,
-      duplicate_mask,
-      num_training_neg,
-      match_mlperf,
-      use_tpu_spec)
-
-  if use_tpu_spec:
-    return tf.contrib.tpu.TPUEstimatorSpec(
-        mode=tf.estimator.ModeKeys.EVAL,
-        loss=cross_entropy,
-        eval_metrics=(metric_fn, [in_top_k, ndcg, metric_weights]))
-
-  return tf.estimator.EstimatorSpec(
-      mode=tf.estimator.ModeKeys.EVAL,
-      loss=cross_entropy,
-      eval_metric_ops=metric_fn(in_top_k, ndcg, metric_weights)
-  )
-
-
-def compute_eval_loss_and_metrics_helper(logits,              # type: tf.Tensor
-                                         softmax_logits,      # type: tf.Tensor
-                                         duplicate_mask,      # type: tf.Tensor
-                                         num_training_neg,    # type: int
-                                         match_mlperf=False,  # type: bool
-                                         use_tpu_spec=False   # type: bool
-                                        ):
-  """Model evaluation with HR and NDCG metrics.
-
-  The evaluation protocol is to rank the test interacted item (truth items)
-  among the randomly chosen 999 items that are not interacted by the user.
-  The performance of the ranked list is judged by Hit Ratio (HR) and Normalized
-  Discounted Cumulative Gain (NDCG).
-
-  For evaluation, the ranked list is truncated at 10 for both metrics. As such,
-  the HR intuitively measures whether the test item is present on the top-10
-  list, and the NDCG accounts for the position of the hit by assigning higher
-  scores to hits at top ranks. Both metrics are calculated for each test user,
-  and the average scores are reported.
-
-  If `match_mlperf` is True, then the HR and NDCG computations are done in a
-  slightly unusual way to match the MLPerf reference implementation.
-  Specifically, if the evaluation negatives contain duplicate items, it will be
-  treated as if the item only appeared once. Effectively, for duplicate items in
-  a row, the predicted score for all but one of the items will be set to
-  -infinity
-
-  For example, suppose we have that following inputs:
-  logits_by_user:     [[ 2,  3,  3],
-                       [ 5,  4,  4]]
-
-  items_by_user:     [[10, 20, 20],
-                      [30, 40, 40]]
-
-  # Note: items_by_user is not explicitly present. Instead the relevant \
-          information is contained within `duplicate_mask`
-
-  top_k: 2
-
-  Then with match_mlperf=True, the HR would be 2/2 = 1.0. With
-  match_mlperf=False, the HR would be 1/2 = 0.5. This is because each user has
-  predicted scores for only 2 unique items: 10 and 20 for the first user, and 30
-  and 40 for the second. Therefore, with match_mlperf=True, it's guaranteed the
-  first item's score is in the top 2. With match_mlperf=False, this function
-  would compute the first user's first item is not in the top 2, because item 20
-  has a higher score, and item 20 occurs twice.
-
-  Args:
-    logits: A tensor containing the predicted logits for each user. The shape
-      of logits is (num_users_per_batch * (1 + NUM_EVAL_NEGATIVES),) Logits
-      for a user are grouped, and the last element of the group is the true
-      element.
-
-    softmax_logits: The same tensor, but with zeros left-appended.
-
-    duplicate_mask: A vector with the same shape as logits, with a value of 1
-      if the item corresponding to the logit at that position has already
-      appeared for that user.
-
-    num_training_neg: The number of negatives per positive during training.
-
-    match_mlperf: Use the MLPerf reference convention for computing rank.
-
-    use_tpu_spec: Should a TPUEstimatorSpec be returned instead of an
-      EstimatorSpec. Required for TPUs and if XLA is done on a GPU. Despite its
-      name, TPUEstimatorSpecs work with GPUs
-
-  Returns:
-    cross_entropy: the loss
-    metric_fn: the metrics function
-    in_top_k: hit rate metric
-    ndcg: ndcg metric
-    metric_weights: metric weights
-  """
-  in_top_k, ndcg, metric_weights, logits_by_user = compute_top_k_and_ndcg(
-      logits, duplicate_mask, match_mlperf)
-
-  # Examples are provided by the eval Dataset in a structured format, so eval
-  # labels can be reconstructed on the fly.
-  eval_labels = tf.reshape(shape=(-1,), tensor=tf.one_hot(
-      tf.zeros(shape=(logits_by_user.shape[0],), dtype=tf.int32) +
-      rconst.NUM_EVAL_NEGATIVES, logits_by_user.shape[1], dtype=tf.int32))
-
-  eval_labels_float = tf.cast(eval_labels, tf.float32)
-
-  # During evaluation, the ratio of negatives to positives is much higher
-  # than during training. (Typically 999 to 1 vs. 4 to 1) By adjusting the
-  # weights for the negative examples we compute a loss which is consistent with
-  # the training data. (And provides apples-to-apples comparison)
-  negative_scale_factor = num_training_neg / rconst.NUM_EVAL_NEGATIVES
-  example_weights = (
-      (eval_labels_float + (1 - eval_labels_float) * negative_scale_factor) *
-      (1 + rconst.NUM_EVAL_NEGATIVES) / (1 + num_training_neg))
-
-  # Tile metric weights back to logit dimensions
-  expanded_metric_weights = tf.reshape(tf.tile(
-      metric_weights[:, tf.newaxis], (1, rconst.NUM_EVAL_NEGATIVES + 1)), (-1,))
-
-  # ignore padded examples
-  example_weights *= tf.cast(expanded_metric_weights, tf.float32)
-
-  cross_entropy = tf.compat.v1.losses.sparse_softmax_cross_entropy(
-      logits=softmax_logits, labels=eval_labels, weights=example_weights)
-
-  def metric_fn(top_k_tensor, ndcg_tensor, weight_tensor):
-    return {
-        rconst.HR_KEY: tf.compat.v1.metrics.mean(top_k_tensor,
-                                                 weights=weight_tensor,
-                                                 name=rconst.HR_METRIC_NAME),
-        rconst.NDCG_KEY: tf.compat.v1.metrics.mean(ndcg_tensor,
-                                                   weights=weight_tensor,
-                                                   name=rconst.NDCG_METRIC_NAME)
-    }
-
-  return cross_entropy, metric_fn, in_top_k, ndcg, metric_weights
-
-
-def compute_top_k_and_ndcg(logits,              # type: tf.Tensor
-                           duplicate_mask,      # type: tf.Tensor
-                           match_mlperf=False   # type: bool
-                          ):
-  """Compute inputs of metric calculation.
-
-  Args:
-    logits: A tensor containing the predicted logits for each user. The shape
-      of logits is (num_users_per_batch * (1 + NUM_EVAL_NEGATIVES),) Logits
-      for a user are grouped, and the first element of the group is the true
-      element.
-    duplicate_mask: A vector with the same shape as logits, with a value of 1
-      if the item corresponding to the logit at that position has already
-      appeared for that user.
-    match_mlperf: Use the MLPerf reference convention for computing rank.
-
-  Returns:
-    is_top_k, ndcg and weights, all of which has size (num_users_in_batch,), and
-    logits_by_user which has size
-    (num_users_in_batch, (rconst.NUM_EVAL_NEGATIVES + 1)).
-  """
-  logits_by_user = tf.reshape(logits, (-1, rconst.NUM_EVAL_NEGATIVES + 1))
-  duplicate_mask_by_user = tf.cast(
-      tf.reshape(duplicate_mask, (-1, rconst.NUM_EVAL_NEGATIVES + 1)),
-      logits_by_user.dtype)
-
-  if match_mlperf:
-    # Set duplicate logits to the min value for that dtype. The MLPerf
-    # reference dedupes during evaluation.
-    logits_by_user *= (1 - duplicate_mask_by_user)
-    logits_by_user += duplicate_mask_by_user * logits_by_user.dtype.min
-
-  # Determine the location of the first element in each row after the elements
-  # are sorted.
-  sort_indices = tf.argsort(
-      logits_by_user, axis=1, direction="DESCENDING")
-
-  # Use matrix multiplication to extract the position of the true item from the
-  # tensor of sorted indices. This approach is chosen because both GPUs and TPUs
-  # perform matrix multiplications very quickly. This is similar to np.argwhere.
-  # However this is a special case because the target will only appear in
-  # sort_indices once.
-  one_hot_position = tf.cast(tf.equal(sort_indices, rconst.NUM_EVAL_NEGATIVES),
-                             tf.int32)
-  sparse_positions = tf.multiply(
-      one_hot_position, tf.range(logits_by_user.shape[1])[tf.newaxis, :])
-  position_vector = tf.reduce_sum(sparse_positions, axis=1)
-
-  in_top_k = tf.cast(tf.less(position_vector, rconst.TOP_K), tf.float32)
-  ndcg = tf.math.log(2.) / tf.math.log(
-      tf.cast(position_vector, tf.float32) + 2)
-  ndcg *= in_top_k
-
-  # If a row is a padded row, all but the first element will be a duplicate.
-  metric_weights = tf.not_equal(tf.reduce_sum(duplicate_mask_by_user, axis=1),
-                                rconst.NUM_EVAL_NEGATIVES)
-
-  return in_top_k, ndcg, metric_weights, logits_by_user
diff --git a/models/recommendation/tensorflow/wide_deep_large_ds/training/__init__.py b/models/recommendation/tensorflow/wide_deep_large_ds/training/__init__.py
deleted file mode 100644
index b43c21f99..000000000
--- a/models/recommendation/tensorflow/wide_deep_large_ds/training/__init__.py
+++ /dev/null
@@ -1,20 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-# SPDX-License-Identifier: EPL-2.0
-#
-
diff --git a/models/recommendation/tensorflow/wide_deep_large_ds/training/train.py b/models/recommendation/tensorflow/wide_deep_large_ds/training/train.py
deleted file mode 100644
index 576fc08f0..000000000
--- a/models/recommendation/tensorflow/wide_deep_large_ds/training/train.py
+++ /dev/null
@@ -1,266 +0,0 @@
-#
-# -*- coding: utf-8 -*-
-#
-# Copyright (c) 2018 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-#
-'''This script used to create and train wide and deep model on Kaggle's Criteo Dataset'''
-
-import time
-import argparse
-import tensorflow as tf
-import math
-import collections
-import numpy as np
-import os.path
-from os import path
-import sys
-# Set to INFO for tracking training, default is WARN. ERROR for least messages
-tf.compat.v1.logging.set_verbosity(tf.compat.v1.logging.INFO)
-print("Using TensorFlow version %s" % (tf.__version__))
-CONTINUOUS_COLUMNS = ["I"+str(i) for i in range(1, 14)]  # 1-13 inclusive
-CATEGORICAL_COLUMNS = ["C"+str(i) for i in range(1, 27)]  # 1-26 inclusive
-LABEL_COLUMN = ["clicked"]
-TRAIN_DATA_COLUMNS = LABEL_COLUMN + CONTINUOUS_COLUMNS + CATEGORICAL_COLUMNS
-FEATURE_COLUMNS = CONTINUOUS_COLUMNS + CATEGORICAL_COLUMNS
-
-
-def generate_input_fn(filename, batch_size, num_epochs):
-    def parse_csv(value):
-        tf.compat.v1.logging.info('Parsing {}'.format(filename))
-        cont_defaults = [[0.0] for i in range(1, 14)]
-        cate_defaults = [[" "] for i in range(1, 27)]
-        label_defaults = [[0]]
-        column_headers = TRAIN_DATA_COLUMNS
-        record_defaults = label_defaults + cont_defaults + cate_defaults
-        columns = tf.io.decode_csv(value, record_defaults=record_defaults)
-        all_columns = collections.OrderedDict(zip(column_headers, columns))
-        labels = all_columns.pop(LABEL_COLUMN[0])
-        features = all_columns
-        return features, labels
-
-    # Extract lines from input files using the Dataset API.
-    dataset = tf.data.TextLineDataset(filename)
-    dataset = dataset.shuffle(buffer_size=20000)
-    dataset = dataset.repeat(num_epochs)
-    dataset = dataset.prefetch(batch_size)
-    dataset = dataset.batch(batch_size)
-    dataset = dataset.map(parse_csv, num_parallel_calls=28)
-    dataset = dataset.prefetch(1)
-    return dataset
-
-
-def build_feature_cols(train_file_path,test_file_path):
-    # compute statistics(min,max,range) of train dataset
-    print('****Computing statistics of train dataset*****')
-    with open(train_file_path, 'r') as f, open(test_file_path, 'r') as f1:
-        nums = [line.strip('\n').split(',') for line in f.readlines(
-        )]+[line.strip('\n').split(',') for line in f1.readlines()]
-        numpy_arr = np.array(nums)
-        mins_list, max_list, range_list = [], [], []
-        for i in range(len(TRAIN_DATA_COLUMNS)):
-            if TRAIN_DATA_COLUMNS[i] in CONTINUOUS_COLUMNS:
-                col_min = numpy_arr[:, i].astype(np.float32).min()
-                col_max = numpy_arr[:, i].astype(np.float32).max()
-                mins_list.append(col_min)
-                max_list.append(col_max)
-                range_list.append(col_max-col_min)
-
-    def numeric_column_normalized(column_name, normalizer_fn):
-        return tf.feature_column.numeric_column(column_name, normalizer_fn=normalizer_fn)
-
-    def make_minmaxscaler(min, range):
-        def minmaxscaler(col):
-            return (col - min)/range
-        return minmaxscaler
-    deep_columns = []
-    for i in range(len(CONTINUOUS_COLUMNS)):
-        normalizer_fn = None
-        col_min = mins_list[i]
-        col_range = range_list[i]
-        normalizer_fn = make_minmaxscaler(col_min, col_range)
-        deep_columns.append(numeric_column_normalized(
-            CONTINUOUS_COLUMNS[i], normalizer_fn))
-    wide_columns = []
-    for name in CATEGORICAL_COLUMNS:
-        wide_columns.append(tf.feature_column.categorical_column_with_hash_bucket(
-            name, hash_bucket_size=1000))
-    for col in wide_columns:
-        deep_columns.append(
-            tf.feature_column.embedding_column(col, dimension=32))
-    return wide_columns, deep_columns
-
-
-def build_model(model_type, model_dir, wide_columns, deep_columns):
-    runconfig = tf.estimator.RunConfig(
-        save_checkpoints_steps=500
-    )
-    m = None
-    # Linear Classifier
-    if model_type == 'WIDE':
-        m = tf.estimator.LinearClassifier(
-            config=runconfig,
-            model_dir=model_dir,
-            feature_columns=wide_columns)
-
-    # Deep Neural Net Classifier
-    elif model_type == 'DEEP':
-        m = tf.estimator.DNNClassifier(
-            config=runconfig,
-            model_dir=model_dir,
-            feature_columns=deep_columns,
-            hidden_units=[1024, 512, 256])
-
-    # Combined Linear and Deep Classifier
-    elif model_type == 'WIDE_AND_DEEP':
-        m = tf.estimator.DNNLinearCombinedClassifier(
-            config=runconfig,
-            model_dir=model_dir,
-            linear_feature_columns=wide_columns,
-            dnn_feature_columns=deep_columns,
-            dnn_hidden_units=[1024, 512, 256])
-
-    return m
-
-
-def build_estimator(model_type='WIDE_AND_DEEP', model_dir=None, train_file_path=None,test_file_path=None):
-    if model_dir is None:
-        model_dir = 'models/model_' + model_type + '_' + str(int(time.time()))
-        print("Model directory = %s" % model_dir)
-
-    wide_columns, deep_columns = build_feature_cols(train_file_path,test_file_path)
-    m = build_model(model_type, model_dir, wide_columns, deep_columns)
-    print('estimator built')
-    return m
-
-
-# All categorical columns are strings for this dataset
-def column_to_dtype(column):
-    if column in CATEGORICAL_COLUMNS:
-        return tf.string
-    else:
-        return tf.float32
-
-
-"""
-  This function maps input columns (feature_placeholders) to 
-  tensors that can be inputted into the graph 
-  (similar in purpose to the output of our input functions)
-  In this particular case, we need to accomodate the sparse fields (strings)
-  so we have to do a slight modification to expand their dimensions, 
-  just like in the input functions
-"""
-
-
-def serving_input_fn():
-    feature_placeholders = {
-        column: tf.compat.v1.placeholder(column_to_dtype(column), [None])
-        for column in FEATURE_COLUMNS
-    }
-    # DNNCombinedLinearClassifier expects rank 2 Tensors,
-    # but inputs should be rank 1, so that we can provide
-    # scalars to the server
-    features = {
-        key: tf.expand_dims(tensor, -1)
-        for key, tensor in feature_placeholders.items()
-    }
-    return tf.estimator.export.ServingInputReceiver(
-        features,  # input into graph
-        feature_placeholders  # tensor input converted from request
-    )
-
-
-def train_and_eval():
-    print("Begin training and evaluation")
-    train_file = args.data_location+'/train.csv'
-    test_file = args.data_location+'/eval.csv'
-    if (not path.exists(train_file)) or (not path.exists(test_file)):
-        print('------------------------------------------------------------------------------------------')
-        print("train.csv or eval.csv does not exist in the given data_location. Please provide valid path")
-        print('------------------------------------------------------------------------------------------')
-        sys.exit() 
-    no_of_training_examples = sum(1 for line in open(train_file))
-    no_of_test_examples = sum(1 for line in open(test_file))
-    batch_size = args.batch_size
-    if args.steps == 0:
-        no_of_epochs = 10
-        train_steps = math.ceil(
-        (float(no_of_epochs)*no_of_training_examples)/batch_size)
-    else:
-        no_of_epochs = math.ceil(
-        (float(batch_size)*args.steps)/no_of_training_examples)
-        train_steps = args.steps
-    test_steps = math.ceil(float(no_of_test_examples)/batch_size)
-    model_type = 'WIDE_AND_DEEP'
-    model_dir = path.join(args.output_dir, 'model_' + model_type + '_' + str(int(time.time())))
-    checkpoint_dir = args.checkpoint if args.checkpoint else model_dir
-    print("Saving model checkpoints to " + checkpoint_dir)
-    export_dir = args.output_dir
-    m = build_estimator(model_type, checkpoint_dir, train_file, test_file)
-    m.train(input_fn=lambda: generate_input_fn(
-        train_file, batch_size, int(no_of_epochs)),steps=int(train_steps))
-    print('fit done')
-    results = m.evaluate(input_fn=lambda: generate_input_fn(
-        test_file, batch_size, 1), steps=test_steps)
-    print('evaluate done')
-
-    export_folder = m.export_saved_model(
-        export_dir,
-        serving_input_fn
-    )
-    print('Model exported to ' + export_dir)
-    print('Accuracy: %s' % results['accuracy'])
-
-
-def get_arg_parser():
-    parser = argparse.ArgumentParser()
-    parser.add_argument(
-        '--data_location',
-        help='Full path of train data',
-        required=True
-    )
-    parser.add_argument(
-        '--steps',
-        help='set the number of steps on train dataset.Default is will set to 1 epoch',
-        type=int,
-        default=0
-    )
-    parser.add_argument(
-        '--batch_size',
-        help='Batch size to train. Default is 512',
-        type=int,
-        default=512
-    )
-    parser.add_argument(
-        '--output_dir',
-        help='Full path to output directory for logs and saved model',
-        required=True
-    )
-    parser.add_argument(
-        '--checkpoint',
-        help='Full path to the input/output directory for checkpoints',
-        required=False
-    )
-    return parser
-
-
-if __name__ == "__main__":
-    parser = get_arg_parser()
-    args = parser.parse_args()
-    main_start = time.time()
-    train_and_eval()
-    main_end = time.time()
-    print("Total time:", main_end-main_start)
diff --git a/models_v2/common/parse_result.py b/models_v2/common/parse_result.py
new file mode 100644
index 000000000..11e5385e4
--- /dev/null
+++ b/models_v2/common/parse_result.py
@@ -0,0 +1,238 @@
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+#!/urs/bin/env python
+# coding: utf-8
+
+
+"""
+Models log parser.
+"""
+
+
+import argparse
+import importlib.util
+import re
+from typing import Optional
+import os
+# from common.utils import check_python_version
+
+
+def get_functional_status(log_path: str, pattern_dict: dict) -> str:
+    """Check if log contains functional line."""
+    with open(log_path, "r", encoding="UTF-8") as file:
+        for line in file:
+            match = re.search(pattern_dict["pattern"], line)
+            if match:
+                return "pass"
+    return "fail"
+
+
+def get_error(log_path: str) -> str:
+    """Get error message from log."""
+    with open(log_path, "r", encoding="UTF-8") as file:
+        for line in file:
+            match = re.search(r"\S+Error", line)
+            if match:
+                return match.group(0)
+    return "?"
+
+
+def parse_log_total(log_path: str, pattern_dict: dict) -> Optional[float]:
+    """Parse log and return matched value as total."""
+    value = None
+    with open(log_path, "r", encoding="UTF-8") as file:
+        for line in file:
+            match = re.search(pattern_dict["pattern"], line)
+            if match:
+                value = float(match.group(1))
+    return value
+
+
+# Parse results for models that leverage Torch DDP Horovod backend
+def parse_log_total_for_hvd_ddp(log_path: str, pattern_dict: dict) -> Optional[float]:
+    """Parse log and return matched value as total."""
+    sum_value = []
+    with open(log_path, "r", encoding="UTF-8") as file:
+        for line in file:
+            match = re.search(pattern_dict["pattern"], line)
+            if match:
+                sum_value.append(float(match.group(1)))
+    if sum_value:
+        return float(sum(sum_value[-len(sum_value):]))
+    return None
+
+
+def parse_log_min_for_hvd_ddp(log_path: str, pattern_dict: dict) -> Optional[float]:
+    """Parse log and return matched value as total."""
+    min_value = []
+    with open(log_path, "r", encoding="UTF-8") as file:
+        for line in file:
+            match = re.search(pattern_dict["pattern"], line)
+            if match:
+                min_value.append(float(match.group(1)))
+    if min_value:
+        return float(min(min_value))
+    return None
+
+
+def parse_log_max_for_hvd_ddp(log_path: str, pattern_dict: dict) -> Optional[float]:
+    """Parse log and return matched value as total."""
+    max_value = []
+    with open(log_path, "r", encoding="UTF-8") as file:
+        for line in file:
+            match = re.search(pattern_dict["pattern"], line)
+            if match:
+                max_value.append(float(match.group(1)))
+    if max_value:
+        return float(max(max_value))
+    return None
+
+
+def parse_log_mean(log_path: str, pattern_dict: dict) -> Optional[float]:
+    """Parse log and return mean of matched values."""
+    partials = []
+    with open(log_path, "r", encoding="UTF-8") as file:
+        for line in file:
+            match = re.search(pattern_dict["pattern"], line)
+            if match:
+                partials.append(float(match.group(1)))
+    number_of_partials = pattern_dict.get("number_of_partials", len(partials))
+    if partials:
+        return float(sum(partials[-number_of_partials:]) / number_of_partials)
+    return None
+
+
+def parse_log(args) -> None:
+    """Parse log and save results."""
+    spec = importlib.util.spec_from_file_location(
+        args.model_name, os.path.abspath(args.model_name+".py"))
+    patterns = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(patterns)
+    acc_pattern = patterns.ACC
+    perf_pattern = patterns.PERF
+    functional_pattern = patterns.FUNCTIONAL
+
+    # parse accuracy
+    accuracy = "N/A"
+    accuracy_unit = ""
+    if acc_pattern.get("unit"):
+        accuracy_unit = acc_pattern['unit']
+        if args.hvd or args.ddp:
+            if acc_pattern.get("type") == "total":
+                accuracy = parse_log_total_for_hvd_ddp(
+                    args.log_dir, acc_pattern)
+            elif acc_pattern.get("type") == "mean":
+                accuracy = parse_log_mean(args.log_dir, acc_pattern)
+        else:
+            if acc_pattern.get("type") == "total":
+                accuracy = parse_log_total(args.log_dir, acc_pattern)
+            elif acc_pattern.get("type") == "mean":
+                accuracy = parse_log_mean(args.log_dir, acc_pattern)
+
+    # parse performance
+    performance = "N/A"
+    performance_unit = ""
+    if perf_pattern.get("unit"):
+        performance_unit = perf_pattern["unit"]
+        if args.hvd or args.ddp:
+            performance_min = parse_log_min_for_hvd_ddp(
+                args.log_dir, perf_pattern)
+            if perf_pattern["type"] == "total":
+                performance = parse_log_total_for_hvd_ddp(
+                    args.log_dir, perf_pattern)
+            elif perf_pattern["type"] == "mean":
+                performance = parse_log_mean(args.log_dir, perf_pattern)
+            elif perf_pattern["type"] == "max":
+                performance = parse_log_max_for_hvd_ddp(
+                    args.log_dir, perf_pattern)
+        else:
+            if perf_pattern["type"] == "total":
+                performance = parse_log_total(args.log_dir, perf_pattern)
+            elif perf_pattern["type"] == "mean":
+                performance = parse_log_mean(args.log_dir, perf_pattern)
+        # inverse if needed
+        if perf_pattern.get("inverse") and performance:
+            performance = 1 / performance
+        # multiply if needed
+        if perf_pattern.get("multiply") and performance:
+            multiply = perf_pattern["multiply"]
+            performance = performance * multiply
+        # use batch size if needed
+        if perf_pattern.get("use_batch_size") and performance:
+            performance = performance * int(args.batch_size)
+
+    # get latency
+    latency = "N/A"
+    if args.hvd or args.ddp:
+        latency = "N/A"
+    else:
+        if perf_pattern.get("unit") and isinstance(performance, float):
+            performance_unit = perf_pattern["unit"]
+            if bool(re.search('/s$', performance_unit)) or performance_unit == "fps":
+                latency = int(args.batch_size) / performance
+
+    # get functional status
+    functional_status = get_functional_status(args.log_dir, functional_pattern)
+
+    performance = f"{float(f'{performance:.8g}')}" if isinstance(
+        performance, float) else performance
+    if args.hvd or args.ddp:
+        performance_min = f"{float(f'{performance_min:.8g}')}" if isinstance(
+            performance_min, float) else performance_min
+    accuracy = f"{accuracy:.3f}" if isinstance(accuracy, float) else accuracy
+
+    # save results to log file
+    result_dir = args.log_dir.replace('_raw.log', '.log')
+    result_file = open(result_dir, 'w')
+    result_file.write(f"Batch Size: {args.batch_size}")
+    if args.hvd or args.ddp:
+        result_file.write(
+            f"\nSum Performance: {performance} {performance_unit}")
+        result_file.write(
+            f"\nMin Performance: {performance_min} {performance_unit}")
+    else:
+        result_file.write(f"\nPerformance: {performance} {performance_unit}")
+    result_file.write(f"\nLatency: {latency} s")
+    result_file.write(f"\nAccuracy: {accuracy_unit} {accuracy}")
+    result_file.write(f"\nFunctional: {functional_status}")
+    if functional_status == "fail":
+        error = get_error(args.log_dir)
+        result_file.write(f"\nError: {error}")
+    result_file.write(f"\n")
+    result_file.close()
+
+
+def main():
+    """
+    Main function of models results parser.
+    """
+    # check_python_version()
+    parser = argparse.ArgumentParser(description=f"Model result parser")
+    parser.add_argument('--model_name', '-m', type=str, required=True,
+                        help="Test scope type")
+    parser.add_argument('--ddp', action='store_true',
+                        help="parse results for models that leverage Torch DDP backend")
+    parser.add_argument('--hvd', action='store_true',
+                        help="parse results for models that leverage Horovod backend")
+    parser.add_argument('--batch_size', '-b', required=True, help='Batch size')
+    parser.add_argument('--log_dir', '-l', required=True, help='Log directory')
+    args = parser.parse_args()
+    parse_log(args)
+
+
+if __name__ == "__main__":
+    main()
diff --git a/models_v2/pytorch/3d_unet/inference/gpu/3d-unet/Makefile b/models_v2/pytorch/3d_unet/inference/gpu/3d-unet/Makefile
new file mode 100644
index 000000000..baa1d9d30
--- /dev/null
+++ b/models_v2/pytorch/3d_unet/inference/gpu/3d-unet/Makefile
@@ -0,0 +1,251 @@
+# Copyright (c) 2020, NVIDIA CORPORATION.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+SHELL := /bin/bash
+
+MAKEFILE_NAME := $(lastword $(MAKEFILE_LIST))
+UNAME := $(shell whoami)
+UID := $(shell id -u `whoami`)
+GROUPNAME := $(shell id -gn `whoami`)
+GROUPID := $(shell id -g `whoami`)
+
+HOST_VOL ?= ${PWD}
+CONTAINER_VOL ?= /workspace
+
+BUILD_DIR := build
+ifndef DOWNLOAD_DATA_DIR
+	export DOWNLOAD_DATA_DIR := $(HOST_VOL)/$(BUILD_DIR)/MICCAI_BraTS_2019_Data_Training
+endif
+RAW_DATA_DIR := $(BUILD_DIR)/raw_data
+PREPROCESSED_DATA_DIR := $(BUILD_DIR)/preprocessed_data
+POSTPROCESSED_DATA_DIR := $(BUILD_DIR)/postprocessed_data
+#MODEL_DIR := $(BUILD_DIR)/model
+RESULT_DIR := $(BUILD_DIR)/result
+MLPERF_CONF := $(BUILD_DIR)/mlperf.conf
+PYTORCH_MODEL := $(RESULT_DIR)/fold_1.zip
+#ONNX_MODEL := $(MODEL_DIR)/224_224_160.onnx
+#ONNX_DYNAMIC_BS_MODEL := $(MODEL_DIR)/224_224_160_dynamic_bs.onnx
+#TF_MODEL := $(MODEL_DIR)/224_224_160.pb
+#OPENVINO_MODEL := $(MODEL_DIR)/brats_model_checkpoint_final_fold1_H224_W224_D160_C4.bin
+#OPENVINO_MODEL_METADATA := $(MODEL_DIR)/brats_model_checkpoint_final_fold1_H224_W224_D160_C4.xml
+
+# Env variables needed by nnUnet
+export nnUNet_raw_data_base=$(RAW_DATA_DIR)
+export nnUNet_preprocessed=$(PREPROCESSED_DATA_DIR)
+export RESULTS_FOLDER=$(RESULT_DIR)
+
+HAS_GPU := $(shell command -v nvidia-smi 2> /dev/null)
+
+ifeq ($(HAS_GPU),)
+	DOCKER_RUN_CMD := docker run
+else
+	# Handle different nvidia-docker version
+	ifneq ($(wildcard /usr/bin/nvidia-docker),)
+		DOCKER_RUN_CMD := nvidia-docker run
+	else
+		DOCKER_RUN_CMD := docker run --gpus=all
+	endif
+endif
+
+.PHONY: setup
+setup: check_download_data_dir create_directories
+	@echo "Running basic setup..."
+#	@if [ ! -e $(MLPERF_CONF) ]; then \
+#		cp ../../../mlperf.conf $(MLPERF_CONF); \
+#	fi
+#	@$(MAKE) -f $(MAKEFILE_NAME) init_submodule
+	@$(MAKE) -f $(MAKEFILE_NAME) download_model
+
+.PHONY: check_download_data_dir
+check_download_data_dir:
+	@if [ ! -e $(DOWNLOAD_DATA_DIR) ]; then \
+		echo "Please set environment variable DOWNLOAD_DATA_DIR to <path/to/MICCAI_BraTS_2019_Data_Training>" && false ; \
+	fi
+
+.PHONY: create_directories
+create_directories:
+	@if [ ! -e $(BUILD_DIR) ]; then \
+		mkdir $(BUILD_DIR); \
+	fi
+	#@if [ ! -e $(MODEL_DIR) ]; then \
+	#	mkdir $(MODEL_DIR); \
+	#fi
+	@if [ ! -e $(RESULT_DIR) ]; then \
+		mkdir $(RESULT_DIR); \
+	fi
+
+#.PHONY: init_submodule
+#init_submodule:
+#	@echo "Initialize nnUnet submodule.."
+#	@git submodule update --init nnUnet
+
+.PHONY: download_model
+download_model:
+	@echo "Download models..."
+	@$(MAKE) -f $(MAKEFILE_NAME) download_pytorch_model
+#	@$(MAKE) -f $(MAKEFILE_NAME) download_onnx_model
+#	@$(MAKE) -f $(MAKEFILE_NAME) download_tf_model
+#	@$(MAKE) -f $(MAKEFILE_NAME) download_openvino_model
+
+.PHONY: download_pytorch_model
+download_pytorch_model: create_directories
+	@echo "Downloading PyTorch model from Zenodo..."
+	@if [ ! -e $(PYTORCH_MODEL) ]; then \
+		wget -O $(PYTORCH_MODEL) https://zenodo.org/record/3904106/files/fold_1.zip?download=1; \
+	fi
+	@cd $(RESULT_DIR) && unzip -o fold_1.zip;
+
+#.PHONY: download_onnx_model
+#download_onnx_model: create_directories
+#	@echo "Downloading ONNX model from Zenodo..."
+#	@if [ ! -e $(ONNX_MODEL) ]; then \
+#		wget -O $(ONNX_MODEL) https://zenodo.org/record/3928973/files/224_224_160.onnx?download=1; \
+#	fi
+#	@if [ ! -e $(ONNX_DYNAMIC_BS_MODEL) ]; then \
+#		wget -O $(ONNX_DYNAMIC_BS_MODEL) https://zenodo.org/record/3928973/files/224_224_160_dyanmic_bs.onnx?download=1; \
+#	fi
+#
+#.PHONY: download_tf_model
+#download_tf_model: create_directories
+#	@echo "Downloading TF model from Zenodo..."
+#	@if [ ! -e $(TF_MODEL) ]; then \
+#		wget -O $(TF_MODEL) https://zenodo.org/record/3928991/files/224_224_160.pb?download=1; \
+#	fi
+#
+#.PHONY: download_openvino_model
+#download_openvino_model: create_directories
+#	@echo "Downloading OpenVINO model from Zenodo..."
+#	@if [ ! -e $(OPENVINO_MODEL) ]; then \
+#		wget -O $(OPENVINO_MODEL) https://zenodo.org/record/3929002/files/brats_model_checkpoint_final_fold1_H224_W224_D160_C4.bin?download=1; \
+#	fi
+#	@if [ ! -e $(OPENVINO_MODEL_METADATA) ]; then \
+#		wget -O $(OPENVINO_MODEL_METADATA) https://zenodo.org/record/3929002/files/brats_model_checkpoint_final_fold1_H224_W224_D160_C4.xml?download=1; \
+#	fi
+#
+#.PHONY: convert_onnx_model
+#convert_onnx_model: download_pytorch_model
+#	@echo "Converting PyTorch model to ONNX model..."
+#	@if [ ! -e $(ONNX_MODEL) ]; then \
+#		python3 unet_pytorch_to_onnx.py; \
+#	fi
+#
+#.PHONY: convert_tf_model
+#convert_tf_model: convert_onnx_model
+#	@echo "Converting ONNX model to TF model..."
+#	@if [ ! -e $(TF_MODEL) ]; then \
+#		python3 unet_onnx_to_tf.py; \
+#	fi
+#
+#.PHONY: build_docker
+#build_docker:
+#	@echo "Building docker image..."
+#	@docker pull nvcr.io/nvidia/pytorch:20.03-py3
+#	@docker build --build-arg GID=$(GROUPID) --build-arg UID=$(UID) --build-arg GROUP=$(GROUPNAME) --build-arg USER=$(UNAME) \
+#	--build-arg http_proxy=$(HTTP_PROXY) --build-arg https_proxy=$(HTTPS_PROXY) --build-arg HTTPS_PROXY=$(HTTPS_PROXY) --build-arg HTTP_PROXY=$(HTTP_PROXY) \
+#        --build-arg BASE_IMAGE=nvcr.io/nvidia/pytorch:20.03-py3 -t mlperf-inference-3d-unet -f Dockerfile .
+#
+#.PHONY: launch_docker
+#launch_docker: check_download_data_dir
+#	@mkdir -p $(POSTPROCESSED_DATA_DIR)
+#	@echo "Launching docker container..."
+#	@$(DOCKER_RUN_CMD) --rm -it -w $(CONTAINER_VOL) -v $(HOST_VOL):$(CONTAINER_VOL) -v ${HOME}:/mnt/${HOME} \
+#		-v $(DOWNLOAD_DATA_DIR):/downloaded_data_dir \
+#		--shm-size=1g --ulimit memlock=-1 --ulimit stack=67108864 \
+#		-v /etc/timezone:/etc/timezone:ro -v /etc/localtime:/etc/localtime:ro \
+#		--security-opt apparmor=unconfined --security-opt seccomp=unconfined \
+#		--name mlperf-inference-3d-unet-$(UNAME) -h mlperf-inference-3d-unet-$(UNAME) --add-host mlperf-inference-3d-unet-$(UNAME):127.0.0.1 \
+#		--user $(UID):$(GROUPID) --net host --device /dev/fuse --cap-add SYS_ADMIN $(DOCKER_ARGS) mlperf-inference-3d-unet
+#
+.PHONY: preprocess_data
+preprocess_data: create_directories
+	@echo "Restructuring raw data to $(RAW_DATA_DIR)..."
+	@if [ ! -e $(RAW_DATA_DIR) ]; then \
+		mkdir $(RAW_DATA_DIR); \
+	fi
+	@python3 Task043_BraTS_2019.py --downloaded_data_dir $(DOWNLOAD_DATA_DIR) 
+	@echo "Preprocessing and saving preprocessed data to $(PREPROCESSED_DATA_DIR)..."
+	@if [ ! -e $(PREPROCESSED_DATA_DIR) ]; then \
+		mkdir $(PREPROCESSED_DATA_DIR); \
+	fi
+	@python3 preprocess.py --model_dir $(RESULT_DIR)/nnUNet/3d_fullres/Task043_BraTS2019/nnUNetTrainerV2__nnUNetPlansv2.mlperf.1 --raw_data_dir $(RAW_DATA_DIR)/nnUNet_raw_data/Task043_BraTS2019/imagesTr --preprocessed_data_dir $(PREPROCESSED_DATA_DIR)
+
+.PHONY: mkdir_postprocessed_data
+mkdir_postprocessed_data:
+	@if [ ! -e $(POSTPROCESSED_DATA_DIR) ]; then \
+		mkdir -p $(POSTPROCESSED_DATA_DIR); \
+	fi
+
+.PHONY: run_pytorch_performance
+run_pytorch_performance:
+	@python3 run.py --backend=pytorch --model_dir $(RESULT_DIR)/nnUNet/3d_fullres/Task043_BraTS2019/nnUNetTrainerV2__nnUNetPlansv2.mlperf.1 --preprocessed_data_dir $(PREPROCESSED_DATA_DIR) --mlperf_conf $(BUILD_DIR)/mlperf.conf --build_dir $(BUILD_DIR)
+
+.PHONY: run_pytorch_accuracy
+run_pytorch_accuracy: mkdir_postprocessed_data
+	@python3 run.py --backend=pytorch --accuracy --model_dir $(RESULT_DIR)/nnUNet/3d_fullres/Task043_BraTS2019/nnUNetTrainerV2__nnUNetPlansv2.mlperf.1 --preprocessed_data_dir $(PREPROCESSED_DATA_DIR) --mlperf_conf $(BUILD_DIR)/mlperf.conf --build_dir $(BUILD_DIR)
+
+.PHONY: run_pytorch_performance_fp16
+run_pytorch_performance_fp16:
+	@python3 run.py --backend=pytorch --run_fp16 --model_dir $(RESULT_DIR)/nnUNet/3d_fullres/Task043_BraTS2019/nnUNetTrainerV2__nnUNetPlansv2.mlperf.1 --preprocessed_data_dir $(PREPROCESSED_DATA_DIR) --mlperf_conf $(BUILD_DIR)/mlperf.conf --build_dir $(BUILD_DIR)
+
+.PHONY: run_pytorch_accuracy_fp16
+run_pytorch_accuracy_fp16: mkdir_postprocessed_data
+	@python3 run.py --backend=pytorch --accuracy --run_fp16 --model_dir $(RESULT_DIR)/nnUNet/3d_fullres/Task043_BraTS2019/nnUNetTrainerV2__nnUNetPlansv2.mlperf.1 --preprocessed_data_dir $(PREPROCESSED_DATA_DIR) --mlperf_conf $(BUILD_DIR)/mlperf.conf --build_dir $(BUILD_DIR)
+
+.PHONY: run_pytorch_performance_int8
+run_pytorch_performance_int8:
+	@python3 run.py --backend pytorch --run_int8 --calib_iters 1 --model_dir $(RESULT_DIR)/nnUNet/3d_fullres/Task043_BraTS2019/nnUNetTrainerV2__nnUNetPlansv2.mlperf.1 --preprocessed_data_dir $(PREPROCESSED_DATA_DIR) --mlperf_conf $(BUILD_DIR)/mlperf.conf --build_dir $(BUILD_DIR)
+
+.PHONY: run_pytorch_accuracy_symm_u8
+run_pytorch_accuracy_symm_u8: mkdir_postprocessed_data
+	@python3 run.py --backend pytorch --accuracy --run_int8 --uint8 --calib_iters 1 --model_dir $(RESULT_DIR)/nnUNet/3d_fullres/Task043_BraTS2019/nnUNetTrainerV2__nnUNetPlansv2.mlperf.1 --preprocessed_data_dir $(PREPROCESSED_DATA_DIR) --mlperf_conf $(BUILD_DIR)/mlperf.conf --build_dir $(BUILD_DIR)
+
+.PHONY: run_pytorch_accuracy_symm_s8
+run_pytorch_accuracy_symm_s8: mkdir_postprocessed_data
+	@python3 run.py --backend pytorch --accuracy --run_int8 --calib_iters 1 --model_dir $(RESULT_DIR)/nnUNet/3d_fullres/Task043_BraTS2019/nnUNetTrainerV2__nnUNetPlansv2.mlperf.1 --preprocessed_data_dir $(PREPROCESSED_DATA_DIR) --mlperf_conf $(BUILD_DIR)/mlperf.conf --build_dir $(BUILD_DIR)
+
+.PHYONY: run_pytorch_accuracy_asymm_u8
+run_pytorch_accuracy_asymm_u8: mkdir_postprocessed_data
+	@python3 run.py --backend pytorch --accuracy --asymm --uint8 --run_int8 --calib_iters 1 --model_dir $(RESULT_DIR)/nnUNet/3d_fullres/Task043_BraTS2019/nnUNetTrainerV2__nnUNetPlansv2.mlperf.1 --preprocessed_data_dir $(PREPROCESSED_DATA_DIR) --mlperf_conf $(BUILD_DIR)/mlperf.conf --build_dir $(BUILD_DIR)
+
+.PHYONY: run_pytorch_accuracy_asymm_s8
+run_pytorch_accuracy_asymm_s8: mkdir_postprocessed_data
+	@python3 run.py --backend pytorch --accuracy --asymm --run_int8 --calib_iters 1 --model_dir $(RESULT_DIR)/nnUNet/3d_fullres/Task043_BraTS2019/nnUNetTrainerV2__nnUNetPlansv2.mlperf.1 --preprocessed_data_dir $(PREPROCESSED_DATA_DIR) --mlperf_conf $(BUILD_DIR)/mlperf.conf --build_dir $(BUILD_DIR)
+
+.PHYONY: run_pytorch_accuracy_int8
+run_pytorch_accuracy_int8:
+	@$(MAKE) -f $(MAKEFILE_NAME) run_pytorch_accuracy_symm_s8
+
+#.PHONY: run_onnxruntime_performance
+#run_onnxruntime_performance:
+#	@python3 run.py --backend=onnxruntime --model=build/model/224_224_160.onnx
+#
+#.PHONY: run_onnxruntime_accuracy
+#run_onnxruntime_accuracy: mkdir_postprocessed_data
+#	@python3 run.py --backend=onnxruntime --model=build/model/224_224_160.onnx --accuracy
+#
+#.PHONY: run_tf_performance
+#run_tf_performance:
+#	@python3 run.py --backend=tf --model=build/model/224_224_160.pb
+#
+#.PHONY: run_tf_accuracy
+#run_tf_accuracy: mkdir_postprocessed_data
+#	@python3 run.py --backend=tf --model=build/model/224_224_160.pb --accuracy
+#
+.PHONY: evaluate
+evaluate:
+	@python3 accuracy-brats.py
+
+.PHONY: clean
+clean:
+	@rm -rf build
diff --git a/models_v2/pytorch/3d_unet/inference/gpu/3d-unet/Task043_BraTS_2019.py b/models_v2/pytorch/3d_unet/inference/gpu/3d-unet/Task043_BraTS_2019.py
new file mode 100644
index 000000000..6c196077b
--- /dev/null
+++ b/models_v2/pytorch/3d_unet/inference/gpu/3d-unet/Task043_BraTS_2019.py
@@ -0,0 +1,135 @@
+#    Copyright (c) 2020, NVIDIA CORPORATION.  All rights reserved.
+#    Copyright 2020 Division of Medical Image Computing, German Cancer Research Center (DKFZ), Heidelberg, Germany
+#
+#    Licensed under the Apache License, Version 2.0 (the "License");
+#    you may not use this file except in compliance with the License.
+#    You may obtain a copy of the License at
+#
+#        http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS,
+#    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#    See the License for the specific language governing permissions and
+#    limitations under the License.
+
+# This file is copied from nnUnet/nnunet/dataset_conversion/Task043_BraTS_2019.py, except that
+# the validation/test set part is removed and downloaded_data_dir is now configurable.
+
+import argparse
+import numpy as np
+from collections import OrderedDict
+import os
+import sys
+
+sys.path.insert(0, os.path.join(os.getcwd(), "nnUnet"))
+
+from batchgenerators.utilities.file_and_folder_operations import *
+from nnunet.paths import nnUNet_raw_data
+import SimpleITK as sitk
+import shutil
+
+def get_args():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--downloaded_data_dir", default="build/MICCAI_BraTS_2019_Data_Training", help="path to MICCAI_BraTS_2019_Data_Training")
+    args = parser.parse_args()
+    return args
+
+def copy_BraTS_segmentation_and_convert_labels(in_file, out_file):
+    # use this for segmentation only!!!
+    # nnUNet wants the labels to be continuous. BraTS is 0, 1, 2, 4 -> we make that into 0, 1, 2, 3
+    img = sitk.ReadImage(in_file)
+    img_npy = sitk.GetArrayFromImage(img)
+
+    uniques = np.unique(img_npy)
+    for u in uniques:
+        if u not in [0, 1, 2, 4]:
+            raise RuntimeError('unexpected label')
+
+    seg_new = np.zeros_like(img_npy)
+    seg_new[img_npy == 4] = 3
+    seg_new[img_npy == 2] = 1
+    seg_new[img_npy == 1] = 2
+    img_corr = sitk.GetImageFromArray(seg_new)
+    img_corr.CopyInformation(img)
+    sitk.WriteImage(img_corr, out_file)
+
+def main():
+
+    args = get_args()
+
+    """
+    REMEMBER TO CONVERT LABELS BACK TO BRATS CONVENTION AFTER PREDICTION!
+    """
+
+    task_name = "Task043_BraTS2019"
+    downloaded_data_dir = args.downloaded_data_dir
+
+    target_base = join(nnUNet_raw_data, task_name)
+    target_imagesTr = join(target_base, "imagesTr")
+    target_imagesVal = join(target_base, "imagesVal")
+    target_imagesTs = join(target_base, "imagesTs")
+    target_labelsTr = join(target_base, "labelsTr")
+
+    maybe_mkdir_p(target_imagesTr)
+    maybe_mkdir_p(target_imagesVal)
+    maybe_mkdir_p(target_imagesTs)
+    maybe_mkdir_p(target_labelsTr)
+
+    patient_names = []
+    for tpe in ["HGG", "LGG"]:
+        cur = join(downloaded_data_dir, tpe)
+        for p in subdirs(cur, join=False):
+            patdir = join(cur, p)
+            patient_name = tpe + "__" + p
+            patient_names.append(patient_name)
+            t1 = join(patdir, p + "_t1.nii.gz")
+            t1c = join(patdir, p + "_t1ce.nii.gz")
+            t2 = join(patdir, p + "_t2.nii.gz")
+            flair = join(patdir, p + "_flair.nii.gz")
+            seg = join(patdir, p + "_seg.nii.gz")
+
+            assert all([
+                isfile(t1),
+                isfile(t1c),
+                isfile(t2),
+                isfile(flair),
+                isfile(seg)
+            ]), "%s" % patient_name
+
+            shutil.copy(t1, join(target_imagesTr, patient_name + "_0000.nii.gz"))
+            shutil.copy(t1c, join(target_imagesTr, patient_name + "_0001.nii.gz"))
+            shutil.copy(t2, join(target_imagesTr, patient_name + "_0002.nii.gz"))
+            shutil.copy(flair, join(target_imagesTr, patient_name + "_0003.nii.gz"))
+
+            copy_BraTS_segmentation_and_convert_labels(seg, join(target_labelsTr, patient_name + ".nii.gz"))
+
+    json_dict = OrderedDict()
+    json_dict['name'] = "BraTS2019"
+    json_dict['description'] = "nothing"
+    json_dict['tensorImageSize'] = "4D"
+    json_dict['reference'] = "see BraTS2019"
+    json_dict['licence'] = "see BraTS2019 license"
+    json_dict['release'] = "0.0"
+    json_dict['modality'] = {
+        "0": "T1",
+        "1": "T1ce",
+        "2": "T2",
+        "3": "FLAIR"
+    }
+    json_dict['labels'] = {
+        "0": "background",
+        "1": "edema",
+        "2": "non-enhancing",
+        "3": "enhancing",
+    }
+    json_dict['numTraining'] = len(patient_names)
+    json_dict['numTest'] = 0
+    json_dict['training'] = [{'image': "./imagesTr/%s.nii.gz" % i, "label": "./labelsTr/%s.nii.gz" % i} for i in
+                             patient_names]
+    json_dict['test'] = []
+
+    save_json(json_dict, join(target_base, "dataset.json"))
+
+if __name__ == "__main__":
+    main()
diff --git a/models_v2/pytorch/3d_unet/inference/gpu/3d-unet/accuracy-brats.py b/models_v2/pytorch/3d_unet/inference/gpu/3d-unet/accuracy-brats.py
new file mode 100644
index 000000000..82c9ce6c7
--- /dev/null
+++ b/models_v2/pytorch/3d_unet/inference/gpu/3d-unet/accuracy-brats.py
@@ -0,0 +1,156 @@
+# coding=utf-8
+# Copyright (c) 2020 NVIDIA CORPORATION. All rights reserved.
+# Copyright 2020 Division of Medical Image Computing, German Cancer Research Center (DKFZ), Heidelberg, Germany
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import argparse
+import json
+import numpy as np
+import os
+import pickle
+import sys
+
+sys.path.insert(0, os.path.join(os.path.dirname(__file__), "nnUnet"))
+
+from multiprocessing import Pool
+from nnunet.evaluation.region_based_evaluation import evaluate_regions, get_brats_regions
+from nnunet.inference.segmentation_export import save_segmentation_nifti_from_softmax
+
+dtype_map = {
+    "int8": np.int8,
+    "int16": np.int16,
+    "int32": np.int32,
+    "int64": np.int64,
+    "float16": np.float16,
+    "float32": np.float32,
+    "float64": np.float64
+}
+
+def get_args():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--log_file", default="build/logs/mlperf_log_accuracy.json", help="Path to accuracy log json file")
+    parser.add_argument("--output_dtype", default="float16", choices=dtype_map.keys(), help="Output data type")
+    parser.add_argument("--preprocessed_data_dir", default="build/preprocessed_data", help="Path to the directory containing preprocessed data")
+    parser.add_argument("--postprocessed_data_dir", default="build/postprocessed_data", help="Path to the directory containing postprocessed data")
+    parser.add_argument("--label_data_dir", default="build/raw_data/nnUNet_raw_data/Task043_BraTS2019/labelsTr",
+        help="Path to the directory containing ground truth labels")
+    parser.add_argument("--num_threads_nifti_save", type=int, default=12, help="Number of threads to run the postprocessing with")
+    args = parser.parse_args()
+    return args
+
+def save_predictions_MLPerf(predictions, output_folder, output_files, dictionaries, num_threads_nifti_save, all_in_gpu, force_separate_z=None, interp_order=3, interp_order_z=0):
+    print("Saving predictions...")
+    pool = Pool(num_threads_nifti_save)
+    results = []
+    for i, output_filename in enumerate(output_files):
+        print(i, "/", len(output_files))
+        output_filename = os.path.join(output_folder, output_filename + ".nii.gz")
+        softmax_mean = predictions[i]
+        dct = dictionaries[i]
+        bytes_per_voxel = 4
+        if all_in_gpu:
+            bytes_per_voxel = 2  # if all_in_gpu then the return value is half (float16)
+        if np.prod(softmax_mean.shape) > (2e9 / bytes_per_voxel * 0.85):  # * 0.85 just to be save
+            print(
+                "This output is too large for python process-process communication. Saving output temporarily to disk")
+            np.save(output_filename[:-7] + ".npy", softmax_mean)
+            softmax_mean = output_filename[:-7] + ".npy"
+
+        results.append(pool.starmap_async(save_segmentation_nifti_from_softmax,
+                                          ((softmax_mean, output_filename, dct, interp_order, None, None, None,
+                                            None, None, force_separate_z, interp_order_z),)
+                                          ))
+    _ = [i.get() for i in results]
+
+    pool.close()
+    pool.join()
+
+    del predictions
+
+def load_loadgen_log(log_file, result_dtype, dictionaries):
+    with open(log_file) as f:
+        predictions = json.load(f)
+
+    assert len(predictions) == len(dictionaries), "Number of predictions does not match number of samples in validation set!"
+
+    padded_shape = [224, 224, 160]
+    results = [None for i in range(len(predictions))]
+    for prediction in predictions:
+        qsl_idx = prediction["qsl_idx"]
+        assert qsl_idx >= 0 and qsl_idx < len(predictions), "Invalid qsl_idx!"
+        raw_shape = list(dictionaries[qsl_idx]["size_after_cropping"])
+        # Remove the padded part
+        pad_before = [(p - r) // 2 for p, r in zip(padded_shape, raw_shape)]
+        pad_after = [-(p - r - b) for p, r, b in zip(padded_shape, raw_shape, pad_before)]
+        result_shape = (4,) + tuple(padded_shape)
+        result = np.frombuffer(bytes.fromhex(prediction["data"]), result_dtype).reshape(result_shape).astype(np.float16)
+        results[qsl_idx] = result[:, pad_before[0]:pad_after[0], pad_before[1]:pad_after[1], pad_before[2]:pad_after[2]]
+
+    assert all([i is not None for i in results]), "Missing some results!"
+
+    return results
+
+def main():
+    args = get_args()
+    log_file = args.log_file
+    preprocessed_data_dir = args.preprocessed_data_dir
+    output_folder = args.postprocessed_data_dir
+    ground_truths = args.label_data_dir
+    output_dtype = dtype_map[args.output_dtype]
+    num_threads_nifti_save = args.num_threads_nifti_save
+    all_in_gpu = "None"
+    force_separate_z = None
+    interp_order = 3
+    interp_order_z = 0
+
+    # Load necessary metadata.
+    print("Loading necessary metadata...")
+    with open(os.path.join(preprocessed_data_dir, "preprocessed_files.pkl"), "rb") as f:
+        preprocessed_files = pickle.load(f)
+    dictionaries = []
+    for preprocessed_file in preprocessed_files:
+        with open(os.path.join(preprocessed_data_dir, preprocessed_file + ".pkl"), "rb") as f:
+            dct = pickle.load(f)[1]
+            dictionaries.append(dct)
+
+    # Load predictions from loadgen accuracy log.
+    print("Loading loadgen accuracy log...")
+    predictions = load_loadgen_log(log_file, output_dtype, dictionaries)
+
+    # Save predictions
+    # This runs in multiprocess
+    print("Running postprocessing with multiple threads...")
+    save_predictions_MLPerf(predictions, output_folder, preprocessed_files, dictionaries, num_threads_nifti_save, all_in_gpu, force_separate_z, interp_order, interp_order_z)
+
+    # Run evaluation
+    print("Running evaluation...")
+    evaluate_regions(output_folder, ground_truths, get_brats_regions())
+
+    # Load evaluation summary
+    print("Loading evaluation summary...")
+    with open(os.path.join(output_folder, "summary.csv")) as f:
+        for line in f:
+            words = line.split(",")
+            if words[0] == "mean":
+                whole = float(words[1])
+                core = float(words[2])
+                enhancing = float(words[3])
+                mean = (whole + core + enhancing) / 3
+                print("Accuracy: mean = {:.5f}, whole tumor = {:.4f}, tumor core = {:.4f}, enhancing tumor = {:.4f}".format(mean, whole, core, enhancing))
+                break
+
+    print("Done!")
+
+if __name__ == "__main__":
+    main()
diff --git a/models_v2/pytorch/3d_unet/inference/gpu/3d-unet/brats_QSL.py b/models_v2/pytorch/3d_unet/inference/gpu/3d-unet/brats_QSL.py
new file mode 100644
index 000000000..7be8e1e3c
--- /dev/null
+++ b/models_v2/pytorch/3d_unet/inference/gpu/3d-unet/brats_QSL.py
@@ -0,0 +1,58 @@
+# coding=utf-8
+# Copyright (c) 2020 NVIDIA CORPORATION. All rights reserved.
+# Copyright 2020 Division of Medical Image Computing, German Cancer Research Center (DKFZ), Heidelberg, Germany
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+import pickle
+import sys
+sys.path.insert(0, os.getcwd())
+
+import mlperf_loadgen as lg
+
+sys.path.insert(0, os.path.join(os.getcwd(), "nnUnet"))
+from nnUnet.nnunet.inference.predict import preprocess_multithreaded
+
+class BraTS_2019_QSL():
+    def __init__(self, preprocessed_data_dir, perf_count):
+        print("Constructing QSL...")
+        self.preprocessed_data_dir = preprocessed_data_dir
+        with open(os.path.join(self.preprocessed_data_dir, "preprocessed_files.pkl"), "rb") as f:
+            self.preprocess_files = pickle.load(f)
+
+        self.count = len(self.preprocess_files)
+        self.perf_count = perf_count if perf_count is not None else self.count
+        print("Found {:d} preprocessed files".format(self.count))
+        print("Using performance count = {:d}".format(self.perf_count))
+
+        self.loaded_files = {}
+        self.qsl = lg.ConstructQSL(self.count, self.perf_count, self.load_query_samples, self.unload_query_samples)
+        print("Finished constructing QSL.")
+
+    def load_query_samples(self, sample_list):
+        for sample_id in sample_list:
+            file_name = self.preprocess_files[sample_id]
+            print("Loading file {:}".format(file_name))
+            with open(os.path.join(self.preprocessed_data_dir, "{:}.pkl".format(file_name)), "rb") as f:
+                self.loaded_files[sample_id] = pickle.load(f)[0]
+
+    def unload_query_samples(self, sample_list):
+        for sample_id in sample_list:
+            del self.loaded_files[sample_id]
+
+    def get_features(self, sample_id):
+        return self.loaded_files[sample_id]
+
+def get_brats_QSL(preprocessed_data_dir="build/preprocessed_data", perf_count=None):
+    return BraTS_2019_QSL(preprocessed_data_dir, perf_count)
diff --git a/models_v2/pytorch/3d_unet/inference/gpu/3d-unet/preprocess.py b/models_v2/pytorch/3d_unet/inference/gpu/3d-unet/preprocess.py
new file mode 100644
index 000000000..39bac4e0a
--- /dev/null
+++ b/models_v2/pytorch/3d_unet/inference/gpu/3d-unet/preprocess.py
@@ -0,0 +1,123 @@
+# coding=utf-8
+# Copyright (c) 2020 NVIDIA CORPORATION. All rights reserved.
+# Copyright 2020 Division of Medical Image Computing, German Cancer Research Center (DKFZ), Heidelberg, Germany
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import argparse
+import numpy
+import os
+import pickle
+import sys
+import torch
+
+sys.path.insert(0, os.path.join(os.getcwd(), "nnUnet"))
+
+from batchgenerators.augmentations.utils import pad_nd_image
+from batchgenerators.utilities.file_and_folder_operations import subfiles
+from nnunet.training.model_restore import load_model_and_checkpoint_files
+from nnunet.inference.predict import preprocess_multithreaded
+
+def get_args():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--model_dir", default="build/result/nnUNet/3d_fullres/Task043_BraTS2019/nnUNetTrainerV2__nnUNetPlansv2.mlperf.1",
+        help="Path to the directory containing plans.pkl")
+    parser.add_argument("--raw_data_dir", default="build/raw_data/nnUNet_raw_data/Task043_BraTS2019/imagesTr",
+        help="Path to the directory containing raw nii.gz files")
+    parser.add_argument("--preprocessed_data_dir", default="build/preprocessed_data", help="Path to the directory containing preprocessed data")
+    parser.add_argument("--validation_fold_file", default="folds/fold1_validation.txt", help="Path to the txt file storing all the sample names for the validation fold")
+    parser.add_argument("--num_threads_preprocessing", type=int, default=12, help="Number of threads to run the preprocessing with")
+    args = parser.parse_args()
+    return args
+
+def preprocess_MLPerf(model, checkpoint_name, folds, fp16, list_of_lists, output_filenames, preprocessing_folder, num_threads_preprocessing):
+    assert len(list_of_lists) == len(output_filenames)
+
+    print("loading parameters for folds", folds)
+    trainer, params = load_model_and_checkpoint_files(model, folds, mixed_precision=fp16, checkpoint_name=checkpoint_name)
+
+    print("starting preprocessing generator")
+    preprocessing = preprocess_multithreaded(trainer, list_of_lists, output_filenames, num_threads_preprocessing, None)
+    print("Preprocessing images...")
+    all_output_files = []
+
+    for preprocessed in preprocessing:
+        output_filename, (d, dct) = preprocessed
+
+        all_output_files.append(output_filename)
+        if isinstance(d, str):
+            data = np.load(d)
+            os.remove(d)
+            d = data
+
+        # Pad to the desired full volume
+        d = pad_nd_image(d, trainer.patch_size, "constant", None, False, None)
+
+        with open(os.path.join(preprocessing_folder, output_filename+ ".pkl"), "wb") as f:
+            pickle.dump([d, dct], f)
+        f.close()
+
+    return  all_output_files
+
+
+def main():
+    args = get_args()
+
+    print("Preparing for preprocessing data...")
+
+    # Validation set is fold 1
+    fold = 1
+    validation_fold_file = args.validation_fold_file
+
+    # Make sure the model exists
+    model_dir = args.model_dir
+    model_path = os.path.join(model_dir, "plans.pkl")
+    assert os.path.isfile(model_path), "Cannot find the model file {:}!".format(model_path)
+    checkpoint_name = "model_final_checkpoint"
+
+    # Other settings
+    fp16 = False
+    num_threads_preprocessing = args.num_threads_preprocessing
+    raw_data_dir = args.raw_data_dir
+    preprocessed_data_dir = args.preprocessed_data_dir
+
+    # Open list containing validation images from specific fold (e.g. 1)
+    validation_files = []
+    with open(validation_fold_file) as f:
+        for line in f:
+            validation_files.append(line.rstrip())
+
+    # Create output and preprocessed directory
+    if not os.path.isdir(preprocessed_data_dir):
+        os.makedirs(preprocessed_data_dir)
+
+    # Create list of images locations (i.e. 4 images per case => 4 modalities)
+    all_files = subfiles(raw_data_dir, suffix=".nii.gz", join=False, sort=True)
+    list_of_lists = [[os.path.join(raw_data_dir, i) for i in all_files if i[:len(j)].startswith(j) and
+                      len(i) == (len(j) + 12)] for j in validation_files]
+
+    # Preprocess images, returns filenames list
+    # This runs in multiprocess
+    print("Acually preprocessing data...")
+    preprocessed_files = preprocess_MLPerf(model_dir, checkpoint_name, fold, fp16, list_of_lists,
+        validation_files, preprocessed_data_dir, num_threads_preprocessing)
+
+    print("Saving metadata of the preprocessed data...")
+    with open(os.path.join(preprocessed_data_dir, "preprocessed_files.pkl"), "wb") as f:
+        pickle.dump(preprocessed_files, f)
+
+    print("Preprocessed data saved to {:}".format(preprocessed_data_dir))
+    print("Done!")
+
+if __name__ == "__main__":
+    main()
diff --git a/models_v2/pytorch/3d_unet/inference/gpu/3d-unet/pytorch_SUT.py b/models_v2/pytorch/3d_unet/inference/gpu/3d-unet/pytorch_SUT.py
new file mode 100644
index 000000000..48bff76f7
--- /dev/null
+++ b/models_v2/pytorch/3d_unet/inference/gpu/3d-unet/pytorch_SUT.py
@@ -0,0 +1,290 @@
+# coding=utf-8
+# Copyright (c) 2020 NVIDIA CORPORATION. All rights reserved.
+# Copyright 2020 Division of Medical Image Computing, German Cancer Research Center (DKFZ), Heidelberg, Germany
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import array
+import json
+import os
+import sys
+import time
+sys.path.insert(0, os.getcwd())
+
+import mlperf_loadgen as lg
+import numpy as np
+import torch
+import intel_extension_for_pytorch
+import torch.nn as nn
+import torch.nn.functional as F
+
+from torch.jit._recursive import wrap_cpp_module
+from torch.quantization.quantize_jit import(
+    convert_jit,
+    prepare_jit
+)
+from torch.quantization import default_qconfig
+torch._C._jit_set_profiling_mode(False)
+torch._C._jit_set_profiling_executor(False)
+from brats_QSL import get_brats_QSL
+
+sys.path.insert(0, os.path.join(os.getcwd(), "nnUnet"))
+from nnunet.training.model_restore import load_model_and_checkpoint_files
+
+traced = False
+
+class _3DUNET_PyTorch_SUT():
+    def __init__(self, model_dir, preprocessed_data_dir, performance_count,
+        run_fp16, run_int8, calib_iters, channels_last, asymm, uint8, profiling, folds,
+        checkpoint_name):
+        print("Loading PyTorch model...")
+        model_path = os.path.join(model_dir, "plans.pkl")
+        assert os.path.isfile(model_path), "Cannot find the model file {:}!".format(model_path)
+        self.trainer, params = load_model_and_checkpoint_files(model_dir, folds, mixed_precision=False, checkpoint_name=checkpoint_name)
+        self.trainer.load_checkpoint_ram(params[0], False)
+        self.device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
+
+        print("Constructing SUT...")
+        self.sut = lg.ConstructSUT(self.issue_queries, self.flush_queries)
+        print("Finished constructing SUT.")
+        self.warmup_iter = 3
+        self.qsl = get_brats_QSL(preprocessed_data_dir, performance_count)
+        self.run_fp16 = run_fp16
+        self.run_int8 = run_int8
+        self.calib_iters = calib_iters
+        self.channels_last = channels_last
+        self.profiling = profiling
+        self.asymm = asymm
+        self.uint8  = uint8
+
+    def jit_calib(self, query_samples):
+        img = self.qsl.get_features(query_samples[0].index)
+        print("tracing sample id {:d} with shape ={:}".format(query_samples[0].index, img.shape))
+        img = torch.from_numpy(img[np.newaxis, ...]).float()
+        self.trainer.network.eval()
+        img = img.to("xpu")
+        model = self.trainer.network.to("xpu")
+        if self.channels_last:
+            model = model.to(memory_format=torch.channels_last_3d)
+            img = img.to(memory_format=torch.channels_last_3d)
+        print("start jit tracing")
+        model = model.to("cpu")
+        img = img.to("cpu")
+        modelJit = torch.jit.trace(model, img)
+        print("cpu trace finished")
+        modelJit = modelJit.to("xpu")
+        modelJit = wrap_cpp_module(modelJit._c) #No folding here
+
+        qscheme = torch.per_tensor_affine if self.asymm else torch.per_tensor_symmetric
+        dtype = torch.quint8 if self.uint8 else torch.qint8
+
+        qconfig = torch.quantization.QConfig(
+            activation=torch.quantization.observer.MinMaxObserver.with_args(
+                qscheme=qscheme,
+                reduce_range=False,
+                dtype=dtype
+            ),
+            weight=torch.quantization.default_weight_observer
+        )
+        modelJit = prepare_jit(modelJit, {'': qconfig}, True)
+        modelJit = modelJit.to('xpu')
+
+        print("calib_iters:", self.calib_iters)
+        for i in range(len(query_samples)):
+            data = self.qsl.get_features(query_samples[i].index)
+            print("calibrating through jit path, sample id {:d} with shape ={:}".format(query_samples[i].index, data.shape))
+            image = torch.from_numpy(data[np.newaxis, ...]).float()
+            calib = image.to("xpu")
+            if self.channels_last:
+                calib = calib.to(memory_format=torch.channels_last_3d)
+            output = modelJit(calib)
+            if i == self.calib_iters - 1:
+                break
+        print("start converting model")
+        modelJit = convert_jit(modelJit)
+        print("converting model ends")
+
+        self.trainer.network = modelJit
+
+        # setting global flag to avoid multiple times of tracing
+        global traced
+        traced = True
+
+    def issue_queries(self, query_samples):
+        if self.run_int8:
+            if not traced:
+                self.jit_calib(query_samples)
+        total_time = 0
+        counter = 0
+        if self.run_int8:
+            with torch.no_grad():
+                for i in range(len(query_samples)):
+                    print("evaluating", i, "th image")
+                    data = self.qsl.get_features(query_samples[i].index)
+                    print("Processing sample id {:d} with shape = {:}".format(query_samples[i].index, data.shape))
+
+                    image = torch.from_numpy(data[np.newaxis,...]).float()
+                    for layer in self.trainer.network.modules():
+                        if isinstance(layer, torch.nn.InstanceNorm3d):
+                            layer.float()
+                    if self.channels_last:
+                        image = image.to(memory_format=torch.channels_last_3d)
+                        print("image to channels last")
+                    try:
+                        import memory_check
+                        memory_check.display_mem("xpu:0")
+                    except:
+                        pass
+                    if self.profiling and (i > self.warmup_iter):
+                        with torch.autograd.profiler_legacy.profile(use_xpu=True, record_shapes=False) as prof:
+                                image = image.to("xpu")
+                                output = self.trainer.network(image)
+                                output = output[0].cpu()
+                                torch.xpu.synchronize()
+                        # output profiling statiscs
+                        profiling_path = os.path.abspath('../../') + '/report/'
+                        torch.save(prof.key_averages().table(), profiling_path + '3dunet_inference_profiling.pt')
+                        prof.export_chrome_trace(profiling_path + '3dunet_inference_profiling.json')
+                        print(prof.key_averages().table(sort_by="self_cpu_time_total"))
+                        print(prof.key_averages(group_by_input_shape=True).table())
+                        print(prof.table(sort_by="id", row_limit=100000))
+                        sys.exit(0)
+                    else:
+                        t0 = time.time()
+                        image = image.to("xpu")
+                        output = self.trainer.network(image)
+                        output = output[0].cpu()
+                        torch.xpu.synchronize()
+                        t1 = time.time()
+                        if i > self.warmup_iter:
+                            total_time += (t1 - t0)
+                            counter += 1
+                    print("evaluated", i, "th image")
+                    output = output.numpy().astype(np.float16)
+
+                    transpose_forward = self.trainer.plans.get("transpose_forward")
+                    transpose_backward = self.trainer.plans.get("transpose_backward")
+                    assert transpose_forward == [0, 1, 2], "Unexpected transpose_forward {:}".format(transpose_forward)
+                    assert transpose_backward == [0, 1, 2], "Unexpected transpose_backward {:}".format(transpose_backward)
+
+                    response_array = array.array("B", output.tobytes())
+                    bi = response_array.buffer_info()
+                    response = lg.QuerySampleResponse(query_samples[i].id, bi[0], bi[1])
+                    lg.QuerySamplesComplete([response])
+            if counter > 0:
+                latency = total_time / counter
+                throughput = counter / total_time
+                print('3dunet_inf throughput: ', throughput, ' sample/s')
+                print('3dunet_inf latency: ', latency, ' s')
+            
+        else:      
+            model = self.trainer.network.to("xpu")
+            model.eval()
+            dtype = torch.float16 if self.run_fp16 else torch.float32
+            model = torch.xpu.optimize(model=model, dtype=dtype, level="O1")
+            for i in range(len(query_samples)):
+                data = self.qsl.get_features(query_samples[i].index)
+                print("calibrating through jit path, sample id {:d} with shape ={:}".format(query_samples[i].index, data.shape))
+            use_autocast = False
+            if self.run_fp16:
+                print('using autocast')
+                use_autocast = True
+
+            autocast_dtype = torch.float32
+            if self.run_fp16:
+                autocast_dtype = torch.float16
+
+            trace_input = torch.randn(1, 4, 224, 224, 160).to("xpu")
+            with torch.no_grad():
+                with torch.xpu.amp.autocast(enabled=use_autocast, dtype=autocast_dtype):
+                    modelJit = torch.jit.trace(model, trace_input)
+                    modelJit = torch.jit.freeze(modelJit)
+
+            with torch.no_grad():
+                for i in range(len(query_samples)):
+                    print("evaluating", i, "th image")
+                    data = self.qsl.get_features(query_samples[i].index)
+                    print("Processing sample id {:d} with shape = {:}".format(query_samples[i].index, data.shape))
+
+                    image = torch.from_numpy(data[np.newaxis,...]).float()
+                    if self.run_fp16:
+                        image = image.half()
+                        self.trainer.network.half()
+                    for layer in self.trainer.network.modules():
+                        if isinstance(layer, torch.nn.InstanceNorm3d):
+                            layer.float()
+                    if self.channels_last:
+                        image = image.to(memory_format=torch.channels_last_3d)
+                        print("image to channels last")
+                    try:
+                        import memory_check
+                        memory_check.display_mem("xpu:0")
+                    except:
+                        pass
+                    if self.profiling and (i > self.warmup_iter):
+                        with torch.autograd.profiler_legacy.profile(use_xpu=True, record_shapes=False) as prof:
+                                image = image.to("xpu")
+                                with torch.xpu.amp.autocast(enabled=use_autocast, dtype=autocast_dtype):
+                                    output = modelJit(image)
+                                output = output[0].cpu()
+                                torch.xpu.synchronize()
+                        # output profiling statiscs
+                        profiling_path = os.path.abspath('../../') + '/report/'
+                        torch.save(prof.key_averages().table(), profiling_path + '3dunet_inference_profiling.pt')
+                        prof.export_chrome_trace(profiling_path + '3dunet_inference_profiling.json')
+                        print(prof.key_averages().table(sort_by="self_cpu_time_total"))
+                        print(prof.key_averages(group_by_input_shape=True).table())
+                        print(prof.table(sort_by="id", row_limit=100000))
+                        sys.exit(0)
+                    else:
+                        t0 = time.time()
+                        image = image.to("xpu")
+                        with torch.xpu.amp.autocast(enabled=use_autocast, dtype=autocast_dtype):
+                            output = modelJit(image)
+                        output = output[0].cpu()
+                        torch.xpu.synchronize()
+                        t1 = time.time()
+                        if i > self.warmup_iter:
+                            total_time += (t1 - t0)
+                            counter += 1
+                    print("evaluated", i, "th image")
+                    output = output.numpy().astype(np.float16)
+
+                    transpose_forward = self.trainer.plans.get("transpose_forward")
+                    transpose_backward = self.trainer.plans.get("transpose_backward")
+                    assert transpose_forward == [0, 1, 2], "Unexpected transpose_forward {:}".format(transpose_forward)
+                    assert transpose_backward == [0, 1, 2], "Unexpected transpose_backward {:}".format(transpose_backward)
+
+                    response_array = array.array("B", output.tobytes())
+                    bi = response_array.buffer_info()
+                    response = lg.QuerySampleResponse(query_samples[i].id, bi[0], bi[1])
+                    lg.QuerySamplesComplete([response])
+            if counter > 0:
+                latency = total_time / counter
+                throughput = counter / total_time
+                print('3dunet_inf throughput: ', throughput, ' sample/s')
+                print('3dunet_inf latency: ', latency, ' s')
+
+    def flush_queries(self):
+        pass
+
+    def process_latencies(self, latencies_ns):
+        pass
+
+
+def get_pytorch_sut(model_dir, preprocessed_data_dir, performance_count, run_fp16,
+    run_int8, calib_iters, channels_last, asymm, uint8, profiling=False, folds=1,
+    checkpoint_name="model_final_checkpoint"):
+    return _3DUNET_PyTorch_SUT(model_dir, preprocessed_data_dir,
+        performance_count, run_fp16, run_int8, calib_iters, channels_last, asymm, uint8,
+        profiling, folds, checkpoint_name)
diff --git a/models_v2/pytorch/3d_unet/inference/gpu/3d-unet/run.py b/models_v2/pytorch/3d_unet/inference/gpu/3d-unet/run.py
new file mode 100644
index 000000000..2c6802a53
--- /dev/null
+++ b/models_v2/pytorch/3d_unet/inference/gpu/3d-unet/run.py
@@ -0,0 +1,181 @@
+# coding=utf-8
+# Copyright (c) 2020 NVIDIA CORPORATION. All rights reserved.
+# Copyright (c) 2020 INTEL CORPORATION. All rights reserved.
+# Copyright 2020 Division of Medical Image Computing, German Cancer Research Center (DKFZ), Heidelberg, Germany
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+import sys
+sys.path.insert(0, os.getcwd())
+
+import argparse
+try:
+    # FIXME: this is a WA: It will fail in Ubuntu 22.04 if mlperf_loadgen is
+    # imported before IPEX.
+    import torch
+    import intel_extension_for_pytorch
+except ImportError:
+    pass
+import mlperf_loadgen as lg
+import subprocess
+
+
+def get_args():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--backend",
+                        choices=["pytorch", "onnxruntime", "tf", "ov"],
+                        default="pytorch",
+                        help="Backend")
+    parser.add_argument(
+        "--scenario",
+        choices=["SingleStream", "Offline", "Server", "MultiStream"],
+        default="Offline",
+        help="Scenario")
+    parser.add_argument("--accuracy",
+                        action="store_true",
+                        help="enable accuracy pass")
+    parser.add_argument("--mlperf_conf",
+                        default="build/mlperf.conf",
+                        help="mlperf rules config")
+    parser.add_argument("--user_conf",
+                        default="user.conf",
+                        help="user config for user LoadGen settings such as target QPS")
+    parser.add_argument(
+        "--model_dir",
+        default=
+        "build/result/nnUNet/3d_fullres/Task043_BraTS2019/nnUNetTrainerV2__nnUNetPlansv2.mlperf.1",
+        help="Path to the directory containing plans.pkl")
+    parser.add_argument("--model", help="Path to the ONNX, OpenVINO, or TF model")
+    parser.add_argument("--preprocessed_data_dir",
+                        default="build/preprocessed_data",
+                        help="path to preprocessed data")
+    parser.add_argument("--performance_count",
+                        type=int,
+                        default=16,
+                        help="performance count")
+    parser.add_argument("--run_fp16",
+                        required=False,
+                        help="Flag for inference in FP16",
+                        action="store_true")
+    parser.add_argument("--run_int8",
+                        required=False,
+                        help="Flag for inference in INT8",
+                        action="store_true")
+    parser.add_argument("--calib_iters",
+                        required=False,
+                        type=int,
+                        help="number of iterations during calibration",
+                        default=1)
+    parser.add_argument("--channels_last",
+                        required=False,
+                        help="Flag to enable chanels-last",
+                        action="store_true")
+    parser.add_argument("--profiling",
+                        required=False,
+                        help="Flag to enabling profiling",
+                        action="store_true")
+    parser.add_argument("--build_dir",
+                        default="build",
+                        help="Path to the build directory")
+    parser.add_argument("--asymm", required=False, action="store_true", help="Flag for asymm quant")
+    parser.add_argument("--uint8", required=False, action="store_true", help="Use uint8 dtype for int8 inference")
+
+    args = parser.parse_args()
+    return args
+
+
+scenario_map = {
+    "SingleStream": lg.TestScenario.SingleStream,
+    "Offline": lg.TestScenario.Offline,
+    "Server": lg.TestScenario.Server,
+    "MultiStream": lg.TestScenario.MultiStream
+}
+
+
+def main():
+    args = get_args()
+
+    if args.backend == "pytorch":
+        from pytorch_SUT import get_pytorch_sut
+        calib_args = None
+        if os.environ.get("PROFILE", "OFF").upper() in ["1", "Y", "ON", "YES", "TRUE"]:
+                args.profiling = True
+        print(args)
+        sut = get_pytorch_sut(args.model_dir, args.preprocessed_data_dir,
+                              args.performance_count, args.run_fp16,
+                              args.run_int8, args.calib_iters,
+                              args.channels_last, args.asymm, args.uint8, args.profiling)
+    elif args.backend == "onnxruntime":
+        from onnxruntime_SUT import get_onnxruntime_sut
+        sut = get_onnxruntime_sut(args.model, args.preprocessed_data_dir,
+                                  args.performance_count)
+    elif args.backend == "tf":
+        from tf_SUT import get_tf_sut
+        sut = get_tf_sut(args.model, args.preprocessed_data_dir,
+                         args.performance_count)
+    elif args.backend == "ov":
+        from ov_SUT import get_ov_sut
+        sut = get_ov_sut(args.model, args.preprocessed_data_dir,
+                         args.performance_count)
+    else:
+        raise ValueError("Unknown backend: {:}".format(args.backend))
+
+    settings = lg.TestSettings()
+    settings.scenario = scenario_map[args.scenario]
+    settings.FromConfig(args.mlperf_conf, "3d-unet", args.scenario)
+    settings.FromConfig(args.user_conf, "3d-unet", args.scenario)
+
+    if args.accuracy:
+        settings.mode = lg.TestMode.AccuracyOnly
+    else:
+        settings.mode = lg.TestMode.PerformanceOnly
+
+    log_path = os.path.join(args.build_dir, "logs")
+    if not os.path.exists(log_path):
+        os.makedirs(log_path)
+    log_output_settings = lg.LogOutputSettings()
+    log_output_settings.outdir = log_path
+    log_output_settings.copy_summary_to_stdout = True
+    log_settings = lg.LogSettings()
+    log_settings.log_output = log_output_settings
+
+    print("Running Loadgen test...")
+    lg.StartTestWithLogSettings(sut.sut, sut.qsl.qsl, settings, log_settings)
+
+    if args.accuracy and (not args.profiling):
+        print("Running accuracy script...")
+        accuracy_log_file = os.path.join(args.build_dir, "logs/mlperf_log_accuracy.json")
+        accuracy_preprocessed_data = os.path.join(args.build_dir, "preprocessed_data")
+        accuracy_postprocessed_data = os.path.join(args.build_dir, "postprocessed_data")
+        accuracy_label_data = os.path.join(args.build_dir, "raw_data/nnUNet_raw_data/Task043_BraTS2019/labelsTr")
+        cmd = "python3 accuracy-brats.py --log_file {} " \
+              "--preprocessed_data_dir {} " \
+              "--postprocessed_data_dir {} " \
+              "--label_data_dir {}".format(accuracy_log_file,
+                                           accuracy_preprocessed_data,
+                                           accuracy_postprocessed_data,
+                                           accuracy_label_data)
+        subprocess.check_call(cmd, shell=True)
+
+    print("Done!")
+
+    print("Destroying SUT...")
+    lg.DestroySUT(sut.sut)
+
+    print("Destroying QSL...")
+    lg.DestroyQSL(sut.qsl.qsl)
+
+
+if __name__ == "__main__":
+    main()
diff --git a/models_v2/pytorch/3d_unet/inference/gpu/3d-unet/user.conf b/models_v2/pytorch/3d_unet/inference/gpu/3d-unet/user.conf
new file mode 100644
index 000000000..17cd838d2
--- /dev/null
+++ b/models_v2/pytorch/3d_unet/inference/gpu/3d-unet/user.conf
@@ -0,0 +1,6 @@
+# Please set these fields depending on the performance of your system to
+# override default LoadGen settings.
+*.SingleStream.target_latency = 10
+*.Server.target_qps = 1.0
+*.Offline.target_qps = 1.0
+*.MultiStream.samples_per_query = 4
diff --git a/models_v2/pytorch/3d_unet/inference/gpu/3dunet.py b/models_v2/pytorch/3d_unet/inference/gpu/3dunet.py
new file mode 100644
index 000000000..99d8b1da4
--- /dev/null
+++ b/models_v2/pytorch/3d_unet/inference/gpu/3dunet.py
@@ -0,0 +1,35 @@
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Consts for 3dunet inference."""
+
+ACC = {
+    "type": "total",
+    "pattern": r"Accuracy: mean = (\d.\d+)",
+    "unit": "mean",
+}
+
+PERF = {
+    "type": "total",
+    "pattern": r"3dunet_inf throughput:  (\d+.\d+)  sample/s",
+    "inverse": False,
+    "multiply": False,
+    "use_batch_size": False,
+    "unit": "sample/s",
+}
+
+FUNCTIONAL = {
+    "pattern": r"No warnings encountered during test."
+}
diff --git a/models_v2/pytorch/3d_unet/inference/gpu/README.md b/models_v2/pytorch/3d_unet/inference/gpu/README.md
new file mode 100644
index 000000000..a3c60db2e
--- /dev/null
+++ b/models_v2/pytorch/3d_unet/inference/gpu/README.md
@@ -0,0 +1,103 @@
+# 3D-UNet Inference
+
+3D-UNet Inference best known configurations with Intel® Extension for PyTorch.
+
+## Model Information
+
+| **Use Case** | **Framework** | **Model Repo** | **Branch/Commit/Tag** | **Optional Patch** |
+|:---:| :---: |:--------------:|:---------------------:|:------------------:|
+|  Inference   |    Pytorch    |       https://github.com/mlcommons/inference/tree/master/vision/medical_imaging/3d-unet-brats19        |           -           |         -          |
+
+# Pre-Requisite
+* Host has Intel® Data Center GPU Max Series - [Intel® Data Center GPU Max Series](https://ark.intel.com/content/www/us/en/ark/products/series/232874/intel-data-center-gpu-max-series.html)
+* Host has installed latest Intel® Data Center GPU Max Series Drivers https://dgpu-docs.intel.com/driver/installation.html
+* The following Intel® oneAPI Base Toolkit components are required:
+  - Intel® oneAPI DPC++ Compiler (Placeholder DPCPPROOT as its installation path)
+  - Intel® oneAPI Math Kernel Library (oneMKL) (Placeholder MKLROOT as its installation path)
+  - Intel® oneAPI MPI Library
+  - Intel® oneAPI TBB Library
+
+  Follow instructions at [Intel® oneAPI Base Toolkit Download page](https://www.intel.com/content/www/us/en/developer/tools/oneapi/base-toolkit-download.html?operatingsystem=linux) to setup the package manager repository.
+
+
+# Prepare Dataset
+* Please download [BraTS 2019](https://www.med.upenn.edu/cbica/brats2019/data.html) separately and unzip the dataset.
+
+## Prepare data file
+* Download the data file from https://github.com/mlcommons/inference/tree/master/vision/medical_imaging/3d-unet-brats19/folds, put them under the folder models_v2/pytorch/3d_unet/inference/gpu/folds 
+
+## Inference
+1. `git clone https://github.com/IntelAI/models.git`
+2. `cd models/models_v2/pytorch/3d_unet/inference/gpu`
+3. Create virtual environment `venv` and activate it:
+    ```
+    python3 -m venv venv
+    . ./venv/bin/activate
+    ```
+4. Run setup.sh
+    ```
+    ./setup.sh
+    ```
+5. Install the latest GPU versions of [torch, torchvision and intel_extension_for_pytorch](https://intel.github.io/intel-extension-for-pytorch/index.html#installation):
+    ```
+    python -m pip install torch==<torch_version> torchvision==<torchvision_version> intel-extension-for-pytorch==<ipex_version> --extra-index-url https://pytorch-extension.intel.com/release-whl-aitools/
+    ```
+6. Set environment variables for Intel® oneAPI Base Toolkit: 
+    Default installation location `{ONEAPI_ROOT}` is `/opt/intel/oneapi` for root account, `${HOME}/intel/oneapi` for other accounts
+    ```bash
+    source {ONEAPI_ROOT}/compiler/latest/env/vars.sh
+    source {ONEAPI_ROOT}/mkl/latest/env/vars.sh
+    source {ONEAPI_ROOT}/tbb/latest/env/vars.sh
+    source {ONEAPI_ROOT}/mpi/latest/env/vars.sh
+    source {ONEAPI_ROOT}/ccl/latest/env/vars.sh
+    ```
+5. Setup required environment paramaters
+
+| **Parameter**                |                                  **export command**                                  |
+|:---------------------------:|:------------------------------------------------------------------------------------:|
+| **MULTI_TILE**               | `export MULTI_TILE=True` (True or False)                                             |
+| **PLATFORM**                 | `export PLATFORM=Max` (Max)                                                 |
+| **DATASET_DIR**              |                               `export DATASET_DIR=`                                  |
+| **PRECISION** (optional)     |                               `export PRECISION=FP16` (FP16, INT8 and FP32 for Max)                               |
+| **OUTPUT_DIR** (optional)    |                               `export OUTPUT_DIR=$PWD`                               |
+6. Run `run_model.sh`
+
+## Output
+
+Single-tile output will typically looks like:
+
+```
+3dunet_inf throughput:  12.63259639639566  sample/s
+3dunet_inf latency:  0.07916029045979182  s
+Done!
+Destroying SUT...
+Destroying QSL...
+```
+
+Multi-tile output will typically looks like:
+```
+3dunet_inf throughput:  12.63259639639566  sample/s
+3dunet_inf latency:  0.07916029045979182  s
+Done!
+Destroying SUT...
+Destroying QSL...
+3dunet_inf throughput:  12.638104577393317  sample/s
+3dunet_inf latency:  0.07912578930457433  s
+Done!
+Destroying SUT...
+Destroying QSL...
+```
+
+Final results of the inference run can be found in `results.yaml` file.
+```
+results:
+ - key: throughput
+   value: 25.2707
+   unit: sample/s
+ - key: latency
+   value: 0.079143
+   unit: s
+ - key: accuracy
+   value: None
+   unit: mean
+```
diff --git a/models_v2/pytorch/3d_unet/inference/gpu/requirements.txt b/models_v2/pytorch/3d_unet/inference/gpu/requirements.txt
new file mode 100644
index 000000000..f8abfaec5
--- /dev/null
+++ b/models_v2/pytorch/3d_unet/inference/gpu/requirements.txt
@@ -0,0 +1,7 @@
+batchgenerators==0.23
+SimpleITK
+pandas
+tqdm
+absl-py
+nnunet==1.7.1
+pybind11==2.11.1
diff --git a/models_v2/pytorch/3d_unet/inference/gpu/run_model.sh b/models_v2/pytorch/3d_unet/inference/gpu/run_model.sh
new file mode 100755
index 000000000..5ca83df2e
--- /dev/null
+++ b/models_v2/pytorch/3d_unet/inference/gpu/run_model.sh
@@ -0,0 +1,153 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+
+#!/bin/bash
+
+# Create an array of input directories that are expected and then verify that they exist
+declare -A input_envs
+input_envs[DATASET_DIR]=${DATASET_DIR}
+input_envs[MULTI_TILE]=${MULTI_TILE}
+input_envs[PLATFORM]=${PLATFORM}
+
+for i in "${!input_envs[@]}"; do
+  var_name=$i
+  env_param=${input_envs[$i]}
+
+  if [[ -z $env_param ]]; then
+    echo "The required environment variable $var_name is not set" >&2
+    exit 1
+  fi
+done
+
+OUTPUT_DIR=${OUTPUT_DIR:-$PWD}
+
+if [[ "${PLATFORM}" == "Max" ]]; then
+    PRECISION=${PRECISION:-FP16}
+elif [[ "${PLATFORM}" == "Flex" ]]; then
+    echo "Only support Max for platform"
+fi
+
+if [[ ! -d "${DATASET_DIR}" ]]; then
+    echo "The DATASET_DIR '${DATASET_DIR}' does not exist"
+    exit 1
+else
+    export DOWNLOAD_DATA_DIR=${DATASET_DIR}
+fi
+BATCH_SIZE=1
+git clone --recurse-submodules https://github.com/mlcommons/inference.git mlperf_inference
+cd mlperf_inference/loadgen
+python setup.py install
+cd ../../3d-unet
+git clone https://github.com/MIC-DKFZ/nnUNet.git nnUnet -b v1.7.1
+
+make setup
+make preprocess_data
+make mkdir_postprocessed_data
+
+
+# known issue
+if [[ "${MULTI_TILE}" == "True" ]]; then
+    export ZE_FLAT_DEVICE_HIERARCHY=COMPOSITE
+fi
+
+echo 'Running with parameters:'
+echo " PLATFORM: ${PLATFORM}"
+echo " DATASET_DIR: ${DATASET_DIR}"
+echo " OUTPUT_DIR: ${OUTPUT_DIR}"
+echo " PRECISION: ${PRECISION}"
+echo " BATCH_SIZE: ${BATCH_SIZE}"
+echo " MULTI_TILE: ${MULTI_TILE}"
+
+if [[ "${PRECISION}" == "INT8" ]]; then
+    flag="run_pytorch_performance_int8"
+    IPEX_XPU_ONEDNN_LAYOUT=1
+elif [[ "${PRECISION}" == "FP32" ]]; then
+    flag="run_pytorch_performance"
+elif [[ "${PRECISION}" == "FP16" ]]; then
+    flag="run_pytorch_performance_fp16"
+else
+    echo -e "Invalid input! Only FP32 FP16 INT8 are supported."
+    exit 1
+fi
+echo "3DUNet ${PRECISION} inference plain MultiTile=${MULTI_TILE} BS=${BATCH_SIZE}"
+
+# Create the output directory, if it doesn't already exist
+mkdir -p $OUTPUT_DIR
+
+sum_log_analysis() {
+    if [ -f $2 ]; then
+        rm -f $2
+    fi
+    if diff /dev/null ${1}_t0.log |tail -l | grep '^\\ No newline' > /dev/null;then echo >> ${1}_t0.log; fi
+    if diff /dev/null ${1}_t1.log |tail -l | grep '^\\ No newline' > /dev/null;then echo >> ${1}_t1.log; fi
+    bs=$(cat ${1}_t1.log |grep Batch |awk '{print $3}')
+    echo -e "Batch Size: $bs" >$2
+    cat ${1}"_t0.log" ${1}"_t1.log" |grep "Performance" |awk -v tag=$(cat ${1}"_t0.log" ${1}"_t1.log" |grep "Performance" |awk '{sum+=$2} END {printf "%.4f\n",sum}') '{if ( $2=="None" ) {sum="None";nextfile}else  sum=tag} ;END{print "Sum "$1" "sum " "$3}' >> $2
+    cat ${1}"_t0.log" ${1}"_t1.log" |grep "Performance" |awk -v tag=$(cat ${1}"_t0.log" ${1}"_t1.log" |grep "Performance" |awk 'BEGIN {min=1234567890123} {if ($2 <min) {min=$2}}END {printf "%.4f\n",min}') '{if ( $2=="None" ) {min="None";nextfile}else  min=tag} ;END{print "Min "$1" "min " "$3}' >> $2
+    cat ${1}"_t0.log" ${1}"_t1.log" |grep "Latency" |awk '{if ( $2=="N/A" ){avg="N/A";nextfile}else avg=((sum+=$2/2))};END{print "Avg "$1" "avg " "$3}'  >> $2
+    cat ${1}"_t0.log" ${1}"_t1.log" |grep "Accuracy" |awk -v avg=$(cat ${1}"_t0.log" ${1}"_t1.log" |grep "Accuracy" |awk '{sum+=$3}END{printf "%.4f\n",sum/NR}') '{if ( $3=="None" || $2=="N/A" || $3=="nan" || $3=="N/A"){avg="None";nextfile}else avg=avg};END{print "Avg "$1" "$2 " "avg}' >> $2
+    cat ${1}"_t0.log" ${1}"_t1.log" |grep "Functional" | awk -v fail=$(cat ${1}"_t0.log" ${1}"_t1.log" |grep "Functional" |awk '{for(i=1;i<=NF;++i) if($i=="fail") ++sum}END{print sum}') '{if ( fail >= 1 ) tag="fail ";else tag="pass"};END{print $1" "tag}'  >> $2
+    cat ${1}"_t0.log" ${1}"_t1.log" |grep "Error" |awk '{if(a[$1]){a[$1]=a[$1]";"$2}else{a[$1]=$2}}END{for(i in a)print $1" " a[i]}' >> $2
+}
+
+modelname=3dunet
+if [[ ${MULTI_TILE} == "False" ]]; then
+    rm ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0_raw.log
+    make ${flag} 2>&1 | tee ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0_raw.log
+    cd ..
+    python common/parse_result.py -m $modelname -l ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0_raw.log -b ${BATCH_SIZE}
+    throughput=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0.log | grep Performance | awk -F ' ' '{print $2}')
+    throughput_unit=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0.log | grep Performance | awk -F ' ' '{print $3}')
+    latency=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0.log | grep Latency | awk -F ' ' '{print $2}')
+    acc=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0.log | grep Accuracy | awk -F ' ' '{print $3}')
+    acc_unit=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0.log | grep Accuracy | awk -F ' ' '{print $2}')
+else
+    rm ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0_raw.log
+    rm ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t1_raw.log
+    ZE_AFFINITY_MASK=0.0 make ${flag} 2>&1 | tee ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0_raw.log &
+    ZE_AFFINITY_MASK=0.1 make ${flag} 2>&1 | tee ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t1_raw.log
+    wait
+    cd ..
+    python common/parse_result.py -m $modelname -l ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0_raw.log -b ${BATCH_SIZE}
+    python common/parse_result.py -m $modelname -l ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t1_raw.log -b ${BATCH_SIZE}
+    sum_log_analysis ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf.log
+    throughput=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf.log | grep "Sum Performance" | awk -F ' ' '{print $3}')
+    throughput_unit=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf.log | grep "Sum Performance" | awk -F ' ' '{print $4}')
+    latency=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf.log | grep Latency | awk -F ' ' '{print $3}')
+    acc=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf.log | grep Accuracy | awk -F ' ' '{print $4}')
+    acc_unit=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf.log | grep Accuracy | awk -F ' ' '{print $3}')
+fi
+
+yaml_content=$(cat <<EOF
+results:
+ - key: throughput
+   value: $throughput
+   unit: $throughput_unit
+ - key: latency
+   value: $latency
+   unit: s
+ - key: accuracy
+   value: $acc
+   unit: $acc_unit
+EOF
+)
+
+# Write the content to a YAML file
+echo "$yaml_content" >  ${OUTPUT_DIR}/results.yaml
+echo "YAML file created."
diff --git a/models_v2/pytorch/3d_unet/inference/gpu/setup.sh b/models_v2/pytorch/3d_unet/inference/gpu/setup.sh
new file mode 100755
index 000000000..5d0a40aab
--- /dev/null
+++ b/models_v2/pytorch/3d_unet/inference/gpu/setup.sh
@@ -0,0 +1,33 @@
+#!/usr/bin/env bash
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# setup.sh
+#  - install OS pkgs
+#  - should create virtual env & install pip  requirement.txt
+#  - git clones & applying patches
+
+set -e
+apt-get update && apt-get install -y python3-venv protobuf-compiler
+
+pip install -r requirements.txt
+
+mkdir -p 3d-unet/build/result
+cd 3d-unet/build/result
+wget https://zenodo.org/record/3904106/files/fold_1.zip
+cd ../../..
+
+cp -r ../../../../common .
diff --git a/models_v2/pytorch/bert_large/inference/gpu/.gitignore b/models_v2/pytorch/bert_large/inference/gpu/.gitignore
new file mode 100644
index 000000000..c769faa86
--- /dev/null
+++ b/models_v2/pytorch/bert_large/inference/gpu/.gitignore
@@ -0,0 +1,2 @@
+huggingface/
+*.tar.gz
diff --git a/models_v2/pytorch/bert_large/inference/gpu/CONTAINER.md b/models_v2/pytorch/bert_large/inference/gpu/CONTAINER.md
new file mode 100644
index 000000000..ae3f03007
--- /dev/null
+++ b/models_v2/pytorch/bert_large/inference/gpu/CONTAINER.md
@@ -0,0 +1,108 @@
+# Running BERT Large inference on Intel® Data Center GPU Max Series using Intel® Extension for PyTorch*
+
+## Description
+
+This document has instructions for running BERT Large Inference with FP16, BF16 and FP32 precisions using Intel(R) Extension for PyTorch on Intel Max Series GPU. 
+
+
+## Requirements
+| Item | Detail |
+| ------ | ------- |
+| Host machine  | [Intel® Data Center GPU Max Series](https://ark.intel.com/content/www/us/en/ark/products/series/232874/intel-data-center-gpu-max-series.html) |
+| Drivers | GPU-compatible drivers need to be installed: Download [Driver](https://dgpu-docs.intel.com/driver/installation.html) |
+| Software | Docker* |
+
+## Quick Start Scripts
+
+| Script name | Description |
+|-------------|-------------|
+| `run_model.sh` |  BERT Large FP16, BF16 and FP32 inference on single tile and two tiles |
+
+## Datasets
+
+### SQuAD dataset
+
+Download the [SQuAD 1.0 dataset](https://github.com/huggingface/transformers/tree/v4.0.0/examples/question-answering#fine-tuning-bert-on-squad10).
+Set the `DATASET_DIR` to point to the directory where the files are located before running the BERT quickstart scripts. Your dataset directory should look something
+like this:
+```bash
+<DATASET_DIR>/
+├── dev-v1.1.json
+├── evaluate-v1.1.py
+└── train-v1.1.json
+```
+The setup assumes the dataset is downloaded to the current directory. 
+
+## Pre-trained Model
+
+Download the `config.json` and fine tuned model from huggingface and set the `BERT_WEIGHT` environment variable to point to the directory that has both files:
+
+```bash
+mkdir squad_large_finetuned_checkpoint
+wget -c https://huggingface.co/bert-large-uncased-whole-word-masking-finetuned-squad/resolve/main/config.json -O squad_large_finetuned_checkpoint/config.json
+wget -c https://huggingface.co/bert-large-uncased-whole-word-masking-finetuned-squad/resolve/main/pytorch_model.bin  -O squad_large_finetuned_checkpoint/pytorch_model.bin
+wget -c https://huggingface.co/bert-large-uncased-whole-word-masking-finetuned-squad/resolve/main/tokenizer.json -O squad_large_finetuned_checkpoint/tokenizer.json
+wget -c https://huggingface.co/bert-large-uncased-whole-word-masking-finetuned-squad/resolve/main/tokenizer_config.json -O squad_large_finetuned_checkpoint/tokenizer_config.json
+wget -c https://huggingface.co/bert-large-uncased-whole-word-masking-finetuned-squad/resolve/main/vocab.txt -O squad_large_finetuned_checkpoint/vocab.txt
+
+BERT_WEIGHT=$(pwd)/squad_large_finetuned_checkpoint
+```
+
+## Docker pull Command
+
+```bash
+docker pull intel/language-modeling:pytorch-max-gpu-bert-large-inference
+```
+
+The BERT Large inference container includes scripts, models,libraries needed to run inference To run the `fp16_inference_plain_format.sh` quick start script follow the instructions below.
+
+```bash
+#Optional
+export PRECISION=<provide FP16 ,BF16 or FP32,otherwise (default:BF16)>
+export BATCH_SIZE=<provide batch size,otherwise (default: 256)>
+export NUM_ITERATIONS=<provide num_iterations,otherwise (default: 1)>
+
+#Required
+export MULTI_TILE=<provide True or False to enable/disable multi-tile inference>
+export DATASET_DIR=<path to dataset>
+export OUTPUT_DIR=<path to output log files>
+export BERT_WEIGHT=$(pwd)/squad_large_finetuned_checkpoint
+export PLATFORM=Max
+
+
+DOCKER_ARGS="--rm -it"
+IMAGE_NAME=intel/language-modeling:pytorch-max-gpu-bert-large-inference
+SCRIPT=run_model.sh
+
+docker run \
+  --privileged \
+  --device=/dev/dri \
+  --ipc=host \
+  --env DATASET_DIR=${DATASET_DIR} \
+  --env BERT_WEIGHT=${BERT_WEIGHT} \
+  --env OUTPUT_DIR=${OUTPUT_DIR} \
+  --env NUM_ITERATIONS=${NUM_ITERATIONS} \
+  --env PRECISION=${PRECISION} \
+  --env BATCH_SIZE=${BATCH_SIZE} \
+  --env PLATFORM=${PLATFORM} \
+  --env MULTI_TILE=${MULTI_TILE} \
+  --env http_proxy=${http_proxy} \
+  --env https_proxy=${https_proxy} \
+  --env no_proxy=${no_proxy} \
+  --volume ${DATASET_DIR}:${DATASET_DIR} \
+  --volume ${BERT_WEIGHT}:${BERT_WEIGHT} \
+  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
+  ${DOCKER_ARGS} \
+  $IMAGE_NAME \
+  /bin/bash $SCRIPT
+  ```
+## Documentation and Sources
+
+[GitHub* Repository](https://github.com/IntelAI/models/tree/master/docker/max-gpu)
+
+## Support
+Support for Intel® Extension for PyTorch* is found via the [Intel® AI Analytics Toolkit.](https://www.intel.com/content/www/us/en/developer/tools/oneapi/ai-analytics-toolkit.html#gs.qbretz) Additionally, the Intel® Extension for PyTorch* team tracks both bugs and enhancement requests using [GitHub issues](https://github.com/intel/intel-extension-for-pytorch/issues). Before submitting a suggestion or bug report, please search the GitHub issues to see if your issue has already been reported.
+
+## License Agreement
+
+LEGAL NOTICE: By accessing, downloading or using this software and any required dependent software (the “Software Package”), you agree to the terms and conditions of the software license agreements for the Software Package, which may also include notices, disclaimers, or license terms for third party software included with the Software Package. Please refer to the [license file](https://github.com/IntelAI/models/tree/master/third_party) for additional details.
diff --git a/models_v2/pytorch/bert_large/inference/gpu/README.md b/models_v2/pytorch/bert_large/inference/gpu/README.md
new file mode 100644
index 000000000..7abd98eea
--- /dev/null
+++ b/models_v2/pytorch/bert_large/inference/gpu/README.md
@@ -0,0 +1,121 @@
+# Bert Large Inference
+
+BERT Large Inference best known configurations with Intel® Extension for PyTorch.
+
+## Model Information
+
+| **Use Case** | **Framework** | **Model Repo** | **Branch/Commit/Tag** | **Optional Patch** |
+|:---:| :---: |:--------------:|:---------------------:|:------------------:|
+|  Inference   |    PyTorch    |       https://github.com/huggingface/transformers/tree/main/src/transformers/models/bert        |           -           |         -          |
+
+# Pre-Requisite
+* Host has one of the following GPUs:
+  * **Arc Series** - [Intel® Arc™ A-Series Graphics](https://ark.intel.com/content/www/us/en/ark/products/series/227957/intel-arc-a-series-graphics.html)
+  * **Max Series** - [Intel® Data Center GPU Max Series](https://ark.intel.com/content/www/us/en/ark/products/series/232874/intel-data-center-gpu-max-series.html)
+* Host has installed latest Intel® Data Center GPU Max & ARC Series Drivers https://dgpu-docs.intel.com/driver/installation.html
+* The following Intel® oneAPI Base Toolkit components are required:
+  - Intel® oneAPI DPC++ Compiler (Placeholder DPCPPROOT as its installation path)
+  - Intel® oneAPI Math Kernel Library (oneMKL) (Placeholder MKLROOT as its installation path)
+  - Intel® oneAPI MPI Library
+  - Intel® oneAPI TBB Library
+  - Intel® oneAPI CCL Library
+
+  Follow instructions at [Intel® oneAPI Base Toolkit Download page](https://www.intel.com/content/www/us/en/developer/tools/oneapi/base-toolkit-download.html?operatingsystem=linux) to setup the package manager repository.
+
+
+## Datasets
+
+### SQuAD dataset
+
+Download the [SQuAD 1.0 dataset](https://github.com/huggingface/transformers/tree/v4.0.0/examples/question-answering#fine-tuning-bert-on-squad10).
+Set the `DATASET_DIR` to point to the directory where the files are located before running the BERT quickstart scripts. Your dataset directory should look something
+like this:
+```bash
+<DATASET_DIR>/
+├── dev-v1.1.json
+├── evaluate-v1.1.py
+└── train-v1.1.json
+```
+The setup assumes the dataset is downloaded to the current directory. 
+
+## Pre-trained Model
+
+Download the `config.json` and fine tuned model from huggingface and set the `BERT_WEIGHT` environment variable to point to the directory that has both files:
+
+```bash
+mkdir squad_large_finetuned_checkpoint
+wget -c https://huggingface.co/bert-large-uncased-whole-word-masking-finetuned-squad/resolve/main/config.json -O squad_large_finetuned_checkpoint/config.json
+wget -c https://huggingface.co/bert-large-uncased-whole-word-masking-finetuned-squad/resolve/main/pytorch_model.bin  -O squad_large_finetuned_checkpoint/pytorch_model.bin
+wget -c https://huggingface.co/bert-large-uncased-whole-word-masking-finetuned-squad/resolve/main/tokenizer.json -O squad_large_finetuned_checkpoint/tokenizer.json
+wget -c https://huggingface.co/bert-large-uncased-whole-word-masking-finetuned-squad/resolve/main/tokenizer_config.json -O squad_large_finetuned_checkpoint/tokenizer_config.json
+wget -c https://huggingface.co/bert-large-uncased-whole-word-masking-finetuned-squad/resolve/main/vocab.txt -O squad_large_finetuned_checkpoint/vocab.txt
+
+BERT_WEIGHT=<path_to_BERT_WEIGHT_directory>/squad_large_finetuned_checkpoint
+```
+
+## Inference
+1. `git clone https://github.com/IntelAI/models.git`
+2. `cd models/models_v2/pytorch/bert_large/inference/gpu`
+3. Create virtual environment `venv` and activate it:
+    ```
+    python3 -m venv venv
+    . ./venv/bin/activate
+    ```
+4. Run setup.sh
+    ```
+    ./setup.sh
+    ```
+5. Install the latest GPU versions of [torch, torchvision and intel_extension_for_pytorch](https://intel.github.io/intel-extension-for-pytorch/index.html#installation):
+    ```
+    python -m pip install torch==<torch_version> torchvision==<torchvvision_version> intel-extension-for-pytorch==<ipex_version> --extra-index-url https://pytorch-extension.intel.com/release-whl-aitools/
+    ```
+6. Set environment variables for Intel® oneAPI Base Toolkit: 
+    Default installation location `{ONEAPI_ROOT}` is `/opt/intel/oneapi` for root account, `${HOME}/intel/oneapi` for other accounts
+    ```bash
+    source {ONEAPI_ROOT}/compiler/latest/env/vars.sh
+    source {ONEAPI_ROOT}/mkl/latest/env/vars.sh
+    source {ONEAPI_ROOT}/tbb/latest/env/vars.sh
+    source {ONEAPI_ROOT}/mpi/latest/env/vars.sh
+    source {ONEAPI_ROOT}/ccl/latest/env/vars.sh
+    ```
+7. Setup required environment paramaters
+
+| **Parameter**                |                                  **export command**                                  |
+|:---------------------------:|:------------------------------------------------------------------------------------:|
+| **MULTI_TILE**               | `export MULTI_TILE=True` (True or False)                                             |
+| **PLATFORM**                 | `export PLATFORM=Max` (Max or ARC)                                                 |
+| **BERT_WEIGHT**              | `export BERT_WEIGHT=<path_to_BERT_WEIGHT_directory>/squad_large_finetuned_checkpoint` |
+| **DATASET_DIR**              | `export DATASET_DIR=<path/to/dataset>`                  |
+| **BATCH_SIZE** (optional)    |                               `export BATCH_SIZE=256`                                |
+| **PRECISION** (optional)     |                  `export PRECISION=BF16` (BF16, FP32 and FP16 are supported for Max and FP16 for ARC) |
+| **OUTPUT_DIR** (optional)    |                               `export OUTPUT_DIR=$PWD`                               |
+|**NUM_ITERATIONS** (optional) |                               `export NUM_ITERATIONS=-1`                             |
+8. Run `run_model.sh`
+
+## Output
+
+Single-tile output will typically looks like:
+
+```
+2023-11-15 06:22:47,398 - __main__ - INFO - Results: {'exact': 87.01040681173131, 'f1': 93.17865304772475, 'total': 10570, 'HasAns_exact': 87.01040681173131, 'HasAns_f1': 93.17865304772475, 'HasAns_total': 10570, 'best_exact': 87.01040681173131, 'best_exact_thresh': 0.0, 'best_f1': 93.17865304772475, 'best_f1_thresh': 0.0}
+```
+
+Multi-tile output will typically looks like:
+```
+2023-11-15 06:29:34,737 - __main__ - INFO - Results: {'exact': 87.01040681173131, 'f1': 93.17865304772475, 'total': 10570, 'HasAns_exact': 87.01040681173131, 'HasAns_f1': 93.17865304772475, 'HasAns_total': 10570, 'best_exact': 87.01040681173131, 'best_exact_thresh': 0.0, 'best_f1': 93.17865304772475, 'best_f1_thresh': 0.0}
+2023-11-15 06:29:35,599 - __main__ - INFO - Results: {'exact': 87.01040681173131, 'f1': 93.17865304772475, 'total': 10570, 'HasAns_exact': 87.01040681173131, 'HasAns_f1': 93.17865304772475, 'HasAns_total': 10570, 'best_exact': 87.01040681173131, 'best_exact_thresh': 0.0, 'best_f1': 93.17865304772475, 'best_f1_thresh': 0.0}
+```
+
+Final results of the inference run can be found in `results.yaml` file.
+```
+results:
+ - key: throughput
+   value: 405.9567
+   unit: sent/s
+ - key: latency
+   value: 0.15765228112538657
+   unit: s
+ - key: accuracy
+   value: 93.179
+   unit: f1
+```
diff --git a/models_v2/pytorch/bert_large/inference/gpu/bertsquad.py b/models_v2/pytorch/bert_large/inference/gpu/bertsquad.py
new file mode 100644
index 000000000..ee3a77222
--- /dev/null
+++ b/models_v2/pytorch/bert_large/inference/gpu/bertsquad.py
@@ -0,0 +1,35 @@
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Consts for bertsquad inference."""
+
+ACC = {
+    "type": "total",
+    "pattern": r"\'f1\': (\d+.\d+)",
+    "unit": "f1",
+}
+
+PERF = {
+    "type": "total",
+    "pattern": r"bert_inf throughput:  (\d+.\d+)  sentences/s",
+    "inverse": False,
+    "multiply": False,
+    "use_batch_size": False,
+    "unit": "sent/s",
+}
+
+FUNCTIONAL = {
+    "pattern": r"\'f1\': \d+.\d+"
+}
diff --git a/models_v2/pytorch/bert_large/inference/gpu/cmd_infer.sh b/models_v2/pytorch/bert_large/inference/gpu/cmd_infer.sh
new file mode 100755
index 000000000..82c961722
--- /dev/null
+++ b/models_v2/pytorch/bert_large/inference/gpu/cmd_infer.sh
@@ -0,0 +1,113 @@
+#!/bin/bash
+# Copyright (c) 2022-2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+function Parser() {
+    while [ $# -ne 0 ]; do
+        case $1 in
+            -m)
+                shift
+                MODEL="$1"
+                ;;
+            -d)
+                shift
+                DEVICE="$1"
+                ;;
+            -b)
+                shift
+                BATCH="$1"
+                ;;
+            -t)
+                shift
+                DTYPE="$1"
+                ;;
+            -n)
+                shift
+                if [[ $1 -gt 0 ]];then
+                   NUM_ITER="$1"
+                fi
+                ;;
+            -o)
+                shift
+                OUTPUT_DIR="$1"
+                ;;
+            -g)
+                GDB_ARGS="gdb --args "
+                ;;
+            -h | --help)
+                echo "Usage: cmd_infer.sh [OPTION...] PAGE..."
+                echo "-m, Optional    Specify the model type[bert_base or bert_large]. The default value is bert_base"
+                echo "-d, Optional    Specify the device[cpu, xpu]. The default device is cpu"
+                echo "-b, Optional    Specify the batch size. The default value is 32"
+                echo "-t, Optional    Specify the dtype[FP32, FP16...]. The default value is FP32"
+                echo "-n, Optional    Specify the number of iterations to run evaluation"
+                echo "-o, Optional    Specify the output dir. The default value is /tmp/debug_squad/"
+                echo "-g, Optional    use gdb"
+                exit
+                ;;
+            --*|-*)
+                echo ">>> New param: <$1>"
+                ;;
+            *)
+                echo ">>> Parsing mismatch: $1"
+                ;;
+        esac
+        shift
+    done
+}
+
+MODEL="bert_base"
+DEVICE=cpu
+BATCH=32
+DTYPE=FP32
+NUM_ITER=-1
+OUTPUT_DIR=$OUTPUT_DIR
+GDB_ARGS=""
+NUMA_ARGS=""
+
+Parser $@
+
+if command -v numactl >& /dev/null ; then
+if [ "x$MPI_LOCALRANKID" != "x" ] ; then
+  REAL_NUM_NUMA_NODES=`lscpu | grep "NUMA node(s):" | awk '{print $NF}'`
+  PPNUMA=$(( MPI_LOCALNRANKS / REAL_NUM_NUMA_NODES ))
+  if [ $PPNUMA -eq 0 ] ; then 
+    if [ "x$SINGLE_SOCKET_ONLY" == "x1" ] ; then 
+      NUMA_ARGS="numactl -m 0 "
+    fi
+  else
+    NUMARANK=$(( MPI_LOCALRANKID / PPNUMA ))
+    NUMA_ARGS="$NUMA_ARGS $GDB_ARGS "
+  fi
+  NUM_RANKS=$PMI_SIZE
+else
+  NUMA_ARGS="numactl -m 0 "
+  NUM_RANKS=1
+fi
+fi
+
+$NUMA_RAGS $GDB_ARGS python -u run_squad.py \
+  --model_type bert \
+  --model_name_or_path $BERT_WEIGHT \
+  --do_eval \
+  --do_lower_case \
+  --do_jit \
+  --device_choice ${DEVICE} \
+  --dtype ${DTYPE}    \
+  --predict_file $DATASET_DIR/dev-v1.1.json \
+  --per_gpu_eval_batch_size ${BATCH} \
+  --max_seq_length 384 \
+  --doc_stride 128 \
+  --num_iterations ${NUM_ITER} \
+  --output_dir ${OUTPUT_DIR}
diff --git a/models_v2/pytorch/bert_large/inference/gpu/requirements.txt b/models_v2/pytorch/bert_large/inference/gpu/requirements.txt
new file mode 100644
index 000000000..589a226c5
--- /dev/null
+++ b/models_v2/pytorch/bert_large/inference/gpu/requirements.txt
@@ -0,0 +1,2 @@
+transformers==4.36.2
+tensorboardX
diff --git a/models_v2/pytorch/bert_large/inference/gpu/run_model.sh b/models_v2/pytorch/bert_large/inference/gpu/run_model.sh
new file mode 100755
index 000000000..b4c45e8a2
--- /dev/null
+++ b/models_v2/pytorch/bert_large/inference/gpu/run_model.sh
@@ -0,0 +1,168 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+
+#!/bin/bash
+
+# Create an array of input directories that are expected and then verify that they exist
+declare -A input_envs
+input_envs[MULTI_TILE]=${MULTI_TILE}
+input_envs[PLATFORM]=${PLATFORM}
+input_envs[BERT_WEIGHT]=${BERT_WEIGHT}
+input_envs[DATASET_DIR]=${DATASET_DIR}
+input_envs[OUTPUT_DIR]=${OUTPUT_DIR}
+
+for i in "${!input_envs[@]}"; do
+  var_name=$i
+  env_param=${input_envs[$i]}
+
+  if [[ -z $env_param ]]; then
+    echo "The required environment variable $var_name is not set" >&2
+    exit 1
+  fi
+done
+
+if [[ ! -d "${BERT_WEIGHT}" ]]; then
+  echo "The DATASET_DIR '${BERT_WEIGHT}' does not exist"
+  exit 1
+fi
+
+if [[ ! -d "${DATASET_DIR}" ]]; then
+  echo "The DATASET_DIR '${DATASET_DIR}' does not exist"
+  exit 1
+fi
+
+mkdir -p ${OUTPUT_DIR}
+
+if [[ "${PLATFORM}" == "Max" ]]; then
+    BATCH_SIZE=${BATCH_SIZE:-256}
+    PRECISION=${PRECISION:-BF16}
+    NUM_ITERATIONS=${NUM_ITERATIONS:--1}
+elif [[ "${PLATFORM}" == "Flex" ]]; then
+    echo "only support Max for platform"
+elif [[ "${PLATFORM}" == "ARC" ]]; then
+    if [[ "${MULTI_TILE}" == "True" ]]; then
+        echo "Only support MULTI_TILE=False when in arc platform"
+        exit 1
+    fi
+    BATCH_SIZE=${BATCH_SIZE:-64}
+    PRECISION=${PRECISION:-FP16}
+    NUM_ITERATIONS=${NUM_ITERATIONS:--1}
+fi
+
+# known issue
+if [[ "${MULTI_TILE}" == "True" ]]; then
+    export ZE_FLAT_DEVICE_HIERARCHY=COMPOSITE
+fi
+
+echo 'Running with parameters:'
+echo " PLATFORM: ${PLATFORM}"
+echo " OUTPUT_DIR: ${OUTPUT_DIR}"
+echo " PRECISION: ${PRECISION}"
+echo " BATCH_SIZE: ${BATCH_SIZE}"
+echo " NUM_ITERATIONS: ${NUM_ITERATIONS}"
+echo " MULTI_TILE: ${MULTI_TILE}"
+
+if [[ "${PRECISION}" != "BF16" ]] && [[ "${PRECISION}" != "FP32" ]] && [[ "${PRECISION}" != "FP16" ]]; then
+    echo -e "Invalid input! Only BF16 FP32 FP16 are supported."
+    exit 1
+fi
+echo "bert-large ${PRECISION} inference plain MultiTile=${MULTI_TILE} BS=${BATCH_SIZE} Iter=${NUM_ITERATIONS}"
+
+# Create the output directory, if it doesn't already exist
+mkdir -p $OUTPUT_DIR
+
+sum_log_analysis() {
+    if [ -f $2 ]; then
+        rm -f $2
+    fi
+    if diff /dev/null ${1}_t0.log |tail -l | grep '^\\ No newline' > /dev/null;then echo >> ${1}_t0.log; fi
+    if diff /dev/null ${1}_t1.log |tail -l | grep '^\\ No newline' > /dev/null;then echo >> ${1}_t1.log; fi
+    bs=$(cat ${1}_t1.log |grep Batch |awk '{print $3}')
+    echo -e "Batch Size: $bs" >$2
+    cat ${1}"_t0.log" ${1}"_t1.log" |grep "Performance" |awk -v tag=$(cat ${1}"_t0.log" ${1}"_t1.log" |grep "Performance" |awk '{sum+=$2} END {printf "%.4f\n",sum}') '{if ( $2=="None" ) {sum="None";nextfile}else  sum=tag} ;END{print "Sum "$1" "sum " "$3}' >> $2
+    cat ${1}"_t0.log" ${1}"_t1.log" |grep "Performance" |awk -v tag=$(cat ${1}"_t0.log" ${1}"_t1.log" |grep "Performance" |awk 'BEGIN {min=1234567890123} {if ($2 <min) {min=$2}}END {printf "%.4f\n",min}') '{if ( $2=="None" ) {min="None";nextfile}else  min=tag} ;END{print "Min "$1" "min " "$3}' >> $2
+    cat ${1}"_t0.log" ${1}"_t1.log" |grep "Latency" |awk '{if ( $2=="N/A" ){avg="N/A";nextfile}else avg=((sum+=$2/2))};END{print "Avg "$1" "avg " "$3}'  >> $2
+    cat ${1}"_t0.log" ${1}"_t1.log" |grep "Accuracy" |awk -v avg=$(cat ${1}"_t0.log" ${1}"_t1.log" |grep "Accuracy" |awk '{sum+=$3}END{printf "%.4f\n",sum/NR}') '{if ( $3=="None" || $2=="N/A" || $3=="nan" || $3=="N/A"){avg="None";nextfile}else avg=avg};END{print "Avg "$1" "$2 " "avg}' >> $2
+    cat ${1}"_t0.log" ${1}"_t1.log" |grep "Functional" | awk -v fail=$(cat ${1}"_t0.log" ${1}"_t1.log" |grep "Functional" |awk '{for(i=1;i<=NF;++i) if($i=="fail") ++sum}END{print sum}') '{if ( fail >= 1 ) tag="fail ";else tag="pass"};END{print $1" "tag}'  >> $2
+    cat ${1}"_t0.log" ${1}"_t1.log" |grep "Error" |awk '{if(a[$1]){a[$1]=a[$1]";"$2}else{a[$1]=$2}}END{for(i in a)print $1" " a[i]}' >> $2
+}
+
+
+modelname=bertsquad
+
+if [[ ${MULTI_TILE} == "False" ]]; then
+    rm ${OUTPUT_DIR}/${modelname}${PRECISION}_inf_t0_raw.log 
+    bash cmd_infer.sh \
+        -m bert_large \
+        -b ${BATCH_SIZE} \
+        -d xpu \
+        -t ${PRECISION} \
+        -o ${OUTPUT_DIR} \
+        -n ${NUM_ITERATIONS} 2>&1 | tee ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0_raw.log
+    python common/parse_result.py -m $modelname -l ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0_raw.log -b ${BATCH_SIZE}
+    throughput=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0.log | grep Performance | awk -F ' ' '{print $2}')
+    throughput_unit=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0.log | grep Performance | awk -F ' ' '{print $3}')
+    latency=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0.log | grep Latency | awk -F ' ' '{print $2}')
+    acc=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0.log | grep Accuracy | awk -F ' ' '{print $3}')
+    acc_unit=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0.log | grep Accuracy | awk -F ' ' '{print $2}')
+else
+    rm ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0_raw.log
+    rm ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t1_raw.log
+    ZE_AFFINITY_MASK=0.0 bash cmd_infer.sh \
+	    -m bert_large \
+	    -b ${BATCH_SIZE} \
+	    -d xpu \
+	    -t ${PRECISION} \
+	    -o ${OUTPUT_DIR} \
+	    -n ${NUM_ITERATIONS} 2>&1 | tee ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0_raw.log &
+    ZE_AFFINITY_MASK=0.1 bash cmd_infer.sh \
+	    -m bert_large \
+	    -b ${BATCH_SIZE} \
+	    -d xpu \
+	    -t ${PRECISION} \
+	    -o ${OUTPUT_DIR} \
+	    -n ${NUM_ITERATIONS} 2>&1 | tee ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t1_raw.log
+    wait
+    python common/parse_result.py -m $modelname -l ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0_raw.log -b ${BATCH_SIZE}
+    python common/parse_result.py -m $modelname -l ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t1_raw.log -b ${BATCH_SIZE}
+    sum_log_analysis ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf.log
+    throughput=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf.log | grep "Sum Performance" | awk -F ' ' '{print $3}')
+    throughput_unit=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf.log | grep "Sum Performance" | awk -F ' ' '{print $4}')
+    latency=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf.log | grep Latency | awk -F ' ' '{print $3}')
+    acc=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf.log | grep Accuracy | awk -F ' ' '{print $4}')
+    acc_unit=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf.log | grep Accuracy | awk -F ' ' '{print $3}')
+fi
+
+yaml_content=$(cat <<EOF
+results:
+ - key: throughput
+   value: $throughput
+   unit: $throughput_unit
+ - key: latency
+   value: $latency
+   unit: s
+ - key: accuracy
+   value: $acc
+   unit: $acc_unit
+EOF
+)
+
+# Write the content to a YAML file
+echo "$yaml_content" >  $OUTPUT_DIR/results.yaml
+echo "YAML file created."
diff --git a/models_v2/pytorch/bert_large/inference/gpu/run_squad.py b/models_v2/pytorch/bert_large/inference/gpu/run_squad.py
new file mode 100644
index 000000000..1284893f1
--- /dev/null
+++ b/models_v2/pytorch/bert_large/inference/gpu/run_squad.py
@@ -0,0 +1,1422 @@
+# coding=utf-8
+# Copyright (c) 2022-2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Copyright 2018 The Google AI Language Team Authors and The HuggingFace Inc. team.
+# Copyright (c) 2018, NVIDIA CORPORATION.  All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+""" Finetuning the library models for question-answering on SQuAD (DistilBERT, Bert, XLM, XLNet)."""
+
+
+import argparse
+import glob
+import logging
+import os
+import random
+import timeit
+import time
+
+import numpy as np
+import torch
+import torch.nn as nn
+from torch.utils.data import DataLoader, RandomSampler, SequentialSampler
+from torch.utils.data.distributed import DistributedSampler
+from tqdm import tqdm, trange
+
+import intel_extension_for_pytorch
+import transformers
+from transformers import (
+    MODEL_FOR_QUESTION_ANSWERING_MAPPING,
+    WEIGHTS_NAME,
+    AdamW,
+    AutoConfig,
+    AutoModelForQuestionAnswering,
+    AutoTokenizer,
+    get_linear_schedule_with_warmup,
+    squad_convert_examples_to_features,
+)
+from transformers.data.metrics.squad_metrics import (
+    compute_predictions_log_probs,
+    compute_predictions_logits,
+    squad_evaluate,
+)
+
+from transformers.modeling_outputs import (
+    QuestionAnsweringModelOutput,
+)
+
+from transformers.data.processors.squad import (
+    SquadResult,
+    SquadV1Processor,
+    SquadV2Processor,
+)
+from transformers.trainer_utils import is_main_process
+
+
+try:
+    from torch.utils.tensorboard import SummaryWriter
+except ImportError:
+    from tensorboardX import SummaryWriter
+
+logger = logging.getLogger(__name__)
+
+MODEL_CONFIG_CLASSES = list(MODEL_FOR_QUESTION_ANSWERING_MAPPING.keys())
+MODEL_TYPES = tuple(conf.model_type for conf in MODEL_CONFIG_CLASSES)
+
+MAP_TORCH_DTYPE = {"FP32": torch.float32, "FP16": torch.float16,
+                   "BF16": torch.bfloat16, "FP64": torch.float64,
+                   "CF64": torch.cfloat, "CF128": torch.cdouble,
+                   "UINT8": torch.uint8, "INT8": torch.int8,
+                   "INT16": torch.int16, "INT": torch.int32,
+                   "INT64": torch.int64, "BOOL": torch.bool,
+                  }
+
+hub = os.path.expanduser("~/.cache/torch/intel")
+if not os.path.exists(hub):
+    os.makedirs(hub)
+
+
+def set_seed(args):
+    random.seed(args.seed)
+    np.random.seed(args.seed)
+    torch.manual_seed(args.seed)
+    if args.use_pcl:
+        pcl_bert.set_rnd_seed(args.seed)
+    if (args.n_gpu > 0) and (args.device_choice == "cuda"):
+        torch.cuda.manual_seed_all(args.seed)
+    elif (args.n_gpu > 0) and (args.device_choice == "xpu"):
+        torch.xpu.manual_seed_all(args.seed)
+
+
+def to_list(tensor):
+    return tensor.detach().cpu().tolist()
+
+
+def load_model(args, config = None, checkpoint = None):
+    model = None
+    if checkpoint and args.use_pcl:
+        with pcl_bert.pcl_impl(args.use_pcl, args.pcl_bf16, args.unpad):
+            model = AutoModelForQuestionAnswering.from_pretrained(
+                checkpoint
+            )  # , force_download=True)
+
+    elif checkpoint and (not args.use_pcl):
+        model = AutoModelForQuestionAnswering.from_pretrained(
+            checkpoint
+        )  # , force_download=True)
+
+    elif (not checkpoint) and args.use_pcl:
+        with pcl_bert.pcl_impl(args.use_pcl, args.pcl_bf16, args.unpad):
+            model = AutoModelForQuestionAnswering.from_pretrained(
+                args.model_name_or_path,
+                from_tf=bool(".ckpt" in args.model_name_or_path),
+                config=config,
+                cache_dir=args.cache_dir if args.cache_dir else None,
+            )
+    else:
+        model = AutoModelForQuestionAnswering.from_pretrained(
+            args.model_name_or_path,
+            from_tf=bool(".ckpt" in args.model_name_or_path),
+            config=config,
+            cache_dir=args.cache_dir if args.cache_dir else None,
+        )
+    return model
+
+def load_jit_model(model, inputs, dtype, device, jit_trace_path, use_jit_cache):
+    jit_model = None
+    if (os.path.isfile(jit_trace_path)) and use_jit_cache:
+        print("load trace model ...")
+        jit_model = torch.jit.load(jit_trace_path)
+        print("load trace model done")
+    else:
+        print("create trace model")
+        in_1 = torch.unsqueeze(inputs["input_ids"][0].clone(), 0)
+        in_2 = torch.unsqueeze(inputs["token_type_ids"][0].clone(), 0)
+        in_3 = torch.unsqueeze(inputs["attention_mask"][0].clone(), 0)
+        with torch.xpu.amp.autocast(enabled=True, dtype=dtype, cache_enabled=False):
+            jit_model = torch.jit.trace(model,
+                (in_1.to(device),
+                in_2.to(device),
+                in_3.to(device)),
+                strict = False)
+        jit_model.save(jit_trace_path)
+
+    return jit_model
+
+def mkdir(path):
+    if not os.path.exists(path):
+        os.makedirs(path)
+
+
+def collate_fn_(batch, device=None, dtype=None):
+    for key, value in batch.items():
+        if device:
+            batch[key] = value.to(device)
+        if ((isinstance(value, torch.FloatTensor)\
+            or isinstance(value, torch.cuda.FloatTensor)\
+            or isinstance(value, torch.xpu.FloatTensor))\
+            and (dtype != None)\
+            and (dtype != "FP32")):
+            batch[key] = value.to(MAP_TORCH_DTYPE[dtype])
+    return batch
+
+def model_cast(model, device=None, dtype=None):
+    # Set device type
+    if (device):
+        model.to(device)
+
+    # Set data type
+    if (dtype == None):
+        pass
+    elif (dtype == "FP16"):
+        model.half()
+    elif (dtype == "BF16"):
+        model.bfloat16()
+    elif (dtype == "FP64"):
+        model.double()
+    elif (dtype == "FP32"):
+        model.float()
+    else:
+        logger.error("The datatype for model casting not yet supported by pytorch")
+
+def get_device(device_choice):
+    if device_choice == "xpu":
+        device = torch.device(
+            "xpu" if torch.xpu.is_available() else "cpu"
+        )
+        n_gpu = 1 if torch.xpu.is_available() else 0
+    elif device_choice == "cuda":
+        device = torch.device(
+            "cuda" if torch.cuda.is_available() else "cpu"
+        )
+        n_gpu = torch.cuda.device_count() if torch.cuda.is_available() else 0
+    else :
+        device = torch.device("cpu")
+        n_gpu = 0
+    return device, n_gpu
+
+
+def train(args, train_dataset, model, tokenizer):
+    """Train the model"""
+    if args.local_rank in [-1, 0]:
+        tb_writer = SummaryWriter()
+
+    args.train_batch_size = args.per_gpu_train_batch_size * max(1, args.n_gpu)
+    train_sampler = (
+        RandomSampler(train_dataset)
+        if args.local_rank == -1
+        else DistributedSampler(train_dataset)
+    )
+    train_dataloader = DataLoader(
+        train_dataset, sampler=train_sampler, batch_size=args.train_batch_size
+    )
+
+    if args.max_steps > 0:
+        t_total = args.max_steps
+        args.num_train_epochs = (
+            args.max_steps
+            // (len(train_dataloader) // args.gradient_accumulation_steps)
+            + 1
+        )
+    else:
+        t_total = (
+            len(train_dataloader)
+            // args.gradient_accumulation_steps
+            * args.num_train_epochs
+        )
+
+    # Prepare optimizer and schedule (linear warmup and decay)
+    no_decay = ["bias", "LayerNorm.weight"]
+    optimizer_grouped_parameters = [
+        {
+            "params": [
+                p
+                for n, p in model.named_parameters()
+                if not any(nd in n for nd in no_decay)
+            ],
+            "weight_decay": args.weight_decay,
+        },
+        {
+            "params": [
+                p
+                for n, p in model.named_parameters()
+                if any(nd in n for nd in no_decay)
+            ],
+            "weight_decay": 0.0,
+        },
+    ]
+    if args.use_pcl:
+        optimizer = pcl_bert.AdamW(
+            optimizer_grouped_parameters, lr=args.learning_rate, eps=args.adam_epsilon
+        )
+    else:
+        optimizer = AdamW(
+            optimizer_grouped_parameters, lr=args.learning_rate, eps=args.adam_epsilon
+        )
+    scheduler = get_linear_schedule_with_warmup(
+        optimizer, num_warmup_steps=args.warmup_steps, num_training_steps=t_total
+    )
+
+    # Check if saved optimizer or scheduler states exist
+    if os.path.isfile(
+        os.path.join(args.model_name_or_path, "optimizer.pt")
+    ) and os.path.isfile(os.path.join(args.model_name_or_path, "scheduler.pt")):
+        # Load in optimizer and scheduler states
+        optimizer.load_state_dict(
+            torch.load(os.path.join(args.model_name_or_path, "optimizer.pt"))
+        )
+        scheduler.load_state_dict(
+            torch.load(os.path.join(args.model_name_or_path, "scheduler.pt"))
+        )
+
+    if (args.dtype == "FP16") and (args.device_choice == "cuda"):
+        try:
+            from apex import amp
+        except ImportError:
+            raise ImportError(
+                "Please install apex from https://www.github.com/nvidia/apex to use fp16 training."
+            )
+
+        model, optimizer = amp.initialize(
+            model, optimizer, opt_level=args.fp16_opt_level
+        )
+
+    # multi-gpu training (should be after apex fp16 initialization)
+    if args.n_gpu > 1:
+        model = torch.nn.DataParallel(model)
+
+    # Distributed training (should be after apex fp16 initialization)
+    if args.local_rank != -1:
+        if args.n_gpu == 0:
+            model = torch.nn.parallel.DistributedDataParallel(
+                model, find_unused_parameters=True
+            )
+        else:
+            model = torch.nn.parallel.DistributedDataParallel(
+                model,
+                device_ids=[args.local_rank],
+                output_device=args.local_rank,
+                find_unused_parameters=True,
+            )
+
+    # Train!
+    logger.info("***** Running training *****")
+    logger.info("  Num examples = %d", len(train_dataset))
+    logger.info("  Num Epochs = %d", args.num_train_epochs)
+    logger.info(
+        "  Instantaneous batch size per GPU = %d", args.per_gpu_train_batch_size
+    )
+    logger.info(
+        "  Total train batch size (w. parallel, distributed & accumulation) = %d",
+        args.train_batch_size
+        * args.gradient_accumulation_steps
+        * (torch.distributed.get_world_size() if args.local_rank != -1 else 1),
+    )
+    logger.info("  Gradient Accumulation steps = %d", args.gradient_accumulation_steps)
+    logger.info("  Total optimization steps = %d", t_total)
+
+    global_step = 1
+    epochs_trained = 0
+    steps_trained_in_current_epoch = 0
+    # Check if continuing training from a checkpoint
+    if os.path.exists(args.model_name_or_path):
+        try:
+            # set global_step to gobal_step of last saved checkpoint from model path
+            checkpoint_suffix = args.model_name_or_path.split("-")[-1].split("/")[0]
+            global_step = int(checkpoint_suffix)
+            epochs_trained = global_step // (
+                len(train_dataloader) // args.gradient_accumulation_steps
+            )
+            steps_trained_in_current_epoch = global_step % (
+                len(train_dataloader) // args.gradient_accumulation_steps
+            )
+
+            logger.info(
+                "  Continuing training from checkpoint, will skip to saved global_step"
+            )
+            logger.info("  Continuing training from epoch %d", epochs_trained)
+            logger.info("  Continuing training from global step %d", global_step)
+            logger.info(
+                "  Will skip the first %d steps in the first epoch",
+                steps_trained_in_current_epoch,
+            )
+        except ValueError:
+            logger.info("  Starting fine-tuning.")
+
+    tr_loss, logging_loss = 0.0, 0.0
+    model.zero_grad()
+    train_iterator = trange(
+        epochs_trained,
+        int(args.num_train_epochs),
+        desc="Epoch",
+        disable=args.local_rank not in [-1, 0],
+    )
+    # Added here for reproductibility
+    set_seed(args)
+
+    for _ in train_iterator:
+        epoch_iterator = tqdm(
+            train_dataloader, desc="Iteration", disable=True
+        )  # args.local_rank not in [-1, 0])
+        # epoch_iterator = tqdm(train_dataloader, desc="Iteration", disable=args.local_rank not in [-1, 0])
+        end_time = timeit.default_timer()
+        for step, batch in enumerate(epoch_iterator):
+            record_shapes = True
+            with torch.autograd.profiler.profile(
+                enabled=args.profile,
+                use_cuda=(args.n_gpu > 0),
+                record_shapes=record_shapes,
+            ) as prof:
+                # Skip past any already trained steps if resuming training
+                try:
+                    import memory_check
+                    memory_check.display_mem("xpu:0")
+                except:
+                    pass
+                if steps_trained_in_current_epoch > 0:
+                    steps_trained_in_current_epoch -= 1
+                    continue
+
+                if prof and args.use_pcl:
+                    pcl_bert.reset_debug_timers()
+                start_fwd_time = timeit.default_timer()
+                model.train()
+                batch = tuple(t.to(args.device) for t in batch)
+
+                inputs = {
+                    "input_ids": batch[0],
+                    "attention_mask": batch[1],
+                    "token_type_ids": batch[2],
+                    "start_positions": batch[3],
+                    "end_positions": batch[4],
+                }
+
+                if args.model_type in [
+                    "xlm",
+                    "roberta",
+                    "distilbert",
+                    "camembert",
+                    "bart",
+                    "longformer",
+                ]:
+                    del inputs["token_type_ids"]
+
+                if args.model_type in ["xlnet", "xlm"]:
+                    inputs.update({"cls_index": batch[5], "p_mask": batch[6]})
+                    if args.version_2_with_negative:
+                        inputs.update({"is_impossible": batch[7]})
+                    if hasattr(model, "config") and hasattr(model.config, "lang2id"):
+                        inputs.update(
+                            {
+                                "langs": (
+                                    torch.ones(batch[0].shape, dtype=torch.int64)
+                                    * args.lang_id
+                                ).to(args.device)
+                            }
+                        )
+
+                outputs = model(**inputs)
+                # model outputs are always tuple in transformers (see doc)
+                loss = outputs[0]
+
+                if args.n_gpu > 1:
+                    loss = (
+                        loss.mean()
+                    )  # mean() to average on multi-gpu parallel (not distributed) training
+                if args.gradient_accumulation_steps > 1:
+                    loss = loss / args.gradient_accumulation_steps
+
+                start_bwd_time = timeit.default_timer()
+                if args.dtype == "FP16":
+                    with amp.scale_loss(loss, optimizer) as scaled_loss:
+                        scaled_loss.backward()
+                else:
+                    loss.backward()
+
+                # tr_loss += loss.item()
+                step_loss = loss.item()
+                tr_loss += step_loss
+                start_opt_time = timeit.default_timer()
+                if (step + 1) % args.gradient_accumulation_steps == 0:
+                    with torch.autograd.profiler.record_function("clip_grad_norm"):
+                        if args.dtype == "FP16":
+                            torch.nn.utils.clip_grad_norm_(
+                                amp.master_params(optimizer), args.max_grad_norm
+                            )
+                        elif args.use_pcl and args.pcl_bf16:
+                            pcl_bert.clip_grad_norm_(
+                                model.parameters(), args.max_grad_norm
+                            )
+                        else:
+                            torch.nn.utils.clip_grad_norm_(
+                                model.parameters(), args.max_grad_norm
+                            )
+
+                    with torch.autograd.profiler.record_function("optimizer"):
+                        optimizer.step()
+                        scheduler.step()  # Update learning rate schedule
+                        # model.zero_grad()
+                        for p in model.parameters():
+                            p.grad = None
+                    global_step += 1
+
+                    # Log metrics
+                    if (
+                        args.local_rank in [-1, 0]
+                        and args.logging_steps > 0
+                        and global_step % args.logging_steps == 0
+                    ):
+                        # Only evaluate when single GPU otherwise metrics may not average well
+                        if args.local_rank == -1 and args.evaluate_during_training:
+                            results = evaluate(args, model, tokenizer)
+                            for key, value in results.items():
+                                tb_writer.add_scalar(
+                                    "eval_{}".format(key), value, global_step
+                                )
+                        tb_writer.add_scalar("lr", scheduler.get_lr()[0], global_step)
+                        tb_writer.add_scalar(
+                            "loss",
+                            (tr_loss - logging_loss) / args.logging_steps,
+                            global_step,
+                        )
+                        logging_loss = tr_loss
+
+                    # Save model checkpoint
+                    if (
+                        args.local_rank in [-1, 0]
+                        and args.save_steps > 0
+                        and global_step % args.save_steps == 0
+                    ):
+                        output_dir = os.path.join(
+                            args.output_dir, "checkpoint-{}".format(global_step)
+                        )
+                        # Take care of distributed/parallel training
+                        model_to_save = (
+                            model.module if hasattr(model, "module") else model
+                        )
+                        model_to_save.save_pretrained(output_dir)
+                        tokenizer.save_pretrained(output_dir)
+
+                        torch.save(args, os.path.join(output_dir, "training_args.bin"))
+                        logger.info("Saving model checkpoint to %s", output_dir)
+
+                        torch.save(
+                            optimizer.state_dict(),
+                            os.path.join(output_dir, "optimizer.pt"),
+                        )
+                        torch.save(
+                            scheduler.state_dict(),
+                            os.path.join(output_dir, "scheduler.pt"),
+                        )
+                        logger.info(
+                            "Saving optimizer and scheduler states to %s", output_dir
+                        )
+
+                data_time = start_fwd_time - end_time
+                end_time = timeit.default_timer()
+                if args.local_rank in [-1, 0]:
+                    print(
+                        f"Step: {global_step-1}, loss: {step_loss:6g}  tr_loss: {tr_loss/(global_step-1):6g} DT: {data_time*1e3:6g} FT: {(start_bwd_time-start_fwd_time)*1e3:6g} BT: {(start_opt_time-start_bwd_time)*1e3:6g} OT: {(end_time-start_opt_time)*1e3:6g} TT: {(end_time-start_fwd_time+data_time)*1e3:6g}"
+                    )
+                if prof and args.use_pcl:
+                    pcl_bert.print_debug_timers()
+                if args.max_steps > 0 and global_step > args.max_steps:
+                    epoch_iterator.close()
+                    break
+            if prof:
+                file_prefix = "squad_time%s" % (
+                    "_r%d" % args.local_rank if args.local_rank >= 0 else ""
+                )
+                with open("%s.prof" % file_prefix, "w") as prof_f:
+                    prof_f.write(
+                        prof.key_averages(group_by_input_shape=record_shapes).table(
+                            sort_by="cpu_time_total"
+                        )
+                    )
+                try:
+                    with open("%s.nested.prof" % file_prefix, "w") as prof_f:
+                        # prof_f.write(prof.nested_key_averages().table(sort_by="cpu_time_total"))
+                        prof_f.write(
+                            prof.nested_key_averages().table(
+                                sort_by=None, row_limit=1000
+                            )
+                        )
+                    with open("%s.top_level.prof" % file_prefix, "w") as prof_f:
+                        prof_f.write(
+                            prof.nested_key_averages(only_top_level=True).table(
+                                sort_by="cpu_time_total"
+                            )
+                        )
+                    prof.print_op_timings(prof, prefix=file_prefix)
+                except:
+                    pass
+                end_time = timeit.default_timer()
+        if args.max_steps > 0 and global_step > args.max_steps:
+            train_iterator.close()
+            break
+
+    if args.local_rank in [-1, 0]:
+        tb_writer.close()
+
+    return global_step, tr_loss / global_step
+
+
+def evaluate(args, model, tokenizer, prefix=""):
+    model_cast(model, device=args.device)
+    model.eval()
+
+    dataset, examples, features = load_and_cache_examples(
+        args, tokenizer, evaluate=True, output_examples=True
+    )
+
+    if not os.path.exists(args.output_dir) and args.local_rank in [-1, 0]:
+        os.makedirs(args.output_dir)
+
+    args.eval_batch_size = args.per_gpu_eval_batch_size * max(1, args.n_gpu)
+
+    # Note that DistributedSampler samples randomly
+    eval_sampler = SequentialSampler(dataset)
+    eval_dataloader = DataLoader(
+        dataset, sampler=eval_sampler, batch_size=args.eval_batch_size
+    )
+
+    # multi-gpu evaluate
+    if args.n_gpu > 1 and not isinstance(model, torch.nn.DataParallel):
+        model = torch.nn.DataParallel(model)
+
+    # Eval!
+    logger.info("***** Running evaluation {} *****".format(prefix))
+    logger.info("  Num examples = %d", len(dataset))
+    logger.info("  Batch size = %d", args.eval_batch_size)
+
+    all_results = []
+
+    jit_model = None
+    jit_trace_path = os.path.join(hub, args.model_type + "_" + args.model_name_or_path.split("_")[1] + "_trace_" + args.dtype + ".zip")
+
+    do_profiling = os.environ.get("PROFILE", "OFF").upper() in ["1", "Y", "ON", "YES", "TRUE"]
+
+    start_time = time.time()
+    iter_num = 0
+    time_collect = []
+    for batch in tqdm(eval_dataloader, desc="Evaluating"):
+        inputs = {
+            "input_ids": batch[0],
+            "attention_mask": batch[1],
+            "token_type_ids": batch[2],
+        }
+
+        if args.model_type in [
+            "xlm",
+            "roberta",
+            "distilbert",
+            "camembert",
+            "bart",
+            "longformer",
+        ]:
+            del inputs["token_type_ids"]
+
+        feature_indices = batch[3]
+
+        # XLNet and XLM use more arguments for their predictions
+        if args.model_type in ["xlnet", "xlm"]:
+            inputs.update({"cls_index": batch[4], "p_mask": batch[5]})
+            # for lang_id-sensitive xlm models
+            if hasattr(model, "config") and hasattr(model.config, "lang2id"):
+                inputs.update(
+                    {
+                        "langs": (
+                            torch.ones(batch[0].shape, dtype=torch.int64)
+                            * args.lang_id
+                        ).to(args.device)
+                    }
+                )
+
+        if iter_num == 0:
+           model = torch.xpu.optimize(model=model, dtype=MAP_TORCH_DTYPE[args.dtype], level="O1", weights_prepack = False)
+           if args.do_jit:
+               jit_model = load_jit_model(model, inputs, MAP_TORCH_DTYPE[args.dtype], args.device, jit_trace_path, args.jit_cache)
+        with torch.no_grad():
+            outputs = None
+            batch_start = None
+            batch_end = None
+            with torch.autograd.profiler_legacy.profile(do_profiling, use_xpu=True, record_shapes=False) as prof:
+                batch_start = time.time()
+                inputs = collate_fn_(inputs, device = args.device)
+                if args.do_jit:
+                    outputs = jit_model(**inputs)
+                else:
+                    outputs = model(**inputs)
+                # torch.xpu.synchronize() does not take effect
+                # torch.xpu.synchronize()
+                for k, v in outputs.items():
+                    v = v.to(torch.float32).to("cpu")
+                batch_end = time.time()
+            print("local latency is:", (batch_end - batch_start), ' s')
+            print("local throughput is:", args.eval_batch_size/(batch_end - batch_start), ' sentences/s')
+            if iter_num >= 10 and iter_num <= (len(dataset)/args.eval_batch_size - 10):
+                time_collect.append(batch_end - batch_start)
+
+            if args.do_jit:
+                outputs = QuestionAnsweringModelOutput(start_logits = outputs["start_logits"],
+                    end_logits = outputs["end_logits"])
+            if iter_num == 10:
+                if do_profiling:
+                    profiling_path = os.getenv('PROFILE_PATH')
+                    # if no set PROFILE_PATH, use current dir
+                    if profiling_path == None:
+                        profiling_path = './'
+                    if (args.profiling_sub_file != ""):
+                        profiling_path += args.profiling_sub_file
+                    mkdir(profiling_path)
+                    torch.save(prof.key_averages().table(sort_by="self_xpu_time_total"), \
+                        profiling_path + '/bert_inference_profile.pt')
+                    print(prof.key_averages().table(sort_by="self_xpu_time_total"))
+                    torch.save(prof.table(sort_by="id", row_limit=100000), \
+                        profiling_path + '/bert_inference_profile_detailed.pt')
+                    prof.export_chrome_trace(profiling_path + '/bert_inference_profile.json')
+
+        for i, feature_index in enumerate(feature_indices):
+            eval_feature = features[feature_index.item()]
+            unique_id = int(eval_feature.unique_id)
+
+            output = [to_list(output[i]) for output in outputs.to_tuple()]
+
+            # Some models (XLNet, XLM) use 5 arguments for their predictions, while the other "simpler"
+            # models only use two.
+            if len(output) >= 5:
+                start_logits = output[0]
+                start_top_index = output[1]
+                end_logits = output[2]
+                end_top_index = output[3]
+                cls_logits = output[4]
+
+                result = SquadResult(
+                    unique_id,
+                    start_logits,
+                    end_logits,
+                    start_top_index=start_top_index,
+                    end_top_index=end_top_index,
+                    cls_logits=cls_logits,
+                )
+
+            else:
+                start_logits, end_logits = output
+                result = SquadResult(unique_id, start_logits, end_logits)
+
+            all_results.append(result)
+        iter_num += 1
+        if args.num_iterations != -1 and iter_num >= args.num_iterations:
+            break;
+
+    evalTime = time.time() - start_time
+    if len(time_collect) > 0:
+        avg_time = sum(time_collect)/len(time_collect)
+        print("bert_inf latency: ", avg_time, ' s')
+        print("bert_inf throughput: ", args.eval_batch_size/avg_time, ' sentences/s')
+    if args.num_iterations != -1 and args.num_iterations < (len(dataset)/args.eval_batch_size + 1):
+        exit()
+    logger.info(
+        "  Evaluation done in total %f secs (%f sec per example)",
+        evalTime,
+        evalTime / len(dataset),
+    )
+
+    # Compute predictions
+    output_prediction_file = os.path.join(
+        args.output_dir, "predictions_{}.json".format(prefix)
+    )
+    output_nbest_file = os.path.join(
+        args.output_dir, "nbest_predictions_{}.json".format(prefix)
+    )
+
+    if args.version_2_with_negative:
+        output_null_log_odds_file = os.path.join(
+            args.output_dir, "null_odds_{}.json".format(prefix)
+        )
+    else:
+        output_null_log_odds_file = None
+
+    # XLNet and XLM use a more complex post-processing procedure
+    if args.model_type in ["xlnet", "xlm"]:
+        start_n_top = (
+            model.config.start_n_top
+            if hasattr(model, "config")
+            else model.module.config.start_n_top
+        )
+        end_n_top = (
+            model.config.end_n_top
+            if hasattr(model, "config")
+            else model.module.config.end_n_top
+        )
+
+        predictions = compute_predictions_log_probs(
+            examples,
+            features,
+            all_results,
+            args.n_best_size,
+            args.max_answer_length,
+            output_prediction_file,
+            output_nbest_file,
+            output_null_log_odds_file,
+            start_n_top,
+            end_n_top,
+            args.version_2_with_negative,
+            tokenizer,
+            args.verbose_logging,
+        )
+    else:
+        predictions = compute_predictions_logits(
+            examples,
+            features,
+            all_results,
+            args.n_best_size,
+            args.max_answer_length,
+            args.do_lower_case,
+            output_prediction_file,
+            output_nbest_file,
+            output_null_log_odds_file,
+            args.verbose_logging,
+            args.version_2_with_negative,
+            args.null_score_diff_threshold,
+            tokenizer,
+        )
+
+    # Compute the F1 and exact scores.
+    results = squad_evaluate(examples, predictions)
+    return results
+
+
+def load_and_cache_examples(args, tokenizer, evaluate=False, output_examples=False):
+    if args.local_rank not in [-1, 0] and not evaluate:
+        # Make sure only the first process in distributed training process the dataset, and the others will use the cache
+        torch.distributed.barrier()
+
+    # Load data features from cache or dataset file
+    input_dir = args.data_dir if args.data_dir else "."
+    cached_features_file = os.path.join(
+        input_dir,
+        "cached_{}_{}_{}".format(
+            "dev" if evaluate else "train",
+            list(filter(None, args.model_name_or_path.split("/"))).pop(),
+            str(args.max_seq_length),
+        ),
+    )
+
+    # Init features and dataset from cache if it exists
+    if os.path.exists(cached_features_file) and not args.overwrite_cache:
+        logger.info("Loading features from cached file %s", cached_features_file)
+        features_and_dataset = torch.load(cached_features_file)
+        features, dataset, examples = (
+            features_and_dataset["features"],
+            features_and_dataset["dataset"],
+            features_and_dataset["examples"],
+        )
+    else:
+        logger.info("Creating features from dataset file at %s", input_dir)
+
+        if not args.data_dir and (
+            (evaluate and not args.predict_file)
+            or (not evaluate and not args.train_file)
+        ):
+            try:
+                import tensorflow_datasets as tfds
+            except ImportError:
+                raise ImportError(
+                    "If not data_dir is specified, tensorflow_datasets needs to be installed."
+                )
+
+            if args.version_2_with_negative:
+                logger.warn("tensorflow_datasets does not handle version 2 of SQuAD.")
+
+            tfds_examples = tfds.load("squad")
+            examples = SquadV1Processor().get_examples_from_dataset(
+                tfds_examples, evaluate=evaluate
+            )
+        else:
+            processor = (
+                SquadV2Processor()
+                if args.version_2_with_negative
+                else SquadV1Processor()
+            )
+            if evaluate:
+                examples = processor.get_dev_examples(
+                    args.data_dir, filename=args.predict_file
+                )
+            else:
+                examples = processor.get_train_examples(
+                    args.data_dir, filename=args.train_file
+                )
+
+        features, dataset = squad_convert_examples_to_features(
+            examples=examples,
+            tokenizer=tokenizer,
+            max_seq_length=args.max_seq_length,
+            doc_stride=args.doc_stride,
+            max_query_length=args.max_query_length,
+            is_training=not evaluate,
+            return_dataset="pt",
+            threads=args.threads,
+        )
+
+        if args.local_rank in [-1, 0]:
+            logger.info("Saving features into cached file %s", cached_features_file)
+            torch.save(
+                {"features": features, "dataset": dataset, "examples": examples},
+                cached_features_file,
+            )
+
+    if args.local_rank == 0 and not evaluate:
+        # Make sure only the first process in distributed training process the dataset, and the others will use the cache
+        torch.distributed.barrier()
+
+    if output_examples:
+        return dataset, examples, features
+    return dataset
+
+
+def main():
+    parser = argparse.ArgumentParser()
+
+    # Required parameters
+    parser.add_argument(
+        "--model_type",
+        default=None,
+        type=str,
+        required=True,
+        help="Model type selected in the list: " + ", ".join(MODEL_TYPES),
+    )
+    parser.add_argument(
+        "--model_name_or_path",
+        default=None,
+        type=str,
+        required=True,
+        help="Path to pretrained model or model identifier from huggingface.co/models",
+    )
+    parser.add_argument(
+        "--output_dir",
+        default=None,
+        type=str,
+        required=True,
+        help="The output directory where the model checkpoints and predictions will be written.",
+    )
+
+    # Other parameters
+    parser.add_argument(
+        "--data_dir",
+        default=None,
+        type=str,
+        help="The input data dir. Should contain the .json files for the task."
+        + "If no data dir or train/predict files are specified, will run with tensorflow_datasets.",
+    )
+    parser.add_argument(
+        "--train_file",
+        default=None,
+        type=str,
+        help="The input training file. If a data dir is specified, will look for the file there"
+        + "If no data dir or train/predict files are specified, will run with tensorflow_datasets.",
+    )
+    parser.add_argument(
+        "--predict_file",
+        default=None,
+        type=str,
+        help="The input evaluation file. If a data dir is specified, will look for the file there"
+        + "If no data dir or train/predict files are specified, will run with tensorflow_datasets.",
+    )
+    parser.add_argument(
+        "--config_name",
+        default="",
+        type=str,
+        help="Pretrained config name or path if not the same as model_name",
+    )
+    parser.add_argument(
+        "--tokenizer_name",
+        default="",
+        type=str,
+        help="Pretrained tokenizer name or path if not the same as model_name",
+    )
+    parser.add_argument(
+        "--cache_dir",
+        default="",
+        type=str,
+        help="Where do you want to store the pre-trained models downloaded from huggingface.co",
+    )
+
+    parser.add_argument(
+        "--dtype",
+        default="FP32",
+        type=str,
+        choices=["FP32", "BF16", "FP16"],
+        help= "Specify precision to use",
+    )
+
+    parser.add_argument(
+        "--version_2_with_negative",
+        action="store_true",
+        help="If true, the SQuAD examples contain some that do not have an answer.",
+    )
+    parser.add_argument(
+        "--null_score_diff_threshold",
+        type=float,
+        default=0.0,
+        help="If null_score - best_non_null is greater than the threshold predict null.",
+    )
+
+    parser.add_argument(
+        "--max_seq_length",
+        default=384,
+        type=int,
+        help="The maximum total input sequence length after WordPiece tokenization. Sequences "
+        "longer than this will be truncated, and sequences shorter than this will be padded.",
+    )
+    parser.add_argument(
+        "--doc_stride",
+        default=128,
+        type=int,
+        help="When splitting up a long document into chunks, how much stride to take between chunks.",
+    )
+    parser.add_argument(
+        "--max_query_length",
+        default=64,
+        type=int,
+        help="The maximum number of tokens for the question. Questions longer than this will "
+        "be truncated to this length.",
+    )
+    parser.add_argument(
+        "--do_train", action="store_true", help="Whether to run training."
+    )
+    parser.add_argument(
+        "--do_eval", action="store_true", help="Whether to run eval on the dev set."
+    )
+    parser.add_argument(
+        "--do_jit", action="store_true", help="Whether to run eval with jit on the dev set."
+    )
+    parser.add_argument(
+        "--jit_cache",
+        action="store_true",
+        default=True,
+        help="Whether to use jit model cache",
+    )
+    parser.add_argument(
+        "--evaluate_during_training",
+        action="store_true",
+        help="Run evaluation during training at each logging step.",
+    )
+    parser.add_argument(
+        "--do_lower_case",
+        action="store_true",
+        help="Set this flag if you are using an uncased model.",
+    )
+    parser.add_argument(
+        "--per_gpu_train_batch_size",
+        default=8,
+        type=int,
+        help="Batch size per GPU/CPU for training.",
+    )
+    parser.add_argument(
+        "--per_gpu_eval_batch_size",
+        default=8,
+        type=int,
+        help="Batch size per GPU/CPU for evaluation.",
+    )
+    parser.add_argument(
+        "--learning_rate",
+        default=5e-5,
+        type=float,
+        help="The initial learning rate for Adam.",
+    )
+    parser.add_argument(
+        "--gradient_accumulation_steps",
+        type=int,
+        default=1,
+        help="Number of updates steps to accumulate before performing a backward/update pass.",
+    )
+    parser.add_argument(
+        "--num_iterations",
+        type=int,
+        default=-1,
+        help="Number of iterations to run evaluation.",
+    )
+    parser.add_argument(
+        "--weight_decay", default=0.0, type=float, help="Weight decay if we apply some."
+    )
+    parser.add_argument(
+        "--adam_epsilon", default=1e-8, type=float, help="Epsilon for Adam optimizer."
+    )
+    parser.add_argument(
+        "--max_grad_norm", default=1.0, type=float, help="Max gradient norm."
+    )
+    parser.add_argument(
+        "--num_train_epochs",
+        default=3.0,
+        type=float,
+        help="Total number of training epochs to perform.",
+    )
+    parser.add_argument(
+        "--max_steps",
+        default=-1,
+        type=int,
+        help="If > 0: set total number of training steps to perform. Override num_train_epochs.",
+    )
+    parser.add_argument(
+        "--warmup_steps", default=0, type=int, help="Linear warmup over warmup_steps."
+    )
+    parser.add_argument(
+        "--n_best_size",
+        default=20,
+        type=int,
+        help="The total number of n-best predictions to generate in the nbest_predictions.json output file.",
+    )
+    parser.add_argument(
+        "--max_answer_length",
+        default=30,
+        type=int,
+        help="The maximum length of an answer that can be generated. This is needed because the start "
+        "and end predictions are not conditioned on one another.",
+    )
+    parser.add_argument(
+        "--verbose_logging",
+        action="store_true",
+        help="If true, all of the warnings related to data processing will be printed. "
+        "A number of warnings are expected for a normal SQuAD evaluation.",
+    )
+    parser.add_argument(
+        "--lang_id",
+        default=0,
+        type=int,
+        help="language id of input for language-specific xlm models (see tokenization_xlm.PRETRAINED_INIT_CONFIGURATION)",
+    )
+
+    parser.add_argument(
+        "--logging_steps", type=int, default=500, help="Log every X updates steps."
+    )
+    parser.add_argument(
+        "--save_steps",
+        type=int,
+        default=500,
+        help="Save checkpoint every X updates steps.",
+    )
+    parser.add_argument(
+        "--eval_all_checkpoints",
+        action="store_true",
+        help="Evaluate all checkpoints starting with the same prefix as model_name ending and ending with step number",
+    )
+    parser.add_argument(
+        "--device_choice",
+        default="cpu",
+        type=str,
+        choices=["cpu", "xpu", "cuda"],
+        help= "Specify device to use when available",
+    )
+    parser.add_argument(
+        "--profile",
+        action="store_true",
+        help="Whether to use PCL Fused impl when available",
+    )
+    parser.add_argument(
+        "--use_pcl",
+        action="store_true",
+        help="Whether to use PCL Fused impl when available",
+    )
+    parser.add_argument(
+        "--pcl_bf16",
+        action="store_true",
+        help="Whether to use PCL Fused impl when available",
+    )
+    parser.add_argument(
+        "--unpad",
+        action="store_true",
+        help="Whether to use PCL Fused impl when available",
+    )
+    parser.add_argument(
+        "--dist_backend",
+        type=str,
+        default="ccl",
+        help="Specify distributed backend to use.",
+    )
+    parser.add_argument(
+        "--profiling_sub_file",
+        type=str,
+        default="",
+        help="Path to saved torch profiler",
+    )
+    parser.add_argument(
+        "--overwrite_output_dir",
+        action="store_true",
+        help="Overwrite the content of the output directory",
+    )
+    parser.add_argument(
+        "--overwrite_cache",
+        action="store_true",
+        help="Overwrite the cached training and evaluation sets",
+    )
+    parser.add_argument(
+        "--seed", type=int, default=42, help="random seed for initialization"
+    )
+
+    parser.add_argument(
+        "--local_rank",
+        type=int,
+        default=-1,
+        help="local_rank for distributed training on gpus",
+    )
+    parser.add_argument(
+        "--fp16_opt_level",
+        type=str,
+        default="O1",
+        help="For fp16: Apex AMP optimization level selected in ['O0', 'O1', 'O2', and 'O3']."
+        "See details at https://nvidia.github.io/apex/amp.html",
+    )
+    parser.add_argument(
+        "--server_ip", type=str, default="", help="Can be used for distant debugging."
+    )
+    parser.add_argument(
+        "--server_port", type=str, default="", help="Can be used for distant debugging."
+    )
+
+    parser.add_argument(
+        "--threads",
+        type=int,
+        default=1,
+        help="multiple threads for converting example to features",
+    )
+    args = parser.parse_args()
+
+    if args.doc_stride >= args.max_seq_length - args.max_query_length:
+        logger.warning(
+            "WARNING - You've set a doc stride which may be superior to the document length in some "
+            "examples. This could result in errors when building features from the examples. Please reduce the doc "
+            "stride or increase the maximum length to ensure the features are correctly built."
+        )
+
+    if (
+        os.path.exists(args.output_dir)
+        and os.listdir(args.output_dir)
+        and args.do_train
+        and not args.overwrite_output_dir
+    ):
+        raise ValueError(
+            "Output directory ({}) already exists and is not empty. Use --overwrite_output_dir to overcome.".format(
+                args.output_dir
+            )
+        )
+
+    if args.use_pcl:
+        try:
+            from pcl_pytorch_extension import bert as pcl_bert
+        except:
+            print("CCL backend requested but import torch_ccl failed")
+            raise
+    # Setup distant debugging if needed
+    if args.server_ip and args.server_port:
+        # Distant debugging - see https://code.visualstudio.com/docs/python/debugging#_attach-to-a-local-script
+        import ptvsd
+
+        print("Waiting for debugger attach")
+        ptvsd.enable_attach(
+            address=(args.server_ip, args.server_port), redirect_output=True
+        )
+        ptvsd.wait_for_attach()
+
+    # Setup CUDA, GPU & distributed training
+    if int(os.environ.get("PMI_SIZE", "0")) > 1:
+        if args.dist_backend == "ccl":
+            try:
+                import torch_ccl
+            except:
+                print("CCL backend requested but import torch_ccl failed")
+                raise
+        elif args.dist_backend == "mpi":
+            if not torch.distributed.is_mpi_available():
+                try:
+                    import torch_mpi
+                except:
+                    print(
+                        "MPI backend requested but not available try installing torch_mpi module"
+                    )
+                    raise
+        else:
+            raise ValueError(f"{args.dist_backend} backend requested but not supported")
+
+        os.environ["RANK"] = os.environ.get("PMI_RANK", "0")
+        os.environ["WORLD_SIZE"] = os.environ.get("PMI_SIZE", "1")
+        torch.distributed.init_process_group(backend=args.dist_backend)
+        device = torch.device("cpu")
+        args.n_gpu = 0
+        args.local_rank = torch.distributed.get_rank()
+        print(
+            f"Using {args.dist_backend.upper()} dist run with {torch.distributed.get_world_size()} ranks"
+        )
+    elif args.local_rank == -1:
+        device, args.n_gpu = get_device(args.device_choice)
+    else:  # Initializes the distributed backend which will take care of sychronizing nodes/GPUs
+        torch.cuda.set_device(args.local_rank)
+        device = torch.device("cuda", args.local_rank)
+        torch.distributed.init_process_group(backend="nccl")
+        args.n_gpu = 1
+    args.device = device
+
+    # Setup logging
+    logging.basicConfig(
+        format="%(asctime)s - %(levelname)s - %(name)s -   %(message)s",
+        datefmt="%m/%d/%Y %H:%M:%S",
+        level=logging.INFO if args.local_rank in [-1, 0] else logging.WARN,
+    )
+    logger.warning(
+        "Process rank: %s, device: %s, n_gpu: %s, distributed training: %s, dtype: %s",
+        args.local_rank,
+        device,
+        args.n_gpu,
+        bool(args.local_rank != -1),
+        args.dtype,
+    )
+    # Set the verbosity to info of the Transformers logger (on main process only):
+    if is_main_process(args.local_rank):
+        transformers.utils.logging.set_verbosity_info()
+        transformers.utils.logging.enable_default_handler()
+        transformers.utils.logging.enable_explicit_format()
+    # Set seed
+    set_seed(args)
+
+    # Load pretrained model and tokenizer
+    if args.local_rank not in [-1, 0]:
+        # Make sure only the first process in distributed training will download model & vocab
+        torch.distributed.barrier()
+
+    args.model_type = args.model_type.lower()
+    config = AutoConfig.from_pretrained(
+        args.config_name if args.config_name else args.model_name_or_path,
+        cache_dir=args.cache_dir if args.cache_dir else None,
+    )
+    tokenizer = AutoTokenizer.from_pretrained(
+        args.tokenizer_name if args.tokenizer_name else args.model_name_or_path,
+        do_lower_case=args.do_lower_case,
+        cache_dir=args.cache_dir if args.cache_dir else None,
+        use_fast=False,  # SquadDataset is not compatible with Fast tokenizers which have a smarter overflow handeling
+    )
+    model = load_model(args, config = config)
+
+    for m in model.modules():
+        if hasattr(m, "maybe_block_params"):
+            m.maybe_block_params()
+            # if args.pcl_bf16: m.to(torch.bfloat16)
+
+    if args.local_rank == 0:
+        # Make sure only the first process in distributed training will download model & vocab
+        torch.distributed.barrier()
+
+    model.to(args.device)
+
+    logger.info("Training/evaluation parameters %s", args)
+
+    # Before we do anything with models, we want to ensure that we get fp16 execution of torch.einsum if args.dtype is float16.
+    # Otherwise it'll default to "promote" mode, and we'll get fp32 operations. Note that running `--fp16_opt_level="O2"` will
+    # remove the need for this code, but it is still valid.
+    if (args.dtype == "FP16") and (args.device_choice == "cuda"):
+        try:
+            import apex
+
+            apex.amp.register_half_function(torch, "einsum")
+        except ImportError:
+            raise ImportError(
+                "Please install apex from https://www.github.com/nvidia/apex to use fp16 training."
+            )
+
+    # Training
+    if args.do_train:
+        train_dataset = load_and_cache_examples(
+            args, tokenizer, evaluate=False, output_examples=False
+        )
+        global_step, tr_loss = train(args, train_dataset, model, tokenizer)
+        logger.info(" global_step = %s, average loss = %s", global_step, tr_loss)
+
+    # Save the trained model and the tokenizer
+    if args.do_train and (args.local_rank == -1 or torch.distributed.get_rank() == 0):
+        logger.info("Saving model checkpoint to %s", args.output_dir)
+        # Save a trained model, configuration and tokenizer using `save_pretrained()`.
+        # They can then be reloaded using `from_pretrained()`
+        # Take care of distributed/parallel training
+        model_to_save = model.module if hasattr(model, "module") else model
+        model_to_save.save_pretrained(args.output_dir)
+        tokenizer.save_pretrained(args.output_dir)
+
+        # Good practice: save your training arguments together with the trained model
+        torch.save(args, os.path.join(args.output_dir, "training_args.bin"))
+
+        # Load a trained model and vocabulary that you have fine-tuned
+        model = AutoModelForQuestionAnswering.from_pretrained(
+            args.output_dir
+        )  # , force_download=True)
+
+        # SquadDataset is not compatible with Fast tokenizers which have a smarter overflow handeling
+        # So we use use_fast=False here for now until Fast-tokenizer-compatible-examples are out
+        tokenizer = AutoTokenizer.from_pretrained(
+            args.output_dir, do_lower_case=args.do_lower_case, use_fast=False
+        )
+        model.to(args.device)
+
+    # Evaluation - we can ask to evaluate all the checkpoints (sub-directories) in a directory
+    results = {}
+    if args.do_eval and args.local_rank in [-1, 0]:
+        if args.do_train:
+            logger.info("Loading checkpoints saved during training for evaluation")
+            checkpoints = [args.output_dir]
+            if args.eval_all_checkpoints:
+                checkpoints = list(
+                    os.path.dirname(c)
+                    for c in sorted(
+                        glob.glob(
+                            args.output_dir + "/**/" + WEIGHTS_NAME, recursive=True
+                        )
+                    )
+                )
+
+        else:
+            logger.info("Loading checkpoint %s for evaluation", args.model_name_or_path)
+            checkpoints = [args.model_name_or_path]
+
+        logger.info("Evaluate the following checkpoints: %s", checkpoints)
+
+        for checkpoint in checkpoints:
+            # Reload the model
+            global_step = checkpoint.split("-")[-1] if len(checkpoints) > 1 else ""
+            model = load_model(args, checkpoint = checkpoint)
+
+            # Evaluate
+            result = evaluate(args, model, tokenizer, prefix=global_step)
+
+            result = dict(
+                (k + ("_{}".format(global_step) if global_step else ""), v)
+                for k, v in result.items()
+            )
+            results.update(result)
+
+    logger.info("Results: {}".format(results))
+
+    return results
+
+
+if __name__ == "__main__":
+    main()
diff --git a/models_v2/pytorch/bert_large/inference/gpu/setup.sh b/models_v2/pytorch/bert_large/inference/gpu/setup.sh
new file mode 100644
index 000000000..75a3f37b0
--- /dev/null
+++ b/models_v2/pytorch/bert_large/inference/gpu/setup.sh
@@ -0,0 +1,49 @@
+#
+<<<<<<<< HEAD:datasets/cloud_data_connector/cloud_data_connector/azure/__init__.py
+# -*- coding: utf-8 -*-
+#
+========
+>>>>>>>> r3.1:models_v2/pytorch/bert_large/inference/gpu/setup.sh
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+__path__ = __import__("pkgutil").extend_path(__path__, __name__)  # type: ignore
+
+<<<<<<<< HEAD:datasets/cloud_data_connector/cloud_data_connector/azure/__init__.py
+from .connector import Connector
+from .downloader import Downloader
+from .uploader import Uploader
+from .ml_uploader import MLUploader
+
+connect = Connector().connect
+
+__all__=[
+    'Connector',
+    'Downloader',
+    'Uploader',
+    'MLUploader'
+]
+========
+# setup.sh
+#  - install OS pkgs
+#  - should create virtual env & install pip  requirement.txt
+#  - git clones & applying patches
+
+set -e
+apt-get update && apt-get install -y python3-venv protobuf-compiler
+
+pip install -r requirements.txt
+
+cp -r ../../../../common/parse_result.py common/parse_result.py 
+>>>>>>>> r3.1:models_v2/pytorch/bert_large/inference/gpu/setup.sh
diff --git a/models_v2/pytorch/bert_large/training/gpu/.gitignore b/models_v2/pytorch/bert_large/training/gpu/.gitignore
new file mode 100644
index 000000000..d1b4845dc
--- /dev/null
+++ b/models_v2/pytorch/bert_large/training/gpu/.gitignore
@@ -0,0 +1,2 @@
+bert/
+*.tar.gz
diff --git a/models_v2/pytorch/bert_large/training/gpu/CONTAINER.md b/models_v2/pytorch/bert_large/training/gpu/CONTAINER.md
new file mode 100644
index 000000000..cb3adaab1
--- /dev/null
+++ b/models_v2/pytorch/bert_large/training/gpu/CONTAINER.md
@@ -0,0 +1,76 @@
+# Running BERT Large Training on Intel® Data Center GPU Max Series using Intel® Extension for PyTorch*
+
+## Description
+This document has instructions for running BERT Large training using BF16,TF32 and FP32 precisions using Intel(R) Extension for PyTorch on Intel Max Series GPU. 
+
+## Quick Start Scripts
+| Script name | Description |
+|-------------|-------------|
+| `run_model.sh` | Runs BERT Large training on single and two tiles |
+
+## Requirements
+| Item | Detail |
+| ------ | ------- |
+| Host machine  | [Intel® Data Center GPU Max Series](https://ark.intel.com/content/www/us/en/ark/products/series/232874/intel-data-center-gpu-max-series.html)  |
+| Drivers | GPU-compatible drivers need to be installed: Download [Driver](https://dgpu-docs.intel.com/driver/installation.html) |
+| Software | Docker* |
+
+## Datasets
+### Download and Extract the Dataset
+Refer to the [link](README.md#dataset) to download and prepare dataset for training. Set `DATASET_DIR` to point to the dataset directory. 
+
+## Docker pull Command
+```
+docker pull intel/language-modeling:pytorch-max-gpu-bert-large-training
+```
+
+The BERT Large training container includes scripts, models,libraries needed to run BF16,TF32 and FP32 training. To run the `run_model.sh` script, you will need to provide an output directory where log files will be written. 
+
+```bash
+#optional
+export PRECISION=<provide FP32,TF32 or BF16 otherwise, (default: BF16)>
+export NUM_ITERATIONS=<provide num_iterations,otherwise (default: 10)>
+export BATCH_SIZE=<provide batch size,otherwise (default: 16)>
+
+#required
+export OUTPUT_DIR=<path to output directory>
+export DATASET_DIR=<path to dataset directory>
+export MULTI_TILE=<provide True or False to enable/disable multi-tile inference>
+export PLATFORM=Max
+
+DOCKER_ARGS="--rm --init -it"
+IMAGE_NAME=intel/language-modeling:pytorch-max-gpu-bert-large-training
+SCRIPT=run_model.sh
+
+docker run \
+  --privileged \
+  --device=/dev/dri \
+  --ipc host \
+  --env DATASET_DIR=${DATASET_DIR} \
+  --env BATCH_SIZE=${BATCH_SIZE} \
+  --env PRECISION=${PRECISION} \
+  --env NUM_ITERATIONS=${NUM_ITERATIONS} \
+  --env MULTI_TILE=${MULTI_TILE} \
+  --env PLATFORM=${PLATFORM} \
+  --env OUTPUT_DIR=${OUTPUT_DIR} \
+  --env http_proxy=${http_proxy} \
+  --env https_proxy=${https_proxy} \
+  --env no_proxy=${no_proxy} \
+  --volume ${DATASET_DIR}:${DATASET_DIR} \
+  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
+  --volume /dev/dri:/dev/dri/ \
+  ${DOCKER_ARGS} \
+  $IMAGE_NAME \
+  /bin/bash $SCRIPT
+  ```
+
+## Documentation and Sources
+
+[GitHub* Repository](https://github.com/IntelAI/models/tree/master/docker/max-gpu)
+
+## Support
+Support for Intel® Extension for PyTorch* is found via the [Intel® AI Analytics Toolkit.](https://www.intel.com/content/www/us/en/developer/tools/oneapi/ai-analytics-toolkit.html#gs.qbretz) Additionally, the Intel® Extension for PyTorch* team tracks both bugs and enhancement requests using [GitHub issues](https://github.com/intel/intel-extension-for-pytorch/issues). Before submitting a suggestion or bug report, please search the GitHub issues to see if your issue has already been reported.
+
+## License Agreement
+
+LEGAL NOTICE: By accessing, downloading or using this software and any required dependent software (the “Software Package”), you agree to the terms and conditions of the software license agreements for the Software Package, which may also include notices, disclaimers, or license terms for third party software included with the Software Package. Please refer to the [license file](https://github.com/IntelAI/models/tree/master/third_party) for additional details.
diff --git a/models_v2/pytorch/bert_large/training/gpu/README.md b/models_v2/pytorch/bert_large/training/gpu/README.md
new file mode 100644
index 000000000..74a830c47
--- /dev/null
+++ b/models_v2/pytorch/bert_large/training/gpu/README.md
@@ -0,0 +1,142 @@
+# BERT Large Training
+
+BERT Large training best known configurations with Intel® Extension for PyTorch.
+
+## Model Information
+
+| **Use Case** | **Framework** | **Model Repo** | **Branch/Commit/Tag** | **Optional Patch** |
+|:---:| :---: |:--------------:|:---------------------:|:------------------:|
+|  Training   |    PyTorch    |       https://github.com/huggingface/transformers/tree/main/src/transformers/models/bert        |           -           |         -          |
+
+# Pre-Requisite
+* Host has one of the following GPUs:
+  * **Arc Series** - [Intel® Arc™ A-Series Graphics](https://ark.intel.com/content/www/us/en/ark/products/series/227957/intel-arc-a-series-graphics.html)
+  * **Max Series** - [Intel® Data Center GPU Max Series](https://ark.intel.com/content/www/us/en/ark/products/series/232874/intel-data-center-gpu-max-series.html)
+* Host has installed latest Intel® Data Center GPU Max & ARC Series Drivers https://dgpu-docs.intel.com/driver/installation.html
+* The following Intel® oneAPI Base Toolkit components are required:
+  - Intel® oneAPI DPC++ Compiler (Placeholder DPCPPROOT as its installation path)
+  - Intel® oneAPI Math Kernel Library (oneMKL) (Placeholder MKLROOT as its installation path)
+  - Intel® oneAPI MPI Library
+  - Intel® oneAPI TBB Library
+  - Intel® oneAPI CCL Library
+
+  Follow instructions at [Intel® oneAPI Base Toolkit Download page](https://www.intel.com/content/www/us/en/developer/tools/oneapi/base-toolkit-download.html?operatingsystem=linux) to setup the package manager repository.
+
+# Prepare Dataset
+## Dataset: 
+Please refer to https://github.com/mlcommons/training_results_v2.1/tree/main/NVIDIA/benchmarks/bert/implementations/pytorch-22.09#download-and-prepare-the-data
+
+the dataset should be like below
+|_hdf5  
+      |_ eval                               # evaluation chunks in binary hdf5 format fixed length (not used in training, can delete after data   preparation)  
+      |_ eval_varlength                     # evaluation chunks in binary hdf5 format variable length *used for training*
+      |_ training                           # 500 chunks in binary hdf5 format 
+      |_ training_4320                      # 
+      |_ hdf5_4320_shards_uncompressed   # sharded data in hdf5 format fixed length (not used in training, can delete after data   preparation)
+      |_ hdf5_4320_shards_varlength      # sharded data in hdf5 format variable length *used for training*
+we are using  hdf5/hdf5_4320_shards_varlength as the our dataset.
+
+## Training
+1. `git clone https://github.com/IntelAI/models.git`
+2. `cd models/models_v2/pytorch/bert_large/training/gpu`
+3. Create virtual environment `venv` and activate it:
+    ```
+    python3 -m venv venv
+    . ./venv/bin/activate
+    ```
+4. Run setup.sh
+    ```
+    ./setup.sh
+    ```
+5. Install the latest GPU versions of [torch, torchvision and intel_extension_for_pytorch](https://intel.github.io/intel-extension-for-pytorch/index.html#installation):
+    ```
+    python -m pip install torch==<torch_version> torchvision==<torchvvision_version> intel-extension-for-pytorch==<ipex_version> --extra-index-url https://pytorch-extension.intel.com/release-whl-aitools/
+    ```
+6. Setup required environment paramaters
+
+| **Parameter**                |                                  **export command**                                  |
+|:---------------------------:|:------------------------------------------------------------------------------------:|
+| **MULTI_TILE**               | `export MULTI_TILE=True` (True or False)                                             |
+| **PLATFORM**                 | `export PLATFORM=Max` (Max or ARC)                                                 |
+| **DATASET_DIR**                 | `export DATASET_DIR=`                                                                                  |
+| **BATCH_SIZE** (optional)    |                               `export BATCH_SIZE=16`                                |
+| **PRECISION** (optional)     |`export PRECISION=BF16` (BF16 FP8 FP32 and TF32 are supported for Max and BF16 for ARC )               |
+| **OUTPUT_DIR** (optional)    |                               `export OUTPUT_DIR=$PWD`                               |
+|**NUM_ITERATIONS** (optional) |                               `export NUM_ITERATIONS=20`                             |
+6. Run `run_model.sh`
+
+## Output
+
+Single-tile output will typically looks like:
+
+```
+[info] construct file from initialization
+[info] input dir =  /home/gta/Cosim_test/dataset/hdf5
+[info] num files =  2
+epoch: 1
+Loaded 193485 samples from datafile: /home/gta/Cosim_test/dataset/hdf5/pretrain-part-01.hdf5
+bert_train latency:  0.24147300720214843  s
+bert_train throughput:  66.25999396531161  sentences/s
+perplexity = 11.020857810974121
+```
+
+Multi-tile output will typically looks like:
+```
+Model to device:  xpu:0
+using adamw
+Doing torch xpu optimize, dtype:  torch.bfloat16
+Torch distributed is available.
+Torch distributed is initialized.
+[info] Setting seed:  123 . worker seed:  224899942
+
+
+found num checkpoints:  0
+resume from checkpoints:  False
+resume checkpoint:  None
+[info] construct file from initialization
+[info] input dir =  /home/gta/Cosim_test/dataset/hdf5
+[info] num files =  2
+epoch: 1
+Loaded 194779 samples from datafile: /home/gta/Cosim_test/dataset/hdf5/pretrain-part-00.hdf5
+bert_train latency:  0.2703933477401733  s
+bert_train throughput:  59.17305338212218  sentences/s
+perplexity = 11.018452644348145
+Setting seed to ensure same model master weight at the beginning.
+world_size:2, rank:1, device:xpu:1
+args.world_size=2, args.rank=1
+Get config from config_name bert_config.json
+Set different weight_decay for model parameters
+GroupSizes:  [335869952, 356156]
+Model to device:  xpu:1
+using adamw
+Doing torch xpu optimize, dtype:  torch.bfloat16
+Torch distributed is available.
+Torch distributed is initialized.
+[info] Setting seed:  123 . worker seed:  1749090055
+
+
+found num checkpoints:  0
+resume from checkpoints:  False
+resume checkpoint:  None
+[info] construct file from initialization
+[info] input dir =  /home/gta/Cosim_test/dataset/hdf5
+[info] num files =  2
+Loaded 193485 samples from datafile: /home/gta/Cosim_test/dataset/hdf5/pretrain-part-01.hdf5
+bert_train latency:  0.2730390548706055  s
+bert_train throughput:  58.599675447831  sentences/s
+perplexity = 11.12635612487793
+```
+
+Final results of the inference run can be found in `results.yaml` file.
+```
+results:
+ - key: throughput
+   value: 66.259994
+   unit: sent/s
+ - key: latency
+   value: 0.2414730072021484
+   unit: s
+ - key: accuracy
+   value: 11.021
+   unit: perplexity
+```
diff --git a/models_v2/pytorch/bert_large/training/gpu/bert_config.json b/models_v2/pytorch/bert_large/training/gpu/bert_config.json
new file mode 100644
index 000000000..14c938cd2
--- /dev/null
+++ b/models_v2/pytorch/bert_large/training/gpu/bert_config.json
@@ -0,0 +1,14 @@
+{
+    "attention_probs_dropout_prob": 0.1,
+    "hidden_act": "gelu",
+    "hidden_dropout_prob": 0.1,
+    "hidden_size": 1024,
+    "initializer_range": 0.02,
+    "intermediate_size": 4096,
+    "max_position_embeddings": 512,
+    "num_attention_heads": 16,
+    "num_hidden_layers": 24,
+    "type_vocab_size": 2,
+    "vocab_size": 30522,
+    "model_type": "bert"
+}
diff --git a/models_v2/pytorch/bert_large/training/gpu/bert_converge.sh b/models_v2/pytorch/bert_large/training/gpu/bert_converge.sh
new file mode 100755
index 000000000..8a05340ed
--- /dev/null
+++ b/models_v2/pytorch/bert_large/training/gpu/bert_converge.sh
@@ -0,0 +1,78 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+#converge
+export Train_DATASET=/pytorch/bert_dataset_from_mlperf_scp/hdf5/training-4320/hdf5_4320_shards_varlength
+export Eval_DATASET=/pytorch/bert_dataset_from_mlperf_scp/hdf5/eval_varlength
+export Output_RESULT=./result
+# ckpt trained on phase1, load it to run pretrain phase2
+export MODEL_NAME_PATH=/pytorch/bert_dataset_from_mlperf_scp/checkpoint
+export CKPT_PATH=./result
+
+echo "Bert single tile convergence"
+echo "[info] Train_DATASET:"
+printenv Train_DATASET
+echo "[info] Eval_DATASET:"
+printenv Eval_DATASET
+echo "[info] Output_RESULT:"
+printenv Output_RESULT
+echo "[info] MODEL_NAME_PATH:"
+printenv MODEL_NAME_PATH
+echo "[info] CKPT_PATH:"
+printenv CKPT_PATH
+
+# max_steps means how many steps this training will run
+# max_steps_for_scheduler means how many steps the scheduler will use to decrease the lr
+# min_learning_rate means the minial lr the scheduler will decrease the lr to
+# learning_rate means the initial lr
+
+# [watch out] min learning rate means the minimal learning rate the scheduler will decay lr to
+python -u run_pretrain_mlperf.py \
+    --input_dir=$Train_DATASET \
+    --output_dir=$Output_RESULT \
+    --eval_dir=$Eval_DATASET \
+    --model_name_or_path=$MODEL_NAME_PATH \
+    --device=xpu \
+    --do_train \
+    --train_batch_size=16 \
+    --eval_batch_size=16 \
+    --gradient_accumulation_steps=1 \
+    --bf16 \
+    --lamb \
+    --phase2 \
+    --workers 4 \
+    --amp \
+    --converge \
+    --opt_lamb_beta_1=0.9 \
+    --opt_lamb_beta_2=0.999 \
+    --warmup_steps=0 \
+    --start_warmup_step=0 \
+    --learning_rate=2e-4 \
+    --min_learning_rate=9e-6 \
+    --weight_decay_rate=0.01 \
+    --max_steps=1313280 \
+    --max_steps_for_scheduler=1094400 \
+    --max_samples_termination=21012480 \
+    --eval_iter_start_samples=50000 \
+    --eval_iter_samples=50000 \
+    --num_samples_per_checkpoint 50000 \
+    --min_samples_to_start_checkpoints 10000 \
+    --keep_n_most_recent_checkpoints 10 \
+    --seed 123
+    # --resume_from_checkpoint \
+    # --resume_checkpoint $CKPT_PATH
diff --git a/models_v2/pytorch/bert_large/training/gpu/ddp-bert.py b/models_v2/pytorch/bert_large/training/gpu/ddp-bert.py
new file mode 100644
index 000000000..adfa47ea0
--- /dev/null
+++ b/models_v2/pytorch/bert_large/training/gpu/ddp-bert.py
@@ -0,0 +1,35 @@
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Consts for ddp-bert training."""
+
+ACC = {
+    "type": "mean",
+    "pattern": r"perplexity = (\d+.\d+)",
+    "unit": "perplexity",
+}
+
+PERF = {
+    "type": "total",
+    "pattern": r"bert_train throughput:  (\d+.\d+)  sentences/s",
+    "inverse": False,
+    "multiply": False,
+    "use_batch_size": False,
+    "unit": "sent/s",
+}
+
+FUNCTIONAL = {
+    "pattern": r"perplexity = (.*)",
+}
diff --git a/models_v2/pytorch/bert_large/training/gpu/gen_wiki_dataset.sh b/models_v2/pytorch/bert_large/training/gpu/gen_wiki_dataset.sh
new file mode 100755
index 000000000..dea57ad0c
--- /dev/null
+++ b/models_v2/pytorch/bert_large/training/gpu/gen_wiki_dataset.sh
@@ -0,0 +1,28 @@
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+pip install -r requirements.txt
+
+WIKI_BASE_DIR=$1
+
+python ./data/wiki_downloader.py --language=en --save_path=${WIKI_BASE_DIR}
+python ./data/WikiExtractor.py ${WIKI_BASE_DIR}wikicorpus_en/wikicorpus_en.xml -o ${WIKI_BASE_DIR}wikicorpus_en/text
+cd data/wikicleaner/
+bash run.sh "${WIKI_BASE_DIR}wikicorpus_en/text/*/wiki_??" ${WIKI_BASE_DIR}wikicorpus_en/results
+cd ..
+python vocab_downloader.py --type=bert-base-uncased
+mv bert-base-uncased-vocab.txt ${WIKI_BASE_DIR}wikicorpus_en
+export VOCAB_FILE=${WIKI_BASE_DIR}wikicorpus_en/bert-base-uncased-vocab.txt
+bash parallel_create_pretraining_data.sh ${WIKI_BASE_DIR}wikicorpus_en/results/
diff --git a/models_v2/pytorch/bert_large/training/gpu/lamb.py b/models_v2/pytorch/bert_large/training/gpu/lamb.py
new file mode 100644
index 000000000..4b4609269
--- /dev/null
+++ b/models_v2/pytorch/bert_large/training/gpu/lamb.py
@@ -0,0 +1,189 @@
+#
+# Copyright (c) 2022-2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Lamb optimizer."""
+
+import collections
+import math
+
+import torch
+from tensorboardX import SummaryWriter
+from torch.optim import Optimizer
+
+
+def log_lamb_rs(optimizer: Optimizer, event_writer: SummaryWriter, token_count: int):
+    """Log a histogram of trust ratio scalars in across layers."""
+    results = collections.defaultdict(list)
+    for group in optimizer.param_groups:
+        for p in group["params"]:
+            state = optimizer.state[p]
+            for i in ("weight_norm", "adam_norm", "trust_ratio"):
+                if i in state:
+                    results[i].append(state[i])
+
+    for k, v in results.items():
+        event_writer.add_histogram(f"lamb/{k}", torch.tensor(v), token_count)
+
+
+class Lamb(Optimizer):
+    r"""Implements Lamb algorithm.
+
+    It has been proposed in `Large Batch Optimization for Deep Learning: Training BERT in 76 minutes`_.
+
+    Arguments:
+        params (iterable): iterable of parameters to optimize or dicts defining
+            parameter groups
+        lr (float, optional): learning rate (default: 1e-3)
+        betas (Tuple[float, float], optional): coefficients used for computing
+            running averages of gradient and its square (default: (0.9, 0.999))
+        eps (float, optional): term added to the denominator to improve
+            numerical stability (default: 1e-8)
+        weight_decay (float, optional): weight decay (L2 penalty) (default: 0)
+        adam (bool, optional): always use trust ratio = 1, which turns this into
+            Adam. Useful for comparison purposes.
+
+    .. _Large Batch Optimization for Deep Learning: Training BERT in 76 minutes:
+        https://arxiv.org/abs/1904.00962
+    """
+
+    def __init__(
+        self,
+        params,
+        lr=1e-3,
+        betas=(0.9, 0.999),
+        eps=1e-6,
+        weight_decay=0,
+        adam=False,
+        bias_correction=True,
+        perform_allreduce=False,
+    ):
+        if not 0.0 <= lr:
+            raise ValueError("Invalid learning rate: {}".format(lr))
+        if not 0.0 <= eps:
+            raise ValueError("Invalid epsilon value: {}".format(eps))
+        if not 0.0 <= betas[0] < 1.0:
+            raise ValueError("Invalid beta parameter at index 0: {}".format(betas[0]))
+        if not 0.0 <= betas[1] < 1.0:
+            raise ValueError("Invalid beta parameter at index 1: {}".format(betas[1]))
+        defaults = dict(lr=lr, betas=betas, eps=eps, weight_decay=weight_decay)
+        self.adam = adam
+        self.bias_correction = bias_correction
+        self.perform_allreduce = perform_allreduce
+        self.distributed = (
+            torch.distributed.is_initialized()
+            and torch.distributed.get_world_size() > 1
+        )
+        super(Lamb, self).__init__(params, defaults)
+
+    def sync_params(self):
+        if not self.distributed:
+            return
+        for group in self.param_groups:
+            for p in group["params"]:
+                torch.distributed.broadcase(p.data, 0)
+
+    def sync_grads(self):
+        if not self.distributed:
+            return
+        world_size = torch.distributed.get_world_size()
+        for group in self.param_groups:
+            for p in group["params"]:
+                p.grad.data.div_(world_size)
+                torch.distributed.all_reduce(p.grad.data)
+
+    def step(self, closure=None):
+        """Performs a single optimization step.
+
+        Arguments:
+            closure (callable, optional): A closure that reevaluates the model
+                and returns the loss.
+        """
+        loss = None
+        if closure is not None:
+            loss = closure()
+
+        if self.perform_allreduce:
+            self.sync_grads()
+
+        for group in self.param_groups:
+            for p in group["params"]:
+                if p.grad is None:
+                    continue
+                bf16_param = p.data.dtype == torch.bfloat16
+                grad = p.grad.data
+                data = p.data
+                if grad.is_sparse:
+                    raise RuntimeError(
+                        "Lamb does not support sparse gradients, consider SparseAdam instad."
+                    )
+
+                state = self.state[p]
+                # State initialization
+                if len(state) == 0:
+                    state["step"] = 0
+                    # Exponential moving average of gradient values
+                    state["exp_avg"] = torch.zeros_like(p.data, dtype=torch.float32)
+                    # Exponential moving average of squared gradient values
+                    state["exp_avg_sq"] = torch.zeros_like(p.data, dtype=torch.float32)
+                    if bf16_param:
+                        # additional fp32 version of master weights
+                        state["data_fp32"] = p.data.to(torch.float32)
+
+                exp_avg, exp_avg_sq = state["exp_avg"], state["exp_avg_sq"]
+                beta1, beta2 = group["betas"]
+                if bf16_param:
+                    grad = grad.to(torch.float32)
+                    data = state["data_fp32"]
+
+                state["step"] += 1
+
+                # Decay the first and second moment running average coefficient
+                # m_t
+                exp_avg.mul_(beta1).add_(grad, alpha=(1 - beta1))
+                # v_t
+                exp_avg_sq.mul_(beta2).addcmul_(1 - beta2, grad, grad)
+
+                step_size = group["lr"]
+                if self.bias_correction:
+                    # Paper v3 does not use debiasing.
+                    exp_avg_hat = exp_avg / (1 - beta1 ** state["step"])
+                    exp_avg_sq_hat = exp_avg_sq / (1 - beta2 ** state["step"])
+                    # Apply bias to lr to avoid broadcast.
+                else:
+                    exp_avg_hat = exp_avg
+                    exp_avg_sq_hat = exp_avg_sq
+
+                adam_step = exp_avg_hat / exp_avg_sq_hat.sqrt().add(group["eps"])
+                trust_ratio = 1
+                if group["weight_decay"] != 0:
+                    adam_step.add_(data, alpha=group["weight_decay"])
+
+                    weight_norm = data.pow(2).sum().sqrt()  # .clamp(0, 10)
+                    adam_norm = adam_step.pow(2).sum().sqrt()
+                    if weight_norm == 0 or adam_norm == 0:
+                        trust_ratio = 1
+                    else:
+                        trust_ratio = weight_norm / adam_norm
+                    if self.adam:
+                        trust_ratio = 1
+                    state["weight_norm"] = weight_norm
+                    state["adam_norm"] = adam_norm
+                    state["trust_ratio"] = trust_ratio
+
+                data.add_(adam_step, alpha=-step_size * trust_ratio)
+                if bf16_param:
+                    p.data = data.to(torch.bfloat16)
+
+        return loss
diff --git a/models_v2/pytorch/bert_large/training/gpu/process_mini_data.sh b/models_v2/pytorch/bert_large/training/gpu/process_mini_data.sh
new file mode 100755
index 000000000..2133d5364
--- /dev/null
+++ b/models_v2/pytorch/bert_large/training/gpu/process_mini_data.sh
@@ -0,0 +1,19 @@
+# Copyright (c) 2022-2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+pip install -r requirements.txt
+cd data
+python vocab_downloader.py --type=bert-base-uncased
+export VOCAB_FILE=bert-base-uncased-vocab.txt
+bash parallel_create_pretraining_data.sh ../miniwiki
diff --git a/models_v2/pytorch/bert_large/training/gpu/reference.py b/models_v2/pytorch/bert_large/training/gpu/reference.py
new file mode 100644
index 000000000..4f2bf8760
--- /dev/null
+++ b/models_v2/pytorch/bert_large/training/gpu/reference.py
@@ -0,0 +1,1691 @@
+#!/usr/bin/env python
+# coding=utf-8
+# Copyright 2021 The HuggingFace Inc. team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+Fine-tuning the library models for masked language modeling (BERT, ALBERT, RoBERTa...)
+on a text file or a dataset without using HuggingFace Trainer.
+
+Here is the full list of checkpoints on the hub that can be fine-tuned by this script:
+https://huggingface.co/models?filter=masked-lm
+"""
+# You can also adapt this script on your own mlm task. Pointers for this are left as comments.
+
+"""BERT Pretraining"""
+
+import argparse
+import csv
+import h5py
+import os
+import glob
+import numpy as np
+import torch
+from torch.utils.data import DataLoader, RandomSampler, SequentialSampler, Dataset
+from torch.utils.data.distributed import DistributedSampler
+import logging
+import math
+import multiprocessing
+import random
+import re
+import time
+import sys
+
+from collections import OrderedDict
+from concurrent.futures import ProcessPoolExecutor
+
+from schedulers import LinearWarmupPolyDecayScheduler
+
+import utils
+
+import torch.nn.functional as F
+import argparse
+
+
+import transformers
+import modeling_bert_patched
+
+# from accelerate import Accelerator, DistributedType
+from transformers import (
+    CONFIG_MAPPING,
+    MODEL_MAPPING,
+    AutoConfig,
+    AutoModelForPreTraining,
+    SchedulerType,
+    get_scheduler,
+    set_seed,
+)
+
+from pcl_pytorch_extension import bert as pcl_bert
+from schedulers import LinearWarmUpScheduler, LinearWarmupPolyDecayScheduler
+import mlperf_logger
+from lamb import Lamb
+
+from pcl_pytorch_extension.optim import DistLamb
+
+ref_time = 0
+
+
+def get_time():
+    global ref_time
+    t = time.time()
+    return (t - ref_time) * 1000.0
+
+
+def sync_clock():
+    global ref_time
+    t = time.time()
+    if torch.distributed.is_initialized():
+        torch.distributed.barrier()
+        t = time.time()
+        t = torch.tensor([t], dtype=torch.double)
+        torch.distributed.broadcast(t, 0)
+        torch.distributed.barrier()
+        t = t.item()
+    ref_time = t
+
+
+logger = logging.getLogger(__name__)
+MODEL_CONFIG_CLASSES = list(MODEL_MAPPING.keys())
+MODEL_TYPES = tuple(conf.model_type for conf in MODEL_CONFIG_CLASSES)
+
+
+class WorkerInitObj(object):
+    def __init__(self, seed):
+        self.seed = seed
+
+    def __call__(self, id):
+        np.random.seed(seed=self.seed + id)
+        random.seed(self.seed + id)
+
+
+def get_eval_batchsize_per_worker(args):
+    if torch.distributed.is_initialized():
+        chunk_size = args.num_eval_examples // args.world_size
+        rank = args.local_rank
+        remainder = args.num_eval_examples % args.world_size
+        if rank < remainder:
+            return chunk_size + 1
+        else:
+            return chunk_size
+
+
+def create_pretraining_dataset(
+    input_file, max_pred_length, shared_list, args, worker_init_fn
+):
+    train_data = pretraining_dataset(
+        input_file=input_file, max_pred_length=max_pred_length
+    )
+    train_sampler = RandomSampler(train_data)
+    train_dataloader = DataLoader(
+        train_data, sampler=train_sampler, batch_size=args.train_batch_size
+    )
+
+    return train_dataloader, input_file
+
+
+def create_eval_dataset(args, worker_init_fn):
+    eval_data = []
+    for eval_file in sorted(os.listdir(args.eval_dir)):
+        eval_file_path = os.path.join(args.eval_dir, eval_file)
+
+        if os.path.isfile(eval_file_path) and "part" in eval_file_path:
+            eval_data.extend(
+                pretraining_dataset(
+                    eval_file_path, max_pred_length=args.max_predictions_per_seq
+                )
+            )
+            if len(eval_data) > args.num_eval_examples:
+                eval_data = eval_data[: args.num_eval_examples]
+                break
+    if torch.distributed.is_initialized():
+        chunk_size = args.num_eval_examples // args.world_size
+        rank = args.local_rank
+        remainder = args.num_eval_examples % args.world_size
+        if rank < remainder:
+            eval_data = eval_data[
+                (chunk_size + 1) * rank : (chunk_size + 1) * (rank + 1)
+            ]
+        else:
+            eval_data = eval_data[
+                chunk_size * rank + remainder : chunk_size * (rank + 1) + remainder
+            ]
+
+    eval_sampler = SequentialSampler(eval_data)
+    eval_dataloader = DataLoader(
+        eval_data, sampler=eval_sampler, batch_size=args.eval_batch_size, num_workers=0
+    )
+    return eval_dataloader
+
+
+class pretraining_dataset(Dataset):
+    def __init__(self, input_file, max_pred_length):
+        self.input_file = input_file
+        self.max_pred_length = max_pred_length
+        f = h5py.File(input_file, "r")
+        keys = [
+            "input_ids",
+            "input_mask",
+            "segment_ids",
+            "masked_lm_positions",
+            "masked_lm_ids",
+            "next_sentence_labels",
+        ]
+        self.inputs = [np.asarray(f[key][:]) for key in keys]
+        print(f"Loaded {len(self.inputs[0]):d} samples from datafile: {input_file}")
+        f.close()
+
+    def __len__(self):
+        "Denotes the total number of samples"
+        return len(self.inputs[0])
+
+    def __getitem__(self, index):
+        [
+            input_ids,
+            input_mask,
+            segment_ids,
+            masked_lm_positions,
+            masked_lm_ids,
+            next_sentence_labels,
+        ] = [
+            torch.from_numpy(input[index].astype(np.int64))
+            if indice < 5
+            else torch.from_numpy(np.asarray(input[index].astype(np.int64)))
+            for indice, input in enumerate(self.inputs)
+        ]
+        masked_lm_labels = torch.zeros(input_ids.shape, dtype=torch.long) - 100
+        index = self.max_pred_length
+        masked_token_count = torch.count_nonzero(masked_lm_positions)
+        if masked_token_count != 0:
+            index = masked_token_count
+        masked_lm_labels[masked_lm_positions[:index]] = masked_lm_ids[:index]
+
+        return [
+            input_ids,
+            segment_ids,
+            input_mask,
+            masked_lm_labels,
+            next_sentence_labels,
+        ]
+
+
+def parse_args():
+    parser = argparse.ArgumentParser(
+        description="Finetune a transformers model on a Masked Language Modeling task"
+    )
+
+    ## Required parameters
+    parser.add_argument(
+        "--input_dir",
+        default=None,
+        type=str,
+        required=True,
+        help="The input data dir. Should contain .hdf5 files  for the task.",
+    )
+    parser.add_argument(
+        "--output_dir",
+        default=None,
+        type=str,
+        required=True,
+        help="The output directory where the model checkpoints will be written.",
+    )
+    parser.add_argument(
+        "--eval_dir",
+        default=None,
+        type=str,
+        help="The eval data dir. Should contain .hdf5 files  for the task.",
+    )
+    parser.add_argument(
+        "--eval_iter_start_samples",
+        default=3000000,
+        type=int,
+        help="Sample to begin performing eval.",
+    )
+    parser.add_argument(
+        "--eval_iter_samples",
+        default=-1,
+        type=int,
+        help="If set to -1, disable eval, \
+                        else evaluate every eval_iter_samples during training",
+    )
+    parser.add_argument(
+        "--num_eval_examples",
+        default=10000,
+        type=int,
+        help="number of eval examples to run eval on",
+    )
+    parser.add_argument(
+        "--init_checkpoint",
+        default=None,
+        type=str,
+        help="The initial checkpoint to start training from.",
+    )
+    parser.add_argument(
+        "--init_tf_checkpoint",
+        default=None,
+        type=str,
+        help="The initial TF checkpoint to start training from.",
+    )
+    parser.add_argument(
+        "--train_file",
+        type=str,
+        default=None,
+        help="A csv or a json file containing the training data.",
+    )
+    parser.add_argument(
+        "--validation_file",
+        type=str,
+        default=None,
+        help="A csv or a json file containing the validation data.",
+    )
+    parser.add_argument(
+        "--model_name_or_path",
+        type=str,
+        help="Path to pretrained model or model identifier from huggingface.co/models.",
+        required=True,
+    )
+    parser.add_argument(
+        "--config_name",
+        type=str,
+        default=None,
+        help="Pretrained config name or path if not the same as model_name",
+    )
+    parser.add_argument(
+        "--per_device_train_batch_size",
+        type=int,
+        default=8,
+        help="Batch size (per device) for the training dataloader.",
+    )
+    parser.add_argument(
+        "--per_device_eval_batch_size",
+        type=int,
+        default=8,
+        help="Batch size (per device) for the evaluation dataloader.",
+    )
+    parser.add_argument(
+        "--learning_rate",
+        type=float,
+        default=5e-5,
+        help="Initial learning rate (after the potential warmup period) to use.",
+    )
+    parser.add_argument(
+        "--max_predictions_per_seq",
+        default=76,
+        type=int,
+        help="The maximum total of masked tokens in input sequence",
+    )
+    parser.add_argument(
+        "--train_batch_size",
+        default=18,
+        type=int,
+        help="Total batch size for training.",
+    )
+    parser.add_argument(
+        "--eval_batch_size",
+        default=128,
+        type=int,
+        help="Total batch size for training.",
+    )
+    parser.add_argument(
+        "--weight_decay_rate",
+        default=0.01,
+        type=float,
+        help="weight decay rate for LAMB.",
+    )
+    parser.add_argument(
+        "--opt_lamb_beta_1", default=0.9, type=float, help="LAMB beta1."
+    )
+    parser.add_argument(
+        "--opt_lamb_beta_2", default=0.999, type=float, help="LAMB beta2."
+    )
+    parser.add_argument(
+        "--max_steps",
+        default=1536,
+        type=float,
+        help="Total number of training steps to perform.",
+    )
+    parser.add_argument(
+        "--max_samples_termination",
+        default=14000000,
+        type=float,
+        help="Total number of training samples to run.",
+    )
+    parser.add_argument(
+        "--warmup_proportion",
+        default=0.01,
+        type=float,
+        help="Proportion of optimizer update steps to perform linear learning rate warmup for. "
+        "Typically 1/8th of steps for Phase2",
+    )
+    parser.add_argument(
+        "--warmup_steps",
+        default=0,
+        type=float,
+        help="Number of optimizer update steps to perform linear learning rate warmup for. "
+        "Typically 1/8th of steps for Phase2",
+    )
+    parser.add_argument(
+        "--start_warmup_step", default=0, type=float, help="Starting step for warmup. "
+    )
+    parser.add_argument(
+        "--log_freq",
+        type=float,
+        default=10000.0,
+        help="frequency of logging loss. If not positive, no logging is provided for training loss",
+    )
+    parser.add_argument(
+        "--checkpoint_activations",
+        default=False,
+        action="store_true",
+        help="Whether to use gradient checkpointing",
+    )
+    parser.add_argument(
+        "--resume_from_checkpoint",
+        default=False,
+        action="store_true",
+        help="Whether to resume training from checkpoint. If set, precedes init_checkpoint/init_tf_checkpoint",
+    )
+    parser.add_argument(
+        "--keep_n_most_recent_checkpoints",
+        type=int,
+        default=20,
+        help="Number of checkpoints to keep (rolling basis).",
+    )
+    parser.add_argument(
+        "--num_samples_per_checkpoint",
+        type=int,
+        default=500000,
+        help="Number of update steps until a model checkpoint is saved to disk.",
+    )
+    parser.add_argument(
+        "--min_samples_to_start_checkpoints",
+        type=int,
+        default=3000000,
+        help="Number of update steps until model checkpoints start saving to disk.",
+    )
+    parser.add_argument(
+        "--skip_checkpoint",
+        default=False,
+        action="store_true",
+        help="Whether to save checkpoints",
+    )
+    parser.add_argument(
+        "--phase2",
+        default=False,
+        action="store_true",
+        help="Only required for checkpoint saving format",
+    )
+    parser.add_argument(
+        "--do_train",
+        default=False,
+        action="store_true",
+        help="Whether to run training.",
+    )
+    parser.add_argument(
+        "--bert_config_path",
+        type=str,
+        default="/workspace/phase1",
+        help="Path bert_config.json is located in",
+    )
+    parser.add_argument(
+        "--target_mlm_accuracy",
+        type=float,
+        default=0.72,
+        help="Stop training after reaching this Masked-LM accuracy",
+    )
+    parser.add_argument(
+        "--train_mlm_accuracy_window_size",
+        type=int,
+        default=0,
+        help="Average accuracy over this amount of batches before performing a stopping criterion test",
+    )
+    parser.add_argument(
+        "--num_epochs_to_generate_seeds_for",
+        type=int,
+        default=2,
+        help="Number of epochs to plan seeds for. Same set across all workers.",
+    )
+    parser.add_argument(
+        "--use_gradient_as_bucket_view",
+        default=False,
+        action="store_true",
+        help="Turn ON gradient_as_bucket_view optimization in native DDP.",
+    )
+    parser.add_argument(
+        "--dense_seq_output",
+        default=False,
+        action="store_true",
+        help="Whether to run with optimizations.",
+    )
+    parser.add_argument(
+        "--no_ddp", default=False, action="store_true", help="Whether to use DDP."
+    )
+    parser.add_argument(
+        "--dist_lamb",
+        default=False,
+        action="store_true",
+        help="Whether to use DistLamb from pcl_bert",
+    )
+
+    parser.add_argument(
+        "--weight_decay", type=float, default=0.0, help="Weight decay to use."
+    )
+    parser.add_argument(
+        "--num_train_epochs",
+        type=int,
+        default=3,
+        help="Total number of training epochs to perform.",
+    )
+    parser.add_argument(
+        "--gradient_accumulation_steps",
+        type=int,
+        default=1,
+        help="Number of updates steps to accumulate before performing a backward/update pass.",
+    )
+    parser.add_argument(
+        "--lr_scheduler_type",
+        type=SchedulerType,
+        default="linear",
+        help="The scheduler type to use.",
+        choices=[
+            "linear",
+            "cosine",
+            "cosine_with_restarts",
+            "polynomial",
+            "constant",
+            "constant_with_warmup",
+        ],
+    )
+    parser.add_argument(
+        "--seed", type=int, default=42, help="A seed for reproducible training."
+    )
+    parser.add_argument(
+        "--model_type",
+        type=str,
+        default=None,
+        help="Model type to use if training from scratch.",
+        choices=MODEL_TYPES,
+    )
+
+    parser.add_argument(
+        "--preprocessing_num_workers",
+        type=int,
+        default=None,
+        help="The number of processes to use for the preprocessing.",
+    )
+    parser.add_argument(
+        "--local_rank", default=0, type=int, help="Total batch size for training."
+    )
+    parser.add_argument(
+        "--world_size", default=1, type=int, help="Total batch size for training."
+    )
+
+    parser.add_argument(
+        "--use_pcl",
+        action="store_true",
+        help="Whether to use PCL Fused impl when available",
+    )
+    parser.add_argument(
+        "--pcl_bf16", action="store_true", help="Whether to use PCL BF16 impl"
+    )
+    parser.add_argument(
+        "--profile", action="store_true", help="Whether to enable profiling"
+    )
+
+    parser.add_argument(
+        "--skip_exchange_padding",
+        default=False,
+        action="store_true",
+        help="Whether to skip padding exchange.",
+    )
+    parser.add_argument(
+        "--unpad",
+        default=False,
+        action="store_true",
+        help="Whether to run with unpadding.",
+    )
+    parser.add_argument(
+        "--benchmark_steps",
+        type=int,
+        default=0,
+        help="Number of steps to run for benchmark.",
+    )
+    parser.add_argument(
+        "--dist_backend",
+        type=str,
+        default="ccl",
+        help="Specify distributed backend to use.",
+    )
+    parser.add_argument(
+        "--multi_instance",
+        action="store_true",
+        help="Skip torch.distributed initialization to Run multiple instance independently",
+    )
+    parser.add_argument(
+        "--dist_profile",
+        action="store_true",
+        help="Whether to enable distributed timing profile",
+    )
+
+    args = parser.parse_args()
+
+    if args.output_dir is not None:
+        os.makedirs(args.output_dir, exist_ok=True)
+    # assert args.init_checkpoint is not None or args.init_tf_checkpoint is not None or found_resume_checkpoint(args), \
+    #    "Must specify --init_checkpoint, --init_tf_checkpoint or have ckpt to resume from in --output_dir of the form *.pt"
+
+    # assert not (args.init_checkpoint is not None and args.init_tf_checkpoint is not None), \
+    #        "Can only specify one of --init_checkpoint and --init_tf_checkpoint"
+    return args
+
+
+def found_resume_checkpoint(args):
+    if args.phase2:
+        checkpoint_str = "phase2_ckpt*.pt"
+    else:
+        checkpoint_str = "phase1_ckpt*.pt"
+    return (
+        args.resume_from_checkpoint
+        and len(glob.glob(os.path.join(args.output_dir, checkpoint_str))) > 0
+    )
+
+
+def setup_training(args):
+    device = torch.device("cpu")
+    if int(os.environ.get("PMI_SIZE", "0")) > 1 and not args.multi_instance:
+        if args.dist_backend == "ccl":
+            try:
+                import torch_ccl
+            except:
+                print("CCL backend requested but import torch_ccl failed")
+                raise
+        elif args.dist_backend == "mpi":
+            if not torch.distributed.is_mpi_available():
+                try:
+                    import torch_mpi
+                except:
+                    print(
+                        "MPI backend requested but not available try installing torch_mpi module"
+                    )
+                    raise
+        else:
+            raise ValueError(f"{args.dist_backend} backend requested but not supported")
+
+        os.environ["RANK"] = os.environ.get("PMI_RANK", "0")
+        os.environ["WORLD_SIZE"] = os.environ.get("PMI_SIZE", "1")
+        torch.distributed.init_process_group(backend=args.dist_backend)
+        device = torch.device("cpu")
+        args.local_rank = torch.distributed.get_rank()
+        args.world_size = torch.distributed.get_world_size()
+        if args.local_rank == 0:
+            print(
+                f"##################Using {args.dist_backend.upper()} dist run with {args.world_size} ranks",
+                flush=True,
+            )
+    if args.gradient_accumulation_steps < 1:
+        raise ValueError(
+            "Invalid gradient_accumulation_steps parameter: {}, should be >= 1".format(
+                args.gradient_accumulation_steps
+            )
+        )
+    if args.train_batch_size % args.gradient_accumulation_steps != 0:
+        raise ValueError(
+            "Invalid gradient_accumulation_steps parameter: {}, batch size {} should be divisible".format(
+                args.gradient_accumulation_steps, args.train_batch_size
+            )
+        )
+
+    args.train_batch_size = args.train_batch_size // args.gradient_accumulation_steps
+
+    if not (args.do_train or (args.eval_dir and args.eval_iter_samples <= 0)):
+        raise ValueError(" `do_train`  or should be in offline eval mode")
+
+    if not args.resume_from_checkpoint or not os.path.exists(args.output_dir):
+        os.makedirs(args.output_dir, exist_ok=True)
+    return device, args
+
+
+def prepare_model_and_optimizer(args, device):
+    global_step = 0
+    args.resume_step = 0
+    checkpoint = None
+    # download model & vocab.
+    if args.config_name:
+        config = AutoConfig.from_pretrained(args.config_name)
+    elif args.model_name_or_path:
+        config = AutoConfig.from_pretrained(args.model_name_or_path)
+    else:
+        config = CONFIG_MAPPING[args.model_type]()
+        logger.warning("You are instantiating a new config instance from scratch.")
+
+    config.dense_seq_output = args.dense_seq_output
+    with pcl_bert.pcl_impl(args.use_pcl, args.pcl_bf16, args.unpad):
+        if args.model_name_or_path:
+            model = AutoModelForPreTraining.from_pretrained(
+                args.model_name_or_path,
+                from_tf=bool(".ckpt" in args.model_name_or_path),
+                config=config,
+            )
+        else:
+            logger.info("Training new model from scratch")
+            model = AutoModelForPreTraining.from_config(config)
+    if args.use_pcl:
+        pcl_bert.block(model)
+
+    param_optimizer = list(model.named_parameters())
+
+    no_decay = ["bias", "gamma", "beta", "LayerNorm"]
+
+    optimizer_grouped_parameters = [
+        {
+            "params": [
+                p for n, p in param_optimizer if not any(nd in n for nd in no_decay)
+            ],
+            "weight_decay": args.weight_decay_rate,
+        },
+        {
+            "params": [
+                p for n, p in param_optimizer if any(nd in n for nd in no_decay)
+            ],
+            "weight_decay": 0.0,
+        },
+    ]
+
+    print(
+        "GroupSizes: ",
+        [sum([p.numel() for p in g["params"]]) for g in optimizer_grouped_parameters],
+    )
+    if args.dist_lamb:
+        optimizer = DistLamb(
+            optimizer_grouped_parameters,
+            lr=args.learning_rate,
+            betas=(args.opt_lamb_beta_1, args.opt_lamb_beta_2),
+        )
+    else:
+        optimizer = Lamb(
+            optimizer_grouped_parameters,
+            lr=args.learning_rate,
+            betas=(args.opt_lamb_beta_1, args.opt_lamb_beta_2),
+        )
+
+    mlperf_logger.log_event(
+        key=mlperf_logger.constants.OPT_BASE_LR, value=args.learning_rate, sync=False
+    )
+    mlperf_logger.log_event(
+        key="opt_epsilon", value=optimizer.defaults["eps"], sync=False
+    )
+    b1, b2 = optimizer.defaults["betas"]
+    mlperf_logger.log_event(key="opt_lamb_beta_1", value=b1, sync=False)
+    mlperf_logger.log_event(key="opt_lamb_beta_2", value=b2, sync=False)
+    mlperf_logger.log_event(
+        key="opt_lamb_weight_decay_rate",
+        value=optimizer.defaults["weight_decay"],
+        sync=False,
+    )
+
+    if args.warmup_steps == 0:
+        warmup_steps = int(args.max_steps * args.warmup_proportion)
+        warmup_start = 0
+    else:
+        warmup_steps = args.warmup_steps
+        warmup_start = args.start_warmup_step
+
+    lr_scheduler = LinearWarmupPolyDecayScheduler(
+        optimizer,
+        start_warmup_steps=warmup_start,
+        warmup_steps=warmup_steps,
+        total_steps=args.max_steps,
+        end_learning_rate=0.0,
+        degree=1.0,
+    )
+    return model, optimizer, lr_scheduler, checkpoint, global_step
+
+
+def take_optimizer_step(args, optimizer, model, overflow_buf, global_step):
+    global skipped_steps
+    optimizer.step()
+    global_step += 1
+    return global_step
+
+
+def run_eval(
+    model,
+    eval_dataloader,
+    device,
+    num_eval_examples,
+    args,
+    first_eval=False,
+    use_cache=False,
+):
+    model.eval()
+    total_eval_loss, total_eval_mlm_acc = 0.0, 0.0
+    total_masked = 0
+    with torch.no_grad():
+        for batch in eval_dataloader:
+            (
+                input_ids,
+                segment_ids,
+                input_mask,
+                masked_lm_labels,
+                next_sentence_labels,
+            ) = batch
+            outputs = model(
+                input_ids=input_ids,
+                token_type_ids=segment_ids,
+                attention_mask=input_mask,
+                labels=masked_lm_labels,
+                next_sentence_label=next_sentence_labels,
+            )
+            mlm_acc, num_masked = calc_mlm_acc(
+                outputs, masked_lm_labels, args.dense_seq_output
+            )
+            total_eval_loss += outputs.loss.item() * num_masked
+            total_eval_mlm_acc += mlm_acc * num_masked
+            total_masked += num_masked
+    model.train()
+    total_masked = torch.tensor(total_masked, device=device, dtype=torch.int64)
+    total_eval_loss = torch.tensor(total_eval_loss, device=device, dtype=torch.float64)
+    if torch.distributed.is_initialized():
+        # Collect total scores from all ranks
+        torch.distributed.all_reduce(
+            total_eval_mlm_acc, op=torch.distributed.ReduceOp.SUM
+        )
+        torch.distributed.all_reduce(total_eval_loss, op=torch.distributed.ReduceOp.SUM)
+        torch.distributed.all_reduce(total_masked, op=torch.distributed.ReduceOp.SUM)
+
+    # Average by number of examples
+    total_eval_mlm_acc /= total_masked
+    total_eval_loss /= total_masked
+
+    return total_eval_loss, total_eval_mlm_acc
+
+
+def global_batch_size(args):
+    return args.train_batch_size * args.gradient_accumulation_steps * args.world_size
+
+
+def calc_mlm_acc(outputs, masked_lm_labels, dense_seq_output=False):
+    prediction_scores = outputs.prediction_logits
+    masked_lm_labels_flat = masked_lm_labels.view(-1)
+    mlm_labels = masked_lm_labels_flat[masked_lm_labels_flat != -100]
+    if not dense_seq_output:
+        prediction_scores_flat = prediction_scores.view(-1, prediction_scores.shape[-1])
+        mlm_predictions_scores = prediction_scores_flat[masked_lm_labels_flat != -100]
+        mlm_predictions = mlm_predictions_scores.argmax(dim=-1)
+    else:
+        mlm_predictions = prediction_scores.argmax(dim=-1)
+
+    num_masked = mlm_labels.numel()
+    mlm_acc = (mlm_predictions == mlm_labels).sum(dtype=torch.float) / num_masked
+
+    return mlm_acc, num_masked
+
+
+def calc_accuracy(outputs, masked_lm_labels, next_sentence_label, args):
+    loss = outputs.loss.item()
+    prediction_logits = outputs.prediction_logits
+    seq_relationship_logits = outputs.seq_relationship_logits
+    mlm_acc, num_masked = calc_mlm_acc(outputs, masked_lm_labels, args.dense_seq_output)
+    seq_acc_t = (
+        torch.argmax(seq_relationship_logits, dim=-1)
+        .eq(next_sentence_label.view([-1]))
+        .to(torch.float)
+    )
+    seq_acc_true, seq_tot = seq_acc_t.sum().item(), seq_acc_t.numel()
+    seq_acc = seq_acc_true / seq_tot
+    return loss, mlm_acc, num_masked, seq_acc, seq_tot
+
+
+def exchange_padding_fast(
+    input_ids,
+    segment_ids,
+    input_mask,
+    masked_lm_labels,
+    next_sentence_labels,
+    max_batch_size,
+):
+    device = "cpu"
+    comm_dtype = torch.int32
+    pad_size = max_batch_size - input_ids.shape[0]
+    if pad_size > 0:
+        input_ids = F.pad(input_ids, (0, 0, 0, pad_size))
+        segment_ids = F.pad(segment_ids, (0, 0, 0, pad_size))
+        input_mask = F.pad(input_mask, (0, 0, 0, pad_size))
+        masked_lm_labels = F.pad(masked_lm_labels, (0, 0, 0, pad_size))
+        next_sentence_labels = F.pad(next_sentence_labels, (0, pad_size))
+    nranks = torch.distributed.get_world_size()
+    nseqs = input_mask.shape[0]
+    ntokensperseq = input_mask.shape[1]
+    my_rank = torch.distributed.get_rank()
+
+    flattened_length_seq = nseqs * ntokensperseq
+    flattened_length_nsp = nseqs
+
+    def get_local_packet_size():
+        return 4 * flattened_length_seq + flattened_length_nsp
+
+    # Storing tensors in same order as arguments
+    def encode_packet(
+        input_ids, segment_ids, input_mask, masked_lm_labels, next_sentence_labels
+    ):
+
+        packet = torch.zeros([get_local_packet_size()], device=device, dtype=comm_dtype)
+
+        curr_pos = 0
+
+        packet[curr_pos : curr_pos + flattened_length_seq] = input_ids.view(-1)[:]
+        curr_pos += flattened_length_seq
+
+        packet[curr_pos : curr_pos + flattened_length_seq] = segment_ids.view(-1)[:]
+        curr_pos += flattened_length_seq
+
+        packet[curr_pos : curr_pos + flattened_length_seq] = input_mask.view(-1)[:]
+        curr_pos += flattened_length_seq
+
+        packet[curr_pos : curr_pos + flattened_length_seq] = masked_lm_labels.view(-1)[
+            :
+        ]
+        curr_pos += flattened_length_seq
+
+        packet[curr_pos : curr_pos + flattened_length_nsp] = next_sentence_labels.view(
+            -1
+        )[:]
+
+        return packet
+
+    def decode_packet(flat_packet):
+        packet = flat_packet.view(nranks, get_local_packet_size())
+
+        curr_pos = 0
+
+        input_ids_ = (
+            packet[:, curr_pos : curr_pos + flattened_length_seq]
+            .contiguous()
+            .view(nranks, nseqs, ntokensperseq)
+        )
+        curr_pos += flattened_length_seq
+
+        segment_ids_ = (
+            packet[:, curr_pos : curr_pos + flattened_length_seq]
+            .contiguous()
+            .view(nranks, nseqs, ntokensperseq)
+        )
+        curr_pos += flattened_length_seq
+
+        input_mask_ = (
+            packet[:, curr_pos : curr_pos + flattened_length_seq]
+            .contiguous()
+            .view(nranks, nseqs, ntokensperseq)
+        )
+        curr_pos += flattened_length_seq
+
+        masked_lm_labels_ = (
+            packet[:, curr_pos : curr_pos + flattened_length_seq]
+            .contiguous()
+            .view(nranks, nseqs, ntokensperseq)
+        )
+        curr_pos += flattened_length_seq
+
+        next_sentence_labels_ = (
+            packet[:, curr_pos : curr_pos + flattened_length_nsp]
+            .contiguous()
+            .view(nranks, nseqs)
+        )
+
+        return (
+            input_ids_,
+            segment_ids_,
+            input_mask_,
+            masked_lm_labels_,
+            next_sentence_labels_,
+        )
+
+    tensors = encode_packet(
+        input_ids, segment_ids, input_mask, masked_lm_labels, next_sentence_labels
+    )
+
+    tensors_ = torch.zeros(
+        [nranks, get_local_packet_size()], device=device, dtype=comm_dtype
+    )
+    tensors_ = list(torch.split(tensors_, 1))
+
+    # torch.distributed.all_gather(tensors_, tensors.view(torch.float16))
+    torch.distributed.all_gather(tensors_, tensors)
+
+    tensors_ = torch.stack(tensors_).view(comm_dtype).long()
+    (
+        input_ids_,
+        segment_ids_,
+        input_mask_,
+        masked_lm_labels_,
+        next_sentence_labels_,
+    ) = decode_packet(tensors_)
+
+    seqlens_, indices = torch.sort(input_mask_.sum(dim=2).view(-1), descending=True)
+
+    if pad_size > 0:
+        input_ids_sorted = input_ids_.view(nranks * nseqs, ntokensperseq)[indices[:], :]
+        segment_ids_sorted = segment_ids_.view(nranks * nseqs, ntokensperseq)[
+            indices[:], :
+        ]
+        input_mask_sorted = input_mask_.view(nranks * nseqs, ntokensperseq)[
+            indices[:], :
+        ]
+        masked_lm_labels_sorted = masked_lm_labels_.view(nranks * nseqs, ntokensperseq)[
+            indices[:], :
+        ]
+        next_sentence_labels_sorted = next_sentence_labels_.view(nranks * nseqs)[
+            indices[:]
+        ]
+        # we need to remove the empty sequences we added to the batch
+        valid_idx = seqlens_.view(nseqs, nranks)[:, my_rank] > 0
+        input_ids_sorted = input_ids_sorted.view(nseqs, nranks, ntokensperseq)[
+            valid_idx, my_rank, :
+        ].contiguous()
+        segment_ids_sorted = segment_ids_sorted.view(nseqs, nranks, ntokensperseq)[
+            valid_idx, my_rank, :
+        ].contiguous()
+        input_mask_sorted = input_mask_sorted.view(nseqs, nranks, ntokensperseq)[
+            valid_idx, my_rank, :
+        ].contiguous()
+        masked_lm_labels_sorted = masked_lm_labels_sorted.view(
+            nseqs, nranks, ntokensperseq
+        )[valid_idx, my_rank, :].contiguous()
+        next_sentence_labels_sorted = next_sentence_labels_sorted.view(nseqs, nranks)[
+            valid_idx, my_rank
+        ].contiguous()
+    else:
+        indices_ = indices.view(nseqs, nranks)[:, my_rank]
+        input_ids_sorted = input_ids_.view(nseqs * nranks, ntokensperseq)[
+            indices_, :
+        ].contiguous()
+        segment_ids_sorted = segment_ids_.view(nseqs * nranks, ntokensperseq)[
+            indices_, :
+        ].contiguous()
+        input_mask_sorted = input_mask_.view(nseqs * nranks, ntokensperseq)[
+            indices_, :
+        ].contiguous()
+        masked_lm_labels_sorted = masked_lm_labels_.view(nseqs * nranks, ntokensperseq)[
+            indices_, :
+        ].contiguous()
+        next_sentence_labels_sorted = next_sentence_labels_.view(nseqs * nranks)[
+            indices_
+        ].contiguous()
+
+    return (
+        input_ids_sorted,
+        segment_ids_sorted,
+        input_mask_sorted,
+        masked_lm_labels_sorted,
+        next_sentence_labels_sorted,
+    )
+
+
+def main():
+    args = parse_args()
+    status = "aborted"  # later set to 'success' if termination criteria met
+    mlperf_logger.log_start(
+        key=mlperf_logger.constants.INIT_START, log_all_ranks=True, sync=False
+    )
+    device, args = setup_training(args)
+    print("####################args.world_size={}".format(args.world_size))
+    total_batch_size = global_batch_size(args)
+    mlperf_logger.mlperf_submission_log("bert")
+
+    mlperf_logger.log_event(
+        key=mlperf_logger.constants.SEED, value=args.seed, sync=False
+    )
+    mlperf_logger.log_event(
+        key=mlperf_logger.constants.GLOBAL_BATCH_SIZE,
+        value=global_batch_size(args),
+        sync=False,
+    )
+    mlperf_logger.log_event(key="d_batch_size", value=args.train_batch_size, sync=False)
+    mlperf_logger.log_event(
+        key=mlperf_logger.constants.GRADIENT_ACCUMULATION_STEPS,
+        value=args.gradient_accumulation_steps,
+        sync=False,
+    )
+    mlperf_logger.log_event(
+        key="max_predictions_per_seq", value=args.max_predictions_per_seq, sync=False
+    )
+    mlperf_logger.log_event(
+        key="opt_learning_rate_training_steps", value=args.max_steps, sync=False
+    )
+    mlperf_logger.log_event(
+        key="num_warmup_steps",
+        value=int(args.warmup_proportion * args.max_steps)
+        if args.warmup_steps == 0
+        else args.warmup_steps,
+        sync=False,
+    )
+    # Initialize the accelerator. We will let the accelerator handle device placement for us in this example.
+    # Make one log on every process with the configuration for debugging.
+    logging.basicConfig(
+        format="%(asctime)s - %(levelname)s - %(name)s -   %(message)s",
+        datefmt="%m/%d/%Y %H:%M:%S",
+        level=logging.INFO,
+    )
+    if args.local_rank == 0 or args.local_rank == -1:
+        print("parsed args:")
+        print(args)
+    # Prepare optimizer
+    (
+        model,
+        optimizer,
+        lr_scheduler,
+        checkpoint,
+        global_step,
+    ) = prepare_model_and_optimizer(args, device)
+    worker_seeds, shuffling_seeds = utils.setup_seeds(
+        args.seed, args.num_epochs_to_generate_seeds_for, device
+    )
+    worker_seed = worker_seeds[args.local_rank]
+
+    random.seed(worker_seed)
+    np.random.seed(worker_seed)
+    torch.manual_seed(worker_seed)
+    worker_init = WorkerInitObj(worker_seed)
+    samples_trained = (
+        global_step
+        * args.train_batch_size
+        * args.gradient_accumulation_steps
+        * args.world_size
+    )
+    final_loss = float("inf")
+    train_time_raw = float("inf")
+    raw_train_start = time.time()
+    if args.do_train:
+        model.train()
+        most_recent_ckpts_paths = []
+        average_loss = 0.0  # averaged loss every args.log_freq steps
+        epoch = 1
+        training_steps = 0
+        end_training, converged = False, False
+        samples_trained_prev = 0
+
+        # pre-compute eval boundaries
+        samples_trained_per_step = (
+            args.train_batch_size * args.gradient_accumulation_steps * args.world_size
+        )
+        start, stop, step = (
+            args.eval_iter_start_samples,
+            args.max_samples_termination,
+            args.eval_iter_samples,
+        )
+        eval_steps = [
+            math.ceil(i / samples_trained_per_step)
+            for i in np.arange(start, stop, step)
+        ]
+        eval_count = 0
+        next_eval_step = eval_steps[eval_count]
+        # pool = ProcessPoolExecutor(1)
+
+        if args.target_mlm_accuracy:
+            if args.train_mlm_accuracy_window_size > 0:
+                accuracy_scores = []
+                avg_mlm_accuracy = torch.Tensor([0])
+
+        first_epoch = True
+        if found_resume_checkpoint(args):
+            f_start_id = checkpoint["files"][0]
+            files = checkpoint["files"][1:]
+            num_files = len(files)
+        else:
+            files = [
+                os.path.join(args.input_dir, f)
+                for f in os.listdir(args.input_dir)
+                if os.path.isfile(os.path.join(args.input_dir, f)) and "part" in f
+            ]
+            files.sort()
+            num_files = len(files)
+            random.Random(shuffling_seeds[epoch]).shuffle(files)
+            f_start_id = 0
+    global skipped_steps
+    if torch.distributed.is_initialized() and not args.no_ddp:
+        model = torch.nn.parallel.DistributedDataParallel(
+            model,
+            # bucket_cap_mb=8192,
+            gradient_as_bucket_view=args.use_gradient_as_bucket_view,
+        )
+
+    mlperf_logger.log_end(key=mlperf_logger.constants.INIT_STOP, sync=False)
+    mlperf_logger.log_start(key=mlperf_logger.constants.RUN_START, sync=True)
+    mlperf_logger.barrier()
+
+    now_step, now_skipped, skip_interval = 0, 0, 0
+    # Start prefetching eval dataset
+    # if args.eval_dir:
+    #    eval_dataset_future = pool.submit(create_eval_dataset, args, worker_init_fn=worker_init)
+    # comparing to number of samples in a shard. There are ~38k samples in 4096-way shard, comparing to 10k to be safe
+    need_next_training_shard = (
+        args.train_batch_size * args.gradient_accumulation_steps * args.max_steps
+        > 10000
+    )
+
+    sync_clock()
+
+    while global_step < args.max_steps and not end_training:
+        mlperf_logger.log_start(
+            key=mlperf_logger.constants.EPOCH_START,
+            metadata={"epoch_num": epoch},
+            sync=False,
+        )
+        mlperf_logger.log_start(
+            key=mlperf_logger.constants.BLOCK_START,
+            metadata={"first_epoch_num": epoch, "epoch_count": 1},
+            sync=False,
+        )
+        if args.local_rank == 0 or args.local_rank == -1:
+            now_time = time.time()
+            print("epoch:", epoch)
+
+        thread = None
+
+        # Reshuffle file list on subsequent epochs
+        if not first_epoch:
+            files = [
+                os.path.join(args.input_dir, f)
+                for f in os.listdir(args.input_dir)
+                if os.path.isfile(os.path.join(args.input_dir, f)) and "part" in f
+            ]
+            files.sort()
+            num_files = len(files)
+            random.Random(shuffling_seeds[epoch]).shuffle(files)
+            f_start_id = 0
+
+        first_epoch = False
+
+        shared_file_list = {}
+
+        if torch.distributed.is_initialized() and args.world_size > num_files:
+            remainder = args.world_size % num_files
+            data_file = files[
+                (
+                    f_start_id * args.world_size
+                    + args.local_rank
+                    + remainder * f_start_id
+                )
+                % num_files
+            ]
+        else:
+            data_file = files[
+                (f_start_id * args.world_size + args.local_rank) % num_files
+            ]
+
+        # data_file = '/work/ddkalamk/bert/dataset/2048_shards_uncompressed/part_453_of_2048.hdf5' #files[0]
+        previous_file = data_file
+
+        train_data = pretraining_dataset(data_file, args.max_predictions_per_seq)
+        train_sampler = RandomSampler(train_data)
+        # train_sampler = SequentialSampler(train_data)
+        train_dataloader = DataLoader(
+            train_data, sampler=train_sampler, batch_size=args.train_batch_size
+        )
+        send_lr_in_parallel = False
+        lr_cpu = torch.tensor([0.0], dtype=torch.float32, device="cpu")
+        for f_id in range(f_start_id + 1, len(files)):
+            if args.world_size > num_files:
+                data_file = files[
+                    (f_id * args.world_size + args.local_rank + remainder * f_id)
+                    % num_files
+                ]
+            else:
+                data_file = files[
+                    (f_id * args.world_size + args.local_rank) % num_files
+                ]
+
+            previous_file = data_file
+            if need_next_training_shard:
+                # dataset_future = pool.submit(create_pretraining_dataset, data_file, args.max_predictions_per_seq, shared_file_list, args, worker_init_fn=worker_init)
+                pass
+            t0 = get_time()
+            for step, batch in enumerate(train_dataloader):
+                training_steps += 1
+                if args.profile and args.use_pcl:
+                    pcl_bert.reset_debug_timers()
+                t1 = get_time()
+                (
+                    input_ids,
+                    segment_ids,
+                    input_mask,
+                    masked_lm_labels,
+                    next_sentence_labels,
+                ) = batch
+                if (
+                    args.world_size > 1
+                    and args.unpad
+                    and not args.skip_exchange_padding
+                ):
+                    (
+                        input_ids,
+                        segment_ids,
+                        input_mask,
+                        masked_lm_labels,
+                        next_sentence_labels,
+                    ) = exchange_padding_fast(
+                        input_ids,
+                        segment_ids,
+                        input_mask,
+                        masked_lm_labels,
+                        next_sentence_labels,
+                        args.train_batch_size,
+                    )
+                # print(f"Input shape: {batch['input_ids'].shape}")
+                t2 = get_time()
+                outputs = model(
+                    input_ids=input_ids,
+                    token_type_ids=segment_ids,
+                    attention_mask=input_mask,
+                    labels=masked_lm_labels,
+                    next_sentence_label=next_sentence_labels,
+                )
+                t3 = get_time()
+                loss = outputs.loss
+                # loss = loss / args.gradient_accumulation_steps
+                if (step + 1) % args.gradient_accumulation_steps == 0 or step == len(
+                    train_dataloader
+                ) - 1:
+                    loss.backward()
+                    t4 = get_time()
+                    if hasattr(optimizer, "clip_grad_norm_"):
+                        ggnorm = optimizer.clip_grad_norm_(1.0)
+                    else:
+                        ggnorm = torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
+                    t40 = get_time()
+                    mlperf_logger.barrier()
+                    t41 = get_time()
+                    if args.no_ddp:
+                        optimizer.sync_grads()
+                    t42 = get_time()
+                    optimizer.step()
+                    lr_scheduler.step()
+                    optimizer.zero_grad()
+                    # progress_bar.update(1)
+                else:
+                    # with model.no_sync():
+                    loss.backward()
+                    t4 = get_time()
+                    if hasattr(optimizer, "clip_grad_norm_"):
+                        ggnorm = optimizer.clip_grad_norm_(1.0)
+                    else:
+                        ggnorm = torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
+                    t40 = get_time()
+                    mlperf_logger.barrier()
+                    t42 = t41 = get_time()
+                    optimizer.acc_and_zero_grad()
+                # mlperf_logger.barrier()
+                t5 = get_time()
+                gloss, lm_acc, num_masked, seq_acc, seq_tot = calc_accuracy(
+                    outputs, masked_lm_labels, next_sentence_labels, args
+                )
+                data_time = t1 - t0
+                xchg_time = t2 - t1
+                fwd_time = t3 - t2
+                bwd_time = t4 - t3
+                clip_time = t40 - t4
+                sync_time = t41 - t40
+                allr_time = t42 - t41
+                opt_time = t5 - t42
+                total_time = t5 - t0
+                nnz = input_mask.sum()
+
+                if (
+                    args.dist_profile
+                    and torch.distributed.is_initialized()
+                    and args.world_size > 1
+                ):
+                    l_time = [
+                        nnz,
+                        fwd_time,
+                        bwd_time,
+                        fwd_time + bwd_time,
+                        sync_time,
+                        allr_time,
+                        total_time,
+                        t40,
+                        t41,
+                        t42,
+                    ]
+                    t_time = torch.tensor(l_time)
+                    t_all_time = torch.empty([args.world_size, len(l_time)])
+                    l_all_time = list(t_all_time.split(1))
+                    torch.distributed.all_gather(l_all_time, t_time)
+
+                if args.local_rank == 0:
+                    print(
+                        f"Step {training_steps:5d}: loss: {gloss:6.3f} lm_acc: {lm_acc:.3f} seq_acc: {seq_acc:.3f} lbs: {args.train_batch_size} gbs: {total_batch_size} DT: {data_time:.1f} XT: {xchg_time:.1f} FT: {fwd_time:.1f} BT: {bwd_time:.1f} OT: {opt_time:.1f} TT: {total_time:.1f} LR: {optimizer.param_groups[0]['lr']:.3e} GNorm: {ggnorm:.2f} ST: {sync_time:.1f} AR: {allr_time:.1f} GC: {clip_time:.1f}"
+                    )
+                    if (
+                        args.dist_profile
+                        and torch.distributed.is_initialized()
+                        and args.world_size > 1
+                    ):
+
+                        def print_summary(desc, t):
+                            if isinstance(t, torch.Tensor):
+                                t = t.tolist()
+                            tmin = min(t)
+                            tmin_i = t.index(tmin)
+                            tmax = max(t)
+                            tmax_i = t.index(tmax)
+                            tavg = sum(t) / len(t)
+                            print(
+                                f"{desc} min: {tmin:10.1f} ({tmin_i:2d}) max: {tmax:10.1f} ({tmax_i:2d})  diff: {tmax-tmin:10.1f} ({(tmax-tmin)*100.0/tmin:5.1f}) avg: {tavg:10.1f}"
+                            )
+
+                        print_summary("NNZ:", t_all_time[:, 0])
+                        print_summary("FWD:", t_all_time[:, 1])
+                        print_summary("BWD:", t_all_time[:, 2])
+                        print_summary("FBW:", t_all_time[:, 3])
+                        print_summary("SYN:", t_all_time[:, 4])
+                        print_summary("SYS:", t_all_time[:, 7])
+                        print_summary("ARS:", t_all_time[:, 8])
+                        print_summary("ARE:", t_all_time[:, 9])
+                if args.local_rank == 0 and args.profile and args.use_pcl:
+                    pcl_bert.print_debug_timers()
+                if args.benchmark_steps > 0 and global_step + 1 >= args.benchmark_steps:
+                    mlperf_logger.barrier()
+                    if args.local_rank == 0:
+                        print(f"Done Benchmarking {args.benchmark_steps} steps.")
+                    sys.exit(0)
+
+                update_step = training_steps % args.gradient_accumulation_steps == 0
+                divisor = args.gradient_accumulation_steps
+                if args.log_freq > 0:
+                    average_loss += loss.item()
+                if update_step:
+                    now_lr = optimizer.param_groups[0]["lr"]
+                    # optimizer.step()
+                    global_step += 1
+                    if (
+                        args.eval_dir
+                        and args.eval_iter_samples > 0
+                        and global_step == next_eval_step
+                    ):
+                        # on first eval, get eval_dataloader
+                        if eval_count == 0:
+                            eval_dataloader = create_eval_dataset(
+                                args, worker_init_fn=worker_init
+                            )  # eval_dataset_future.result(timeout=None)
+                        samples_trained = (
+                            global_step
+                            * args.train_batch_size
+                            * args.gradient_accumulation_steps
+                            * args.world_size
+                        )
+                        samples_trained_prev = samples_trained
+                        eval_avg_loss, eval_avg_mlm_accuracy = run_eval(
+                            model,
+                            eval_dataloader,
+                            device,
+                            args.num_eval_examples,
+                            args,
+                            first_eval=(eval_count == 0),
+                        )
+                        if args.local_rank == 0 or args.local_rank == -1:
+                            mlperf_logger.log_event(
+                                key=mlperf_logger.constants.EVAL_ACCURACY,
+                                value=eval_avg_mlm_accuracy.numpy(),
+                                metadata={"epoch_num": epoch},
+                                sync=False,
+                            )
+                            print(
+                                {
+                                    "global_steps": global_step,
+                                    "eval_loss": eval_avg_loss,
+                                    "eval_mlm_accuracy": eval_avg_mlm_accuracy,
+                                }
+                            )
+
+                        if args.target_mlm_accuracy:
+                            if eval_avg_mlm_accuracy >= args.target_mlm_accuracy:
+                                end_training, converged = True, True
+                                if utils.is_main_process():
+                                    print(
+                                        "%f > %f, Target MLM Accuracy reached at %d"
+                                        % (
+                                            eval_avg_mlm_accuracy,
+                                            args.target_mlm_accuracy,
+                                            global_step,
+                                        )
+                                    )
+
+                        eval_count += 1
+                        next_eval_step = eval_steps[eval_count]
+                if args.target_mlm_accuracy and args.train_mlm_accuracy_window_size > 0:
+                    accuracy_scores.append(mlm_acc)
+                    if update_step:
+                        accuracy_scores = accuracy_scores[
+                            -args.train_mlm_accuracy_window_size
+                            * args.gradient_accumulation_steps :
+                        ]
+                        avg_mlm_accuracy[0] = sum(accuracy_scores) / len(
+                            accuracy_scores
+                        )
+                        torch.distributed.all_reduce(
+                            avg_mlm_accuracy, op=torch.distributed.ReduceOp.SUM
+                        )
+                        avg_mlm_accuracy /= args.world_size
+
+                if (
+                    args.log_freq > 0
+                    and training_steps
+                    % (args.log_freq * args.gradient_accumulation_steps)
+                    == 0
+                ):
+                    samples_trained = (
+                        global_step
+                        * args.train_batch_size
+                        * args.gradient_accumulation_steps
+                        * args.world_size
+                    )
+                    if args.local_rank == 0 or args.local_rank == -1:
+                        time_interval = time.time() - now_time
+                        step_interval = global_step - now_step
+                        now_time = time.time()
+                        now_step = global_step
+                        training_perf = (
+                            args.train_batch_size
+                            * args.gradient_accumulation_steps
+                            * args.world_size
+                            * (step_interval + skip_interval)
+                            / time_interval
+                        )
+                        skip_interval = 0
+
+                        if args.train_mlm_accuracy_window_size > 0:
+                            print(
+                                {
+                                    "training_steps": training_steps,
+                                    "average_loss": average_loss
+                                    / (args.log_freq * divisor),
+                                    "step_loss": loss.item()
+                                    * args.gradient_accumulation_steps
+                                    / divisor,
+                                    "learning_rate": now_lr,
+                                    "seq/s": training_perf,
+                                    "global_steps": now_step,
+                                    "samples_trained": samples_trained,
+                                    "skipped_steps": now_skipped,
+                                    "timestamp": now_time,
+                                    "mlm_accuracy": avg_mlm_accuracy[0].item(),
+                                }
+                            )
+                        else:
+                            print(
+                                {
+                                    "training_steps": training_steps,
+                                    "average_loss": average_loss
+                                    / (args.log_freq * divisor),
+                                    "step_loss": loss.item()
+                                    * args.gradient_accumulation_steps
+                                    / divisor,
+                                    "learning_rate": now_lr,
+                                    "seq/s": training_perf,
+                                    "global_steps": now_step,
+                                    "samples_trained": samples_trained,
+                                    "skipped_steps": now_skipped,
+                                    "timestamp": now_time,
+                                }
+                            )
+
+                        # for DLFW CI/CD
+                        mlperf_logger.log_event(
+                            key="tracked_stats",
+                            value={
+                                "seq/sec": training_perf,
+                                "step_loss": loss.item()
+                                * args.gradient_accumulation_steps
+                                / divisor,
+                                "avg_loss": average_loss / (args.log_freq * divisor),
+                                "lr": now_lr,
+                            },
+                            metadata={"step": (epoch, training_steps)},
+                            sync=False,
+                        )
+
+                        mlperf_logger.log_event(key="throughput", value=training_perf)
+
+                    average_loss = 0
+
+                if global_step >= args.max_steps or end_training:
+                    status = "success" if converged else "aborted"
+                    end_training = True
+                    train_time_raw = time.time() - raw_train_start
+                    average_loss = torch.tensor(average_loss, dtype=torch.float32)
+                    if args.log_freq > 0:
+                        last_num_steps = (
+                            int(training_steps / args.gradient_accumulation_steps)
+                            % args.log_freq
+                        )
+                        last_num_steps = (
+                            args.log_freq if last_num_steps == 0 else last_num_steps
+                        )
+                        average_loss = average_loss / (last_num_steps * divisor)
+                    if torch.distributed.is_initialized():
+                        average_loss /= args.world_size
+                        torch.distributed.all_reduce(average_loss)
+                    final_loss = average_loss.item()
+                    if utils.is_main_process():
+                        if args.train_mlm_accuracy_window_size > 0:
+                            print(
+                                (
+                                    epoch,
+                                    training_steps / args.gradient_accumulation_steps,
+                                ),
+                                {
+                                    "final_loss": final_loss,
+                                    "final_mlm_accuracy": avg_mlm_accuracy[0].item(),
+                                },
+                            )
+                        else:
+                            print(
+                                (
+                                    epoch,
+                                    training_steps / args.gradient_accumulation_steps,
+                                ),
+                                {"final_loss": final_loss},
+                            )
+
+                if end_training or (
+                    samples_trained - samples_trained_prev
+                    >= args.num_samples_per_checkpoint
+                    and samples_trained >= args.min_samples_to_start_checkpoints
+                ):
+                    samples_trained_prev = samples_trained
+                    if utils.is_main_process() and not args.skip_checkpoint:
+                        # Save a trained model
+                        model_to_save = (
+                            model.module if hasattr(model, "module") else model
+                        )  # Only save the model it-self
+                        if args.phase2:
+                            output_save_file = os.path.join(
+                                args.output_dir,
+                                "phase2_ckpt_{}.pt".format(samples_trained),
+                            )
+                        else:
+                            output_save_file = os.path.join(
+                                args.output_dir,
+                                "phase1_ckpt_{}.pt".format(samples_trained),
+                            )
+                        if args.do_train:
+                            torch.save(
+                                {
+                                    "model": model_to_save.state_dict(),
+                                    "optimizer": optimizer.state_dict(),
+                                    "files": [f_id] + files,
+                                },
+                                output_save_file,
+                            )
+
+                            most_recent_ckpts_paths.append(output_save_file)
+                            if (
+                                len(most_recent_ckpts_paths)
+                                > args.keep_n_most_recent_checkpoints
+                            ):
+                                ckpt_to_be_removed = most_recent_ckpts_paths.pop(0)
+                                os.remove(ckpt_to_be_removed)
+
+                    if samples_trained >= args.max_samples_termination or end_training:
+                        status = "success" if converged else "aborted"
+                        end_training = True
+                        break
+                t0 = get_time()
+
+            del train_dataloader
+
+            if samples_trained >= args.max_samples_termination or end_training:
+                status = "success" if converged else "aborted"
+                end_training = True
+                break
+
+            if not need_next_training_shard:
+                # dataset_future = pool.submit(create_pretraining_dataset, data_file, args.max_predictions_per_seq, shared_file_list, args, worker_init_fn=worker_init)
+                pass
+            # train_dataloader, data_file = dataset_future.result(timeout=None)
+            train_dataloader, data_file = create_pretraining_dataset(
+                data_file,
+                args.max_predictions_per_seq,
+                shared_file_list,
+                args,
+                worker_init_fn=worker_init,
+            )
+        mlperf_logger.log_end(
+            key=mlperf_logger.constants.BLOCK_STOP,
+            metadata={"first_epoch_num": epoch},
+            sync=False,
+        )
+        mlperf_logger.log_end(
+            key=mlperf_logger.constants.EPOCH_STOP,
+            metadata={"epoch_num": epoch},
+            sync=False,
+        )
+        epoch += 1
+
+    mlperf_logger.log_event(
+        key=mlperf_logger.constants.TRAIN_SAMPLES, value=samples_trained, sync=False
+    )
+    mlperf_logger.log_event(
+        key=mlperf_logger.constants.EVAL_SAMPLES,
+        value=args.num_eval_examples,
+        sync=False,
+    )
+    mlperf_logger.log_end(
+        key=mlperf_logger.constants.RUN_STOP, metadata={"status": status}, sync=False
+    )
+    return args, final_loss, train_time_raw
+
+
+if __name__ == "__main__":
+    main()
diff --git a/models_v2/pytorch/bert_large/training/gpu/requirements.txt b/models_v2/pytorch/bert_large/training/gpu/requirements.txt
new file mode 100644
index 000000000..3eb7a0934
--- /dev/null
+++ b/models_v2/pytorch/bert_large/training/gpu/requirements.txt
@@ -0,0 +1,18 @@
+# Python dependencies required for development
+astunparse
+expecttest
+future
+numpy
+psutil
+pyyaml
+requests
+setuptools
+six
+types-dataclasses
+typing_extensions
+dataclasses; python_version<"3.7"
+transformers
+tensorboardX
+absl-py
+h5py
+nltk
diff --git a/models_v2/pytorch/bert_large/training/gpu/run_model.sh b/models_v2/pytorch/bert_large/training/gpu/run_model.sh
new file mode 100755
index 000000000..5e7d01c1b
--- /dev/null
+++ b/models_v2/pytorch/bert_large/training/gpu/run_model.sh
@@ -0,0 +1,132 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+
+#!/bin/bash
+
+# Create an array of input directories that are expected and then verify that they exist
+declare -A input_envs
+input_envs[DATASET_DIR]=${DATASET_DIR}
+input_envs[MULTI_TILE]=${MULTI_TILE}
+input_envs[PLATFORM]=${PLATFORM}
+input_envs[OUTPUT_DIR]=${OUTPUT_DIR}
+
+for i in "${!input_envs[@]}"; do
+  var_name=$i
+  env_param=${input_envs[$i]}
+
+  if [[ -z $env_param ]]; then
+    echo "The required environment variable $var_name is not set" >&2
+    exit 1
+  fi
+done
+
+if [[ "${PLATFORM}" == "Max" ]]; then
+    BATCH_SIZE=${BATCH_SIZE:-16}
+    PRECISION=${PRECISION:-BF16}
+    NUM_ITERATIONS=${NUM_ITERATIONS:-10}
+elif [[ "${PLATFORM}" == "Flex" ]]; then
+    echo "only support Max series GPU for platform"
+elif [[ "${PLATFORM}" == "ARC" ]]; then
+    if [[ "${MULTI_TILE}" == "True" ]]; then
+	echo "Only support MULTI_TILE=False when in arc platform"
+	exit 1
+    fi
+    BATCH_SIZE=${BATCH_SIZE:-4}
+    PRECISION=${PRECISION:-BF16}
+    NUM_ITERATIONS=${NUM_ITERATIONS:-10}
+fi
+
+
+echo 'Running with parameters:'
+echo " PLATFORM: ${PLATFORM}"
+echo " OUTPUT_DIR: ${OUTPUT_DIR}"
+echo " PRECISION: ${PRECISION}"
+echo " BATCH_SIZE: ${BATCH_SIZE}"
+echo " NUM_ITERATIONS: ${NUM_ITERATIONS}"
+echo " MULTI_TILE: ${MULTI_TILE}"
+
+if [[ "${PRECISION}" == "BF16" ]]; then
+    flag="--bf16 "
+elif [[ "${PRECISION}" == "FP32" ]]; then
+    flag=""
+elif [[ "${PRECISION}" == "TF32" ]]; then
+    export IPEX_FP32_MATH_MODE=1
+    flag=""
+else
+    echo -e "Invalid input! Only BF16 FP32 TF32 are supported."
+    exit 1
+fi
+
+echo "bert-large ${PRECISION} training MultiTile=${MULTI_TILE} BS=${BATCH_SIZE} Iter=${NUM_ITERATIONS}"
+
+# Create the output directory, if it doesn't already exist
+mkdir -p $OUTPUT_DIR
+
+modelname=ddp-bert
+
+if [[ ${MULTI_TILE} == "False" ]]; then
+    rm ${OUTPUT_DIR}/${modelname}${PRECISION}_train_t0_raw.log
+    python run_pretrain_mlperf.py --config_name=bert_config.json --input_dir=${DATASET_DIR} --output_dir=${OUTPUT_DIR} --eval_dir=${DATASET_DIR} --device=xpu --do_train --train_batch_size=${BATCH_SIZE} --gradient_accumulation_steps=1 --adamw --num-iterations ${NUM_ITERATIONS} ${flag} 2>&1 | tee ${OUTPUT_DIR}/${modelname}_${PRECISION}_train_t0_raw.log
+    python common/parse_result.py -m $modelname -l ${OUTPUT_DIR}/${modelname}_${PRECISION}_train_t0_raw.log -b ${BATCH_SIZE}
+    throughput=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_train_t0.log | grep Performance | awk -F ' ' '{print $2}')
+    throughput_unit=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_train_t0.log | grep Performance | awk -F ' ' '{print $3}')
+    latency=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_train_t0.log | grep Latency | awk -F ' ' '{print $2}')
+    acc=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_train_t0.log | grep Accuracy | awk -F ' ' '{print $3}')
+    acc_unit=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_train_t0.log | grep Accuracy | awk -F ' ' '{print $2}')
+else
+    rm ${OUTPUT_DIR}/${modelname}_${PRECISION}_train_raw.log
+    mpiexec -np 2 -ppn 2 python run_pretrain_mlperf.py \
+            --config_name=bert_config.json \
+            --input_dir=${DATASET_DIR} \
+            --output_dir=${OUTPUT_DIR} \
+            --eval_dir=${DATASET_DIR} \
+            --device=xpu \
+            --do_train \
+            --train_batch_size=${BATCH_SIZE} \
+            --gradient_accumulation_steps=1 \
+            ${flag} \
+            --adamw --num-iterations ${NUM_ITERATIONS} \
+            --bucket_cap 1000 --use_gradient_as_bucket_view \
+            --disable-broadcast-buffers \
+            --large-first-bucket \
+            --seed 123 2>&1 | tee ${OUTPUT_DIR}/ddp-${modelname}_${PRECISION}_train_raw.log
+    python common/parse_result.py -m $modelname --ddp -l ${OUTPUT_DIR}/ddp-${modelname}_${PRECISION}_train_raw.log -b ${BATCH_SIZE}
+    throughput=$(cat ${OUTPUT_DIR}/ddp-${modelname}_${PRECISION}_train.log | grep "Sum Performance" | awk -F ' ' '{print $3}')
+    throughput_unit=$(cat ${OUTPUT_DIR}/ddp-${modelname}_${PRECISION}_train.log | grep "Sum Performance" | awk -F ' ' '{print $4}')
+    latency=$(cat ${OUTPUT_DIR}/ddp-${modelname}_${PRECISION}_train.log | grep Latency | awk -F ' ' '{print $2}')
+    acc=$(cat ${OUTPUT_DIR}/ddp-${modelname}_${PRECISION}_train.log | grep Accuracy | awk -F ' ' '{print $3}')
+    acc_unit=$(cat ${OUTPUT_DIR}/ddp-${modelname}_${PRECISION}_train.log | grep Accuracy | awk -F ' ' '{print $2}')
+fi
+
+yaml_content=$(cat <<EOF
+results:
+ - key: throughput
+   value: $throughput
+   unit: $throughput_unit
+ - key: latency
+   value: $latency
+   unit: s
+ - key: accuracy
+   value: $acc
+   unit: $acc_unit
+EOF
+)
+
+# Write the content to a YAML file
+echo "$yaml_content" >  ${OUTPUT_DIR}/results.yaml
+echo "YAML file created."
diff --git a/models_v2/pytorch/bert_large/training/gpu/run_pretrain_mlperf.py b/models_v2/pytorch/bert_large/training/gpu/run_pretrain_mlperf.py
new file mode 100644
index 000000000..45dbe4aee
--- /dev/null
+++ b/models_v2/pytorch/bert_large/training/gpu/run_pretrain_mlperf.py
@@ -0,0 +1,1532 @@
+#!/usr/bin/env python
+# coding=utf-8
+# Copyright 2021 The HuggingFace Inc. team. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+Fine-tuning the library models for masked language modeling (BERT, ALBERT, RoBERTa...)
+on a text file or a dataset without using HuggingFace Trainer.
+
+Here is the full list of checkpoints on the hub that can be fine-tuned by this script:
+https://huggingface.co/models?filter=masked-lm
+"""
+# You can also adapt this script on your own mlm task. Pointers for this are left as comments.
+
+"""BERT Pretraining"""
+import argparse
+import csv
+import h5py
+import os
+import glob
+import numpy as np
+import torch
+from torch.utils.data import DataLoader, RandomSampler, SequentialSampler, Dataset
+from torch.utils.data.distributed import DistributedSampler
+import logging
+import math
+import multiprocessing
+import random
+import re
+import time
+import sys
+
+from collections import OrderedDict
+from concurrent.futures import ProcessPoolExecutor
+
+from schedulers import LinearWarmupPolyDecayScheduler
+
+import utils
+
+import torch.nn.functional as F
+import argparse
+
+import transformers
+from transformers import (
+    CONFIG_MAPPING,
+    MODEL_MAPPING,
+    AutoConfig,
+    AutoModelForPreTraining,
+    SchedulerType,
+    get_scheduler,
+    set_seed,
+)
+
+from schedulers import LinearWarmUpScheduler, LinearWarmupPolyDecayScheduler
+from lamb import Lamb
+
+try:
+    import intel_extension_for_pytorch as ipex
+    from intel_extension_for_pytorch.xpu.fp8.fp8 import fp8_autocast
+    from intel_extension_for_pytorch.xpu.fp8.recipe import DelayedScaling
+    from intel_extension_for_pytorch.nn.utils._fp8_convert import convert_fp8_model
+except:
+    ipex = None
+
+ref_time = 0
+PRINT_ITER_FACTOR = 32
+
+
+def get_time():
+    global ref_time
+    t = time.time()
+    return (t - ref_time) * 1000.0
+
+
+def sync_clock():
+    global ref_time
+    t = time.time()
+    if torch.distributed.is_initialized():
+        torch.distributed.barrier()
+        t = time.time()
+        t = torch.tensor([t], dtype=torch.double)
+        torch.distributed.broadcast(t, 0)
+        torch.distributed.barrier()
+        t = t.item()
+    ref_time = t
+
+
+def synchronize(device='cpu'):
+    if device == 'cuda':
+        if torch.distributed.is_available() and torch.distributed.is_initialized():
+            torch.distributed.all_reduce(torch.cuda.FloatTensor(1))
+            torch.cuda.synchronize()
+        else:
+            torch.cuda.synchronize()
+    elif device == 'xpu':
+        torch.xpu.synchronize()
+
+
+def get_time_sync(device='cpu'):
+    synchronize(device)
+    return get_time()
+
+
+logger = logging.getLogger(__name__)
+MODEL_CONFIG_CLASSES = list(MODEL_MAPPING.keys())
+MODEL_TYPES = tuple(conf.model_type for conf in MODEL_CONFIG_CLASSES)
+
+
+class WorkerInitObj(object):
+    def __init__(self, seed):
+        self.seed = seed
+
+    def __call__(self, id):
+        np.random.seed(seed=self.seed + id)
+        random.seed(self.seed + id)
+
+
+def get_eval_batchsize_per_worker(args):
+    if torch.distributed.is_initialized():
+        chunk_size = args.num_eval_examples // args.world_size
+        rank = args.rank
+        remainder = args.num_eval_examples % args.world_size
+        if rank < remainder:
+            return chunk_size + 1
+        else:
+            return chunk_size
+
+
+class pretraining_dataset_v1(Dataset):
+    def __init__(self, f, input_file, max_pred_length):
+        self.input_file = input_file
+        self.max_pred_length = max_pred_length
+        keys = [
+            "input_ids",
+            "input_mask",
+            "segment_ids",
+            "masked_lm_positions",
+            "masked_lm_ids",
+            "next_sentence_labels",
+        ]
+        self.inputs = [np.asarray(f[key][:]) for key in keys]
+        print(f"Loaded {len(self.inputs[0]):d} samples from datafile: {input_file}")
+        f.close()
+
+    def __len__(self):
+        "Denotes the total number of samples"
+        return len(self.inputs[0])
+
+    def __getitem__(self, index):
+        [
+            input_ids,
+            input_mask,
+            segment_ids,
+            masked_lm_positions,
+            masked_lm_ids,
+            next_sentence_labels,
+        ] = [
+            torch.from_numpy(input[index].astype(np.int64))
+            if indice < 5
+            else torch.from_numpy(np.asarray(input[index].astype(np.int64)))
+            for indice, input in enumerate(self.inputs)
+        ]
+        masked_lm_labels = torch.zeros(input_ids.shape, dtype=torch.long) - 100
+        index = self.max_pred_length
+        masked_token_count = torch.count_nonzero(masked_lm_positions)
+        if masked_token_count != 0:
+            index = masked_token_count
+        masked_lm_labels[masked_lm_positions[:index]] = masked_lm_ids[:index]
+        # print(f"input_mask_len = {torch.count_nonzero(input_ids)}  index = {index}")
+
+        return [
+            input_ids,
+            segment_ids,
+            input_mask,
+            masked_lm_labels,
+            next_sentence_labels,
+        ]
+
+
+class pretraining_dataset_v2(Dataset):
+    def __init__(
+        self, f, input_file, max_pred_length, max_seq_length=512, packed_samples=False
+    ):
+        self.input_file = input_file
+        self.max_pred_length = max_pred_length
+        self.max_seq_length = max_seq_length
+        self.packed_samples = packed_samples
+
+        if not self.packed_samples:
+            keys = [
+                "input_ids",
+                "segment_ids",
+                "masked_lm_positions",
+                "masked_lm_ids",
+                "next_sentence_labels",
+            ]
+        else:
+            keys = [
+                "input_ids",
+                "segment_ids",
+                "masked_lm_positions",
+                "masked_lm_ids",
+                "packed_input_len",
+                "packed_masked_lm_len",
+                "next_sentence_labels",
+            ]
+
+        self.inputs = [np.asarray(f[key][:]) for key in keys]
+        print(f"Loaded {len(self.inputs[0]):d} samples from datafile: {input_file}")
+        f.close()
+
+    def __len__(self):
+        "Denotes the total number of samples"
+        return len(self.inputs[0])
+
+    def __getitem__(self, index):
+        input_ids = np.zeros((self.max_seq_length)).astype(np.int64)
+        input_mask = np.zeros((self.max_seq_length)).astype(np.int64)
+        segment_ids = np.zeros((self.max_seq_length)).astype(np.int64)
+        next_sentence_labels = np.zeros((3)).astype(np.int64)
+        packed_input_len = np.zeros((3)).astype(np.int64)
+
+        if not self.packed_samples:
+            [
+                _input_ids,
+                _segment_ids,
+                _masked_lm_positions,
+                _masked_lm_ids,
+                _next_sentence_labels,
+            ] = [
+                input[index].astype(np.int64)
+                if indice < 4
+                else np.asarray(input[index].astype(np.int64))
+                for indice, input in enumerate(self.inputs)
+            ]
+        else:
+            [
+                _input_ids,
+                _segment_ids,
+                _masked_lm_positions,
+                _masked_lm_ids,
+                _packed_input_len,
+                _packed_masked_lm_len,
+                _next_sentence_labels,
+            ] = [
+                input[index].astype(np.int64)
+                for indice, input in enumerate(self.inputs)
+            ]
+
+        input_mask_len = _input_ids.shape[-1]
+        input_ids[:input_mask_len] = _input_ids
+        input_mask[:input_mask_len] = np.ones((1, input_mask_len)).astype(np.int64)
+        segment_ids[:input_mask_len] = _segment_ids
+        masked_lm_labels = np.zeros(input_ids.shape, dtype=np.int64) - 100
+        masked_lm_labels[_masked_lm_positions] = _masked_lm_ids
+
+        if not self.packed_samples:
+            next_sentence_labels = _next_sentence_labels
+
+            return [
+                torch.from_numpy(input_ids),
+                torch.from_numpy(segment_ids),
+                torch.from_numpy(input_mask),
+                torch.from_numpy(masked_lm_labels),
+                torch.from_numpy(next_sentence_labels),
+            ]
+        else:
+            packed_seqs = _packed_input_len.shape[-1]
+            next_sentence_labels[:packed_seqs] = _next_sentence_labels
+            packed_input_len[:packed_seqs] = _packed_input_len
+
+            return [
+                torch.from_numpy(input_ids),
+                torch.from_numpy(segment_ids),
+                torch.from_numpy(input_mask),
+                torch.from_numpy(masked_lm_labels),
+                torch.from_numpy(next_sentence_labels),
+                torch.from_numpy(packed_input_len),
+            ]
+
+
+def pretraining_dataset(
+    input_file, max_pred_length, max_seq_length=512, packed_samples=False
+):
+    f = h5py.File(input_file, "r")
+    if "input_mask" not in f.keys():
+        return pretraining_dataset_v2(
+            f, input_file, max_pred_length, max_seq_length, packed_samples
+        )
+    else:
+        return pretraining_dataset_v1(f, input_file, max_pred_length)
+
+
+def create_pretraining_dataset(input_file, max_pred_length, shared_list, args, worker_init_fn):
+    train_data = pretraining_dataset(
+        input_file=input_file, max_pred_length=max_pred_length
+    )
+    train_sampler = RandomSampler(train_data)
+    train_dataloader = DataLoader(
+        train_data, sampler=train_sampler, batch_size=args.train_batch_size
+    )
+    return train_dataloader, input_file
+
+
+def create_eval_dataset(args, worker_init_fn):
+    eval_data = []
+    for eval_file in sorted(os.listdir(args.eval_dir)):
+        eval_file_path = os.path.join(args.eval_dir, eval_file)
+
+        if os.path.isfile(eval_file_path) and "part" in eval_file_path:
+            eval_data.extend(
+                pretraining_dataset(
+                    eval_file_path, max_pred_length=args.max_predictions_per_seq
+                )
+            )
+            if len(eval_data) > args.num_eval_examples:
+                eval_data = eval_data[: args.num_eval_examples]
+                break
+    if torch.distributed.is_initialized():
+        chunk_size = args.num_eval_examples // args.world_size
+        rank = args.rank
+        remainder = args.num_eval_examples % args.world_size
+        if rank < remainder:
+            eval_data = eval_data[
+                (chunk_size + 1) * rank: (chunk_size + 1) * (rank + 1)
+            ]
+        else:
+            eval_data = eval_data[
+                chunk_size * rank + remainder: chunk_size * (rank + 1) + remainder
+            ]
+
+    eval_sampler = SequentialSampler(eval_data)
+    eval_dataloader = DataLoader(
+        eval_data, sampler=eval_sampler, batch_size=args.eval_batch_size, num_workers=0, pin_memory=True
+    )
+    return eval_dataloader
+
+
+def parse_args():
+    parser = argparse.ArgumentParser(
+        description="Finetune a transformers model on a Masked Language Modeling task"
+    )
+
+    # Required parameters
+    parser.add_argument("--input_dir",
+                        default=None,
+                        type=str,
+                        required=True,
+                        help="The input data dir. Should contain .hdf5 files  for the task.")
+    parser.add_argument("--output_dir",
+                        default=None,
+                        type=str,
+                        required=True,
+                        help="The output directory where the model checkpoints will be written.")
+    parser.add_argument("--eval_dir",
+                        default=None,
+                        type=str,
+                        help="The eval data dir. Should contain .hdf5 files  for the task.")
+    parser.add_argument("--eval_iter_start_samples",
+                        default=3000000,
+                        type=int,
+                        help="Sample to begin performing eval.")
+    parser.add_argument("--eval_iter_samples",
+                        default=16,
+                        type=int,
+                        help="If set to -1, disable eval, \
+                        else evaluate every eval_iter_samples during training")
+    parser.add_argument("--num_eval_examples",
+                        default=10000,
+                        type=int,
+                        help="number of eval examples to run eval on")
+    parser.add_argument("--resume_checkpoint",
+                        default=None,
+                        type=str,
+                        help="Use to load checkpoint to resume training")
+    parser.add_argument("--init_tf_checkpoint",
+                        default=None,
+                        type=str,
+                        help="The initial TF checkpoint to start training from.")
+    parser.add_argument(
+        "--train_file", type=str, default=None, help="A csv or a json file containing the training data."
+    )
+    parser.add_argument(
+        "--validation_file", type=str, default=None, help="A csv or a json file containing the validation data."
+    )
+    parser.add_argument(
+        "--model_name_or_path",
+        type=str,
+        help="Path to pretrained model or model identifier from huggingface.co/models.",
+    )
+    parser.add_argument(
+        "--config_name",
+        type=str,
+        default=None,
+        help="Pretrained config name or path if not the same as model_name",
+    )
+    parser.add_argument(
+        "--per_device_eval_batch_size",
+        type=int,
+        default=8,
+        help="Batch size (per device) for the evaluation dataloader.",
+    )
+    parser.add_argument(
+        "--learning_rate",
+        type=float,
+        default=5e-5,
+        help="Initial learning rate (after the potential warmup period) to use.",
+    )
+    parser.add_argument(
+        "--min_learning_rate",
+        type=float,
+        default=0.0,
+        help="Minimal learning rate to use.",
+    )
+    parser.add_argument("--max_predictions_per_seq",
+                        default=76,
+                        type=int,
+                        help="The maximum total of masked tokens in input sequence")
+    parser.add_argument("--train_batch_size",
+                        default=8,
+                        type=int,
+                        help="Total batch size for training.")
+    parser.add_argument("--eval_batch_size",
+                        default=16,
+                        type=int,
+                        help="Total batch size for training.")
+    parser.add_argument("--weight_decay_rate",
+                        default=0.01,
+                        type=float,
+                        help="weight decay rate for LAMB.")
+    parser.add_argument("--opt_lamb_beta_1",
+                        default=0.9,
+                        type=float,
+                        help="LAMB beta1.")
+    parser.add_argument("--opt_lamb_beta_2",
+                        default=0.999,
+                        type=float,
+                        help="LAMB beta2.")
+    parser.add_argument("--max_steps",
+                        default=218880,
+                        type=int,
+                        help="Total number of training steps to perform.")
+    parser.add_argument("--max_steps_for_scheduler",
+                        default=218880,
+                        type=int,
+                        help="Total number of training steps for scheduler to decrease lr.")
+    parser.add_argument("--max_samples_termination",
+                        default=14000000,
+                        type=float,
+                        help="Total number of training samples to run.")
+    parser.add_argument("--warmup_proportion",
+                        default=0.01,
+                        type=float,
+                        help="Proportion of optimizer update steps to perform linear learning rate warmup for. "
+                             "Typically 1/8th of steps for Phase2")
+    parser.add_argument("--warmup_steps",
+                        default=0,
+                        type=float,
+                        help="Number of optimizer update steps to perform linear learning rate warmup for. "
+                             "Typically 1/8th of steps for Phase2")
+    parser.add_argument("--start_warmup_step",
+                        default=0,
+                        type=float,
+                        help="Starting step for warmup. ")
+    parser.add_argument('--log_freq',
+                        type=float,
+                        default=10000.0,
+                        help='frequency of logging loss. If not positive, no logging is provided for training loss')
+    parser.add_argument('--checkpoint_activations',
+                        default=False,
+                        action='store_true',
+                        help="Whether to use gradient checkpointing")
+    parser.add_argument("--resume_from_checkpoint",
+                        default=False,
+                        action='store_true',
+                        help="Whether to resume training from checkpoint. If set, precedes init_tf_checkpoint")
+    parser.add_argument('--keep_n_most_recent_checkpoints',
+                        type=int,
+                        default=20,
+                        help="Number of checkpoints to keep (rolling basis).")
+    parser.add_argument('--num_samples_per_checkpoint',
+                        type=int,
+                        default=500000,
+                        help="Number of update steps until a model checkpoint is saved to disk.")
+    parser.add_argument('--min_samples_to_start_checkpoints',
+                        type=int,
+                        default=3000000,
+                        help="Number of update steps until model checkpoints start saving to disk.")
+    parser.add_argument('--skip-checkpoint',
+                        default=False,
+                        action='store_true',
+                        help="Whether to save checkpoints")
+    parser.add_argument('--phase2',
+                        default=False,
+                        action='store_true',
+                        help="Only required for checkpoint saving format")
+    parser.add_argument("--do_train",
+                        default=False,
+                        action='store_true',
+                        help="Whether to run training.")
+    parser.add_argument('--bert_config_path',
+                        type=str,
+                        default="/workspace/phase1",
+                        help="Path bert_config.json is located in")
+    parser.add_argument('--target_mlm_accuracy',
+                        type=float,
+                        default=0.72,
+                        help="Stop training after reaching this Masked-LM accuracy")
+    parser.add_argument('--workers',
+                        type=int,
+                        default=0,
+                        help="Stop training after reaching this Masked-LM accuracy")
+    parser.add_argument('--train_mlm_accuracy_window_size',
+                        type=int,
+                        default=0,
+                        help="Average accuracy over this amount of batches before performing a stopping criterion test")
+    parser.add_argument('--num_epochs_to_generate_seeds_for',
+                        type=int,
+                        default=2,
+                        help="Number of epochs to plan seeds for. Same set across all workers.")
+    parser.add_argument("--use_gradient_as_bucket_view",
+                        default=False,
+                        action='store_true',
+                        help="Turn ON gradient_as_bucket_view optimization in native DDP.")
+    parser.add_argument("--dense_seq_output",
+                        default=False,
+                        action='store_true',
+                        help="Whether to run with optimizations.")
+    parser.add_argument(
+        "--no_ddp", default=False, action="store_true", help="Whether to use DDP."
+    )
+    parser.add_argument(
+        "--converge", default=False, action="store_true", help="Whether to run convergence."
+    )
+    parser.add_argument(
+        "--dist_lamb",
+        default=False,
+        action="store_true",
+        help="Whether to use DistLamb from pcl_bert",
+    )
+
+    # For dtype specific training
+    parser.add_argument("--bf16",
+                        default=False,
+                        action='store_true',
+                        help="Enale BFloat16 training")
+    parser.add_argument("--fp8",
+                        default=False,
+                        action='store_true',
+                        help="Enale FP8 training")
+    parser.add_argument("--bf32",
+                        default=False,
+                        action='store_true',
+                        help="Enale BFloat32 training")
+    parser.add_argument("--weight_decay", type=float,
+                        default=0.0, help="Weight decay to use.")
+    parser.add_argument("--num_train_epochs", type=int, default=3,
+                        help="Total number of training epochs to perform.")
+    parser.add_argument(
+        "--gradient_accumulation_steps",
+        type=int,
+        default=1,
+        help="Number of updates steps to accumulate before performing a backward/update pass.",
+    )
+    parser.add_argument(
+        "--lr_scheduler_type",
+        type=SchedulerType,
+        default="linear",
+        help="The scheduler type to use.",
+        choices=["linear", "cosine", "cosine_with_restarts",
+                 "polynomial", "constant", "constant_with_warmup"],
+    )
+    parser.add_argument("--seed", type=int, default=None,
+                        help="A seed for reproducible training.")
+    parser.add_argument(
+        "--model_type",
+        type=str,
+        default=None,
+        help="Model type to use if training from scratch.",
+        choices=MODEL_TYPES,
+    )
+
+    parser.add_argument(
+        "--preprocessing_num_workers",
+        type=int,
+        default=None,
+        help="The number of processes to use for the preprocessing.",
+    )
+    parser.add_argument('--rank',
+                        default=0,
+                        type=int,
+                        help='node rank for distributed training')
+    parser.add_argument("--world_size",
+                        default=1,
+                        type=int,
+                        help="Total batch size for training.")
+
+    parser.add_argument("--profile", action="store_true",
+                        help="Whether to enable profiling")
+    parser.add_argument("--export_chrome_trace", action="store_true",
+                        help="Exports the collected trace in Chrome JSON format.")
+    parser.add_argument(
+        "--skip_exchange_padding",
+        default=False,
+        action="store_true",
+        help="Whether to skip padding exchange.",
+    )
+    parser.add_argument(
+        "--unpad",
+        default=False,
+        action="store_true",
+        help="Whether to run with unpadding.",
+    )
+    parser.add_argument(
+        "--benchmark_steps",
+        type=int,
+        default=0,
+        help="Number of steps to run for benchmark.",
+    )
+    parser.add_argument(
+        "--dist_backend",
+        type=str,
+        default="ccl",
+        help="Specify distributed backend to use.",
+    )
+    parser.add_argument('--dist-url', default='127.0.0.1', type=str,
+                        help='url used to set up distributed training')
+    parser.add_argument('--dist-port', default='29500', type=str,
+                        help='url port used to set up distributed training')
+    parser.add_argument('--large-first-bucket', action="store_true",
+                        help='Configure a large capacity of the first bucket in DDP for allreduce')
+    parser.add_argument('--disable-broadcast-buffers', action='store_true',
+                        help='disable syncing buffers')
+    parser.add_argument(
+        "--multi_instance",
+        action="store_true",
+        help="Skip torch.distributed initialization to Run multiple instance independently",
+    )
+    parser.add_argument(
+        "--dist_profile",
+        action="store_true",
+        help="Whether to enable distributed timing profile",
+    )
+    parser.add_argument('--bucket_cap', default=25, type=int,
+                        help='controls the bucket size in MegaBytes')
+
+    # choose device
+    parser.add_argument(
+        "--device",
+        type=str,
+        default="cpu",
+        help="backend to run",
+    )
+    parser.add_argument(
+        "--amp",
+        action="store_true",
+        help="Whether to enable autocast",
+    )
+    parser.add_argument("--lamb", action="store_true")
+    parser.add_argument("--adamw", action="store_true")
+    parser.add_argument("--num-iterations", default='10000000000', type=str)
+    parser.add_argument("--info", action="store_true")
+
+    args = parser.parse_args()
+    if os.environ.get("PROFILE", "OFF").upper() in ["1", "Y", "ON", "YES", "TRUE"]:
+        args.profile = True
+    if args.output_dir is not None:
+        os.makedirs(args.output_dir, exist_ok=True)
+    return args
+
+
+def found_resume_checkpoint(args):
+    if args.phase2:
+        checkpoint_str = "phase2_ckpt*.pt"
+    else:
+        checkpoint_str = "phase1_ckpt*.pt"
+    num_ckpts = len(glob.glob(os.path.join(args.output_dir, checkpoint_str)))
+    print('\n\nfound num checkpoints: ', num_ckpts)
+    print('resume from checkpoints: ', args.resume_from_checkpoint)
+    print('resume checkpoint: ', args.resume_checkpoint)
+    if args.resume_from_checkpoint and not os.path.exists(args.resume_checkpoint):
+        raise RuntimeError(
+            "The checkpoint [", args.resume_checkpoint, "] doesn't exist")
+    return args.resume_from_checkpoint and num_ckpts > 0
+
+
+def setup_training(args):
+    device = torch.device(args.device)
+    if (int(os.environ.get("PMI_SIZE", "0")) > 1 or args.world_size > 1) and not args.multi_instance:
+        if args.dist_backend == "ccl":
+            try:
+                import oneccl_bindings_for_pytorch
+            except:
+                print(
+                    "CCL backend requested but import oneccl_bindings_for_pytorch failed")
+                raise
+        elif args.dist_backend == "mpi":
+            if not torch.distributed.is_mpi_available():
+                try:
+                    import torch_mpi
+                except:
+                    print(
+                        "MPI backend requested but not available try installing torch_mpi module"
+                    )
+                    raise
+        else:
+            raise ValueError(
+                f"{args.dist_backend} backend requested but not supported")
+
+        if args.world_size == 1:
+            mpi_world_size = int(os.environ.get('PMI_SIZE', -1))
+            if mpi_world_size > 0:
+                os.environ['MASTER_ADDR'] = args.dist_url  # '127.0.0.1'
+                os.environ['MASTER_PORT'] = args.dist_port  # '29500'
+                os.environ['RANK'] = os.environ.get('PMI_RANK', -1)
+                os.environ['WORLD_SIZE'] = os.environ.get('PMI_SIZE', -1)
+                args.rank = int(os.environ.get('PMI_RANK', -1))
+                args.world_size = int(os.environ.get("WORLD_SIZE", -1))
+        else:
+            if 'PMIX_RANK' in os.environ.keys():  # mpich set
+                os.environ['MASTER_ADDR'] = args.dist_url  # '127.0.0.1'
+                os.environ['MASTER_PORT'] = args.dist_port  # '29500'
+                os.environ['RANK'] = os.environ.get('PMIX_RANK')
+                os.environ['WORLD_SIZE'] = str(args.world_size)
+                args.rank = int(os.environ.get('PMIX_RANK', -1))
+
+        if args.world_size > 1:
+            init_method = 'tcp://' + args.dist_url + ':' + args.dist_port
+            torch.distributed.init_process_group(
+                backend=args.dist_backend, init_method=init_method,  world_size=args.world_size, rank=args.rank)
+            # refer to https://www.open-mpi.org/faq/?category=running#mpi-environmental-variables for MPI ENV
+            if 'MPI_LOCALRANKID' in os.environ.keys():
+                local_rank = os.environ['MPI_LOCALRANKID']
+            elif 'OMPI_COMM_WORLD_LOCAL_RANK' in os.environ.keys():
+                local_rank = os.environ['OMPI_COMM_WORLD_LOCAL_RANK']
+            else:  # mpich set
+                local_rank = os.environ['PALS_LOCAL_RANKID']
+            device = torch.device("xpu", int(local_rank))
+            print('world_size:{}, rank:{}, device:{}'.format(
+                args.world_size, args.rank, device))
+
+        args.rank = torch.distributed.get_rank()
+        args.world_size = torch.distributed.get_world_size()
+        if args.rank == 0:
+            print(
+                f"Using {args.dist_backend.upper()} dist run with {args.world_size} ranks",
+                flush=True,
+            )
+    if args.gradient_accumulation_steps < 1:
+        raise ValueError("Invalid gradient_accumulation_steps parameter: {}, should be >= 1".format(
+            args.gradient_accumulation_steps))
+    if args.train_batch_size % args.gradient_accumulation_steps != 0:
+        raise ValueError("Invalid gradient_accumulation_steps parameter: {}, batch size {} should be divisible".format(
+            args.gradient_accumulation_steps, args.train_batch_size))
+
+    args.train_batch_size = args.train_batch_size // args.gradient_accumulation_steps
+
+    if not (args.do_train or (args.eval_dir and args.eval_iter_samples <= 0)):
+        raise ValueError(" `do_train`  or should be in offline eval mode")
+
+    if not args.resume_from_checkpoint or not os.path.exists(args.output_dir):
+        os.makedirs(args.output_dir, exist_ok=True)
+
+    return device, args
+
+
+optimize_dtype = None
+
+
+def prepare_model_and_optimizer(args, device):
+    global optimize_dtype
+    if args.bf16 or args.fp8:
+        optimize_dtype = torch.bfloat16
+    else:
+        optimize_dtype = torch.float32
+
+    global_step = 0
+    args.resume_step = 0
+    checkpoint = None
+    # download model & vocab.
+    if args.config_name:
+        print('Get config from config_name', args.config_name)
+        config = AutoConfig.from_pretrained(args.config_name)
+    elif args.model_name_or_path:
+        print('Get config from model_name_or_path: ', args.model_name_or_path)
+        config = AutoConfig.from_pretrained(args.model_name_or_path)
+    else:
+        config = CONFIG_MAPPING[args.model_type]()
+        logger.warning(
+            "You are instantiating a new config instance from scratch.")
+
+    config.dense_seq_output = args.dense_seq_output
+    if args.model_name_or_path:
+        print('Load model by AutoModelForPreTraining.from_pretrained')
+        model = AutoModelForPreTraining.from_pretrained(
+            args.model_name_or_path,
+            from_tf=bool(".ckpt" in args.model_name_or_path),
+            config=config,
+        )
+        model.train()
+    else:
+        logger.info("Training new model from scratch")
+        model = AutoModelForPreTraining.from_config(config)
+        model.train()
+
+    if args.resume_from_checkpoint and args.resume_checkpoint is not None:
+        if not os.path.exists(args.resume_checkpoint):
+            raise RuntimeError("The checkpoint ",
+                               args.resume_checkpoint, " doesn't exist")
+        print('load...', args.resume_checkpoint, ' to ',
+              (args.device + ':' + str(args.rank)))
+        map_location = str(args.device) + ':' + str(args.rank)
+        checkpoint = torch.load(args.resume_checkpoint, map_location=map_location)
+
+    print('Set different weight_decay for model parameters')
+    param_optimizer = list(model.named_parameters())
+    no_decay = ['bias', 'gamma', 'beta', 'LayerNorm']
+    optimizer_grouped_parameters = [
+        {'params': [p for n, p in param_optimizer if not any(
+            nd in n for nd in no_decay)], 'weight_decay': args.weight_decay_rate},
+        {'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)], 'weight_decay': 0.0}]
+
+    print(
+        "GroupSizes: ",
+        [sum([p.numel() for p in g["params"]])
+         for g in optimizer_grouped_parameters],
+    )
+
+    print('Model to device: ', device)
+    model = model.to(device).train()
+    if torch.distributed.is_initialized() and not args.no_ddp:
+        torch.xpu.set_device(device)
+        model.xpu(device)
+    if args.dist_lamb:
+        raise NotImplementedError('Not implemeted for dist_lamb')
+    elif args.adamw:
+        print('using adamw')
+        optimizer = torch.optim.AdamW(
+            optimizer_grouped_parameters, lr=args.learning_rate)
+    elif args.lamb:
+        print('using lamb')
+        optimizer = Lamb(
+            optimizer_grouped_parameters,
+            lr=args.learning_rate,
+            betas=(args.opt_lamb_beta_1, args.opt_lamb_beta_2))
+    else:
+        raise NotImplementedError('Please use valid optimizer')
+
+    print("Doing torch xpu optimize, dtype: ", optimize_dtype)
+    model, optimizer = torch.xpu.optimize(
+        model=model, optimizer=optimizer, dtype=optimize_dtype)
+
+    if args.warmup_steps == 0:
+        warmup_steps = int(args.max_steps_for_scheduler *
+                           args.warmup_proportion)
+        warmup_start = 0
+    else:
+        warmup_steps = args.warmup_steps
+        warmup_start = args.start_warmup_step
+
+    if args.converge:
+        print('LinearWarmupPolyDecayScheduler:')
+        print('warmup start = ', warmup_start)
+        print('warmup steps = ', warmup_steps)
+        print('max steps for scheduler = ', args.max_steps_for_scheduler)
+        print('initial learning rate = ', args.learning_rate)
+        print('minimal learning rate = ', args.min_learning_rate)
+    lr_scheduler = LinearWarmupPolyDecayScheduler(
+        optimizer,
+        start_warmup_steps=warmup_start,
+        warmup_steps=warmup_steps,
+        total_steps=args.max_steps_for_scheduler,
+        end_learning_rate=args.min_learning_rate,
+        degree=1.0)
+
+    # load checkpoint for model, optimizer and scheduler
+    if checkpoint is not None:
+        print('[info] loadind model state dict...')
+        model.load_state_dict(checkpoint['model'])
+        print('[info] loadind optimizer state dict...')
+        optimizer.load_state_dict(checkpoint['optimizer'])
+        print('[info] loadind lr_scheduler state dict...')
+        lr_scheduler.load_state_dict(checkpoint['lr_scheduler'])
+
+    # DDP
+    if torch.distributed.is_initialized() and not args.no_ddp:
+        if args.large_first_bucket:
+            # set the first bucket with maximal size to cover all parameters for allreduce
+            torch.distributed._DEFAULT_FIRST_BUCKET_BYTES = sys.maxsize
+        model = torch.nn.parallel.DistributedDataParallel(model,
+                                                          device_ids=[device],
+                                                          find_unused_parameters=True,
+                                                          bucket_cap_mb=args.bucket_cap,
+                                                          broadcast_buffers=False if args.disable_broadcast_buffers else True,
+                                                          gradient_as_bucket_view=args.use_gradient_as_bucket_view)
+
+    return model, optimizer, lr_scheduler, checkpoint, global_step
+
+
+def run_eval(
+    model,
+    eval_dataloader,
+    device,
+    num_eval_examples,
+    args,
+    first_eval=False,
+    use_cache=False,
+):
+    model.eval()
+    total_eval_loss, total_eval_mlm_acc = 0.0, 0.0
+    total_masked = 0
+    with torch.no_grad():
+        for step, batch in enumerate(eval_dataloader):
+            (
+                input_ids,
+                segment_ids,
+                input_mask,
+                masked_lm_labels,
+                next_sentence_labels,
+            ) = batch
+            input_ids = input_ids.to(device)
+            segment_ids = segment_ids.to(device)
+            input_mask = input_mask.to(device)
+            masked_lm_labels = masked_lm_labels.to(device)
+            next_sentence_labels = next_sentence_labels.to(device)
+            with torch.xpu.amp.autocast(enabled=True, dtype=optimize_dtype):
+                outputs = model(
+                    input_ids=input_ids,
+                    token_type_ids=segment_ids,
+                    attention_mask=input_mask,
+                    labels=masked_lm_labels,
+                    next_sentence_label=next_sentence_labels,
+                )
+            mlm_acc, num_masked = calc_mlm_acc(
+                outputs, masked_lm_labels, args.dense_seq_output
+            )
+            total_eval_loss += outputs.loss.item() * num_masked
+            total_eval_mlm_acc += mlm_acc * num_masked
+            total_masked += num_masked
+
+            torch.xpu.synchronize(args.device)
+            if args.converge:
+                print('Eval [', step, '/', len(eval_dataloader), '] loss: ', round(float(total_eval_loss / total_masked), 4), ' acc: ',
+                      mlm_acc.item(), '(avg acc: ', round(float(total_eval_mlm_acc.item() / total_masked), 4), ' num masked: ', num_masked, ')')
+
+    model.train()
+    total_masked = torch.tensor(total_masked, device=device, dtype=torch.int64)
+    total_eval_loss = torch.tensor(
+        total_eval_loss, device=device, dtype=torch.float64)
+    if torch.distributed.is_initialized():
+        # Collect total scores from all ranks
+        torch.distributed.all_reduce(
+            total_eval_mlm_acc, op=torch.distributed.ReduceOp.SUM
+        )
+        torch.distributed.all_reduce(
+            total_eval_loss, op=torch.distributed.ReduceOp.SUM)
+        torch.distributed.all_reduce(
+            total_masked, op=torch.distributed.ReduceOp.SUM)
+        print('finish all reduce for validation loss and accuracy')
+
+    # Average by number of examples
+    total_eval_mlm_acc /= total_masked
+    total_eval_loss /= total_masked
+
+    return total_eval_loss, total_eval_mlm_acc
+
+
+def global_batch_size(args):
+    return args.train_batch_size * args.gradient_accumulation_steps * args.world_size
+
+
+def calc_mlm_acc(outputs, masked_lm_labels, dense_seq_output=False):
+    prediction_scores = outputs.prediction_logits
+    masked_lm_labels_flat = masked_lm_labels.view(-1)
+    mlm_labels = masked_lm_labels_flat[masked_lm_labels_flat != -100]
+    if not dense_seq_output:
+        prediction_scores_flat = prediction_scores.view(
+            -1, prediction_scores.shape[-1])
+        mlm_predictions_scores = prediction_scores_flat[masked_lm_labels_flat != -100]
+        mlm_predictions = mlm_predictions_scores.argmax(dim=-1)
+    else:
+        mlm_predictions = prediction_scores.argmax(dim=-1)
+
+    num_masked = mlm_labels.numel()
+    mlm_acc = (mlm_predictions == mlm_labels).sum(
+        dtype=torch.float) / num_masked
+
+    return mlm_acc, num_masked
+
+
+def calc_accuracy(outputs, masked_lm_labels, next_sentence_label, args):
+    loss = outputs.loss.item()
+    prediction_logits = outputs.prediction_logits
+    seq_relationship_logits = outputs.seq_relationship_logits
+    mlm_acc, num_masked = calc_mlm_acc(
+        outputs, masked_lm_labels, args.dense_seq_output)
+    seq_acc_t = torch.argmax(seq_relationship_logits, dim=-
+                             1).eq(next_sentence_label.view([-1])).to(torch.float)
+    seq_acc_true, seq_tot = seq_acc_t.sum().item(), seq_acc_t.numel()
+    seq_acc = seq_acc_true / seq_tot
+    return loss, mlm_acc, num_masked, seq_acc, seq_tot
+
+
+def main():
+    global_start_time = time.time()
+    latency_list = []
+    args = parse_args()
+
+    # To ensure the same model master weight at initial state, set seed before preparing the model.   
+    if args.seed is not None:
+        print("Setting seed to ensure same model master weight at the beginning.")
+        torch.manual_seed(args.seed)
+        torch.xpu.manual_seed(args.seed)
+
+    if args.converge:
+        print('For convergence verbose show freq, PRINT_ITER_FACTOR = ',
+              PRINT_ITER_FACTOR)
+    status = 'aborted'  # later set to 'success' if termination criteria met
+    device, args = setup_training(args)
+    print("args.world_size={}, args.rank={}".format(
+        args.world_size, args.rank))
+    total_batch_size = global_batch_size(args)
+    # Initialize the accelerator. We will let the accelerator handle device placement for us in this example.
+    # Make one log on every process with the configuration for debugging.
+    logging.basicConfig(
+        format="%(asctime)s - %(levelname)s - %(name)s -   %(message)s",
+        datefmt="%m/%d/%Y %H:%M:%S",
+        level=logging.INFO,
+    )
+    if args.rank == 0 or args.rank == -1:
+        print("parsed args:")
+        print(args)
+    # Prepare optimizer
+    model, optimizer, lr_scheduler, checkpoint, global_step = prepare_model_and_optimizer(
+        args, device)
+    worker_seeds, shuffling_seeds = utils.setup_seeds(
+        args.seed, args.num_epochs_to_generate_seeds_for, device)
+    worker_seed = worker_seeds[args.rank]
+    random.seed(worker_seed)
+    np.random.seed(worker_seed)
+    worker_init = WorkerInitObj(worker_seed)
+    print('[info] Setting seed: ', args.seed, '. worker seed: ', worker_seed)
+
+    samples_trained = global_step * args.train_batch_size * \
+        args.gradient_accumulation_steps * args.world_size
+    final_loss = float("inf")
+    train_time_raw = float("inf")
+    raw_train_start = time.time()
+
+    if args.do_train:
+        model.train()
+        most_recent_ckpts_paths = []
+        average_loss = 0.0  # averaged loss every args.log_freq steps
+        epoch = 1
+        training_steps = 0
+        end_training, converged = False, False
+        samples_trained_prev_for_eval = 0
+        samples_trained_prev_for_save = 0
+
+        # pre-compute eval boundaries
+        samples_trained_per_step = args.train_batch_size * \
+            args.gradient_accumulation_steps * args.world_size
+        start, stop, step = args.eval_iter_start_samples, args.max_samples_termination, args.eval_iter_samples
+        eval_steps = [math.ceil(i/samples_trained_per_step)
+                      for i in np.arange(start, stop, step)]
+        eval_count = 0
+        next_eval_step = eval_steps[eval_count]
+
+        if args.converge:
+            print('max samples termination: ', args.max_samples_termination)
+            print('samples trained per step = ', samples_trained_per_step)
+            print('all eval steps, start: ', eval_steps[0], ', end: ', eval_steps[len(eval_steps) - 1], ', eval times: ', len(eval_steps))
+
+        if args.target_mlm_accuracy:
+            if args.train_mlm_accuracy_window_size > 0:
+                accuracy_scores = []
+                avg_mlm_accuracy = torch.Tensor([0])
+
+        first_epoch = True
+
+        # resume ckpt
+        if found_resume_checkpoint(args) and checkpoint is not None:
+            print('[info] construct file from checkpoint')
+            global_step = checkpoint['global_step']
+            print('[info] loadind global step... ', global_step)
+            f_start_id = checkpoint['files'][0]
+            print('[info] loadind samples file start id... ', f_start_id)
+            files = checkpoint['files'][1:]
+            num_files = len(files)
+            print('[info] loadind samples num files... ', num_files)
+            samples_trained = checkpoint['samples_trained']
+            print('[info] loadind samples trained... ', samples_trained)
+            samples_trained_prev_for_eval = checkpoint['samples_trained_prev_for_eval']
+            print('[info] loadind samples trained prev for eval... ',
+                  samples_trained_prev_for_eval)
+            samples_trained_prev_for_save = checkpoint['samples_trained_prev_for_save']
+            print('[info] loadind samples trained prev for save... ',
+                  samples_trained_prev_for_save)
+            epoch = checkpoint['epoch']
+            print('[info] loadind epoch... ', epoch)
+            eval_count = checkpoint['eval_count']
+            print('[info] loadind eval_count... ', eval_count)
+            eval_steps = checkpoint['eval_steps']
+            print('[info] loadind eval_steps... ')
+            next_eval_step = eval_steps[eval_count]
+            print('[info] next eval step... ')
+        else:
+            print('[info] construct file from initialization')
+            print('[info] input dir = ', args.input_dir)
+            files = [os.path.join(args.input_dir, f) for f in os.listdir(args.input_dir) if
+                     os.path.isfile(os.path.join(args.input_dir, f)) and 'part' in f]
+            files.sort()
+            num_files = len(files)
+            print('[info] num files = ', num_files)
+            random.Random(shuffling_seeds[epoch %
+                                          len(shuffling_seeds)]).shuffle(files)
+            f_start_id = 0
+
+    sync_clock()
+
+    # outer loop to finish max_steps
+    while global_step < args.max_steps and not end_training:
+        if args.converge:
+            print('global step = ', global_step)
+        if args.rank == 0 or args.rank == -1:
+            now_time = time.time()
+            print("epoch:", epoch)
+
+        # Reshuffle file list on subsequent epochs
+        if not first_epoch:
+            files = [os.path.join(args.input_dir, f) for f in os.listdir(args.input_dir) if
+                     os.path.isfile(os.path.join(args.input_dir, f)) and 'part' in f]
+            files.sort()
+            num_files = len(files)
+            random.Random(shuffling_seeds[epoch %
+                                          len(shuffling_seeds)]).shuffle(files)
+            f_start_id = 0
+        first_epoch = False
+
+        shared_file_list = {}
+
+        if torch.distributed.is_initialized() and args.world_size > num_files:
+            remainder = args.world_size % num_files
+            data_file = files[(f_start_id*args.world_size + args.rank +
+                               remainder * f_start_id) % num_files]
+            print('rank: ', args.rank, '. data file: ', data_file)
+        else:
+            data_file = files[(f_start_id*args.world_size +
+                               args.rank) % num_files]
+
+        train_data = pretraining_dataset(
+            data_file, args.max_predictions_per_seq)
+        if torch.distributed.is_initialized() and not args.no_ddp:
+            train_sampler = torch.utils.data.distributed.DistributedSampler(
+                train_data)
+        else:
+            train_sampler = RandomSampler(train_data)
+        train_dataloader = DataLoader(
+            train_data, sampler=train_sampler, batch_size=args.train_batch_size, num_workers=args.workers, pin_memory=True)
+
+        for f_id in range(f_start_id, len(files)):
+
+            # data file for next file loop train
+            if args.world_size > num_files:
+                data_file = files[(f_id*args.world_size + args.rank +
+                                   remainder * f_id) % num_files]
+            else:
+                data_file = files[(f_id*args.world_size +
+                                   args.rank) % num_files]
+
+            # initial
+            gloss = 0.0
+            gloss_list = []
+            acc_list = []
+
+            # this file loop train: f_id
+            for step, batch in enumerate(train_dataloader):
+                if not args.converge:
+                    if training_steps >= 10 + int(args.num_iterations):
+                        latency_list = latency_list[10:]
+                        avg = sum(latency_list) / len(latency_list)
+                        print('bert_train latency:  ' + str(avg) + '  s')
+                        print('bert_train throughput:  ' + str(total_batch_size /
+                                                               args.world_size / avg) + '  sentences/s')
+                        print('perplexity = ' + str(gloss))
+                        return
+                training_steps += 1
+
+                # H2D
+                start_h2d = get_time_sync(args.device)
+                input_ids, segment_ids, input_mask, masked_lm_labels, next_sentence_labels = batch
+                input_ids = input_ids.to(device)
+                segment_ids = segment_ids.to(device)
+                input_mask = input_mask.to(device)
+                masked_lm_labels = masked_lm_labels.to(device)
+                next_sentence_labels = next_sentence_labels.to(device)
+                end_h2d = get_time_sync(args.device)
+
+                info = {
+                    "input_ids": [input_ids.shape, input_ids.dtype],
+                    "segment_ids": [segment_ids.shape, segment_ids.dtype],
+                    "input_mask": [input_mask.shape, input_mask.dtype],
+                    "masked_lm_labels": [masked_lm_labels.shape, masked_lm_labels.dtype],
+                    "next_sentence_labels": [next_sentence_labels.shape, next_sentence_labels.dtype]
+                }
+                if args.info:
+                    print("datainfo", info)
+
+                def step_training():
+                    outputs = None
+                    if args.bf16 and args.amp:
+                        if args.device == 'cpu':
+                            with torch.cpu.amp.autocast():
+                                outputs = model(input_ids=input_ids,
+                                                token_type_ids=segment_ids,
+                                                attention_mask=input_mask,
+                                                labels=masked_lm_labels,
+                                                next_sentence_label=next_sentence_labels)
+                        elif args.device == 'xpu':
+                            with torch.xpu.amp.autocast(enabled=True, dtype=optimize_dtype):
+                                outputs = model(input_ids=input_ids,
+                                                token_type_ids=segment_ids,
+                                                attention_mask=input_mask,
+                                                labels=masked_lm_labels,
+                                                next_sentence_label=next_sentence_labels)
+                        elif args.device == 'cuda':
+                            with torch.cuda.amp.autocast():
+                                outputs = model(input_ids=input_ids,
+                                                token_type_ids=segment_ids,
+                                                attention_mask=input_mask,
+                                                labels=masked_lm_labels,
+                                                next_sentence_label=next_sentence_labels)
+                    elif args.fp8:
+                        # most used path for current xpu OOB solution
+                        with torch.xpu.amp.autocast(enabled=True, dtype=optimize_dtype):
+                            with fp8_autocast(enabled=True, fp8_recipe=DelayedScaling()):
+                                convert_fp8_model(model)
+                                outputs = model(input_ids=input_ids,
+                                                token_type_ids=segment_ids,
+                                                attention_mask=input_mask,
+                                                labels=masked_lm_labels,
+                                                next_sentence_label=next_sentence_labels)
+                    else:
+                        # most used path for current xpu OOB solution
+                        with torch.xpu.amp.autocast(enabled=True, dtype=optimize_dtype):
+                            outputs = model(input_ids=input_ids,
+                                            token_type_ids=segment_ids,
+                                            attention_mask=input_mask,
+                                            labels=masked_lm_labels,
+                                            next_sentence_label=next_sentence_labels)
+                    loss = outputs.loss
+                    loss = loss / args.gradient_accumulation_steps
+                    loss.backward()
+
+                    # clip grad norm
+                    if hasattr(optimizer, "clip_grad_norm_"):
+                        ggnorm = optimizer.clip_grad_norm_(1.0)
+                    else:
+                        ggnorm = torch.nn.utils.clip_grad_norm_(
+                            model.parameters(), 1.0)
+
+                    if (step + 1) % args.gradient_accumulation_steps == 0 or step == len(train_dataloader) - 1:
+                        start_opt = get_time_sync(args.device)
+                        optimizer.step()
+                        lr_scheduler.step()
+                        optimizer.zero_grad()
+                        end_opt = get_time_sync(args.device)
+
+                    return outputs, end_opt - start_opt
+
+
+                # train and profile
+                if args.device == 'xpu':
+                    with torch.autograd.profiler_legacy.profile(args.profile, use_xpu=True) as prof:
+                        try:
+                            import memory_check
+                            memory_check.display_mem("xpu:0")
+                        except:
+                            pass
+                        start_training_time = get_time_sync(args.device)
+                        outputs, opt_time = step_training()
+                    if args.profile:
+                        print(str(prof.key_averages().table(
+                            sort_by="self_xpu_time_total")))
+                        if args.export_chrome_trace and step > 20:
+                            with open('./profile_trace.txt', 'w') as f:
+                                f.write(
+                                    str(prof.table(sort_by="id", row_limit=100000)))
+                            prof.export_chrome_trace('./profile_trace.json')
+                            raise
+                elif args.device == 'cuda':
+                    start_training_time = get_time_sync(args.device)
+                    if args.profile:
+                        with torch.profiler.profile(
+                            activities=[
+                                # torch.profiler.ProfilerActivity.CPU,
+                                torch.profiler.ProfilerActivity.CUDA,
+                            ]
+                        ) as prof:
+                            outputs, opt_time = step_training()
+                    else:
+                        outputs, opt_time = step_training()
+                    if args.profile:
+                        print(str(prof.key_averages().table(
+                            sort_by="self_cuda_time_total")))
+                        if args.export_chrome_trace and step > 20:
+                            prof.export_chrome_trace('./profile_trace.json')
+                            raise
+                end_training_time = get_time_sync(args.device)
+
+                timeinfo = {
+                    'h2d': end_h2d - start_h2d,
+                    'training': end_training_time - start_training_time,
+                    'opt': opt_time,  # training include opt_time
+                }
+
+                timeinfo['total'] = timeinfo['h2d'] + timeinfo['training']
+                timeinfo['total'] = timeinfo['training']
+                latency_list.append(timeinfo['total']/1000.0)
+                if args.info:
+                    print('timeinfo', timeinfo)
+
+                # no need to calculate loss and acc of each iter
+                if args.converge and global_step % PRINT_ITER_FACTOR == 0:
+                    gloss, lm_acc, num_masked, seq_acc, seq_tot = calc_accuracy(
+                        outputs, masked_lm_labels, next_sentence_labels, args)
+                    gloss_list.append(gloss)
+                    acc_list.append(lm_acc.item())
+                    avg_gloss = round(sum(gloss_list) / len(gloss_list), 3)
+                    avg_acc = round(sum(acc_list) / len(acc_list), 3)
+                    print('Global Step [', global_step, '/', args.max_steps, '] perf:', str(round(timeinfo['total']/1000.0, 4)),
+                          ' s/it', ', gloss:', round(gloss, 4), '(', avg_gloss, '), lm acc:', round(lm_acc.item(), 4), '(', avg_acc, ')')
+                elif args.converge:
+                    print('Global Step [', global_step,
+                          '/', args.max_steps, ']')
+                else:
+                    gloss, lm_acc, num_masked, seq_acc, seq_tot = calc_accuracy(
+                        outputs, masked_lm_labels, next_sentence_labels, args)
+
+                if not args.converge:
+                    if args.benchmark_steps > 0 and global_step + 1 >= args.benchmark_steps:
+                        synchronize(args.device)
+                        if args.rank == 0:
+                            print(
+                                f"Done Benchmarking {args.benchmark_steps} steps.")
+                        sys.exit(0)
+
+                if args.info:
+                    info = {
+                        "gloss": gloss,
+                        "lm_acc": lm_acc,
+                        "num_masked": num_masked,
+                        "seq_acc": seq_acc,
+                        "seq_tot": seq_tot
+                    }
+                    print("outinfo", info)
+
+                update_step = training_steps % args.gradient_accumulation_steps == 0
+                if update_step:
+                    now_lr = optimizer.param_groups[0]["lr"]
+                    global_step += 1
+                    if args.converge and global_step % PRINT_ITER_FACTOR == 0:
+                        print('global step = ', global_step, '. next eval step = ',
+                              next_eval_step, '. now lr = ', now_lr)
+                    # global_step == next_eval_step, means this step needs do evaluaion
+                    if (args.eval_dir and args.eval_iter_samples > 0 and global_step == next_eval_step):
+                        # first do validation or load ckpt
+                        if eval_count == 0 or args.resume_from_checkpoint:
+                            eval_dataloader = create_eval_dataset(
+                                args, worker_init_fn=worker_init)
+                        samples_trained = (
+                            global_step * args.train_batch_size * args.gradient_accumulation_steps * args.world_size)
+                        print('\n\nBefore eval, samples trained = ',
+                              samples_trained)
+                        print('Before eval, samples trained prev for eval = ',
+                              samples_trained_prev_for_eval)
+
+                        # update samples trained prev in validation
+                        samples_trained_prev_for_eval = samples_trained
+
+                        print('This step [', global_step, '] begin to do validation')
+                        eval_avg_loss, eval_avg_mlm_accuracy = run_eval(
+                            model,
+                            eval_dataloader,
+                            device,
+                            args.num_eval_examples,
+                            args,
+                            first_eval=(eval_count == 0))
+                        if args.converge:
+                            print('Finish validation, this step [', global_step, '] loss: ', round(eval_avg_loss.item(), 4), \
+                                ' final acc: ', round(eval_avg_mlm_accuracy.item(), 4), '\n\n')
+
+                        if not args.converge and (args.rank == 0 or args.rank == -1):
+                            print(
+                                {
+                                    "global_steps": global_step,
+                                    "eval_loss": eval_avg_loss,
+                                    "eval_mlm_accuracy": eval_avg_mlm_accuracy,
+                                }
+                            )
+
+                        if args.converge:
+                            # judge convergence or not
+                            if args.target_mlm_accuracy:
+                                if eval_avg_mlm_accuracy >= args.target_mlm_accuracy:
+                                    if utils.is_main_process():
+                                        end_training, converged = True, True
+                                        print("%f > %f, [Successfully] Target MLM Accuracy reached at %d" % (
+                                            eval_avg_mlm_accuracy, args.target_mlm_accuracy, global_step))
+                        eval_count += 1
+                        next_eval_step = eval_steps[eval_count]
+
+                # update samples trained each step
+                samples_trained = (global_step * args.train_batch_size *
+                                   args.gradient_accumulation_steps * args.world_size)
+
+                # for save/load ckpt
+                if args.converge:
+                    if args.converge and global_step % PRINT_ITER_FACTOR == 0:
+                        print('samples trained: ', samples_trained, '. samples trained prev for save: ',
+                              samples_trained_prev_for_save, '. this round trained: ', (samples_trained - samples_trained_prev_for_save))
+                    if end_training or (samples_trained - samples_trained_prev_for_save >= args.num_samples_per_checkpoint
+                                        and samples_trained >= args.min_samples_to_start_checkpoints):
+
+                        # update samples trained prev in save ckpt
+                        samples_trained_prev_for_save = samples_trained
+
+                        # main process save ckpt to save disk space
+                        if utils.is_main_process() and not args.skip_checkpoint:
+                            print('Prepare to save model')
+                            # Save a trained model
+                            model_to_save = (
+                                model.module if hasattr(
+                                    model, "module") else model
+                            )  # Only save the model it-self
+
+                            device_info_for_ckpt = "samples_"
+                            if torch.distributed.is_initialized():
+                                device_info_for_ckpt = "rank_" + str(args.rank) + "_samples_"
+                            if end_training and converged:
+                                if args.phase2:
+                                    output_save_file = os.path.join(
+                                        args.output_dir,
+                                        "phase2_ckpt_" + device_info_for_ckpt + str(samples_trained) + ".converged.pt",
+                                    )
+                                else:
+                                    output_save_file = os.path.join(
+                                        args.output_dir,
+                                        "phase1_ckpt_" + device_info_for_ckpt + str(samples_trained) + ".converged.pt",
+                                    )
+                            else:
+                                if args.phase2:
+                                    output_save_file = os.path.join(
+                                        args.output_dir,
+                                        "phase2_ckpt_" + device_info_for_ckpt + str(samples_trained) + ".pt",
+                                    )
+                                else:
+                                    output_save_file = os.path.join(
+                                        args.output_dir,
+                                        "phase1_ckpt_" + device_info_for_ckpt + str(samples_trained) + ".pt",
+                                    )
+                            if args.do_train:
+                                print('\n\n[info] save checkpoint: ',
+                                      output_save_file, '\n')
+                                torch.save({
+                                    "model": model_to_save.state_dict(),
+                                    "optimizer": optimizer.state_dict(),
+                                    "lr_scheduler": lr_scheduler.state_dict(),
+                                    "files": [f_id] + files,
+                                    "epoch": epoch,
+                                    "global_step": global_step,
+                                    "samples_trained": samples_trained,
+                                    "samples_trained_prev_for_eval": samples_trained_prev_for_eval,
+                                    "samples_trained_prev_for_save": samples_trained_prev_for_save,
+                                    "eval_count": eval_count,
+                                    "eval_steps": eval_steps},
+                                    output_save_file)
+
+                                most_recent_ckpts_paths.append(
+                                    output_save_file)
+                                if (len(most_recent_ckpts_paths) > args.keep_n_most_recent_checkpoints):
+                                    print('\n\nkeep ckpt number:',
+                                          args.keep_n_most_recent_checkpoints)
+                                    ckpt_to_be_removed = most_recent_ckpts_paths.pop(
+                                        0)
+                                    print('remove ckpt:',
+                                          ckpt_to_be_removed, '\n\n')
+                                    os.remove(ckpt_to_be_removed)
+
+                        if samples_trained >= args.max_samples_termination or end_training:
+                            status = "success" if converged else "aborted"
+                            end_training = True
+                            break
+
+            del train_dataloader
+            if args.converge:
+                print('\nfinish one file, global step = ', global_step)
+                print('finish one file, samples trained = ', samples_trained)
+                print('finish one file, max samples termination = ',
+                      args.max_samples_termination, '\n')
+            if samples_trained >= args.max_samples_termination or end_training:
+                status = 'success' if converged else 'aborted'
+                print('eventual status = ', status)
+                end_training = True
+                break
+            train_dataloader, data_file = create_pretraining_dataset(
+                data_file,
+                args.max_predictions_per_seq,
+                shared_file_list,
+                args,
+                worker_init_fn=worker_init,
+            )
+
+        epoch += 1
+
+    global_end_time = time.time()
+    print('[info] total consume time: ', ((global_end_time - global_start_time) / 60.0), ' min.')
+    return args, final_loss, train_time_raw
+
+
+if __name__ == "__main__":
+    main()
diff --git a/models_v2/pytorch/bert_large/training/gpu/run_pretraining_demo.sh b/models_v2/pytorch/bert_large/training/gpu/run_pretraining_demo.sh
new file mode 100755
index 000000000..531ee6059
--- /dev/null
+++ b/models_v2/pytorch/bert_large/training/gpu/run_pretraining_demo.sh
@@ -0,0 +1,43 @@
+#
+# Copyright (c) 2022 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+python -u run_pretrain_mlperf.py \
+    --config_name=bert_config.json \
+    --input_dir=miniwiki/hdf5 \
+    --output_dir=result \
+    --eval_dir=miniwiki/hdf5 \
+    --device=xpu \
+    --do_train \
+    --train_batch_size=16 \
+    --gradient_accumulation_steps=1 \
+    --bf16 \
+    --adamw --num-iterations 10
+
+# DDP training
+source $(python -c "import oneccl_bindings_for_pytorch as torch_ccl;print(torch_ccl.cwd)")/env/setvars.sh
+export LD_PRELOAD=$(python -c "import oneccl_bindings_for_pytorch as torch_ccl;print(torch_ccl.cwd)")/lib/libmpi.so
+export ONECCL_BINDINGS_FOR_PYTORCH_ENV_VERBOSE=1
+
+mpiexec -n 2 -l python -u run_pretrain_mlperf.py \
+    --config_name=bert_config.json \
+    --input_dir=miniwiki/hdf5 \
+    --output_dir=result \
+    --eval_dir=miniwiki/hdf5 \
+    --device=xpu \
+    --do_train \
+    --train_batch_size=16 \
+    --gradient_accumulation_steps=1 \
+    --bf16 \
+    --adamw --num-iterations 10
diff --git a/models_v2/pytorch/bert_large/training/gpu/run_pretraining_fp8_demo.sh b/models_v2/pytorch/bert_large/training/gpu/run_pretraining_fp8_demo.sh
new file mode 100755
index 000000000..cccdb64c8
--- /dev/null
+++ b/models_v2/pytorch/bert_large/training/gpu/run_pretraining_fp8_demo.sh
@@ -0,0 +1,26 @@
+#
+# Copyright (c) 2022 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+python -u run_pretrain_mlperf.py \
+    --config_name=bert_config.json \
+    --input_dir=miniwiki/hdf5 \
+    --output_dir=result \
+    --eval_dir=miniwiki/hdf5 \
+    --device=xpu \
+    --do_train \
+    --train_batch_size=16 \
+    --gradient_accumulation_steps=1 \
+    --fp8 \
+    --adamw --num-iterations 10
diff --git a/models_v2/pytorch/bert_large/training/gpu/schedulers.py b/models_v2/pytorch/bert_large/training/gpu/schedulers.py
new file mode 100644
index 000000000..50c5679e9
--- /dev/null
+++ b/models_v2/pytorch/bert_large/training/gpu/schedulers.py
@@ -0,0 +1,119 @@
+# Copyright (c) 2022-2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Copyright (c) 2019 NVIDIA CORPORATION. All rights reserved.
+# Copyright 2018 The Google AI Language Team Authors and The HugginFace Inc. team.
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import math
+import torch
+from torch.optim.optimizer import Optimizer
+from torch.optim.lr_scheduler import _LRScheduler
+
+
+class LRScheduler(_LRScheduler):
+    def __init__(self, optimizer, last_epoch=-1):
+        # Check if using mixed precision training
+        self.mixed_training = False
+        base_optimizer = optimizer
+
+        # Check that optimizer param is valid
+        if not isinstance(optimizer, Optimizer):
+            raise TypeError('{} is not an Optimizer'.format(
+                type(optimizer).__name__))
+
+        super(LRScheduler, self).__init__(base_optimizer, last_epoch)
+
+    def step(self, epoch=None):
+        # Set the current training step
+        # ('epoch' is used to be consistent with _LRScheduler)
+        if self.mixed_training:
+            # The assumption is that the step will be constant
+            state_dict = self.optimizer.state[self.optimizer.param_groups[0]['params'][0]]
+            if 'step' in state_dict:
+                self.last_epoch = state_dict['step'] + 1
+            else:
+                self.last_epoch = 1
+        else:
+            self.last_epoch = epoch if epoch is not None else self.last_epoch + 1
+
+        for param_group, lr in zip(self.optimizer.param_groups, self.get_lr()):
+            param_group['lr'] = lr
+
+
+class LinearWarmUpScheduler(LRScheduler):
+    """
+    Applies a warm up period to the learning rate.
+    """
+
+    def __init__(self, optimizer, warmup, total_steps, last_epoch=-1):
+        self.warmup = warmup
+        self.total_steps = total_steps
+        super(LinearWarmUpScheduler, self).__init__(optimizer, last_epoch)
+
+    def get_lr(self):
+        progress = self.last_epoch / self.total_steps
+        if progress < self.warmup:
+            return [base_lr * progress / self.warmup for base_lr in self.base_lrs]
+        else:
+            return [base_lr * max((progress - 1.0)/(self.warmup - 1.0), 0.) for base_lr in self.base_lrs]
+
+
+class LinearWarmupPolyDecayScheduler(LRScheduler):
+    """
+    Applies a warm up period to the learning rate.
+    """
+
+    def __init__(self, optimizer, start_warmup_steps, warmup_steps, total_steps, end_learning_rate=0.0, degree=1.0, last_epoch=-1):
+        self.num_warmup_updates = warmup_steps
+        self.start_warmup_steps = start_warmup_steps
+        self.total_steps = total_steps
+        self.end_learning_rate = end_learning_rate
+        self.degree = degree
+        self.offset_step = int(self.start_warmup_steps == 0)
+        super(LinearWarmupPolyDecayScheduler,
+              self).__init__(optimizer, last_epoch)
+
+    def step(self, epoch=None):
+        # Instead of optimizer.param_groups['lr'],
+        # update optimizer._lr to avoid sync
+        state_dict = self.optimizer.state[self.optimizer.param_groups[0]['params'][0]]
+        if 'step' in state_dict:
+            self.last_epoch = int(state_dict['step']) + 1
+        else:
+            self.last_epoch = 1
+        lr = self.get_lr()
+        for param_group in self.optimizer.param_groups:
+            param_group['lr'] = lr
+
+    def get_lr(self):
+        mod_step = self.last_epoch - self.offset_step - self.start_warmup_steps
+        cond = mod_step < self.num_warmup_updates
+        progress = (cond * (mod_step / (self.num_warmup_updates + 1e-6))) + \
+            ((1.0 - cond) * (min((self.last_epoch - self.offset_step) / self.total_steps, 1)))
+        base_lr = self.base_lrs[0]
+        lr = (cond * (base_lr * progress)) + \
+            ((1.0 - cond) * ((base_lr - self.end_learning_rate) *
+                             (1-progress) ** self.degree + self.end_learning_rate))
+        return lr
diff --git a/models_v2/pytorch/bert_large/training/gpu/setup.sh b/models_v2/pytorch/bert_large/training/gpu/setup.sh
new file mode 100755
index 000000000..0497e5d23
--- /dev/null
+++ b/models_v2/pytorch/bert_large/training/gpu/setup.sh
@@ -0,0 +1,28 @@
+#!/usr/bin/env bash
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# setup.sh
+#  - install OS pkgs
+#  - should create virtual env & install pip  requirement.txt
+#  - git clones & applying patches
+
+set -e
+apt-get update && apt-get install -y python3-venv protobuf-compiler
+
+pip install -r requirements.txt
+
+cp -r ../../../../common .
diff --git a/models_v2/pytorch/bert_large/training/gpu/utils.py b/models_v2/pytorch/bert_large/training/gpu/utils.py
new file mode 100644
index 000000000..3800669c1
--- /dev/null
+++ b/models_v2/pytorch/bert_large/training/gpu/utils.py
@@ -0,0 +1,137 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Copyright (c) 2019 NVIDIA CORPORATION. All rights reserved.
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import torch
+import torch.distributed as dist
+
+import logging.config
+import random
+
+
+def generate_seeds(rng, size):
+    """
+    Generate list of random seeds
+    :param rng: random number generator
+    :param size: length of the returned list
+    """
+    seeds = [rng.randint(0, 2 ** 32 - 1) for _ in range(size)]
+    return seeds
+
+
+def broadcast_seeds(seeds, device):
+    """
+    Broadcasts random seeds to all distributed workers.
+    Returns list of random seeds (broadcasted from workers with rank 0).
+    :param seeds: list of seeds (integers)
+    :param device: torch.device
+    """
+    if torch.distributed.is_available() and torch.distributed.is_initialized():
+        seeds_tensor = torch.LongTensor(seeds).to(device)
+        torch.distributed.broadcast(seeds_tensor, 0)
+        seeds = seeds_tensor.tolist()
+    return seeds
+
+
+def setup_seeds(master_seed, epochs, device):
+    """
+    Generates seeds from one master_seed.
+    Function returns (worker_seeds, shuffling_seeds), worker_seeds are later
+    used to initialize per-worker random number generators (mostly for
+    dropouts), shuffling_seeds are for RNGs resposible for reshuffling the
+    dataset before each epoch.
+    Seeds are generated on worker with rank 0 and broadcasted to all other
+    workers.
+    :param master_seed: master RNG seed used to initialize other generators
+    :param epochs: number of epochs
+    :param device: torch.device (used for distributed.broadcast)
+    """
+    if master_seed is None:
+        # random master seed, random.SystemRandom() uses /dev/urandom on Unix
+        master_seed = random.SystemRandom().randint(0, 2 ** 32 - 1)
+        if get_rank() == 0:
+            # master seed is reported only from rank=0 worker, it's to avoid
+            # confusion, seeds from rank=0 are later broadcasted to other
+            # workers
+            logging.info(f"Using random master seed: {master_seed}")
+    else:
+        # master seed was specified from command line
+        logging.info(f"Using master seed from command line: {master_seed}")
+
+    # initialize seeding RNG
+    seeding_rng = random.Random(master_seed)
+
+    # generate worker seeds, one seed for every distributed worker
+    worker_seeds = generate_seeds(seeding_rng, get_world_size())
+
+    # generate seeds for data shuffling, one seed for every epoch
+    shuffling_seeds = generate_seeds(seeding_rng, epochs)
+
+    # broadcast seeds from rank=0 to other workers
+    worker_seeds = broadcast_seeds(worker_seeds, device)
+    shuffling_seeds = broadcast_seeds(shuffling_seeds, device)
+    return worker_seeds, shuffling_seeds
+
+
+def get_rank():
+    """
+    Gets distributed rank or returns zero if distributed is not initialized.
+    """
+    if torch.distributed.is_available() and torch.distributed.is_initialized():
+        rank = torch.distributed.get_rank()
+    else:
+        rank = 0
+    return rank
+
+
+def get_world_size():
+    """
+    Gets total number of distributed workers or returns one if distributed is
+    not initialized.
+    """
+
+    if torch.distributed.is_available():
+        print("Torch distributed is available.")
+    else:
+        print("Torch distributed is not available.")
+
+    if torch.distributed.is_initialized():
+        print("Torch distributed is initialized.")
+    else:
+        print("Torch distributed is not initialized.")
+
+    if torch.distributed.is_available() and torch.distributed.is_initialized():
+        world_size = torch.distributed.get_world_size()
+    else:
+        world_size = 1
+    return world_size
+
+
+def is_main_process():
+    return get_rank() == 0
diff --git a/models_v2/pytorch/distilbert/inference/gpu/CONTAINER_FLEX.md b/models_v2/pytorch/distilbert/inference/gpu/CONTAINER_FLEX.md
new file mode 100644
index 000000000..10f61de5d
--- /dev/null
+++ b/models_v2/pytorch/distilbert/inference/gpu/CONTAINER_FLEX.md
@@ -0,0 +1,80 @@
+# Running DistilBERT inference on Intel® Data Center GPU Flex Series using Intel® Extension for PyTorch*
+
+## Overview
+
+This document has instructions for running DistilBERT inference inference using Intel® Extension for PyTorch on Intel® Flex Series GPU.
+
+## Requirements
+| Item | Detail |
+| ------ | ------- |
+| Host machine  | [Intel® Data Center GPU Flex Series](https://ark.intel.com/content/www/us/en/ark/products/series/230021/intel-data-center-gpu-flex-series.html) 170  |
+| Drivers | GPU-compatible drivers need to be installed: Download [Driver](https://dgpu-docs.intel.com/driver/installation.html) |
+| Software | Docker* |
+
+# Datasets
+
+Refer to instructions [here](README.md#dataset) to download and prepare the dataset. Set `DATASET_DIR` to point to the dataset directory. 
+
+## Quick Start Scripts
+
+| Script name | Description |
+|-------------|-------------|
+| `run_model.sh` | Inference with batch size 32 on Flex series 170 |
+
+## Run Using Docker
+
+### Set up Docker Image
+
+```
+docker pull intel/recommendation:pytorch-flex-gpu-distilbert-inference
+```
+
+### Run Docker Image
+The distilbert inference container includes scripts, model and libraries needed to run FP16 and FP32 inference. To run the `run_model.sh` quickstart script using this container, you will need to provide an output directory where log files will be written.
+
+```bash
+#Optional 
+export PRECISION=<provide FP32 or FP16 otherwise (default:FP16)>
+export BATCH_SIZE=<provide batch size otherwise (default:32)>
+export NUM_ITERATIONS=<provide num_iterations otherwise (default:300)>
+
+#Required
+export PLATFORM=Flex
+export MULTI_TILE=False
+export OUTPUT_DIR=<path to output directory>
+export SCRIPT=run_model.sh
+export DATASET_DIR=<path to dataset directory>
+
+IMAGE=intel/recommendation:pytorch-flex-gpu-distilbert-inference
+DOCKER_ARGS="--rm -it"
+
+docker run \
+  --privileged \
+  --device=/dev/dri \
+  --ipc=host \
+  --env PRECISION=${PRECISION} \
+  --env DATASET_DIR=${DATASET_DIR} \
+  --env NUM_ITERATIONS=${NUM_ITERATIONS} \
+  --env OUTPUT_DIR=${OUTPUT_DIR} \
+  --env MULTI_TILE=${MULTI_TILE} \
+  --env PLATFORM=${PLATFORM} \
+  --env http_proxy=${http_proxy} \
+  --env https_proxy=${https_proxy} \
+  --env no_proxy=${no_proxy} \
+  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
+  --volume ${DATASET_DIR}:${DATASET_DIR} \
+  ${DOCKER_ARGS} \
+  ${IMAGE_NAME} \
+  /bin/bash $SCRIPT
+```
+
+## Documentation and Sources
+
+[GitHub* Repository](https://github.com/IntelAI/models/tree/master/docker/flex-gpu)
+
+## Support
+Support for Intel® Extension for PyTorch* is found via the [Intel® AI Analytics Toolkit.](https://www.intel.com/content/www/us/en/developer/tools/oneapi/ai-analytics-toolkit.html#gs.qbretz) Additionally, the Intel® Extension for PyTorch* team tracks both bugs and enhancement requests using [GitHub issues](https://github.com/intel/intel-extension-for-pytorch/issues). Before submitting a suggestion or bug report, please search the GitHub issues to see if your issue has already been reported.
+
+## License Agreement
+
+LEGAL NOTICE: By accessing, downloading or using this software and any required dependent software (the “Software Package”), you agree to the terms and conditions of the software license agreements for the Software Package, which may also include notices, disclaimers, or license terms for third party software included with the Software Package. Please refer to the [license file](https://github.com/IntelAI/models/tree/master/third_party) for additional details.
diff --git a/models_v2/pytorch/distilbert/inference/gpu/CONTAINER_MAX.md b/models_v2/pytorch/distilbert/inference/gpu/CONTAINER_MAX.md
new file mode 100644
index 000000000..fa6c63c2f
--- /dev/null
+++ b/models_v2/pytorch/distilbert/inference/gpu/CONTAINER_MAX.md
@@ -0,0 +1,81 @@
+# Running DistilBERT inference on Intel® Data Center GPU Max Series using Intel® Extension for PyTorch*
+
+## Overview
+
+This document has instructions for running DistilBERT inference inference using Intel® Extension for PyTorch on Intel® Max Series GPU.
+
+## Requirements
+| Item | Detail |
+| ------ | ------- |
+| Host machine  | [Intel® Data Center GPU Max Series](https://ark.intel.com/content/www/us/en/ark/products/series/232874/intel-data-center-gpu-max-series.html) |
+| Drivers | GPU-compatible drivers need to be installed: Download [Driver](https://dgpu-docs.intel.com/driver/installation.html) |
+| Software | Docker* |
+
+# Datasets
+
+Refer to instructions [here](README.md#dataset) to download and prepare the dataset. Set `DATASET_DIR` to point to the dataset directory. 
+
+## Quick Start Scripts
+
+| Script name | Description |
+|-------------|-------------|
+| `run_model.sh` | Inference with batch size 32 on Max series |
+
+## Run Using Docker
+
+### Set up Docker Image
+
+```
+docker pull intel/recommendation:pytorch-max-gpu-distilbert-inference
+```
+
+### Run Docker Image
+The distilbert inference container includes scripts, model and libraries needed to run FP16,BF16 and FP32 inference. To run the `run_model.sh` quickstart script using this container, you will need to provide an output directory where log files will be written.
+
+```bash
+#Optional 
+export PRECISION=<provide FP32, BF16 or FP16 otherwise (default:FP16)>
+export BATCH_SIZE=<provide batch size otherwise (default:32)>
+export NUM_ITERATIONS=<provide num_iterations otherwise (default:300)>
+
+#Required
+export PLATFORM=Max
+export MULTI_TILE=True
+export OUTPUT_DIR=<path to output directory>
+export SCRIPT=run_model.sh
+export DATASET_DIR=<path to dataset directory>
+
+IMAGE=intel/recommendation:pytorch-max-gpu-distilbert-inference
+DOCKER_ARGS="--rm -it"
+
+docker run \
+  --privileged \
+  --device=/dev/dri \
+  --ipc=host \
+  --env BATCH_SIZE=${BATCH_SIZE} \
+  --env PRECISION=${PRECISION} \
+  --env DATASET_DIR=${DATASET_DIR} \
+  --env NUM_ITERATIONS=${NUM_ITERATIONS} \
+  --env OUTPUT_DIR=${OUTPUT_DIR} \
+  --env MULTI_TILE=${MULTI_TILE} \
+  --env PLATFORM=${PLATFORM} \
+  --env http_proxy=${http_proxy} \
+  --env https_proxy=${https_proxy} \
+  --env no_proxy=${no_proxy} \
+  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
+  --volume ${DATASET_DIR}:${DATASET_DIR} \
+  ${DOCKER_ARGS} \
+  ${IMAGE_NAME} \
+  /bin/bash $SCRIPT
+```
+
+## Documentation and Sources
+
+[GitHub* Repository](https://github.com/IntelAI/models/tree/master/docker/max-gpu)
+
+## Support
+Support for Intel® Extension for PyTorch* is found via the [Intel® AI Analytics Toolkit.](https://www.intel.com/content/www/us/en/developer/tools/oneapi/ai-analytics-toolkit.html#gs.qbretz) Additionally, the Intel® Extension for PyTorch* team tracks both bugs and enhancement requests using [GitHub issues](https://github.com/intel/intel-extension-for-pytorch/issues). Before submitting a suggestion or bug report, please search the GitHub issues to see if your issue has already been reported.
+
+## License Agreement
+
+LEGAL NOTICE: By accessing, downloading or using this software and any required dependent software (the “Software Package”), you agree to the terms and conditions of the software license agreements for the Software Package, which may also include notices, disclaimers, or license terms for third party software included with the Software Package. Please refer to the [license file](https://github.com/IntelAI/models/tree/master/third_party) for additional details.
diff --git a/models_v2/pytorch/distilbert/inference/gpu/README.md b/models_v2/pytorch/distilbert/inference/gpu/README.md
new file mode 100644
index 000000000..dd3591bc9
--- /dev/null
+++ b/models_v2/pytorch/distilbert/inference/gpu/README.md
@@ -0,0 +1,113 @@
+# DistilBert Inference
+
+DistilBert Inference best known configurations with Intel® Extension for PyTorch.
+
+## Model Information
+
+| **Use Case** | **Framework** | **Model Repo** | **Branch/Commit/Tag** | **Optional Patch** |
+|:---:| :---: |:--------------:|:---------------------:|:------------------:|
+|  Inference   |    PyTorch    |       https://github.com/huggingface/transformers/tree/main/src/transformers/models/distilbert        |           -           |         -          |
+
+# Pre-Requisite
+* Host has one of the following GPUs:
+  * **Flex Series** - [Intel® Data Center GPU Flex Series](https://ark.intel.com/content/www/us/en/ark/products/series/230021/intel-data-center-gpu-flex-series.html)
+  * **Max Series** - [Intel® Data Center GPU Max Series](https://ark.intel.com/content/www/us/en/ark/products/series/232874/intel-data-center-gpu-max-series.html)
+* Host has installed latest Intel® Data Center GPU Max & Flex Series Drivers https://dgpu-docs.intel.com/driver/installation.html
+* The following Intel® oneAPI Base Toolkit components are required:
+  - Intel® oneAPI DPC++ Compiler (Placeholder DPCPPROOT as its installation path)
+  - Intel® oneAPI Math Kernel Library (oneMKL) (Placeholder MKLROOT as its installation path)
+  - Intel® oneAPI MPI Library
+  - Intel® oneAPI TBB Library
+
+  Follow instructions at [Intel® oneAPI Base Toolkit Download page](https://www.intel.com/content/www/us/en/developer/tools/oneapi/base-toolkit-download.html?operatingsystem=linux) to setup the package manager repository.
+
+# Prepare Dataset
+## Dataset: 
+Please refer to https://github.com/mlcommons/training_results_v2.1/tree/main/NVIDIA/benchmarks/bert/implementations/pytorch-22.09#download-and-prepare-the-data
+
+the dataset should be like below
+|_hdf5  
+      |_ eval                               # evaluation chunks in binary hdf5 format fixed length (not used in training, can delete after data   preparation)  
+      |_ eval_varlength                     # evaluation chunks in binary hdf5 format variable length *used for training*
+      |_ training                           # 500 chunks in binary hdf5 format 
+      |_ training_4320                      # 
+      |_ hdf5_4320_shards_uncompressed   # sharded data in hdf5 format fixed length (not used in training, can delete after data   preparation)
+      |_ hdf5_4320_shards_varlength      # sharded data in hdf5 format variable length *used for training
+
+## Inference
+1. `git clone https://github.com/IntelAI/models.git`
+2. `cd models/models_v2/pytorch/distilbert/inference/gpu`
+3. Create virtual environment `venv` and activate it:
+    ```
+    python3 -m venv venv
+    . ./venv/bin/activate
+    ```
+4. Run setup.sh
+    ```
+    ./setup.sh
+    ```
+5. Install the latest GPU versions of [torch, torchvision and intel_extension_for_pytorch](https://intel.github.io/intel-extension-for-pytorch/index.html#installation):
+  ```
+  python -m pip install torch==<torch_version> torchvision==<torchvvision_version> intel-extension-for-pytorch==<ipex_version> --extra-index-url https://pytorch-extension.intel.com/release-whl-aitools/
+  ```
+6. Set environment variables for Intel® oneAPI Base Toolkit: 
+    Default installation location `{ONEAPI_ROOT}` is `/opt/intel/oneapi` for root account, `${HOME}/intel/oneapi` for other accounts
+    ```bash
+    source {ONEAPI_ROOT}/compiler/latest/env/vars.sh
+    source {ONEAPI_ROOT}/mkl/latest/env/vars.sh
+    source {ONEAPI_ROOT}/tbb/latest/env/vars.sh
+    source {ONEAPI_ROOT}/mpi/latest/env/vars.sh
+    source {ONEAPI_ROOT}/ccl/latest/env/vars.sh
+    ```
+7. Setup required environment paramaters
+
+| **Parameter**                |                                  **export command**                                  |
+|:---------------------------:|:------------------------------------------------------------------------------------:|
+| **MULTI_TILE**               | `export MULTI_TILE=True` (True or False)                                             |
+| **PLATFORM**                 | `export PLATFORM=Max` (Max or Flex)                                                 |
+| **DATASET_DIR**              |                               `export DATASET_DIR=`                                  |
+| **BATCH_SIZE** (optional)    |                               `export BATCH_SIZE=32`                                |
+| **PRECISION** (optional)     | `export PRECISION=BF16` (FP32, BF16, FP16 and TF32 for Max and FP16, FP32 for Flex)|
+| **OUTPUT_DIR** (optional)    |                               `export OUTPUT_DIR=$PWD`                               |
+|**NUM_ITERATIONS** (optional) |                               `export NUM_ITERATIONS=300`                             |
+8. Run `run_model.sh`
+
+> [!NOTE]
+> Refer to [CONTAINER_FLEX.md](CONTAINER_FLEX.md) and [CONTAINER_MAX.md](CONTAINER_MAX.md) for DistilBERT inference instructions using docker containers.
+## Output
+
+Single-tile output will typically looks like:
+
+```
+12/21/2023 14:28:08 - INFO - utils - PID: 148054 -  --- Ending inference
+12/21/2023 14:28:08 - INFO - utils - PID: 148054 -  Results: {'acc': 0.5852613944871974, 'eval_loss': 1.9857747395833334}
+12/21/2023 14:28:08 - INFO - utils - PID: 148054 -  The total_time 8.032912015914917 s, and perf 1115.411196120202 sentences/s for inference
+12/21/2023 14:28:08 - INFO - utils - PID: 148054 -  Let's go get some drinks.
+```
+
+Multi-tile output will typically looks like:
+```
+12/21/2023 14:33:13 - INFO - utils - PID: 148381 -  --- Ending inference
+12/21/2023 14:33:13 - INFO - utils - PID: 148381 -  Results: {'acc': 0.5852613944871974, 'eval_loss': 1.9857747395833334}
+12/21/2023 14:33:13 - INFO - utils - PID: 148381 -  The total_time 8.122166156768799 s, and perf 1103.1539895958633 sentences/s for inference
+-Iter:   5%|▍         | 296/6087 [00:12<03:27, 27.93it/s]12/21/2023 14:33:13 - INFO - utils - PID: 148381 -  Let's go get some drinks.
+-Iter:   5%|▍         | 300/6087 [00:12<03:56, 24.42it/s]
+12/21/2023 14:33:13 - INFO - utils - PID: 148383 -  --- Ending inference
+12/21/2023 14:33:13 - INFO - utils - PID: 148383 -  Results: {'acc': 0.5852613944871974, 'eval_loss': 1.9857747395833334}
+12/21/2023 14:33:13 - INFO - utils - PID: 148383 -  The total_time 8.266947984695435 s, and perf 1083.834084427241 sentences/s for inference
+12/21/2023 14:33:13 - INFO - utils - PID: 148383 -  Let's go get some drinks.
+```
+
+Final results of the inference run can be found in `results.yaml` file.
+```
+results:
+ - key: throughput
+   value: 2186.9881
+   unit: sent/s
+ - key: latency
+   value: 0.0292663
+   unit: s
+ - key: accuracy
+   value: 0.5850
+   unit: acc
+```
diff --git a/models_v2/pytorch/distilbert/inference/gpu/demo-data/create_pretraining_data.py b/models_v2/pytorch/distilbert/inference/gpu/demo-data/create_pretraining_data.py
new file mode 100644
index 000000000..a65649596
--- /dev/null
+++ b/models_v2/pytorch/distilbert/inference/gpu/demo-data/create_pretraining_data.py
@@ -0,0 +1,407 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+import collections
+import random
+import tokenization
+import h5py
+import os
+import argparse
+import numpy as np
+
+hdf5_compression_method = None
+
+
+def parse_args():
+    parser = argparse.ArgumentParser(
+        description="Create pretraining data for bert")
+    parser.add_argument("--vocab_file", type=str, required=True)
+    parser.add_argument("--input_files", type=str, required=True)
+    parser.add_argument("--output_files", type=str, required=True)
+    parser.add_argument("--do_lower_case", action="store_true")
+    parser.add_argument("--seed", type=int, default=12345)
+    parser.add_argument("--max_seq_length", type=int, default=128)
+    parser.add_argument("--dupe_factor", type=int, default=10,
+                        help="Number of times to duplicate the input data (with different masks).")
+    parser.add_argument("--short_seq_prob", type=float, default=0.1,
+                        help="Probability of creating sequences which are shorter than the maximum length.")
+    parser.add_argument("--masked_lm_prob", type=float, default=0.15,
+                        help="Masked LM probability.")
+    parser.add_argument("--max_predictions_per_seq", type=int, default=20,
+                        help="Maximum number of masked LM predictions per sequence.")
+
+    args = parser.parse_args()
+    return args
+
+
+class TrainingInstance(object):
+    """A single training instance (sentence pair)."""
+
+    def __init__(self, tokens, segment_ids, masked_lm_positions, masked_lm_labels,
+                 is_random_next):
+        self.tokens = tokens
+        self.segment_ids = segment_ids
+        self.is_random_next = is_random_next
+        self.masked_lm_positions = masked_lm_positions
+        self.masked_lm_labels = masked_lm_labels
+
+    def __str__(self):
+        s = ""
+        s += "tokens: %s\n" % (" ".join(
+            [tokenization.printable_text(x) for x in self.tokens]))
+        s += "segment_ids: %s\n" % (" ".join([str(x)
+                                              for x in self.segment_ids]))
+        s += "is_random_next: %s\n" % self.is_random_next
+        s += "masked_lm_positions: %s\n" % (" ".join(
+            [str(x) for x in self.masked_lm_positions]))
+        s += "masked_lm_labels: %s\n" % (" ".join(
+            [tokenization.printable_text(x) for x in self.masked_lm_labels]))
+        s += "\n"
+        return s
+
+    def __repr__(self):
+        return self.__str__()
+
+
+def truncate_seq_pair(tokens_a, tokens_b, max_num_tokens, rng):
+    """Truncates a pair of sequences to a maximum sequence length."""
+    while True:
+        total_length = len(tokens_a) + len(tokens_b)
+        if total_length <= max_num_tokens:
+            break
+
+        trunc_tokens = tokens_a if len(tokens_a) > len(tokens_b) else tokens_b
+        assert len(trunc_tokens) >= 1
+
+        # We want to sometimes truncate from the front and sometimes from the
+        # back to add more randomness and avoid biases.
+        if rng.random() < 0.5:
+            del trunc_tokens[0]
+        else:
+            trunc_tokens.pop()
+
+
+MaskedLmInstance = collections.namedtuple(
+    "MaskedLmInstance", ["index", "label"])
+
+
+def create_masked_lm_predictions(tokens, masked_lm_prob,
+                                 max_predictions_per_seq, vocab_words, rng):
+    """Creates the predictions for the masked LM objective."""
+
+    cand_indexes = []
+    for (i, token) in enumerate(tokens):
+        if token == "[CLS]" or token == "[SEP]":
+            continue
+        cand_indexes.append(i)
+
+    rng.shuffle(cand_indexes)
+
+    output_tokens = list(tokens)
+
+    num_to_predict = min(max_predictions_per_seq,
+                         max(1, int(round(len(tokens) * masked_lm_prob))))
+
+    masked_lms = []
+    covered_indexes = set()
+    for index in cand_indexes:
+        if len(masked_lms) >= num_to_predict:
+            break
+        if index in covered_indexes:
+            continue
+        covered_indexes.add(index)
+
+        masked_token = None
+        # 80% of the time, replace with [MASK]
+        if rng.random() < 0.8:
+            masked_token = "[MASK]"
+        else:
+            # 10% of the time, keep original
+            if rng.random() < 0.5:
+                masked_token = tokens[index]
+            # 10% of the time, replace with random word
+            else:
+                masked_token = vocab_words[rng.randint(
+                    0, len(vocab_words) - 1)]
+
+        output_tokens[index] = masked_token
+
+        masked_lms.append(MaskedLmInstance(index=index, label=tokens[index]))
+
+    masked_lms = sorted(masked_lms, key=lambda x: x.index)
+
+    masked_lm_positions = []
+    masked_lm_labels = []
+    for p in masked_lms:
+        masked_lm_positions.append(p.index)
+        masked_lm_labels.append(p.label)
+
+    return (output_tokens, masked_lm_positions, masked_lm_labels)
+
+
+def create_instances_from_document(
+        all_documents, document_index, max_seq_length, short_seq_prob,
+        masked_lm_prob, max_predictions_per_seq, vocab_words, rng):
+    """Creates `TrainingInstance`s for a single document."""
+    document = all_documents[document_index]
+
+    # Account for [CLS], [SEP], [SEP]
+    max_num_tokens = max_seq_length - 3
+
+    # We *usually* want to fill up the entire sequence since we are padding
+    # to `max_seq_length` anyways, so short sequences are generally wasted
+    # computation. However, we *sometimes*
+    # (i.e., short_seq_prob == 0.1 == 10% of the time) want to use shorter
+    # sequences to minimize the mismatch between pre-training and fine-tuning.
+    # The `target_seq_length` is just a rough target however, whereas
+    # `max_seq_length` is a hard limit.
+    target_seq_length = max_num_tokens
+    if rng.random() < short_seq_prob:
+        target_seq_length = rng.randint(2, max_num_tokens)
+
+    # We DON'T just concatenate all of the tokens from a document into a long
+    # sequence and choose an arbitrary split point because this would make the
+    # next sentence prediction task too easy. Instead, we split the input into
+    # segments "A" and "B" based on the actual "sentences" provided by the user
+    # input.
+    instances = []
+    current_chunk = []
+    current_length = 0
+    i = 0
+    while i < len(document):
+        segment = document[i]
+        current_chunk.append(segment)
+        current_length += len(segment)
+        if i == len(document) - 1 or current_length >= target_seq_length:
+            if current_chunk:
+                # `a_end` is how many segments from `current_chunk` go into the `A`
+                # (first) sentence.
+                a_end = 1
+                if len(current_chunk) >= 2:
+                    a_end = rng.randint(1, len(current_chunk) - 1)
+
+                tokens_a = []
+                for j in range(a_end):
+                    tokens_a.extend(current_chunk[j])
+
+                tokens_b = []
+                # Random next
+                is_random_next = False
+                if len(current_chunk) == 1 or rng.random() < 0.5:
+                    is_random_next = True
+                    target_b_length = target_seq_length - len(tokens_a)
+
+                    # This should rarely go for more than one iteration for large
+                    # corpora. However, just to be careful, we try to make sure that
+                    # the random document is not the same as the document
+                    # we're processing.
+                    for _ in range(10):
+                        random_document_index = rng.randint(
+                            0, len(all_documents) - 1)
+                        if random_document_index != document_index:
+                            break
+
+                    random_document = all_documents[random_document_index]
+                    random_start = rng.randint(0, len(random_document) - 1)
+                    for j in range(random_start, len(random_document)):
+                        tokens_b.extend(random_document[j])
+                        if len(tokens_b) >= target_b_length:
+                            break
+                    # We didn't actually use these segments so we "put them back" so
+                    # they don't go to waste.
+                    num_unused_segments = len(current_chunk) - a_end
+                    i -= num_unused_segments
+                # Actual next
+                else:
+                    is_random_next = False
+                    for j in range(a_end, len(current_chunk)):
+                        tokens_b.extend(current_chunk[j])
+                truncate_seq_pair(tokens_a, tokens_b, max_num_tokens, rng)
+
+                assert len(tokens_a) >= 1
+                assert len(tokens_b) >= 1
+
+                tokens = []
+                segment_ids = []
+                tokens.append("[CLS]")
+                segment_ids.append(0)
+                for token in tokens_a:
+                    tokens.append(token)
+                    segment_ids.append(0)
+
+                tokens.append("[SEP]")
+                segment_ids.append(0)
+
+                for token in tokens_b:
+                    tokens.append(token)
+                    segment_ids.append(1)
+                tokens.append("[SEP]")
+                segment_ids.append(1)
+
+                (tokens, masked_lm_positions,
+                 masked_lm_labels) = create_masked_lm_predictions(
+                     tokens, masked_lm_prob, max_predictions_per_seq, vocab_words, rng)
+                instance = TrainingInstance(
+                    tokens=tokens,
+                    segment_ids=segment_ids,
+                    is_random_next=is_random_next,
+                    masked_lm_positions=masked_lm_positions,
+                    masked_lm_labels=masked_lm_labels)
+                instances.append(instance)
+            current_chunk = []
+            current_length = 0
+        i += 1
+
+    return instances
+
+
+def create_training_instances(
+        input_files, tokenizer, max_seq_length,
+        dupe_factor, short_seq_prob, masked_lm_prob,
+        max_predictions_per_seq, rng):
+    all_documents = [[]]
+    for input_file in input_files:
+        print('processing ' + str(input_file))
+        with open(input_file, 'r') as f:
+            while True:
+                line = tokenization.convert_to_unicode(f.readline())
+                if not line:
+                    break
+                line = line.strip()
+                if not line:  # Empty lines are used as document delimiters
+                    all_documents.append([])
+                tokens = tokenizer.tokenize(line)
+                if tokens:
+                    all_documents[-1].append(tokens)
+    all_documents = [x for x in all_documents if x]
+    rng.shuffle(all_documents)
+    vocab_words = list(tokenizer.vocab.keys())
+    instances = []
+    for _ in range(dupe_factor):
+        for document_index in range(len(all_documents)):
+            instances.extend(
+                create_instances_from_document(
+                    all_documents, document_index, max_seq_length, short_seq_prob,
+                    masked_lm_prob, max_predictions_per_seq, vocab_words, rng))
+    rng.shuffle(instances)
+    return instances
+
+
+def write_instance_to_example_files(
+        instances, tokenizer, max_seq_length, max_predictions_per_seq, output_files):
+    writers = []
+    h5_writers = []
+    # Over-allocation to avoid resizing
+    expected_instances_per_file = len(instances) // len(output_files) + 500
+    for output_file in output_files:
+        h5_writers.append({
+            'handle': h5py.File(output_file + ".hdf5", 'w'),
+            'input_ids': np.zeros([expected_instances_per_file, max_seq_length], dtype="int32"),
+            'input_mask': np.zeros([expected_instances_per_file, max_seq_length], dtype="int32"),
+            'segment_ids': np.zeros([expected_instances_per_file, max_seq_length], dtype="int32"),
+            'masked_lm_positions': np.zeros([expected_instances_per_file, max_predictions_per_seq], dtype="int32"),
+            'masked_lm_ids': np.zeros([expected_instances_per_file, max_predictions_per_seq], dtype="int32"),
+            'next_sentence_labels': np.zeros(expected_instances_per_file, dtype="int32"),
+            'len': 0})
+    writer_index = 0
+    total_written = 0
+    features_h5 = collections.OrderedDict()
+    for (inst_index, instance) in enumerate(instances):
+        input_ids = tokenizer.convert_tokens_to_ids(instance.tokens)
+        input_mask = [1] * len(input_ids)
+        segment_ids = list(instance.segment_ids)
+        assert len(input_ids) <= max_seq_length
+
+        while len(input_ids) < max_seq_length:
+            input_ids.append(0)
+            input_mask.append(0)
+            segment_ids.append(0)
+
+        assert len(input_ids) == max_seq_length
+        assert len(input_mask) == max_seq_length
+        assert len(segment_ids) == max_seq_length
+
+        masked_lm_positions = list(instance.masked_lm_positions)
+        masked_lm_ids = tokenizer.convert_tokens_to_ids(
+            instance.masked_lm_labels)
+        masked_lm_weights = [1.0] * len(masked_lm_ids)
+
+        while len(masked_lm_positions) < max_predictions_per_seq:
+            masked_lm_positions.append(0)
+            masked_lm_ids.append(0)
+            masked_lm_weights.append(0.0)
+
+        next_sentence_label = 1 if instance.is_random_next else 0
+
+        h5_writers[writer_index]['input_ids'][inst_index] = input_ids
+        h5_writers[writer_index]['input_mask'][inst_index] = input_mask
+        h5_writers[writer_index]['segment_ids'][inst_index] = segment_ids
+        h5_writers[writer_index]['masked_lm_positions'][inst_index] = masked_lm_positions
+        h5_writers[writer_index]['masked_lm_ids'][inst_index] = masked_lm_ids
+        h5_writers[writer_index]['next_sentence_labels'][inst_index] = next_sentence_label
+        h5_writers[writer_index]['len'] += 1
+
+        writer_index = (writer_index + 1) % len(h5_writers)
+
+        total_written += 1
+
+        if inst_index < 20:
+            print("Example tokens: %s" % " ".join(
+                [tokenization.printable_text(x) for x in instance.tokens]))
+
+    print("saving data")
+    for h5_writer in h5_writers:
+        my_size = h5_writer['len']
+        h5_writer['handle'].create_dataset(
+            'input_ids', data=h5_writer['input_ids'][:my_size], dtype='i4', compression=hdf5_compression_method)
+        h5_writer['handle'].create_dataset(
+            'input_mask', data=h5_writer['input_mask'][:my_size], dtype='i1', compression=hdf5_compression_method)
+        h5_writer['handle'].create_dataset(
+            'segment_ids', data=h5_writer['segment_ids'][:my_size], dtype='i1', compression=hdf5_compression_method)
+        h5_writer['handle'].create_dataset(
+            'masked_lm_positions', data=h5_writer['masked_lm_positions'][:my_size], dtype='i4', compression=hdf5_compression_method)
+        h5_writer['handle'].create_dataset(
+            'masked_lm_ids', data=h5_writer['masked_lm_ids'][:my_size], dtype='i4', compression=hdf5_compression_method)
+        h5_writer['handle'].create_dataset(
+            'next_sentence_labels', data=h5_writer['next_sentence_labels'][:my_size], dtype='i1', compression=hdf5_compression_method)
+        h5_writer['handle'].flush()
+        h5_writer['handle'].close()
+
+    print("Wrote %d total instances", total_written)
+
+
+def main():
+    args = parse_args()
+    tokenizer = tokenization.FullTokenizer(
+        vocab_file=args.vocab_file, do_lower_case=args.do_lower_case)
+    input_files = [item for item in args.input_files.split(
+        ',') if len(item) > 0]
+    rng = random.Random(args.seed)
+    instances = create_training_instances(
+            input_files, tokenizer, args.max_seq_length, args.dupe_factor,
+            args.short_seq_prob, args.masked_lm_prob, args.max_predictions_per_seq,
+            rng)
+    print('Instances generated:', len(instances))
+    output_files = [
+        item for item in args.output_files.split(',') if len(item) > 0]
+
+    basedir = os.path.dirname(output_files[0]) + '/hdf5'
+    os.makedirs(basedir, exist_ok=True)
+    output_files = [os.path.join(basedir, os.path.basename(item)) for item in output_files]
+    write_instance_to_example_files(instances, tokenizer, args.max_seq_length,
+                                    args.max_predictions_per_seq, output_files)
+
+
+if __name__ == '__main__':
+    main()
diff --git a/models_v2/pytorch/distilbert/inference/gpu/demo-data/create_pretraining_data.sh b/models_v2/pytorch/distilbert/inference/gpu/demo-data/create_pretraining_data.sh
new file mode 100644
index 000000000..715a00ab8
--- /dev/null
+++ b/models_v2/pytorch/distilbert/inference/gpu/demo-data/create_pretraining_data.sh
@@ -0,0 +1,22 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+input_files=$1
+
+python create_pretraining_data.py \
+    --vocab_file=${VOCAB_FILE} \
+    --input_file=${input_files} \
+    --output_file=${input_files} \
+    --do_lower_case \
+    --max_seq_length=512 \
diff --git a/models_v2/pytorch/distilbert/inference/gpu/demo-data/parallel_create_pretraining_data.sh b/models_v2/pytorch/distilbert/inference/gpu/demo-data/parallel_create_pretraining_data.sh
new file mode 100644
index 000000000..e014e5832
--- /dev/null
+++ b/models_v2/pytorch/distilbert/inference/gpu/demo-data/parallel_create_pretraining_data.sh
@@ -0,0 +1,22 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# export VOCAB_FILE=/data/wiki/bert-base-uncased-vocab.txt
+cpus=$( ls -d /sys/devices/system/cpu/cpu[[:digit:]]* | wc -w )
+if [ ${cpus} -gt 64 ]; then
+cpus=64
+fi
+echo "Using ${cpus} CPU cores..."
+datadir=$1
+find -L ${datadir} -name "pretrain-part*" | xargs --max-args=1 --max-procs=${cpus} bash create_pretraining_data.sh
diff --git a/models_v2/pytorch/distilbert/inference/gpu/demo-data/requirements.txt b/models_v2/pytorch/distilbert/inference/gpu/demo-data/requirements.txt
new file mode 100644
index 000000000..6ce884aeb
--- /dev/null
+++ b/models_v2/pytorch/distilbert/inference/gpu/demo-data/requirements.txt
@@ -0,0 +1,4 @@
+# Python dependencies required for download demo data
+numpy
+six
+h5py
diff --git a/models_v2/pytorch/distilbert/inference/gpu/demo-data/tokenization.py b/models_v2/pytorch/distilbert/inference/gpu/demo-data/tokenization.py
new file mode 100644
index 000000000..5e9753484
--- /dev/null
+++ b/models_v2/pytorch/distilbert/inference/gpu/demo-data/tokenization.py
@@ -0,0 +1,428 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+import os
+import collections
+import re
+import unicodedata
+import six
+
+
+PRETRAINED_VOCAB_ARCHIVE_MAP = {
+    'bert-base-uncased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-uncased-vocab.txt",
+    'bert-large-uncased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-vocab.txt",
+    'bert-base-cased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-cased-vocab.txt",
+    'bert-large-cased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-cased-vocab.txt",
+    'bert-base-multilingual-uncased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-uncased-vocab.txt",
+    'bert-base-multilingual-cased': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-multilingual-cased-vocab.txt",
+    'bert-base-chinese': "https://s3.amazonaws.com/models.huggingface.co/bert/bert-base-chinese-vocab.txt",
+}
+
+PRETRAINED_VOCAB_POSITIONAL_EMBEDDINGS_SIZE_MAP = {
+    'bert-base-uncased': 512,
+    'bert-large-uncased': 512,
+    'bert-base-cased': 512,
+    'bert-large-cased': 512,
+    'bert-base-multilingual-uncased': 512,
+    'bert-base-multilingual-cased': 512,
+    'bert-base-chinese': 512,
+}
+
+
+def validate_case_matches_checkpoint(do_lower_case, init_checkpoint):
+    """Checks whether the casing config is consistent with the checkpoint name."""
+
+    # The casing has to be passed in by the user and there is no explicit check
+    # as to whether it matches the checkpoint. The casing information probably
+    # should have been stored in the bert_config.json file, but it's not, so
+    # we have to heuristically detect it to validate.
+
+    if not init_checkpoint:
+        return
+
+    m = re.match("^.*?([A-Za-z0-9_-]+)/bert_model.ckpt", init_checkpoint)
+    if m is None:
+        return
+
+    model_name = m.group(1)
+
+    lower_models = [
+        "uncased_L-24_H-1024_A-16", "uncased_L-12_H-768_A-12",
+        "multilingual_L-12_H-768_A-12", "chinese_L-12_H-768_A-12"
+    ]
+
+    cased_models = [
+        "cased_L-12_H-768_A-12", "cased_L-24_H-1024_A-16",
+        "multi_cased_L-12_H-768_A-12"
+    ]
+
+    is_bad_config = False
+    if model_name in lower_models and not do_lower_case:
+        is_bad_config = True
+        actual_flag = "False"
+        case_name = "lowercased"
+        opposite_flag = "True"
+
+    if model_name in cased_models and do_lower_case:
+        is_bad_config = True
+        actual_flag = "True"
+        case_name = "cased"
+        opposite_flag = "False"
+
+    if is_bad_config:
+        raise ValueError(
+            "You passed in `--do_lower_case=%s` with `--init_checkpoint=%s`. "
+            "However, `%s` seems to be a %s model, so you "
+            "should pass in `--do_lower_case=%s` so that the fine-tuning matches "
+            "how the model was pre-training. If this error is wrong, please "
+            "just comment out this check." % (actual_flag, init_checkpoint,
+                                              model_name, case_name, opposite_flag))
+
+
+def convert_to_unicode(text):
+    """Converts `text` to Unicode (if it's not already), assuming utf-8 input."""
+    if six.PY3:
+        if isinstance(text, str):
+            return text
+        elif isinstance(text, bytes):
+            return text.decode("utf-8", "ignore")
+        else:
+            raise ValueError("Unsupported string type: %s" % (type(text)))
+    elif six.PY2:
+        if isinstance(text, str):
+            return text.decode("utf-8", "ignore")
+        elif isinstance(text, unicode):
+            return text
+        else:
+            raise ValueError("Unsupported string type: %s" % (type(text)))
+    else:
+        raise ValueError("Not running on Python2 or Python 3?")
+
+
+def printable_text(text):
+    """Returns text encoded in a way suitable for print or `tf.logging`."""
+
+    # These functions want `str` for both Python2 and Python3, but in one case
+    # it's a Unicode string and in the other it's a byte string.
+    if six.PY3:
+        if isinstance(text, str):
+            return text
+        elif isinstance(text, bytes):
+            return text.decode("utf-8", "ignore")
+        else:
+            raise ValueError("Unsupported string type: %s" % (type(text)))
+    elif six.PY2:
+        if isinstance(text, str):
+            return text
+        elif isinstance(text, unicode):
+            return text.encode("utf-8")
+        else:
+            raise ValueError("Unsupported string type: %s" % (type(text)))
+    else:
+        raise ValueError("Not running on Python2 or Python 3?")
+
+
+def load_vocab(vocab_file):
+    """Loads a vocabulary file into a dictionary."""
+    vocab = collections.OrderedDict()
+    index = 0
+    with open(vocab_file, "r", encoding="utf-8") as reader:
+        while True:
+            token = reader.readline()
+            if not token:
+                break
+            token = token.strip()
+            vocab[token] = index
+            index += 1
+    return vocab
+
+
+def convert_by_vocab(vocab, items):
+    """Converts a sequence of [tokens|ids] using the vocab."""
+    output = []
+    for item in items:
+        output.append(vocab[item])
+    return output
+
+
+def convert_tokens_to_ids(vocab, tokens):
+    return convert_by_vocab(vocab, tokens)
+
+
+def convert_ids_to_tokens(inv_vocab, ids):
+    return convert_by_vocab(inv_vocab, ids)
+
+
+def whitespace_tokenize(text):
+    """Runs basic whitespace cleaning and splitting on a piece of text."""
+    text = text.strip()
+    if not text:
+        return []
+    tokens = text.split()
+    return tokens
+
+
+class FullTokenizer(object):
+    """Runs end-to-end tokenziation."""
+
+    def __init__(self, vocab_file, do_lower_case=True):
+        if not os.path.isfile(vocab_file):
+            raise ValueError(
+                "Can't find a vocabulary file at path '{}'.".format(vocab_file))
+        self.vocab = load_vocab(vocab_file)
+        self.ids_to_tokens = collections.OrderedDict(
+            [(ids, tok) for tok, ids in self.vocab.items()])
+        self.basic_tokenizer = BasicTokenizer(do_lower_case=do_lower_case)
+        self.wordpiece_tokenizer = WordpieceTokenizer(vocab=self.vocab)
+
+    def tokenize(self, text):
+        split_tokens = []
+        for token in self.basic_tokenizer.tokenize(text):
+            for sub_token in self.wordpiece_tokenizer.tokenize(token):
+                split_tokens.append(sub_token)
+        return split_tokens
+
+    def convert_tokens_to_ids(self, tokens):
+        return convert_by_vocab(self.vocab, tokens)
+
+    def convert_ids_to_tokens(self, ids):
+        return convert_by_vocab(self.ids_to_tokens, ids)
+
+
+class BasicTokenizer(object):
+    """Runs basic tokenization (punctuation splitting, lower casing, etc.)."""
+
+    def __init__(self,
+                 do_lower_case=True,
+                 never_split=("[UNK]", "[SEP]", "[PAD]", "[CLS]", "[MASK]")):
+        """Constructs a BasicTokenizer.
+        Args:
+          do_lower_case: Whether to lower case the input.
+          vocab: A container of tokens to not mutate during tokenization.
+        """
+        self.do_lower_case = do_lower_case
+        self.never_split = never_split
+
+    def tokenize(self, text):
+        """Tokenizes a piece of text."""
+        text = convert_to_unicode(text)
+        text = self._clean_text(text)
+
+        # This was added on November 1st, 2018 for the multilingual and Chinese
+        # models. This is also applied to the English models now, but it doesn't
+        # matter since the English models were not trained on any Chinese data
+        # and generally don't have any Chinese data in them (there are Chinese
+        # characters in the vocabulary because Wikipedia does have some Chinese
+        # words in the English Wikipedia.).
+        text = self._tokenize_chinese_chars(text)
+
+        orig_tokens = whitespace_tokenize(text)
+        split_tokens = []
+        for token in orig_tokens:
+            if self.do_lower_case and token not in self.never_split:
+                token = token.lower()
+                token = self._run_strip_accents(token)
+            split_tokens.extend(self._run_split_on_punc(token))
+
+        output_tokens = whitespace_tokenize(" ".join(split_tokens))
+        return output_tokens
+
+    def _run_strip_accents(self, text):
+        """Strips accents from a piece of text."""
+        text = unicodedata.normalize("NFD", text)
+        output = []
+        for char in text:
+            cat = unicodedata.category(char)
+            if cat == "Mn":
+                continue
+            output.append(char)
+        return "".join(output)
+
+    def _run_split_on_punc(self, text):
+        """Splits punctuation on a piece of text."""
+        if text in self.never_split:
+            return [text]
+        chars = list(text)
+        i = 0
+        start_new_word = True
+        output = []
+        while i < len(chars):
+            char = chars[i]
+            if _is_punctuation(char):
+                output.append([char])
+                start_new_word = True
+            else:
+                if start_new_word:
+                    output.append([])
+                start_new_word = False
+                output[-1].append(char)
+            i += 1
+
+        return ["".join(x) for x in output]
+
+    def _tokenize_chinese_chars(self, text):
+        """Adds whitespace around any CJK character."""
+        output = []
+        for char in text:
+            cp = ord(char)
+            if self._is_chinese_char(cp):
+                output.append(" ")
+                output.append(char)
+                output.append(" ")
+            else:
+                output.append(char)
+        return "".join(output)
+
+    def _is_chinese_char(self, cp):
+        """Checks whether CP is the codepoint of a CJK character."""
+        # This defines a "chinese character" as anything in the CJK Unicode block:
+        #   https://en.wikipedia.org/wiki/CJK_Unified_Ideographs_(Unicode_block)
+        #
+        # Note that the CJK Unicode block is NOT all Japanese and Korean characters,
+        # despite its name. The modern Korean Hangul alphabet is a different block,
+        # as is Japanese Hiragana and Katakana. Those alphabets are used to write
+        # space-separated words, so they are not treated specially and handled
+        # like the all of the other languages.
+        if ((cp >= 0x4E00 and cp <= 0x9FFF) or  #
+            (cp >= 0x3400 and cp <= 0x4DBF) or  #
+            (cp >= 0x20000 and cp <= 0x2A6DF) or  #
+            (cp >= 0x2A700 and cp <= 0x2B73F) or  #
+            (cp >= 0x2B740 and cp <= 0x2B81F) or  #
+            (cp >= 0x2B820 and cp <= 0x2CEAF) or
+            (cp >= 0xF900 and cp <= 0xFAFF) or  #
+                (cp >= 0x2F800 and cp <= 0x2FA1F)):  #
+            return True
+
+        return False
+
+    def _clean_text(self, text):
+        """Performs invalid character removal and whitespace cleanup on text."""
+        output = []
+        for char in text:
+            cp = ord(char)
+            if cp == 0 or cp == 0xfffd or _is_control(char):
+                continue
+            if _is_whitespace(char):
+                output.append(" ")
+            else:
+                output.append(char)
+        return "".join(output)
+
+
+class WordpieceTokenizer(object):
+    """Runs WordPiece tokenziation."""
+
+    def __init__(self, vocab, unk_token="[UNK]", max_input_chars_per_word=200):
+        self.vocab = vocab
+        self.unk_token = unk_token
+        self.max_input_chars_per_word = max_input_chars_per_word
+
+    def tokenize(self, text):
+        """Tokenizes a piece of text into its word pieces.
+        This uses a greedy longest-match-first algorithm to perform tokenization
+        using the given vocabulary.
+        For example:
+          input = "unaffable"
+          output = ["un", "##aff", "##able"]
+        Args:
+          text: A single token or whitespace separated tokens. This should have
+            already been passed through `BasicTokenizer.
+        Returns:
+          A list of wordpiece tokens.
+        """
+
+        text = convert_to_unicode(text)
+
+        output_tokens = []
+        for token in whitespace_tokenize(text):
+            chars = list(token)
+            if len(chars) > self.max_input_chars_per_word:
+                output_tokens.append(self.unk_token)
+                continue
+
+            is_bad = False
+            start = 0
+            sub_tokens = []
+            while start < len(chars):
+                end = len(chars)
+                cur_substr = None
+                while start < end:
+                    substr = "".join(chars[start:end])
+                    if start > 0:
+                        substr = "##" + substr
+                    if substr in self.vocab:
+                        cur_substr = substr
+                        break
+                    end -= 1
+                if cur_substr is None:
+                    is_bad = True
+                    break
+                sub_tokens.append(cur_substr)
+                start = end
+
+            if is_bad:
+                output_tokens.append(self.unk_token)
+            else:
+                output_tokens.extend(sub_tokens)
+        return output_tokens
+
+
+def _is_whitespace(char):
+    """Checks whether `chars` is a whitespace character."""
+    # \t, \n, and \r are technically control characters but we treat them
+    # as whitespace since they are generally considered as such.
+    if char == " " or char == "\t" or char == "\n" or char == "\r":
+        return True
+    cat = unicodedata.category(char)
+    if cat == "Zs":
+        return True
+    return False
+
+
+def _is_control(char):
+    """Checks whether `chars` is a control character."""
+    # These are technically control characters but we count them as whitespace
+    # characters.
+    if char == "\t" or char == "\n" or char == "\r":
+        return False
+    cat = unicodedata.category(char)
+    if cat.startswith("C"):
+        return True
+    return False
+
+
+def _is_punctuation(char):
+    """Checks whether `chars` is a punctuation character."""
+    cp = ord(char)
+    # We treat all non-letter/number ASCII as punctuation.
+    # Characters such as "^", "$", and "`" are not in the Unicode
+    # Punctuation class but we treat them as punctuation anyways, for
+    # consistency.
+    if ((cp >= 33 and cp <= 47) or (cp >= 58 and cp <= 64) or
+            (cp >= 91 and cp <= 96) or (cp >= 123 and cp <= 126)):
+        return True
+    cat = unicodedata.category(char)
+    if cat.startswith("P"):
+        return True
+    return False
+
+
+# Just an example to use tokenizer
+if __name__ == '__main__':
+    tokenizer = FullTokenizer('vocab.txt')
+    text = 'We treat all non-letter/number ASCII as punctuation.'
+    tokens = tokenizer.tokenize(text)
+    ids = tokenizer.convert_tokens_to_ids(tokens)
+    print(tokens)
+    print(ids)
diff --git a/models_v2/pytorch/distilbert/inference/gpu/demo-data/vocab_downloader.py b/models_v2/pytorch/distilbert/inference/gpu/demo-data/vocab_downloader.py
new file mode 100644
index 000000000..2a84a51d4
--- /dev/null
+++ b/models_v2/pytorch/distilbert/inference/gpu/demo-data/vocab_downloader.py
@@ -0,0 +1,32 @@
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+import argparse
+import subprocess
+import tokenization
+
+
+def parse_args():
+    parser = argparse.ArgumentParser(description="vocab downloader")
+    parser.add_argument("--type", type=str, required=True,
+        choices=tokenization.PRETRAINED_VOCAB_ARCHIVE_MAP.keys())
+    args = parser.parse_args()
+    return args
+
+
+if __name__ == '__main__':
+    args = parse_args()
+    url = tokenization.PRETRAINED_VOCAB_ARCHIVE_MAP[args.type]
+    subprocess.run(['wget', url])
diff --git a/models_v2/pytorch/distilbert/inference/gpu/distilbert.py b/models_v2/pytorch/distilbert/inference/gpu/distilbert.py
new file mode 100644
index 000000000..1ca6db589
--- /dev/null
+++ b/models_v2/pytorch/distilbert/inference/gpu/distilbert.py
@@ -0,0 +1,35 @@
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Consts for distilbert inf."""
+
+ACC = {
+    "type": "total",
+    "pattern": r"'acc': (.*),",
+    "unit": "acc",
+}
+
+PERF = {
+    "type": "total",
+    "pattern": r"perf (\d+.\d+) sentences/s",
+    "inverse": False,
+    "multiply": False,
+    "use_batch_size": False,
+    "unit": "sent/s",
+}
+
+FUNCTIONAL = {
+    "pattern": r"perf (\d+.\d+) sentences/s",
+}
diff --git a/models_v2/pytorch/distilbert/inference/gpu/distiller.py b/models_v2/pytorch/distilbert/inference/gpu/distiller.py
new file mode 100644
index 000000000..c252af6ce
--- /dev/null
+++ b/models_v2/pytorch/distilbert/inference/gpu/distiller.py
@@ -0,0 +1,928 @@
+# coding=utf-8
+# Copyright 2019-present, the HuggingFace Inc. team and Facebook, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+""" The distiller to distil the student.
+    Adapted in part from Facebook, Inc XLM model (https://github.com/facebookresearch/XLM)
+"""
+import math
+import os
+import time
+import numpy as np
+import pickle
+import psutil
+import torch
+import random
+import h5py
+from torch import nn
+from torch.optim import AdamW
+from torch.utils.data import BatchSampler, DataLoader, RandomSampler, SequentialSampler, Dataset
+from torch.utils.data.distributed import DistributedSampler
+from tqdm import tqdm
+
+from grouped_batch_sampler import GroupedBatchSampler, create_lengths_groups
+from lm_seqs_dataset import LmSeqsDataset
+from transformers import get_linear_schedule_with_warmup
+from utils import logger, MAP_TORCH_DTYPE
+
+
+try:
+    from torch.utils.tensorboard import SummaryWriter
+except ImportError:
+    from tensorboardX import SummaryWriter
+
+
+def calc_mlm_acc(outputs, masked_lm_labels):
+    prediction_scores = outputs[0]
+    masked_lm_labels_flat = masked_lm_labels.view(-1)
+    mlm_labels = masked_lm_labels_flat[masked_lm_labels_flat != -100]
+    prediction_scores_flat = prediction_scores.view(-1, prediction_scores.shape[-1])
+    mlm_predictions_scores = prediction_scores_flat[masked_lm_labels_flat != -100]
+    mlm_predictions = mlm_predictions_scores.argmax(dim=-1)
+
+    num_masked = mlm_labels.numel()
+    mlm_acc = (mlm_predictions == mlm_labels).sum(dtype=torch.float) / num_masked
+    return mlm_acc, num_masked
+
+
+class pretraining_dataset_v1(Dataset):
+    def __init__(self, f, input_file, max_pred_length):
+        self.input_file = input_file
+        self.max_pred_length = max_pred_length
+        # f = h5py.File(input_file, "r")
+        keys = [
+            "input_ids",
+            "input_mask",
+            "segment_ids",
+            "masked_lm_positions",
+            "masked_lm_ids",
+            "next_sentence_labels",
+        ]
+        self.inputs = [np.asarray(f[key][:]) for key in keys]
+        print(f"Loaded {len(self.inputs[0]):d} samples from datafile: {input_file}")
+        f.close()
+
+    def __len__(self):
+        "Denotes the total number of samples"
+        return len(self.inputs[0])
+
+    def __getitem__(self, index):
+        [
+            input_ids,
+            input_mask,
+            segment_ids,
+            masked_lm_positions,
+            masked_lm_ids,
+            next_sentence_labels,
+        ] = [
+            torch.from_numpy(input[index].astype(np.int64))
+            if indice < 5
+            else torch.from_numpy(np.asarray(input[index].astype(np.int64)))
+            for indice, input in enumerate(self.inputs)
+        ]
+        masked_lm_labels = torch.zeros(input_ids.shape, dtype=torch.long) - 100
+        index = self.max_pred_length
+        masked_token_count = torch.count_nonzero(masked_lm_positions)
+        if masked_token_count != 0:
+            index = masked_token_count
+        masked_lm_labels[masked_lm_positions[:index]] = masked_lm_ids[:index]
+        # print(f"input_mask_len = {torch.count_nonzero(input_ids)}  index = {index}")
+
+        return [
+            input_ids,
+            segment_ids,
+            input_mask,
+            masked_lm_labels,
+            next_sentence_labels,
+        ]
+
+
+class pretraining_dataset_v2(Dataset):
+    def __init__(
+        self, f, input_file, max_pred_length, max_seq_length=512, packed_samples=False
+    ):
+        self.input_file = input_file
+        self.max_pred_length = max_pred_length
+        self.max_seq_length = max_seq_length
+        self.packed_samples = packed_samples
+
+        # f = h5py.File(input_file, "r")
+        if not self.packed_samples:
+            keys = [
+                "input_ids",
+                "segment_ids",
+                "masked_lm_positions",
+                "masked_lm_ids",
+                "next_sentence_labels",
+            ]
+        else:
+            keys = [
+                "input_ids",
+                "segment_ids",
+                "masked_lm_positions",
+                "masked_lm_ids",
+                "packed_input_len",
+                "packed_masked_lm_len",
+                "next_sentence_labels",
+            ]
+
+        self.inputs = [np.asarray(f[key][:]) for key in keys]
+        print(f"Loaded {len(self.inputs[0]):d} samples from datafile: {input_file}")
+        f.close()
+
+    def __len__(self):
+        "Denotes the total number of samples"
+        return len(self.inputs[0])
+
+    def __getitem__(self, index):
+        input_ids = np.zeros((self.max_seq_length)).astype(np.int64)
+        input_mask = np.zeros((self.max_seq_length)).astype(np.int64)
+        segment_ids = np.zeros((self.max_seq_length)).astype(np.int64)
+        next_sentence_labels = np.zeros((3)).astype(np.int64)
+        packed_input_len = np.zeros((3)).astype(np.int64)
+
+        if not self.packed_samples:
+            [
+                _input_ids,
+                _segment_ids,
+                _masked_lm_positions,
+                _masked_lm_ids,
+                _next_sentence_labels,
+            ] = [
+                input[index].astype(np.int64)
+                if indice < 4
+                else np.asarray(input[index].astype(np.int64))
+                for indice, input in enumerate(self.inputs)
+            ]
+        else:
+            [
+                _input_ids,
+                _segment_ids,
+                _masked_lm_positions,
+                _masked_lm_ids,
+                _packed_input_len,
+                _packed_masked_lm_len,
+                _next_sentence_labels,
+            ] = [
+                input[index].astype(np.int64)
+                for indice, input in enumerate(self.inputs)
+            ]
+
+        input_mask_len = _input_ids.shape[-1]
+        input_ids[:input_mask_len] = _input_ids
+        input_mask[:input_mask_len] = np.ones((1, input_mask_len)).astype(np.int64)
+        segment_ids[:input_mask_len] = _segment_ids
+        masked_lm_labels = np.zeros(input_ids.shape, dtype=np.int64) - 100
+        masked_lm_labels[_masked_lm_positions] = _masked_lm_ids
+
+        if not self.packed_samples:
+            next_sentence_labels = _next_sentence_labels
+
+            return [
+                torch.from_numpy(input_ids),
+                torch.from_numpy(segment_ids),
+                torch.from_numpy(input_mask),
+                torch.from_numpy(masked_lm_labels),
+                torch.from_numpy(next_sentence_labels),
+            ]
+        else:
+            packed_seqs = _packed_input_len.shape[-1]
+            next_sentence_labels[:packed_seqs] = _next_sentence_labels
+            packed_input_len[:packed_seqs] = _packed_input_len
+
+            return [
+                torch.from_numpy(input_ids),
+                torch.from_numpy(segment_ids),
+                torch.from_numpy(input_mask),
+                torch.from_numpy(masked_lm_labels),
+                torch.from_numpy(next_sentence_labels),
+                torch.from_numpy(packed_input_len),
+            ]
+
+
+def pretraining_dataset(
+    input_file, max_pred_length, max_seq_length=512, packed_samples=False
+):
+    print(input_file)
+    f = h5py.File(input_file, "r")
+    if "input_mask" not in f.keys():
+        return pretraining_dataset_v2(
+            f, input_file, max_pred_length, max_seq_length, packed_samples
+        )
+    else:
+        return pretraining_dataset_v1(f, input_file, max_pred_length)
+
+
+class Distiller:
+    def __init__(
+        self, params: dict, token_probs: torch.tensor, student: nn.Module, teacher: nn.Module
+    ):
+        logger.info("Initializing Distiller")
+        self.params = params
+        self.dump_path = params.dump_path
+        self.multi_gpu = params.multi_gpu
+        self.dtype = params.run_dtype
+        self.device = params.device
+        self.amp = params.amp
+        #self.fp16 = params.fp16
+
+        self.student = student
+        self.teacher = teacher
+
+        self.student_config = student.config
+        self.vocab_size = student.config.vocab_size
+        self.batch_size = params.batch_size
+        self.num_iteration = params.num_iteration
+
+        self.temperature = params.temperature
+        assert self.temperature > 0.0
+
+        self.alpha_ce = params.alpha_ce
+        self.alpha_mlm = params.alpha_mlm
+        self.alpha_clm = params.alpha_clm
+        self.alpha_mse = params.alpha_mse
+        self.alpha_cos = params.alpha_cos
+
+        self.mlm = params.mlm
+        if self.mlm:
+            logger.info("Using MLM loss for LM step.")
+            self.mlm_mask_prop = params.mlm_mask_prop
+            assert 0.0 <= self.mlm_mask_prop <= 1.0
+            assert params.word_mask + params.word_keep + params.word_rand == 1.0
+            self.pred_probs = torch.FloatTensor([params.word_mask, params.word_keep, params.word_rand])
+            #if self.device == "cuda":
+            #    self.pred_probs = self.pred_probs.to(f"cuda:{params.local_rank}") if params.n_gpu > 0 else self.pred_probs
+            #    self.token_probs = token_probs.to(f"cuda:{params.local_rank}") if params.n_gpu > 0 else token_probs
+            #elif self.device == "xpu":
+            #    self.pred_probs = self.pred_probs.to(self.device) if params.n_gpu > 0 else self.pred_probs
+            #    self.token_probs = token_probs.to(self.device) if params.n_gpu > 0 else token_probs
+            self.token_probs = token_probs
+            #if self.dtype == "FP16":
+            #    self.pred_probs = self.pred_probs.half()
+            #    self.token_probs = self.token_probs.half()
+            #elif self.dtype == "BF16":
+            #    self.pred_probs = self.pred_probs.bfloat16()
+            #    self.token_probs = self.token_probs.bfloat16()
+        else:
+            logger.info("Using CLM loss for LM step.")
+
+        self.epoch = 0
+        self.n_iter = 0
+        self.n_total_iter = 0
+        self.n_sequences_epoch = 0
+        self.total_loss_epoch = 0
+        self.last_loss = 0
+        self.last_loss_ce = 0
+        self.last_loss_mlm = 0
+        self.last_loss_clm = 0
+        if self.alpha_mse > 0.0:
+            self.last_loss_mse = 0
+        if self.alpha_cos > 0.0:
+            self.last_loss_cos = 0
+        self.last_log = 0
+
+        self.ce_loss_fct = nn.KLDivLoss(reduction="batchmean")
+        self.lm_loss_fct = nn.CrossEntropyLoss(ignore_index=-100)
+        if self.alpha_mse > 0.0:
+            self.mse_loss_fct = nn.MSELoss(reduction="sum")
+        if self.alpha_cos > 0.0:
+            self.cosine_loss_fct = nn.CosineEmbeddingLoss(reduction="mean")
+
+        logger.info("--- Initializing model optimizer")
+        assert params.gradient_accumulation_steps >= 1
+
+        no_decay = ["bias", "LayerNorm.weight"]
+        optimizer_grouped_parameters = [
+            {
+                "params": [
+                    p for n, p in student.named_parameters() if not any(nd in n for nd in no_decay) and p.requires_grad
+                ],
+                "weight_decay": params.weight_decay,
+            },
+            {
+                "params": [
+                    p for n, p in student.named_parameters() if any(nd in n for nd in no_decay) and p.requires_grad
+                ],
+                "weight_decay": 0.0,
+            },
+        ]
+        logger.info(
+            "------ Number of trainable parameters (student): %i"
+            % sum([p.numel() for p in self.student.parameters() if p.requires_grad])
+        )
+        logger.info("------ Number of parameters (student): %i" % sum([p.numel() for p in self.student.parameters()]))
+        self.optimizer = AdamW(
+            optimizer_grouped_parameters, lr=params.learning_rate, eps=params.adam_epsilon, betas=(0.9, 0.98)
+        )
+
+        if params.run_mode == "train":
+            with open(params.data_file, "rb") as fp:
+                data = pickle.load(fp)
+            dataset = LmSeqsDataset(params=params, data=data)
+
+            if params.n_gpu <= 1:
+                sampler = RandomSampler(dataset)
+            else:
+                sampler = DistributedSampler(dataset)
+
+            if params.group_by_size:
+                groups = create_lengths_groups(lengths=dataset.lengths, k=params.max_model_input_size)
+                sampler = GroupedBatchSampler(sampler=sampler, group_ids=groups, batch_size=params.batch_size)
+            else:
+                sampler = BatchSampler(sampler=sampler, batch_size=params.batch_size, drop_last=False)
+
+            self.dataloader = DataLoader(dataset=dataset, batch_sampler=sampler, collate_fn=dataset.batch_sequences)
+
+            self.num_steps_epoch = len(self.dataloader)
+            num_train_optimization_steps = (
+                int(self.num_steps_epoch / params.gradient_accumulation_steps * params.n_epoch) + 1
+            )
+            warmup_steps = math.ceil(num_train_optimization_steps * params.warmup_prop)
+            self.scheduler = get_linear_schedule_with_warmup(
+                self.optimizer, num_warmup_steps=warmup_steps, num_training_steps=num_train_optimization_steps
+            )
+
+        #if self.fp16:
+        if self.dtype == "FP16" and self.device == "cuda":
+            try:
+                from apex import amp
+            except ImportError:
+                raise ImportError("Please install apex from https://www.github.com/nvidia/apex to use fp16 training.")
+            logger.info(f"Using fp16 training: {self.params.fp16_opt_level} level")
+            self.student, self.optimizer = amp.initialize(
+                self.student, self.optimizer, opt_level=self.params.fp16_opt_level
+            )
+            self.teacher = self.teacher.half()
+
+        if self.multi_gpu:
+            if self.dtype == "FP16" and self.device == "cuda":
+                from apex.parallel import DistributedDataParallel
+
+                logger.info("Using apex.parallel.DistributedDataParallel for distributed training.")
+                self.student = DistributedDataParallel(self.student)
+            else:
+                from torch.nn.parallel import DistributedDataParallel
+
+                logger.info("Using nn.parallel.DistributedDataParallel for distributed training.")
+                self.student = DistributedDataParallel(
+                    self.student,
+                    device_ids=[params.local_rank],
+                    output_device=params.local_rank,
+                    find_unused_parameters=True,
+                )
+
+        self.is_master = params.is_master
+        if self.is_master:
+            logger.info("--- Initializing Tensorboard")
+            self.tensorboard = SummaryWriter(log_dir=os.path.join(self.dump_path, "log", "train"))
+            self.tensorboard.add_text(tag="config/training", text_string=str(self.params), global_step=0)
+            self.tensorboard.add_text(tag="config/student", text_string=str(self.student_config), global_step=0)
+
+    def prepare_batch_mlm(self, batch):
+        """
+        Prepare the batch: from the token_ids and the lengths, compute the attention mask and the masked label for MLM.
+
+        Input:
+        ------
+            batch: `Tuple`
+                token_ids: `torch.tensor(bs, seq_length)` - The token ids for each of the sequence. It is padded.
+                lengths: `torch.tensor(bs)` - The lengths of each of the sequences in the batch.
+
+        Output:
+        -------
+            token_ids: `torch.tensor(bs, seq_length)` - The token ids after the modifications for MLM.
+            attn_mask: `torch.tensor(bs, seq_length)` - The attention mask for the self-attention.
+            mlm_labels: `torch.tensor(bs, seq_length)` - The masked language modeling labels. There is a -100 where there is nothing to predict.
+        """
+        token_ids, lengths = batch
+        token_ids, lengths = self.round_batch(x=token_ids, lengths=lengths)
+        assert token_ids.size(0) == lengths.size(0)
+
+        attn_mask = torch.arange(token_ids.size(1), dtype=torch.long, device=lengths.device) < lengths[:, None]
+
+        bs, max_seq_len = token_ids.size()
+        mlm_labels = token_ids.new(token_ids.size()).copy_(token_ids)
+
+        x_prob = self.token_probs[token_ids.flatten()]
+        n_tgt = math.ceil(self.mlm_mask_prop * lengths.sum().item())
+        tgt_ids = torch.multinomial(x_prob / x_prob.sum(), n_tgt, replacement=False)
+        pred_mask = torch.zeros(
+            bs * max_seq_len, dtype=torch.bool, device=token_ids.device
+        )  # previously `dtype=torch.uint8`, cf pytorch 1.2.0 compatibility
+        pred_mask[tgt_ids] = 1
+        pred_mask = pred_mask.view(bs, max_seq_len)
+
+        pred_mask[token_ids == self.params.special_tok_ids["pad_token"]] = 0
+
+        # mask a number of words == 0 [8] (faster with fp16)
+        if self.dtype == "FP16":
+            n1 = pred_mask.sum().item()
+            if n1 > 8:
+                pred_mask = pred_mask.view(-1)
+                n2 = max(n1 % 8, 8 * (n1 // 8))
+                if n2 != n1:
+                    pred_mask[torch.nonzero(pred_mask).view(-1)[: n1 - n2]] = 0
+                pred_mask = pred_mask.view(bs, max_seq_len)
+                assert pred_mask.sum().item() % 8 == 0, pred_mask.sum().item()
+
+        _token_ids_real = token_ids[pred_mask]
+        _token_ids_rand = _token_ids_real.clone().random_(self.vocab_size)
+        _token_ids_mask = _token_ids_real.clone().fill_(self.params.special_tok_ids["mask_token"])
+        probs = torch.multinomial(self.pred_probs, len(_token_ids_real), replacement=True)
+        _token_ids = (
+            _token_ids_mask * (probs == 0).long()
+            + _token_ids_real * (probs == 1).long()
+            + _token_ids_rand * (probs == 2).long()
+        )
+        token_ids = token_ids.masked_scatter(pred_mask, _token_ids)
+        #if self.params.run_mode == "train":
+        mlm_labels[~pred_mask] = -100  # previously `mlm_labels[1-pred_mask] = -1`, cf pytorch 1.2.0 compatibility
+
+        # sanity checks
+        assert 0 <= token_ids.min() <= token_ids.max() < self.vocab_size
+
+        return token_ids, attn_mask, mlm_labels
+
+    def prepare_batch_clm(self, batch):
+        """
+        Prepare the batch: from the token_ids and the lengths, compute the attention mask and the labels for CLM.
+
+        Input:
+        ------
+            batch: `Tuple`
+                token_ids: `torch.tensor(bs, seq_length)` - The token ids for each of the sequence. It is padded.
+                lengths: `torch.tensor(bs)` - The lengths of each of the sequences in the batch.
+
+        Output:
+        -------
+            token_ids: `torch.tensor(bs, seq_length)` - The token ids after the modifications for MLM.
+            attn_mask: `torch.tensor(bs, seq_length)` - The attention mask for the self-attention.
+            clm_labels: `torch.tensor(bs, seq_length)` - The causal language modeling labels. There is a -100 where there is nothing to predict.
+        """
+        token_ids, lengths = batch
+        token_ids, lengths = self.round_batch(x=token_ids, lengths=lengths)
+        assert token_ids.size(0) == lengths.size(0)
+
+        attn_mask = torch.arange(token_ids.size(1), dtype=torch.long, device=lengths.device) < lengths[:, None]
+        clm_labels = token_ids.new(token_ids.size()).copy_(token_ids)
+        #if self.params.run_mode == "train":
+        clm_labels[~attn_mask] = -100  # previously `clm_labels[1-attn_mask] = -1`, cf pytorch 1.2.0 compatibility
+
+        # sanity checks
+        assert 0 <= token_ids.min() <= token_ids.max() < self.vocab_size
+
+        return token_ids, attn_mask, clm_labels
+
+    def round_batch(self, x: torch.tensor, lengths: torch.tensor):
+        """
+        For float16 only.
+        Sub-sample sentences in a batch, and add padding, so that each dimension is a multiple of 8.
+
+        Input:
+        ------
+            x: `torch.tensor(bs, seq_length)` - The token ids.
+            lengths: `torch.tensor(bs, seq_length)` - The lengths of each of the sequence in the batch.
+
+        Output:
+        -------
+            x:  `torch.tensor(new_bs, new_seq_length)` - The updated token ids.
+            lengths: `torch.tensor(new_bs, new_seq_length)` - The updated lengths.
+        """
+        if not self.dtype == "FP16" or len(lengths) < 8:
+            return x, lengths
+
+        # number of sentences == 0 [8]
+        bs1 = len(lengths)
+        bs2 = 8 * (bs1 // 8)
+        assert bs2 > 0 and bs2 % 8 == 0
+        if bs1 != bs2:
+            idx = torch.randperm(bs1)[:bs2]
+            lengths = lengths[idx]
+            slen = lengths.max().item()
+            x = x[idx, :slen]
+        else:
+            idx = None
+
+        # sequence length == 0 [8]
+        ml1 = x.size(1)
+        if ml1 % 8 != 0:
+            pad = 8 - (ml1 % 8)
+            ml2 = ml1 + pad
+            if self.mlm:
+                pad_id = self.params.special_tok_ids["pad_token"]
+            else:
+                pad_id = self.params.special_tok_ids["unk_token"]
+            padding_tensor = torch.zeros(bs2, pad, dtype=torch.long, device=x.device).fill_(pad_id)
+            x = torch.cat([x, padding_tensor], 1)
+            assert x.size() == (bs2, ml2)
+
+        assert x.size(0) % 8 == 0
+        assert x.size(1) % 8 == 0
+        return x, lengths
+
+    def train(self):
+        """
+        The real training loop.
+        """
+        if self.is_master:
+            logger.info("Starting training")
+        self.last_log = time.time()
+        self.student.train()
+        self.teacher.eval()
+
+        for _ in range(self.params.n_epoch):
+            if self.is_master:
+                logger.info(f"--- Starting epoch {self.epoch}/{self.params.n_epoch-1}")
+            if self.multi_gpu:
+                torch.distributed.barrier()
+
+            iter_bar = tqdm(self.dataloader, desc="-Iter", disable=self.params.local_rank not in [-1, 0])
+            for batch in iter_bar:
+                #if self.params.n_gpu > 0 and self.device == "cuda":
+                #    batch = tuple(t.to(f"cuda:{self.params.local_rank}") for t in batch)
+                #elif self.params.n_gpu > 0 and self.device == "xpu":
+                #    batch = tuple(t.to(self.device) for t in batch)
+
+                if self.mlm:
+                    token_ids, attn_mask, lm_labels = self.prepare_batch_mlm(batch=batch)
+                else:
+                    token_ids, attn_mask, lm_labels = self.prepare_batch_clm(batch=batch)
+
+                self.step(input_ids_=token_ids, attention_mask_=attn_mask, lm_labels_=lm_labels)
+
+                iter_bar.update()
+                iter_bar.set_postfix(
+                    {"Last_loss": f"{self.last_loss:.2f}", "Avg_cum_loss": f"{self.total_loss_epoch/self.n_iter:.2f}"}
+                )
+            iter_bar.close()
+
+            if self.is_master:
+                logger.info(f"--- Ending epoch {self.epoch}/{self.params.n_epoch-1}")
+            self.end_epoch()
+
+        if self.is_master:
+            logger.info("Save very last checkpoint as `pytorch_model.bin`.")
+            self.save_checkpoint(checkpoint_name="pytorch_model.bin")
+            logger.info("Training is finished")
+
+    def step(self, input_ids_: torch.Tensor, attention_mask_: torch.Tensor, lm_labels_: torch.Tensor):
+        """
+        One optimization step: forward of student AND teacher, backward on the loss (for gradient accumulation),
+        and possibly a parameter update (depending on the gradient accumulation).
+
+        Input:
+        ------
+        input_ids: `torch.tensor(bs, seq_length)` - The token ids.
+        attention_mask: `torch.tensor(bs, seq_length)` - The attention mask for self attention.
+        lm_labels: `torch.tensor(bs, seq_length)` - The language modeling labels (mlm labels for MLM and clm labels for CLM).
+        """
+        def run_model(input_ids: torch.Tensor, attention_mask: torch.Tensor, lm_labels: torch.Tensor):
+            if self.params.n_gpu > 0 and self.device == "cuda":
+                #batch = tuple(t.to(f"cuda:{self.params.local_rank}") for t in batch)
+                input_ids = input_ids.to(f"cuda:{self.params.local_rank}")
+                attention_mask = attention_mask.to(f"cuda:{self.params.local_rank}")
+                lm_labels = lm_labels.to(f"cuda:{self.params.local_rank}")
+            elif self.params.n_gpu > 0 and self.device == "xpu":
+                #batch = tuple(t.to(self.device) for t in batch)
+                input_ids = input_ids.to(self.device)
+                attention_mask = attention_mask.to(self.device)
+                lm_labels = lm_labels.to(self.device)
+
+            if self.mlm:
+                student_outputs = self.student(
+                    input_ids=input_ids, attention_mask=attention_mask
+                )  # (bs, seq_length, voc_size)
+                with torch.no_grad():
+                    teacher_outputs = self.teacher(
+                        input_ids=input_ids, attention_mask=attention_mask
+                    )  # (bs, seq_length, voc_size)
+            else:
+                student_outputs = self.student(input_ids=input_ids, attention_mask=None)  # (bs, seq_length, voc_size)
+                with torch.no_grad():
+                    teacher_outputs = self.teacher(input_ids=input_ids, attention_mask=None)  # (bs, seq_length, voc_size)
+            s_logits, s_hidden_states = student_outputs["logits"], student_outputs["hidden_states"]
+            t_logits, t_hidden_states = teacher_outputs["logits"], teacher_outputs["hidden_states"]
+            assert s_logits.size() == t_logits.size()
+
+            # https://github.com/peterliht/knowledge-distillation-pytorch/blob/master/model/net.py#L100
+            # https://github.com/peterliht/knowledge-distillation-pytorch/issues/2
+            if self.params.restrict_ce_to_mask:
+                mask = (lm_labels > -1).unsqueeze(-1).expand_as(s_logits)  # (bs, seq_length, voc_size)
+            else:
+                mask = attention_mask.unsqueeze(-1).expand_as(s_logits)  # (bs, seq_length, voc_size)
+            s_logits_slct = torch.masked_select(s_logits, mask)  # (bs * seq_length * voc_size) modulo the 1s in mask
+            s_logits_slct = s_logits_slct.view(-1, s_logits.size(-1))  # (bs * seq_length, voc_size) modulo the 1s in mask
+            t_logits_slct = torch.masked_select(t_logits, mask)  # (bs * seq_length * voc_size) modulo the 1s in mask
+            t_logits_slct = t_logits_slct.view(-1, s_logits.size(-1))  # (bs * seq_length, voc_size) modulo the 1s in mask
+            assert t_logits_slct.size() == s_logits_slct.size()
+
+            loss_ce = (
+                self.ce_loss_fct(
+                    nn.functional.log_softmax(s_logits_slct / self.temperature, dim=-1),
+                    nn.functional.softmax(t_logits_slct / self.temperature, dim=-1),
+                )
+                * (self.temperature) ** 2
+            )
+            loss = self.alpha_ce * loss_ce
+
+            if self.alpha_mlm > 0.0:
+                loss_mlm = self.lm_loss_fct(s_logits.view(-1, s_logits.size(-1)), lm_labels.view(-1))
+                loss += self.alpha_mlm * loss_mlm
+            if self.alpha_clm > 0.0:
+                shift_logits = s_logits[..., :-1, :].contiguous()
+                shift_labels = lm_labels[..., 1:].contiguous()
+                loss_clm = self.lm_loss_fct(shift_logits.view(-1, shift_logits.size(-1)), shift_labels.view(-1))
+                loss += self.alpha_clm * loss_clm
+
+            if self.alpha_mse > 0.0:
+                loss_mse = self.mse_loss_fct(s_logits_slct, t_logits_slct) / s_logits_slct.size(
+                    0
+                )  # Reproducing batchmean reduction
+                loss += self.alpha_mse * loss_mse
+            if self.alpha_cos > 0.0:
+                s_hidden_states = s_hidden_states[-1]  # (bs, seq_length, dim)
+                t_hidden_states = t_hidden_states[-1]  # (bs, seq_length, dim)
+                mask = attention_mask.unsqueeze(-1).expand_as(s_hidden_states)  # (bs, seq_length, dim)
+                assert s_hidden_states.size() == t_hidden_states.size()
+                dim = s_hidden_states.size(-1)
+
+                s_hidden_states_slct = torch.masked_select(s_hidden_states, mask)  # (bs * seq_length * dim)
+                s_hidden_states_slct = s_hidden_states_slct.view(-1, dim)  # (bs * seq_length, dim)
+                t_hidden_states_slct = torch.masked_select(t_hidden_states, mask)  # (bs * seq_length * dim)
+                t_hidden_states_slct = t_hidden_states_slct.view(-1, dim)  # (bs * seq_length, dim)
+
+                target = s_hidden_states_slct.new(s_hidden_states_slct.size(0)).fill_(1)  # (bs * seq_length,)
+                loss_cos = self.cosine_loss_fct(s_hidden_states_slct, t_hidden_states_slct, target)
+                loss += self.alpha_cos * loss_cos
+            self.total_loss_epoch += loss.item()
+            self.last_loss = loss.item()
+            self.last_loss_ce = loss_ce.item()
+            if self.alpha_mlm > 0.0:
+                self.last_loss_mlm = loss_mlm.item()
+            if self.alpha_clm > 0.0:
+                self.last_loss_clm = loss_clm.item()
+            if self.alpha_mse > 0.0:
+                self.last_loss_mse = loss_mse.item()
+            if self.alpha_cos > 0.0:
+                self.last_loss_cos = loss_cos.item()
+            return loss
+
+        if self.device == "xpu":
+            #with torch.xpu.amp.autocast(enabled=self.amp, dtype=MAP_TORCH_DTYPE[self.dtype]):
+            loss = run_model(input_ids_, attention_mask_, lm_labels_)
+        else:
+            loss = run_model(input_ids_, attention_mask_, lm_labels_)
+
+        self.optimize(loss)
+
+        self.n_sequences_epoch += input_ids_.size(0)
+
+    def optimize(self, loss):
+        """
+        Normalization on the loss (gradient accumulation or distributed training), followed by
+        backward pass on the loss, possibly followed by a parameter update (depending on the gradient accumulation).
+        Also update the metrics for tensorboard.
+        """
+        # Check for NaN
+        if (loss != loss).data.any():
+            logger.error("NaN detected")
+            exit()
+
+        if self.multi_gpu:
+            loss = loss.mean()
+        if self.params.gradient_accumulation_steps > 1:
+            loss = loss / self.params.gradient_accumulation_steps
+
+        if self.dtype == "FP16" and self.device == "cuda":
+            from apex import amp
+
+            with amp.scale_loss(loss, self.optimizer) as scaled_loss:
+                scaled_loss.backward()
+        else:
+            loss.backward()
+
+        self.iter()
+        if self.n_iter % self.params.gradient_accumulation_steps == 0:
+            if self.dtype == "FP16" and self.device == "cuda":
+                nn.utils.clip_grad_norm_(amp.master_params(self.optimizer), self.params.max_grad_norm)
+            else:
+                nn.utils.clip_grad_norm_(self.student.parameters(), self.params.max_grad_norm)
+            self.optimizer.step()
+            self.optimizer.zero_grad()
+            self.scheduler.step()
+
+    def eval(self):
+        files = [
+            os.path.join(self.params.data_file, f)
+            for f in os.listdir(self.params.data_file)
+            if os.path.isfile(os.path.join(self.params.data_file, f)) and "part" in f
+        ]
+        files.sort()
+        num_files = len(files)
+
+        eval_loss = 0
+        nb_eval_steps = 0
+        iter_acc = []
+        iter_num_mask = []
+        time_collect = []
+        self.student.eval()
+
+        do_profiling = os.environ.get("PROFILE", "OFF").upper() in ["1", "Y", "ON", "YES", "TRUE"]
+        if (do_profiling):
+            if self.device == "xpu":
+                sort_key = "self_xpu_time_total"
+            elif self.device == "cuda":
+                sort_key = "self_cuda_time_total"
+            else:
+                sort_key = "self_cpu_time_total"
+
+        if self.device== "xpu":
+            self.student = torch.xpu.optimize(model=self.student, dtype=MAP_TORCH_DTYPE[self.dtype])
+        if self.is_master:
+            logger.info(f"--- Starting inference")
+        with torch.no_grad():
+            for file_iter in range(len(files)):
+                if nb_eval_steps == self.num_iteration:
+                    break
+                data_file = files[file_iter]
+                data = pretraining_dataset(data_file, self.params.max_predictions_per_seq)
+                sampler = RandomSampler(data)
+                dataloader = DataLoader(data, sampler=sampler, batch_size=self.params.batch_size)
+                iter_bar = tqdm(dataloader, desc="-Iter", disable=self.params.local_rank not in [-1, 0])
+                for batch in iter_bar:
+                    if nb_eval_steps == self.num_iteration:
+                        break
+
+                    (token_ids, segment_ids, attn_mask, lm_labels, next_sentence_labels,) = batch
+
+                    with torch.autograd.profiler_legacy.profile(do_profiling, use_xpu=True) if self.device== "xpu" else torch.profiler.profile(activities=[torch.profiler.ProfilerActivity.CUDA]) as prof:
+
+                        if self.device == "xpu":
+                            try:
+                                import memory_check
+                                memory_check.display_mem("xpu:0")
+                            except:
+                                pass
+
+                        start_time = time.time()
+                        if self.params.n_gpu > 0 and self.device == "cuda":
+                            #batch = tuple(t.to(f"cuda:{self.params.local_rank}") for t in batch)
+                            token_ids = token_ids.to(f"cuda:{self.params.local_rank}")
+                            attn_mask = attn_mask.to(f"cuda:{self.params.local_rank}")
+                            lm_labels = lm_labels.to(f"cuda:{self.params.local_rank}")
+                        elif self.params.n_gpu > 0 and self.device == "xpu":
+                            #batch = tuple(t.to(self.device) for t in batch)
+                            token_ids = token_ids.to(self.device)
+                            attn_mask = attn_mask.to(self.device)
+                            lm_labels = lm_labels.to(self.device)
+                        if nb_eval_steps == 0:
+                            if self.params.jit and self.device== "xpu":
+                                with torch.xpu.amp.autocast(enabled=self.amp, dtype=MAP_TORCH_DTYPE[self.dtype]):
+                                    self.student = torch.jit.trace(self.student, (token_ids, attn_mask), strict = False, check_trace=False)
+                                    self.student = torch.jit.freeze(self.student)
+                        student_outputs = self.student(
+                            input_ids=token_ids, attention_mask=attn_mask
+                        )  # (bs, seq_length, voc_size)
+                        torch.xpu.synchronize()
+                    if nb_eval_steps >= 10:
+                        time_collect.append(time.time() - start_time)
+                    loss = self.lm_loss_fct(student_outputs[0].view(-1, student_outputs[0].size(-1)), lm_labels.view(-1))
+                    lm_acc, num_masked = calc_mlm_acc(student_outputs, lm_labels)
+                    lm_acc = lm_acc.item()
+                    iter_acc.append(lm_acc)
+                    iter_num_mask.append(num_masked)
+                    if do_profiling and nb_eval_steps == 150:
+                        torch.save(prof.key_averages().table(sort_by=sort_key),
+                                   './distilbert_profiling.pt')
+                        prof.export_chrome_trace('./distilbert_profiling.json')
+                        if self.device == "xpu":
+                            torch.save(prof.table(sort_by="id", row_limit=100000), './distilbert_profiling_detailed.pt')
+                    eval_loss += loss.item()
+                    nb_eval_steps += 1
+        if self.is_master:
+            logger.info(f"--- Ending inference")
+
+        eval_loss = eval_loss / nb_eval_steps
+        acc = sum([a * b for a, b in zip(iter_acc, iter_num_mask)])/sum(iter_num_mask)
+        result = {}
+        result['acc'] = acc
+        result['eval_loss'] = eval_loss
+        logger.info("Results: {}".format(result))
+        total_time = sum(time_collect[:-10])
+        if len(time_collect) >= 10:
+            logger.info("The total_time {} s, and perf {} sentences/s for inference".format(total_time,  (nb_eval_steps - 20) * self.batch_size/ (total_time)))
+        else:
+            logger.error("The num_iteration must greater then 20")
+
+    def iter(self):
+        """
+        Update global counts, write to tensorboard and save checkpoint.
+        """
+        self.n_iter += 1
+        self.n_total_iter += 1
+
+        if self.n_total_iter % self.params.log_interval == 0:
+            self.log_tensorboard()
+            self.last_log = time.time()
+        if self.n_total_iter % self.params.checkpoint_interval == 0:
+            self.save_checkpoint()
+
+    def log_tensorboard(self):
+        """
+        Log into tensorboard. Only by the master process.
+        """
+        if not self.is_master:
+            return
+
+        for param_name, param in self.student.named_parameters():
+            self.tensorboard.add_scalar(
+                tag="parameter_mean/" + param_name, scalar_value=param.data.mean(), global_step=self.n_total_iter
+            )
+            self.tensorboard.add_scalar(
+                tag="parameter_std/" + param_name, scalar_value=param.data.std(), global_step=self.n_total_iter
+            )
+            if param.grad is None:
+                continue
+            self.tensorboard.add_scalar(
+                tag="grad_mean/" + param_name, scalar_value=param.grad.data.mean(), global_step=self.n_total_iter
+            )
+            self.tensorboard.add_scalar(
+                tag="grad_std/" + param_name, scalar_value=param.grad.data.std(), global_step=self.n_total_iter
+            )
+
+        self.tensorboard.add_scalar(
+            tag="losses/cum_avg_loss_epoch",
+            scalar_value=self.total_loss_epoch / self.n_iter,
+            global_step=self.n_total_iter,
+        )
+        self.tensorboard.add_scalar(tag="losses/loss", scalar_value=self.last_loss, global_step=self.n_total_iter)
+        self.tensorboard.add_scalar(
+            tag="losses/loss_ce", scalar_value=self.last_loss_ce, global_step=self.n_total_iter
+        )
+        if self.alpha_mlm > 0.0:
+            self.tensorboard.add_scalar(
+                tag="losses/loss_mlm", scalar_value=self.last_loss_mlm, global_step=self.n_total_iter
+            )
+        if self.alpha_clm > 0.0:
+            self.tensorboard.add_scalar(
+                tag="losses/loss_clm", scalar_value=self.last_loss_clm, global_step=self.n_total_iter
+            )
+        if self.alpha_mse > 0.0:
+            self.tensorboard.add_scalar(
+                tag="losses/loss_mse", scalar_value=self.last_loss_mse, global_step=self.n_total_iter
+            )
+        if self.alpha_cos > 0.0:
+            self.tensorboard.add_scalar(
+                tag="losses/loss_cos", scalar_value=self.last_loss_cos, global_step=self.n_total_iter
+            )
+        self.tensorboard.add_scalar(
+            tag="learning_rate/lr", scalar_value=self.scheduler.get_lr()[0], global_step=self.n_total_iter
+        )
+
+        self.tensorboard.add_scalar(
+            tag="global/memory_usage",
+            scalar_value=psutil.virtual_memory()._asdict()["used"] / 1_000_000,
+            global_step=self.n_total_iter,
+        )
+        self.tensorboard.add_scalar(
+            tag="global/speed", scalar_value=time.time() - self.last_log, global_step=self.n_total_iter
+        )
+
+    def end_epoch(self):
+        """
+        Finally arrived at the end of epoch (full pass on dataset).
+        Do some tensorboard logging and checkpoint saving.
+        """
+        logger.info(f"{self.n_sequences_epoch} sequences have been trained during this epoch.")
+
+        if self.is_master:
+            self.save_checkpoint(checkpoint_name=f"model_epoch_{self.epoch}.pth")
+            self.tensorboard.add_scalar(
+                tag="epoch/loss", scalar_value=self.total_loss_epoch / self.n_iter, global_step=self.epoch
+            )
+
+        self.epoch += 1
+        self.n_sequences_epoch = 0
+        self.n_iter = 0
+        self.total_loss_epoch = 0
+
+    def save_checkpoint(self, checkpoint_name: str = "checkpoint.pth"):
+        """
+        Save the current state. Only by the master process.
+        """
+        if not self.is_master:
+            return
+        mdl_to_save = self.student.module if hasattr(self.student, "module") else self.student
+        mdl_to_save.config.save_pretrained(self.dump_path)
+        state_dict = mdl_to_save.state_dict()
+        torch.save(state_dict, os.path.join(self.dump_path, checkpoint_name))
diff --git a/models_v2/pytorch/distilbert/inference/gpu/grouped_batch_sampler.py b/models_v2/pytorch/distilbert/inference/gpu/grouped_batch_sampler.py
new file mode 100644
index 000000000..83addc371
--- /dev/null
+++ b/models_v2/pytorch/distilbert/inference/gpu/grouped_batch_sampler.py
@@ -0,0 +1,108 @@
+# coding=utf-8
+# Copyright 2019-present, the HuggingFace Inc. team and Facebook, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+""" Adapted from PyTorch Vision (https://github.com/pytorch/vision/blob/master/references/detection/group_by_aspect_ratio.py)
+"""
+import bisect
+import copy
+from collections import defaultdict
+
+import numpy as np
+from torch.utils.data import BatchSampler, Sampler
+
+from utils import logger
+
+
+def _quantize(x, bins):
+    bins = copy.deepcopy(bins)
+    bins = sorted(bins)
+    quantized = list(map(lambda y: bisect.bisect_right(bins, y), x))
+    return quantized
+
+
+def create_lengths_groups(lengths, k=0):
+    bins = np.arange(start=3, stop=k, step=4).tolist() if k > 0 else [10]
+    groups = _quantize(lengths, bins)
+    # count number of elements per group
+    counts = np.unique(groups, return_counts=True)[1]
+    fbins = [0] + bins + [np.inf]
+    logger.info("Using {} as bins for aspect lengths quantization".format(fbins))
+    logger.info("Count of instances per bin: {}".format(counts))
+    return groups
+
+
+class GroupedBatchSampler(BatchSampler):
+    """
+    Wraps another sampler to yield a mini-batch of indices.
+    It enforces that the batch only contain elements from the same group.
+    It also tries to provide mini-batches which follows an ordering which is
+    as close as possible to the ordering from the original sampler.
+    Arguments:
+        sampler (Sampler): Base sampler.
+        group_ids (list[int]): If the sampler produces indices in range [0, N),
+            `group_ids` must be a list of `N` ints which contains the group id of each sample.
+            The group ids must be a continuous set of integers starting from
+            0, i.e. they must be in the range [0, num_groups).
+        batch_size (int): Size of mini-batch.
+    """
+
+    def __init__(self, sampler, group_ids, batch_size):
+        if not isinstance(sampler, Sampler):
+            raise ValueError(
+                "sampler should be an instance of torch.utils.data.Sampler, but got sampler={}".format(sampler)
+            )
+        self.sampler = sampler
+        self.group_ids = group_ids
+        self.batch_size = batch_size
+
+    def __iter__(self):
+        buffer_per_group = defaultdict(list)
+        samples_per_group = defaultdict(list)
+
+        num_batches = 0
+        for idx in self.sampler:
+            group_id = self.group_ids[idx]
+            buffer_per_group[group_id].append(idx)
+            samples_per_group[group_id].append(idx)
+            if len(buffer_per_group[group_id]) == self.batch_size:
+                yield buffer_per_group[group_id]  # TODO
+                num_batches += 1
+                del buffer_per_group[group_id]
+            assert len(buffer_per_group[group_id]) < self.batch_size
+
+        # now we have run out of elements that satisfy
+        # the group criteria, let's return the remaining
+        # elements so that the size of the sampler is
+        # deterministic
+        expected_num_batches = len(self)
+        num_remaining = expected_num_batches - num_batches
+        if num_remaining > 0:
+            # for the remaining batches, group the batches by similar lengths
+            batch_idx = []
+            for group_id, idxs in sorted(buffer_per_group.items(), key=lambda x: x[0]):
+                batch_idx.extend(idxs)
+                if len(batch_idx) >= self.batch_size:
+                    yield batch_idx[: self.batch_size]
+                    batch_idx = batch_idx[self.batch_size :]
+                    num_remaining -= 1
+            if len(batch_idx) > 0:
+                yield batch_idx
+                num_remaining -= 1
+        assert num_remaining == 0
+
+    def __len__(self):
+        """
+        Return the number of mini-batches rather than the number of samples.
+        """
+        return (len(self.sampler) + self.batch_size - 1) // self.batch_size
diff --git a/models_v2/pytorch/distilbert/inference/gpu/lm_seqs_dataset.py b/models_v2/pytorch/distilbert/inference/gpu/lm_seqs_dataset.py
new file mode 100644
index 000000000..8c00fb22a
--- /dev/null
+++ b/models_v2/pytorch/distilbert/inference/gpu/lm_seqs_dataset.py
@@ -0,0 +1,168 @@
+# coding=utf-8
+# Copyright 2019-present, the HuggingFace Inc. team and Facebook, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+""" Dataset to distilled models
+    adapted in part from Facebook, Inc XLM model (https://github.com/facebookresearch/XLM)
+"""
+import numpy as np
+import torch
+from torch.utils.data import Dataset
+
+from utils import logger
+
+
+class LmSeqsDataset(Dataset):
+    """Custom Dataset wrapping language modeling sequences.
+
+    Each sample will be retrieved by indexing the list of token_ids and their corresponding lengths.
+
+    Input:
+    ------
+        params: `NameSpace` parameters
+        data: `List[np.array[int]]
+    """
+
+    def __init__(self, params, data):
+        self.params = params
+
+        self.token_ids = np.array(data)
+        self.lengths = np.array([len(t) for t in data])
+        #print(self.lengths)
+        #exit()
+
+        self.check()
+        self.remove_long_sequences()
+        self.remove_empty_sequences()
+        self.remove_unknown_sequences()
+        self.check()
+        self.print_statistics()
+
+    def __getitem__(self, index):
+        return (self.token_ids[index], self.lengths[index])
+
+    def __len__(self):
+        return len(self.lengths)
+
+    def check(self):
+        """
+        Some sanity checks
+        """
+        assert len(self.token_ids) == len(self.lengths)
+        assert all(self.lengths[i] == len(self.token_ids[i]) for i in range(len(self.lengths)))
+
+    def remove_long_sequences(self):
+        """
+        Sequences that are too long are split by chunk of max_model_input_size.
+        """
+        max_len = self.params.max_model_input_size
+        indices = self.lengths > max_len
+        logger.info(f"Splitting {sum(indices)} too long sequences.")
+
+        def divide_chunks(l, n):
+            return [l[i : i + n] for i in range(0, len(l), n)]
+
+        new_tok_ids = []
+        new_lengths = []
+        if self.params.mlm:
+            cls_id, sep_id = self.params.special_tok_ids["cls_token"], self.params.special_tok_ids["sep_token"]
+        else:
+            cls_id, sep_id = self.params.special_tok_ids["bos_token"], self.params.special_tok_ids["eos_token"]
+
+        for seq_, len_ in zip(self.token_ids, self.lengths):
+            assert (seq_[0] == cls_id) and (seq_[-1] == sep_id), seq_
+            if len_ <= max_len:
+                new_tok_ids.append(seq_)
+                new_lengths.append(len_)
+            else:
+                sub_seqs = []
+                for sub_s in divide_chunks(seq_, max_len - 2):
+                    if sub_s[0] != cls_id:
+                        sub_s = np.insert(sub_s, 0, cls_id)
+                    if sub_s[-1] != sep_id:
+                        sub_s = np.insert(sub_s, len(sub_s), sep_id)
+                    assert len(sub_s) <= max_len
+                    assert (sub_s[0] == cls_id) and (sub_s[-1] == sep_id), sub_s
+                    sub_seqs.append(sub_s)
+
+                new_tok_ids.extend(sub_seqs)
+                new_lengths.extend([len(l) for l in sub_seqs])
+
+        self.token_ids = np.array(new_tok_ids)
+        self.lengths = np.array(new_lengths)
+
+    def remove_empty_sequences(self):
+        """
+        Too short sequences are simply removed. This could be tuned.
+        """
+        init_size = len(self)
+        indices = self.lengths > 11
+        self.token_ids = self.token_ids[indices]
+        self.lengths = self.lengths[indices]
+        new_size = len(self)
+        logger.info(f"Remove {init_size - new_size} too short (<=11 tokens) sequences.")
+
+    def remove_unknown_sequences(self):
+        """
+        Remove sequences with a (too) high level of unknown tokens.
+        """
+        if "unk_token" not in self.params.special_tok_ids:
+            return
+        else:
+            unk_token_id = self.params.special_tok_ids["unk_token"]
+        init_size = len(self)
+        unk_occs = np.array([np.count_nonzero(a == unk_token_id) for a in self.token_ids])
+        indices = (unk_occs / self.lengths) < 0.5
+        self.token_ids = self.token_ids[indices]
+        self.lengths = self.lengths[indices]
+        new_size = len(self)
+        logger.info(f"Remove {init_size - new_size} sequences with a high level of unknown tokens (50%).")
+
+    def print_statistics(self):
+        """
+        Print some statistics on the corpus. Only the master process.
+        """
+        if not self.params.is_master:
+            return
+        logger.info(f"{len(self)} sequences")
+        # data_len = sum(self.lengths)
+        # nb_unique_tokens = len(Counter(list(chain(*self.token_ids))))
+        # logger.info(f'{data_len} tokens ({nb_unique_tokens} unique)')
+
+        # unk_idx = self.params.special_tok_ids['unk_token']
+        # nb_unknown = sum([(t==unk_idx).sum() for t in self.token_ids])
+        # logger.info(f'{nb_unknown} unknown tokens (covering {100*nb_unknown/data_len:.2f}% of the data)')
+
+    def batch_sequences(self, batch):
+        """
+        Do the padding and transform into torch.tensor.
+        """
+        token_ids = [t[0] for t in batch]
+        lengths = [t[1] for t in batch]
+        assert len(token_ids) == len(lengths)
+
+        # Max for paddings
+        max_seq_len_ = max(lengths)
+
+        # Pad token ids
+        if self.params.mlm:
+            pad_idx = self.params.special_tok_ids["pad_token"]
+        else:
+            pad_idx = self.params.special_tok_ids["unk_token"]
+        tk_ = [list(t.astype(int)) + [pad_idx] * (max_seq_len_ - len(t)) for t in token_ids]
+        assert len(tk_) == len(token_ids)
+        assert all(len(t) == max_seq_len_ for t in tk_)
+
+        tk_t = torch.tensor(tk_)  # (bs, max_seq_len_)
+        lg_t = torch.tensor(lengths)  # (bs)
+        return tk_t, lg_t
diff --git a/models_v2/pytorch/distilbert/inference/gpu/process_mini_data.sh b/models_v2/pytorch/distilbert/inference/gpu/process_mini_data.sh
new file mode 100644
index 000000000..70a9ecea5
--- /dev/null
+++ b/models_v2/pytorch/distilbert/inference/gpu/process_mini_data.sh
@@ -0,0 +1,22 @@
+#!/usr/bin/env bash
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+cd demo-data
+pip install -r requirements.txt
+python vocab_downloader.py --type=bert-base-uncased
+export VOCAB_FILE=bert-base-uncased-vocab.txt
+bash parallel_create_pretraining_data.sh ../miniwiki
diff --git a/models_v2/pytorch/distilbert/inference/gpu/requirements.txt b/models_v2/pytorch/distilbert/inference/gpu/requirements.txt
new file mode 100644
index 000000000..859928626
--- /dev/null
+++ b/models_v2/pytorch/distilbert/inference/gpu/requirements.txt
@@ -0,0 +1,6 @@
+transformers==4.36.0
+gitpython==3.1.41
+tensorboard>=1.14.0
+tensorboardX==1.8
+psutil==5.6.6
+scipy>=1.4.1
diff --git a/models_v2/pytorch/distilbert/inference/gpu/run_model.sh b/models_v2/pytorch/distilbert/inference/gpu/run_model.sh
new file mode 100755
index 000000000..b452ea7cd
--- /dev/null
+++ b/models_v2/pytorch/distilbert/inference/gpu/run_model.sh
@@ -0,0 +1,134 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+
+#!/bin/bash
+
+# Create an array of input directories that are expected and then verify that they exist
+declare -A input_envs
+input_envs[DATASET_DIR]=${DATASET_DIR}
+input_envs[MULTI_TILE]=${MULTI_TILE}
+input_envs[PLATFORM]=${PLATFORM}
+input_envs[OUTPUT_DIR]=${OUTPUT_DIR}
+
+for i in "${!input_envs[@]}"; do
+  var_name=$i
+  env_param=${input_envs[$i]}
+
+  if [[ -z $env_param ]]; then
+    echo "The required environment variable $var_name is not set" >&2
+    exit 1
+  fi
+done
+
+OUTPUT_DIR=${OUTPUT_DIR:-${PWD}}
+if [[ "${PLATFORM}" == "Max" ]]; then
+    BATCH_SIZE=${BATCH_SIZE:-32}
+    PRECISION=${PRECISION:-FP16}
+    NUM_ITERATIONS=${NUM_ITERATIONS:-300}
+elif [[ "${PLATFORM}" == "Flex" ]]; then
+    BATCH_SIZE=${BATCH_SIZE:-32}
+    PRECISION=${PRECISION:-FP16}
+    NUM_ITERATIONS=${NUM_ITERATIONS:-300}
+fi
+
+# known issue
+if [[ "${MULTI_TILE}" == "True" ]]; then
+    export ZE_FLAT_DEVICE_HIERARCHY=COMPOSITE
+fi
+
+echo 'Running with parameters:'
+echo " PLATFORM: ${PLATFORM}"
+echo " OUTPUT_DIR: ${OUTPUT_DIR}"
+echo " PRECISION: ${PRECISION}"
+echo " DATASET_DIR: ${DATASET_DIR}"
+echo " BATCH_SIZE: ${BATCH_SIZE}"
+echo " NUM_ITERATIONS: ${NUM_ITERATIONS}"
+echo " MULTI_TILE: ${MULTI_TILE}"
+
+if [[ "${PRECISION}" != "BF16" ]] && [[ "${PRECISION}" != "FP32" ]] && [[ "${PRECISION}" != "FP16" ]] && [[ "${PRECISION}" != "TF32" ]];  then
+    echo -e "Invalid input! Only BF16 FP16 FP32 TF32 are supported."
+    exit 1
+fi
+echo "DistilBert ${PRECISION} inference plain MultiTile=${MULTI_TILE} BS=${BATCH_SIZE} Iter=${NUM_ITERATIONS}"
+
+# Create the output directory, if it doesn't already exist
+#mkdir -p $OUTPUT_DIR
+
+sum_log_analysis() {
+    if [ -f $2 ]; then
+        rm -f $2
+    fi
+    if diff /dev/null ${1}_t0.log |tail -l | grep '^\\ No newline' > /dev/null;then echo >> ${1}_t0.log; fi
+    if diff /dev/null ${1}_t1.log |tail -l | grep '^\\ No newline' > /dev/null;then echo >> ${1}_t1.log; fi
+    bs=$(cat ${1}_t1.log |grep Batch |awk '{print $3}')
+    echo -e "Batch Size: $bs" >$2
+    cat ${1}"_t0.log" ${1}"_t1.log" |grep "Performance" |awk -v tag=$(cat ${1}"_t0.log" ${1}"_t1.log" |grep "Performance" |awk '{sum+=$2} END {printf "%.4f\n",sum}') '{if ( $2=="None" ) {sum="None";nextfile}else  sum=tag} ;END{print "Sum "$1" "sum " "$3}' >> $2
+    cat ${1}"_t0.log" ${1}"_t1.log" |grep "Performance" |awk -v tag=$(cat ${1}"_t0.log" ${1}"_t1.log" |grep "Performance" |awk 'BEGIN {min=1234567890123} {if ($2 <min) {min=$2}}END {printf "%.4f\n",min}') '{if ( $2=="None" ) {min="None";nextfile}else  min=tag} ;END{print "Min "$1" "min " "$3}' >> $2
+    cat ${1}"_t0.log" ${1}"_t1.log" |grep "Latency" |awk '{if ( $2=="N/A" ){avg="N/A";nextfile}else avg=((sum+=$2/2))};END{print "Avg "$1" "avg " "$3}'  >> $2
+    cat ${1}"_t0.log" ${1}"_t1.log" |grep "Accuracy" |awk -v avg=$(cat ${1}"_t0.log" ${1}"_t1.log" |grep "Accuracy" |awk '{sum+=$3}END{printf "%.4f\n",sum/NR}') '{if ( $3=="None" || $2=="N/A" || $3=="nan" || $3=="N/A"){avg="None";nextfile}else avg=avg};END{print "Avg "$1" "$2 " "avg}' >> $2
+    cat ${1}"_t0.log" ${1}"_t1.log" |grep "Functional" | awk -v fail=$(cat ${1}"_t0.log" ${1}"_t1.log" |grep "Functional" |awk '{for(i=1;i<=NF;++i) if($i=="fail") ++sum}END{print sum}') '{if ( fail >= 1 ) tag="fail ";else tag="pass"};END{print $1" "tag}'  >> $2
+    cat ${1}"_t0.log" ${1}"_t1.log" |grep "Error" |awk '{if(a[$1]){a[$1]=a[$1]";"$2}else{a[$1]=$2}}END{for(i in a)print $1" " a[i]}' >> $2
+}
+
+
+modelname=distilbert
+
+if [[ ${MULTI_TILE} == "False" ]]; then
+    rm ${OUTPUT_DIR}/${modelname}${PRECISION}_inf_t0_raw.log 
+    
+    python train.py --run_mode "eval" --jit --amp --device "xpu" --run_dtype ${PRECISION} --student_type "distilbert" --student_name "distilbert-base-uncased" --student_config "training_configs/distilbert-base-uncased.json" --teacher_type "bert"  --teacher_name "bert-base-uncased" --mlm --freeze_pos_embs --dump_path "${OUTPUT_DIR}/out" --data_file ${DATASET_DIR} --num_iteration ${NUM_ITERATIONS} --batch_size ${BATCH_SIZE} 2>&1 | tee ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0_raw.log
+    python common/parse_result.py -m $modelname -l ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0_raw.log -b ${BATCH_SIZE}
+    throughput=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0.log | grep Performance | awk -F ' ' '{print $2}')
+    throughput_unit=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0.log | grep Performance | awk -F ' ' '{print $3}')
+    latency=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0.log | grep Latency | awk -F ' ' '{print $2}')
+    acc=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0.log | grep Accuracy | awk -F ' ' '{print $3}')
+    acc_unit=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0.log | grep Accuracy | awk -F ' ' '{print $2}')
+else
+    rm ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0_raw.log
+    rm ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t1_raw.log
+    ZE_AFFINITY_MASK=0.0 python train.py --run_mode "eval" --jit --amp --device "xpu" --run_dtype ${PRECISION} --student_type "distilbert" --student_name "distilbert-base-uncased" --student_config "training_configs/distilbert-base-uncased.json" --teacher_type "bert"  --teacher_name "bert-base-uncased" --mlm --freeze_pos_embs --dump_path "${OUTPUT_DIR}/out" --data_file ${DATASET_DIR} --num_iteration ${NUM_ITERATIONS} --batch_size ${BATCH_SIZE} 2>&1 | tee ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0_raw.log &
+    ZE_AFFINITY_MASK=0.1 python train.py --run_mode "eval" --jit --amp --device "xpu" --run_dtype ${PRECISION} --student_type "distilbert" --student_name "distilbert-base-uncased" --student_config "training_configs/distilbert-base-uncased.json" --teacher_type "bert"  --teacher_name "bert-base-uncased" --mlm --freeze_pos_embs --dump_path "${OUTPUT_DIR}/out1" --data_file ${DATASET_DIR} --num_iteration ${NUM_ITERATIONS} --batch_size ${BATCH_SIZE} 2>&1 | tee ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t1_raw.log
+    wait
+    python common/parse_result.py -m $modelname -l ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0_raw.log -b ${BATCH_SIZE}
+    python common/parse_result.py -m $modelname -l ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t1_raw.log -b ${BATCH_SIZE}
+    sum_log_analysis ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf.log
+    throughput=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf.log | grep "Sum Performance" | awk -F ' ' '{print $3}')
+    throughput_unit=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf.log | grep "Sum Performance" | awk -F ' ' '{print $4}')
+    latency=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf.log | grep Latency | awk -F ' ' '{print $3}')
+    acc=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf.log | grep Accuracy | awk -F ' ' '{print $4}')
+    acc_unit=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf.log | grep Accuracy | awk -F ' ' '{print $3}')
+fi
+
+yaml_content=$(cat <<EOF
+results:
+ - key: throughput
+   value: $throughput
+   unit: $throughput_unit
+ - key: latency
+   value: $latency
+   unit: s
+ - key: accuracy
+   value: $acc
+   unit: $acc_unit
+EOF
+)
+
+# Write the content to a YAML file
+echo "$yaml_content" >  ${OUTPUT_DIR}/results.yaml
+echo "YAML file created."
diff --git a/models_v2/pytorch/distilbert/inference/gpu/setup.sh b/models_v2/pytorch/distilbert/inference/gpu/setup.sh
new file mode 100755
index 000000000..c175ea121
--- /dev/null
+++ b/models_v2/pytorch/distilbert/inference/gpu/setup.sh
@@ -0,0 +1,29 @@
+#!/usr/bin/env bash
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# setup.sh
+#  - install OS pkgs
+#  - should create virtual env & install pip  requirement.txt
+#  - git clones & applying patches
+
+set -e
+apt-get update && apt-get install -y python3-venv
+
+pip uninstall google-auth-oauthlib tb-nightly -y
+pip install -r requirements.txt
+
+cp -r ../../../../common .
diff --git a/models_v2/pytorch/distilbert/inference/gpu/train.py b/models_v2/pytorch/distilbert/inference/gpu/train.py
new file mode 100644
index 000000000..6a9413dc0
--- /dev/null
+++ b/models_v2/pytorch/distilbert/inference/gpu/train.py
@@ -0,0 +1,427 @@
+# coding=utf-8
+# Copyright 2019-present, the HuggingFace Inc. team.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+Training the distilled model.
+Supported architectures include: BERT -> DistilBERT, RoBERTa -> DistilRoBERTa, GPT2 -> DistilGPT2.
+"""
+import argparse
+import json
+import os
+import pickle
+import shutil
+
+import numpy
+import numpy as np
+import torch
+from typing import Dict, List, Optional, Set, Tuple, Union
+
+from distiller import Distiller
+from lm_seqs_dataset import LmSeqsDataset
+from transformers import (
+    BertConfig,
+    BertForMaskedLM,
+    BertTokenizer,
+    DistilBertConfig,
+    DistilBertForMaskedLM,
+    DistilBertTokenizer,
+    GPT2Config,
+    GPT2LMHeadModel,
+    GPT2Tokenizer,
+    RobertaConfig,
+    RobertaForMaskedLM,
+    RobertaTokenizer,
+)
+from transformers.modeling_outputs import MaskedLMOutput
+
+from utils import git_log, init_gpu_params, logger, set_seed
+
+class DistilBertForMaskedLM_Trace(DistilBertForMaskedLM):
+
+    def __init__(self, config):
+        super().__init__(config)
+
+    def forward(
+        self,
+        input_ids: Optional[torch.Tensor] = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        head_mask: Optional[torch.Tensor] = None,
+        inputs_embeds: Optional[torch.Tensor] = None,
+        labels: Optional[torch.LongTensor] = None,
+        output_attentions: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+    ) -> Tuple[torch.Tensor, ...]:
+        r"""
+        labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*):
+            Labels for computing the masked language modeling loss. Indices should be in `[-100, 0, ...,
+            config.vocab_size]` (see `input_ids` docstring) Tokens with indices set to `-100` are ignored (masked), the
+            loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`.
+        """
+        return_dict = return_dict if return_dict is not None else self.config.use_return_dict
+
+        dlbrt_output = self.distilbert(
+            input_ids=input_ids,
+            attention_mask=attention_mask,
+            head_mask=head_mask,
+            inputs_embeds=inputs_embeds,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            return_dict=return_dict,
+        )
+        hidden_states = dlbrt_output[0]  # (bs, seq_length, dim)
+        prediction_logits = self.vocab_transform(hidden_states)  # (bs, seq_length, dim)
+        prediction_logits = self.activation(prediction_logits)  # (bs, seq_length, dim)
+        prediction_logits = self.vocab_layer_norm(prediction_logits)  # (bs, seq_length, dim)
+        prediction_logits = self.vocab_projector(prediction_logits)  # (bs, seq_length, vocab_size)
+
+        mlm_loss = None
+        if labels is not None:
+            mlm_loss = self.mlm_loss_fct(prediction_logits.view(-1, prediction_logits.size(-1)), labels.view(-1))
+
+        if not return_dict:
+            output = (prediction_logits,) + dlbrt_output[1:]
+            return ((mlm_loss,) + output) if mlm_loss is not None else output
+
+        if dlbrt_output.attentions == None:
+            dlbrt_output.attentions = torch.empty((0))
+        if mlm_loss == None:
+            mlm_loss = torch.empty((0))
+        return prediction_logits, dlbrt_output.hidden_states, dlbrt_output.attentions, mlm_loss
+
+MODEL_CLASSES = {
+    "distilbert": (DistilBertConfig, DistilBertForMaskedLM, DistilBertTokenizer),
+    "distilbert_infer": (DistilBertConfig, DistilBertForMaskedLM_Trace, DistilBertTokenizer),
+    "roberta": (RobertaConfig, RobertaForMaskedLM, RobertaTokenizer),
+    "bert": (BertConfig, BertForMaskedLM, BertTokenizer),
+    "gpt2": (GPT2Config, GPT2LMHeadModel, GPT2Tokenizer),
+}
+
+def sanity_checks(args):
+    """
+    A bunch of args sanity checks to perform even starting...
+    """
+    if args.run_mode == "train":
+        assert (args.mlm and args.alpha_mlm > 0.0) or (not args.mlm and args.alpha_mlm == 0.0)
+        assert (args.alpha_mlm > 0.0 and args.alpha_clm == 0.0) or (args.alpha_mlm == 0.0 and args.alpha_clm > 0.0)
+        assert args.alpha_ce >= 0.0
+        assert args.alpha_mlm >= 0.0
+        assert args.alpha_clm >= 0.0
+        assert args.alpha_mse >= 0.0
+        assert args.alpha_cos >= 0.0
+        assert args.alpha_ce + args.alpha_mlm + args.alpha_clm + args.alpha_mse + args.alpha_cos > 0.0
+    if args.mlm:
+        if args.run_mode == "train":
+            assert os.path.isfile(args.token_counts)
+        assert (args.student_type in ["roberta", "distilbert"]) and (args.teacher_type in ["roberta", "bert"])
+    else:
+        assert (args.student_type in ["gpt2"]) and (args.teacher_type in ["gpt2"])
+
+    assert args.teacher_type == args.student_type or (
+        args.student_type == "distilbert" and args.teacher_type == "bert"
+    )
+    assert os.path.isfile(args.student_config)
+    if args.student_pretrained_weights is not None:
+        assert os.path.isfile(args.student_pretrained_weights)
+
+    if args.freeze_token_type_embds:
+        assert args.student_type in ["roberta"]
+
+
+
+def freeze_pos_embeddings(student, args):
+    if args.student_type == "roberta":
+        student.roberta.embeddings.position_embeddings.weight.requires_grad = False
+    elif args.student_type == "gpt2":
+        student.transformer.wpe.weight.requires_grad = False
+
+
+def freeze_token_type_embeddings(student, args):
+    if args.student_type == "roberta":
+        student.roberta.embeddings.token_type_embeddings.weight.requires_grad = False
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Training")
+    parser.add_argument("--force", action="store_true", help="Overwrite dump_path if it already exists.")
+
+    parser.add_argument(
+        "--run_mode", choices=["train", "eval"], default="train", help="Runs mode(train, eval)."
+    )
+
+    parser.add_argument(
+        "--run_dtype", choices=["FP32", "BF16", "FP16"], default="FP32", help="Data type (FP32, BF16, FP16)."
+    )
+
+    parser.add_argument(
+        "--device", choices=["cpu", "cuda", "xpu"], default="cpu", help="The running device (cpu, cuda, xpu)."
+    )
+
+    parser.add_argument(
+        "--amp",
+        action="store_true",
+        help="Whether to use mixed precision instead of 32-bit",
+    )
+
+    parser.add_argument(
+        "--jit",
+        action="store_true",
+        help="Whether to use jit trace",
+    )
+
+    parser.add_argument(
+        "--dump_path", type=str, default="./dump_path", help="The output directory (log, checkpoints, parameters, etc.)"
+    )
+    parser.add_argument(
+        "--data_file",
+        type=str,
+        required=True,
+        help="The binarized file (tokenized + tokens_to_ids) and grouped by sequence.",
+    )
+
+    parser.add_argument(
+        "--student_type",
+        type=str,
+        choices=["distilbert", "roberta", "gpt2"],
+        required=True,
+        help="The student type (DistilBERT, RoBERTa).",
+    )
+    parser.add_argument("--student_config", type=str, required=True, help="Path to the student configuration.")
+    parser.add_argument(
+        "--student_pretrained_weights", default=None, type=str, help="Load student initialization checkpoint."
+    )
+
+    parser.add_argument(
+        "--teacher_type", choices=["bert", "roberta", "gpt2"], required=True, help="Teacher type (BERT, RoBERTa)."
+    )
+    parser.add_argument("--teacher_name", type=str, required=True, help="The teacher model.")
+    parser.add_argument("--student_name", type=str, required=True, help="The student model.")
+
+    parser.add_argument("--temperature", default=2.0, type=float, help="Temperature for the softmax temperature.")
+    parser.add_argument(
+        "--alpha_ce", default=0.5, type=float, help="Linear weight for the distillation loss. Must be >=0."
+    )
+    parser.add_argument(
+        "--alpha_mlm",
+        default=0.0,
+        type=float,
+        help="Linear weight for the MLM loss. Must be >=0. Should be used in conjunction with `mlm` flag.",
+    )
+    parser.add_argument("--alpha_clm", default=0.5, type=float, help="Linear weight for the CLM loss. Must be >=0.")
+    parser.add_argument("--alpha_mse", default=0.0, type=float, help="Linear weight of the MSE loss. Must be >=0.")
+    parser.add_argument(
+        "--alpha_cos", default=0.0, type=float, help="Linear weight of the cosine embedding loss. Must be >=0."
+    )
+
+    parser.add_argument(
+        "--mlm", action="store_true", help="The LM step: MLM or CLM. If `mlm` is True, the MLM is used over CLM."
+    )
+    parser.add_argument(
+        "--mlm_mask_prop",
+        default=0.15,
+        type=float,
+        help="Proportion of tokens for which we need to make a prediction.",
+    )
+    parser.add_argument("--word_mask", default=0.8, type=float, help="Proportion of tokens to mask out.")
+    parser.add_argument("--word_keep", default=0.1, type=float, help="Proportion of tokens to keep.")
+    parser.add_argument("--word_rand", default=0.1, type=float, help="Proportion of tokens to randomly replace.")
+    parser.add_argument(
+        "--mlm_smoothing",
+        default=0.7,
+        type=float,
+        help="Smoothing parameter to emphasize more rare tokens (see XLM, similar to word2vec).",
+    )
+    parser.add_argument("--token_counts", type=str, help="The token counts in the data_file for MLM.")
+
+    parser.add_argument(
+        "--restrict_ce_to_mask",
+        action="store_true",
+        help="If true, compute the distillation loss only the [MLM] prediction distribution.",
+    )
+    parser.add_argument(
+        "--freeze_pos_embs",
+        action="store_true",
+        help="Freeze positional embeddings during distillation. For student_type in ['roberta', 'gpt2'] only.",
+    )
+    parser.add_argument(
+        "--freeze_token_type_embds",
+        action="store_true",
+        help="Freeze token type embeddings during distillation if existent. For student_type in ['roberta'] only.",
+    )
+
+    parser.add_argument("--n_epoch", type=int, default=3, help="Number of pass on the whole dataset.")
+    parser.add_argument("--batch_size", type=int, default=64, help="Batch size (for each process).")
+    parser.add_argument("--num_iteration", type=int, default=-1, help="Iteration number.")
+    parser.add_argument(
+        "--group_by_size",
+        action="store_false",
+        help="If true, group sequences that have similar length into the same batch. Default is true.",
+    )
+
+    parser.add_argument("--max_predictions_per_seq",
+                        default=76,
+                        type=int,
+                        help="The maximum total of masked tokens in input sequence")
+
+    parser.add_argument(
+        "--gradient_accumulation_steps",
+        type=int,
+        default=50,
+        help="Gradient accumulation for larger training batches.",
+    )
+    parser.add_argument("--warmup_prop", default=0.05, type=float, help="Linear warmup proportion.")
+    parser.add_argument("--weight_decay", default=0.0, type=float, help="Weight decay if we apply some.")
+    parser.add_argument("--learning_rate", default=5e-4, type=float, help="The initial learning rate for Adam.")
+    parser.add_argument("--adam_epsilon", default=1e-6, type=float, help="Epsilon for Adam optimizer.")
+    parser.add_argument("--max_grad_norm", default=5.0, type=float, help="Max gradient norm.")
+    parser.add_argument("--initializer_range", default=0.02, type=float, help="Random initialization range.")
+
+    #parser.add_argument(
+    #    "--fp16",
+    #    action="store_true",
+    #    help="Whether to use 16-bit (mixed) precision (through NVIDIA apex) instead of 32-bit",
+    #)
+    parser.add_argument(
+        "--fp16_opt_level",
+        type=str,
+        default="O1",
+        help=(
+            "For fp16: Apex AMP optimization level selected in ['O0', 'O1', 'O2', and 'O3']."
+            "See details at https://nvidia.github.io/apex/amp.html"
+        ),
+    )
+    parser.add_argument("--n_gpu", type=int, default=1, help="Number of GPUs in the node.")
+    parser.add_argument("--local_rank", type=int, default=-1, help="Distributed training - Local rank")
+    parser.add_argument("--seed", type=int, default=56, help="Random seed")
+
+    parser.add_argument("--log_interval", type=int, default=500, help="Tensorboard logging interval.")
+    parser.add_argument("--checkpoint_interval", type=int, default=4000, help="Checkpoint interval.")
+    args = parser.parse_args()
+    sanity_checks(args)
+
+    if args.device == "xpu":
+        import intel_extension_for_pytorch
+
+    # ARGS #
+    init_gpu_params(args)
+    set_seed(args)
+    if args.is_master:
+        if os.path.exists(args.dump_path):
+            if not args.force:
+                raise ValueError(
+                    f"Serialization dir {args.dump_path} already exists, but you have not precised wheter to overwrite"
+                    " itUse `--force` if you want to overwrite it"
+                )
+            else:
+                shutil.rmtree(args.dump_path)
+
+        pid = os.getpid()
+        args.dump_path += str(pid)
+        if not os.path.exists(args.dump_path):
+            os.makedirs(args.dump_path)
+        logger.info(f"Experiment will be dumped and logged in {args.dump_path}")
+
+        # SAVE PARAMS #
+        logger.info(f"Param: {args}")
+        with open(os.path.join(args.dump_path, "parameters.json"), "w") as f:
+            json.dump(vars(args), f, indent=4)
+        #git_log(args.dump_path)
+
+    if args.run_mode == "eval":
+        student_config_class, student_model_class, _ = MODEL_CLASSES[args.student_type + "_infer"]
+    else:
+        student_config_class, student_model_class, _ = MODEL_CLASSES[args.student_type]
+    teacher_config_class, teacher_model_class, teacher_tokenizer_class = MODEL_CLASSES[args.teacher_type]
+
+    # TOKENIZER #
+    tokenizer = teacher_tokenizer_class.from_pretrained(args.teacher_name)
+    special_tok_ids = {}
+    for tok_name, tok_symbol in tokenizer.special_tokens_map.items():
+        idx = tokenizer.all_special_tokens.index(tok_symbol)
+        special_tok_ids[tok_name] = tokenizer.all_special_ids[idx]
+    logger.info(f"Special tokens {special_tok_ids}")
+    args.special_tok_ids = special_tok_ids
+    args.max_model_input_size = tokenizer.max_model_input_sizes[args.teacher_name]
+
+    # DATA LOADER #
+    logger.info(f"Loading data from {args.data_file}")
+
+    if args.mlm and args.run_mode == "train":
+        logger.info(f"Loading token counts from {args.token_counts} (already pre-computed)")
+        with open(args.token_counts, "rb") as fp:
+            counts = pickle.load(fp)
+
+        token_probs = np.maximum(counts, 1) ** -args.mlm_smoothing
+        for idx in special_tok_ids.values():
+            token_probs[idx] = 0.0  # do not predict special tokens
+        token_probs = torch.from_numpy(token_probs)
+    else:
+        token_probs = None
+
+    # STUDENT #
+    logger.info(f"Loading student config from {args.student_config}")
+    stu_architecture_config = student_config_class.from_pretrained(args.student_config)
+    stu_architecture_config.output_hidden_states = True
+
+    if args.student_pretrained_weights is not None:
+        logger.info(f"Loading pretrained weights from {args.student_pretrained_weights}")
+        student = student_model_class.from_pretrained(args.student_pretrained_weights, config=stu_architecture_config)
+    else:
+        student = student_model_class.from_pretrained(args.student_name, config=stu_architecture_config)
+        #student = student_model_class(stu_architecture_config)
+
+    if args.n_gpu > 0 and args.device == "cuda":
+        student.to(f"cuda:{args.local_rank}")
+    elif args.n_gpu > 0 and args.device == "xpu": 
+        student.to(args.device)
+    logger.info("Student loaded.")
+
+    # TEACHER #
+    teacher = teacher_model_class.from_pretrained(args.teacher_name, output_hidden_states=True)
+    if args.n_gpu > 0 and args.device == "cuda":
+        teacher.to(f"cuda:{args.local_rank}")
+    elif args.n_gpu > 0 and args.device == "xpu": 
+        teacher.to(args.device)
+    logger.info(f"Teacher loaded from {args.teacher_name}.")
+
+    # FREEZING #
+    if args.freeze_pos_embs:
+        freeze_pos_embeddings(student, args)
+    if args.freeze_token_type_embds:
+        freeze_token_type_embeddings(student, args)
+
+    # SANITY CHECKS #
+    assert student.config.vocab_size == teacher.config.vocab_size
+    assert student.config.hidden_size == teacher.config.hidden_size
+    assert student.config.max_position_embeddings == teacher.config.max_position_embeddings
+    if args.mlm and args.run_mode == "train":
+        assert token_probs.size(0) == stu_architecture_config.vocab_size
+
+    # DISTILLER #
+    if args.device == "cuda":
+        torch.cuda.empty_cache()
+    elif args.device == "xpu":
+        torch.xpu.empty_cache()
+    distiller = Distiller(
+        params=args, token_probs=token_probs, student=student, teacher=teacher
+    )
+    if args.run_mode == "train":
+        distiller.train()
+    elif args.run_mode == "eval":
+        distiller.eval()
+    logger.info("Let's go get some drinks.")
+
+
+if __name__ == "__main__":
+    main()
diff --git a/models_v2/pytorch/distilbert/inference/gpu/training_configs/distilbert-base-cased.json b/models_v2/pytorch/distilbert/inference/gpu/training_configs/distilbert-base-cased.json
new file mode 100644
index 000000000..cef924d1e
--- /dev/null
+++ b/models_v2/pytorch/distilbert/inference/gpu/training_configs/distilbert-base-cased.json
@@ -0,0 +1,15 @@
+{
+	"activation": "gelu",
+	"attention_dropout": 0.1,
+	"dim": 768,
+	"dropout": 0.1,
+	"hidden_dim": 3072,
+	"initializer_range": 0.02,
+	"max_position_embeddings": 512,
+	"n_heads": 12,
+	"n_layers": 6,
+	"sinusoidal_pos_embds": true,
+	"tie_weights_": true,
+	"vocab_size": 28996
+}
+  
diff --git a/models_v2/pytorch/distilbert/inference/gpu/training_configs/distilbert-base-multilingual-cased.json b/models_v2/pytorch/distilbert/inference/gpu/training_configs/distilbert-base-multilingual-cased.json
new file mode 100644
index 000000000..fc5824f81
--- /dev/null
+++ b/models_v2/pytorch/distilbert/inference/gpu/training_configs/distilbert-base-multilingual-cased.json
@@ -0,0 +1,15 @@
+{
+	"activation": "gelu",
+	"attention_dropout": 0.1,
+	"dim": 768,
+	"dropout": 0.1,
+	"hidden_dim": 3072,
+	"initializer_range": 0.02,
+	"max_position_embeddings": 512,
+	"n_heads": 12,
+	"n_layers": 6,
+	"sinusoidal_pos_embds": true,
+	"tie_weights_": true,
+	"vocab_size": 119547
+}
+  
diff --git a/models_v2/pytorch/distilbert/inference/gpu/training_configs/distilbert-base-uncased.json b/models_v2/pytorch/distilbert/inference/gpu/training_configs/distilbert-base-uncased.json
new file mode 100644
index 000000000..914805ea9
--- /dev/null
+++ b/models_v2/pytorch/distilbert/inference/gpu/training_configs/distilbert-base-uncased.json
@@ -0,0 +1,15 @@
+{
+	"activation": "gelu",
+	"attention_dropout": 0.1,
+	"dim": 768,
+	"dropout": 0.1,
+	"hidden_dim": 3072,
+	"initializer_range": 0.02,
+	"max_position_embeddings": 512,
+	"n_heads": 12,
+	"n_layers": 6,
+	"sinusoidal_pos_embds": true,
+	"tie_weights_": true,
+	"vocab_size": 30522
+}
+  
diff --git a/models_v2/pytorch/distilbert/inference/gpu/training_configs/distilgpt2.json b/models_v2/pytorch/distilbert/inference/gpu/training_configs/distilgpt2.json
new file mode 100644
index 000000000..2b187fabe
--- /dev/null
+++ b/models_v2/pytorch/distilbert/inference/gpu/training_configs/distilgpt2.json
@@ -0,0 +1,9 @@
+{
+	"initializer_range": 0.02,
+	"layer_norm_epsilon": 0.00001,
+	"n_embd": 768,
+	"n_head": 12,
+	"n_layer": 6,
+	"n_positions": 1024,
+	"vocab_size": 50257
+}
diff --git a/models_v2/pytorch/distilbert/inference/gpu/training_configs/distilroberta-base.json b/models_v2/pytorch/distilbert/inference/gpu/training_configs/distilroberta-base.json
new file mode 100644
index 000000000..b900b8bc3
--- /dev/null
+++ b/models_v2/pytorch/distilbert/inference/gpu/training_configs/distilroberta-base.json
@@ -0,0 +1,14 @@
+{
+    "vocab_size": 50265,
+    "hidden_size": 768,
+    "num_hidden_layers": 6,
+    "num_attention_heads": 12,
+    "intermediate_size": 3072,
+    "hidden_act": "gelu",
+    "hidden_dropout_prob": 0.1,
+    "attention_probs_dropout_prob": 0.1,
+    "max_position_embeddings": 514,
+    "type_vocab_size": 1,
+    "initializer_range": 0.02,
+    "layer_norm_eps": 0.00001
+}
diff --git a/models_v2/pytorch/distilbert/inference/gpu/utils.py b/models_v2/pytorch/distilbert/inference/gpu/utils.py
new file mode 100644
index 000000000..111c0dca5
--- /dev/null
+++ b/models_v2/pytorch/distilbert/inference/gpu/utils.py
@@ -0,0 +1,147 @@
+# coding=utf-8
+# Copyright 2019-present, the HuggingFace Inc. team and Facebook, Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+""" Utils to train DistilBERT
+    adapted in part from Facebook, Inc XLM model (https://github.com/facebookresearch/XLM)
+"""
+import json
+import logging
+import os
+import socket
+
+import git
+import numpy as np
+import torch
+
+
+logging.basicConfig(
+    format="%(asctime)s - %(levelname)s - %(name)s - PID: %(process)d -  %(message)s",
+    datefmt="%m/%d/%Y %H:%M:%S",
+    level=logging.INFO,
+)
+logger = logging.getLogger(__name__)
+
+MAP_TORCH_DTYPE = {"FP32": torch.float32, "FP16": torch.float16,
+                   "BF16": torch.bfloat16, "FP64": torch.float64,
+                   "CF64": torch.cfloat, "CF128": torch.cdouble,
+                   "UINT8": torch.uint8, "INT8": torch.int8,
+                   "INT16": torch.int16, "INT": torch.int32,
+                   "INT64": torch.int64, "BOOL": torch.bool,
+                  }
+
+
+def git_log(folder_path: str):
+    """
+    Log commit info.
+    """
+    repo = git.Repo(search_parent_directories=True)
+    repo_infos = {
+        "repo_id": str(repo),
+        "repo_sha": str(repo.head.object.hexsha),
+        "repo_branch": str(repo.active_branch),
+    }
+
+    with open(os.path.join(folder_path, "git_log.json"), "w") as f:
+        json.dump(repo_infos, f, indent=4)
+
+
+def init_gpu_params(params):
+    """
+    Handle single and multi-GPU / multi-node.
+    """
+    if params.n_gpu <= 0:
+        params.local_rank = 0
+        params.master_port = -1
+        params.is_master = True
+        params.multi_gpu = False
+        return
+
+    if params.device == "cuda":
+        assert torch.cuda.is_available()
+    elif params.device == "xpu":
+        assert torch.xpu.is_available()
+
+    logger.info("Initializing GPUs")
+    if params.n_gpu > 1:
+        assert params.local_rank != -1
+
+        params.world_size = int(os.environ["WORLD_SIZE"])
+        params.n_gpu_per_node = int(os.environ["N_GPU_NODE"])
+        params.global_rank = int(os.environ["RANK"])
+
+        # number of nodes / node ID
+        params.n_nodes = params.world_size // params.n_gpu_per_node
+        params.node_id = params.global_rank // params.n_gpu_per_node
+        params.multi_gpu = True
+
+        assert params.n_nodes == int(os.environ["N_NODES"])
+        assert params.node_id == int(os.environ["NODE_RANK"])
+
+    # local job (single GPU)
+    else:
+        assert params.local_rank == -1
+
+        params.n_nodes = 1
+        params.node_id = 0
+        params.local_rank = 0
+        params.global_rank = 0
+        params.world_size = 1
+        params.n_gpu_per_node = 1
+        params.multi_gpu = False
+
+    # sanity checks
+    assert params.n_nodes >= 1
+    assert 0 <= params.node_id < params.n_nodes
+    assert 0 <= params.local_rank <= params.global_rank < params.world_size
+    assert params.world_size == params.n_nodes * params.n_gpu_per_node
+
+    # define whether this is the master process / if we are in multi-node distributed mode
+    params.is_master = params.node_id == 0 and params.local_rank == 0
+    params.multi_node = params.n_nodes > 1
+
+    # summary
+    PREFIX = f"--- Global rank: {params.global_rank} - "
+    logger.info(PREFIX + "Number of nodes: %i" % params.n_nodes)
+    logger.info(PREFIX + "Node ID        : %i" % params.node_id)
+    logger.info(PREFIX + "Local rank     : %i" % params.local_rank)
+    logger.info(PREFIX + "World size     : %i" % params.world_size)
+    logger.info(PREFIX + "GPUs per node  : %i" % params.n_gpu_per_node)
+    logger.info(PREFIX + "Master         : %s" % str(params.is_master))
+    logger.info(PREFIX + "Multi-node     : %s" % str(params.multi_node))
+    logger.info(PREFIX + "Multi-GPU      : %s" % str(params.multi_gpu))
+    logger.info(PREFIX + "Hostname       : %s" % socket.gethostname())
+
+    # set GPU device
+    if params.device == "cuda":
+        torch.cuda.set_device(params.local_rank)
+
+    # initialize multi-GPU
+    if params.device == "cuda" and params.multi_gpu:
+        logger.info("Initializing PyTorch distributed")
+        torch.distributed.init_process_group(
+            init_method="env://",
+            backend="nccl",
+        )
+
+
+def set_seed(args):
+    """
+    Set the random seed.
+    """
+    np.random.seed(args.seed)
+    torch.manual_seed(args.seed)
+    if args.n_gpu > 0 and args.device == "cuda":
+        torch.cuda.manual_seed_all(args.seed)
+    elif args.n_gpu > 0 and args.device == "xpu":
+        torch.xpu.manual_seed_all(args.seed)
diff --git a/models_v2/pytorch/dlrm/inference/gpu/CODE_OF_CONDUCT.md b/models_v2/pytorch/dlrm/inference/gpu/CODE_OF_CONDUCT.md
new file mode 100644
index 000000000..0f7ad8bfc
--- /dev/null
+++ b/models_v2/pytorch/dlrm/inference/gpu/CODE_OF_CONDUCT.md
@@ -0,0 +1,5 @@
+# Code of Conduct
+
+Facebook has adopted a Code of Conduct that we expect project participants to adhere to.
+Please read the [full text](https://code.fb.com/codeofconduct/)
+so that you can understand what actions will and will not be tolerated.
diff --git a/models_v2/pytorch/dlrm/inference/gpu/CONTAINER.md b/models_v2/pytorch/dlrm/inference/gpu/CONTAINER.md
new file mode 100644
index 000000000..c6a3cbb30
--- /dev/null
+++ b/models_v2/pytorch/dlrm/inference/gpu/CONTAINER.md
@@ -0,0 +1,84 @@
+# Running DLRM-v1 inference on Intel® Data Center GPU Flex Series using Intel® Extension for PyTorch*
+
+## Overview
+
+This document has instructions for running DLRM-v1 inference using Intel® Extension for PyTorch on Intel® Flex Series GPU.
+
+## Requirements
+| Item | Detail |
+| ------ | ------- |
+| Host machine  | [Intel® Data Center GPU Flex Series](https://ark.intel.com/content/www/us/en/ark/products/series/230021/intel-data-center-gpu-flex-series.html) 170  |
+| Drivers | GPU-compatible drivers need to be installed: Download [Driver](https://dgpu-docs.intel.com/driver/installation.html) |
+| Software | Docker* |
+
+## Quick Start Scripts
+
+| Script name | Description |
+|-------------|-------------|
+| `run_model.sh` | Batch inference with FP16 precision on Flex series 170 |
+
+> [!NOTE]
+> At the moment sample does not support FP32 precision (`export PRECISION=fp16`).
+
+## Datasets and Pre-trained Model
+
+Refer to the [link](README.md#prepare-dataset) to download and prepare datasets and pre-trained models. Set `DATASET_DIR` and `CKPT_DIR` to point to the corresponding directories. 
+
+## Run Using Docker
+
+### Set up Docker Image
+
+```bash
+docker pull intel/recommendation:pytorch-flex-gpu-dlrm-v1-inference
+```
+### Run Docker Image
+The DLRM-v1 inference container includes scripts, model and libraries needed to run FP16 inference. To run the `run_model.sh` quickstart script using this container, you will need to provide an output directory where log files will be written.
+
+```bash
+#Optional
+export PRECISION=fp16
+export BATCH_SIZE=<provide batch size otherwise (default: 32768)>
+export NUM_ITERATIONS=<provide num_iterations otherwise (default: 20)>
+
+#Required
+export OUTPUT_DIR=<path to output directory>
+export SCRIPT=run_model.sh
+export MULTI_TILE=False
+export PLATFORM=Flex
+export DATASET_DIR=<path to processed dataset directory>
+export CKPT_DIR=<path to pre-trained model>
+
+IMAGE_NAME=intel/recommendation:pytorch-flex-gpu-dlrm-v1-inference
+DOCKER_ARGS="--rm -it"
+
+docker run \
+  --privileged \
+  --device=/dev/dri \
+  --ipc=host \
+  --env PRECISION=${PRECISION} \
+  --env NUM_ITERATIONS=${NUM_ITERATIONS} \
+  --env OUTPUT_DIR=${OUTPUT_DIR} \
+  --env MULTI_TILE=${MULTI_TILE} \
+  --env PLATFORM=${PLATFORM} \
+  --env DATASET_DIR=${DATASET_DIR} \
+  --env CKPT_DIR=${CKPT_DIR} \
+  --env http_proxy=${http_proxy} \
+  --env https_proxy=${https_proxy} \
+  --env no_proxy=${no_proxy} \
+  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
+  --volume ${DATASET_DIR}:${DATASET_DIR} \
+  --volume ${CKPT_DIR}:${CKPT_DIR} \
+  ${DOCKER_ARGS} \
+  ${IMAGE_NAME} \
+  /bin/bash $SCRIPT
+```
+## Documentation and Sources
+
+[GitHub* Repository](https://github.com/IntelAI/models/tree/master/docker/flex-gpu)
+
+## Support
+Support for Intel® Extension for PyTorch* is found via the [Intel® AI Analytics Toolkit.](https://www.intel.com/content/www/us/en/developer/tools/oneapi/ai-analytics-toolkit.html#gs.qbretz) Additionally, the Intel® Extension for PyTorch* team tracks both bugs and enhancement requests using [GitHub issues](https://github.com/intel/intel-extension-for-pytorch/issues). Before submitting a suggestion or bug report, please search the GitHub issues to see if your issue has already been reported.
+
+## License Agreement
+
+LEGAL NOTICE: By accessing, downloading or using this software and any required dependent software (the “Software Package”), you agree to the terms and conditions of the software license agreements for the Software Package, which may also include notices, disclaimers, or license terms for third party software included with the Software Package. Please refer to the [license file](https://github.com/IntelAI/models/tree/master/third_party) for additional details.
diff --git a/models_v2/pytorch/dlrm/inference/gpu/CONTRIBUTING.md b/models_v2/pytorch/dlrm/inference/gpu/CONTRIBUTING.md
new file mode 100644
index 000000000..3563e0052
--- /dev/null
+++ b/models_v2/pytorch/dlrm/inference/gpu/CONTRIBUTING.md
@@ -0,0 +1,36 @@
+# Contributing to DLRM
+We want to make contributing to this project as easy and transparent as
+possible.
+
+## Pull Requests
+We actively welcome your pull requests.
+
+1. Fork the repo and create your branch from `main`.
+2. If you've added code that should be tested, add tests.
+3. If you've changed APIs, update the documentation.
+4. Ensure the test suite passes.
+5. Make sure your code lints.
+6. If you haven't already, complete the Contributor License Agreement ("CLA").
+
+## Contributor License Agreement ("CLA")
+In order to accept your pull request, we need you to submit a CLA. You only need
+to do this once to work on any of Facebook's open source projects.
+
+Complete your CLA here: <https://code.facebook.com/cla>
+
+## Issues
+We use GitHub issues to track public bugs. Please ensure your description is
+clear and has sufficient instructions to be able to reproduce the issue.
+
+Facebook has a [bounty program](https://www.facebook.com/whitehat/) for the safe
+disclosure of security bugs. In those cases, please go through the process
+outlined on that page and do not file a public issue.
+
+## Coding Style
+* 4 spaces for indentation rather than tabs
+* 80 character line length
+* in general, please maintain a consistent style with the rest of the code
+
+## License
+By contributing to DLRM, you agree that your contributions will be licensed
+under the LICENSE file in the root directory of this source tree.
diff --git a/models_v2/pytorch/dlrm/inference/gpu/LICENSE b/models_v2/pytorch/dlrm/inference/gpu/LICENSE
new file mode 100644
index 000000000..b96dcb048
--- /dev/null
+++ b/models_v2/pytorch/dlrm/inference/gpu/LICENSE
@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) Facebook, Inc. and its affiliates.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
diff --git a/models_v2/pytorch/dlrm/inference/gpu/README.md b/models_v2/pytorch/dlrm/inference/gpu/README.md
new file mode 100644
index 000000000..ab1139bf0
--- /dev/null
+++ b/models_v2/pytorch/dlrm/inference/gpu/README.md
@@ -0,0 +1,100 @@
+# DLRM v1 Inference
+
+DLRM v1 Inference best known configurations with Intel® Extension for PyTorch.
+
+## Model Information
+
+| **Use Case** | **Framework** | **Model Repo** | **Branch/Commit/Tag** | **Optional Patch** |
+|:---:| :---: |:--------------:|:---------------------:|:------------------:|
+|  Inference   |    PyTorch    |       https://github.com/facebookresearch/dlrm        |           -           |         -          |
+
+# Pre-Requisite
+* Host has [Intel® Data Center GPU Flex Series](https://ark.intel.com/content/www/us/en/ark/products/series/230021/intel-data-center-gpu-flex-series.html)
+* Host has installed latest Intel® Data Center GPU Flex Series Drivers https://dgpu-docs.intel.com/driver/installation.html
+* Host has installed latest Intel® Data Center GPU Max & Flex Series Drivers https://dgpu-docs.intel.com/driver/installation.html
+* The following Intel® oneAPI Base Toolkit components are required:
+  - Intel® oneAPI DPC++ Compiler (Placeholder DPCPPROOT as its installation path)
+  - Intel® oneAPI Math Kernel Library (oneMKL) (Placeholder MKLROOT as its installation path)
+  - Intel® oneAPI MPI Library
+  - Intel® oneAPI TBB Library
+
+  Follow instructions at [Intel® oneAPI Base Toolkit Download page](https://www.intel.com/content/www/us/en/developer/tools/oneapi/base-toolkit-download.html?operatingsystem=linux) to setup the package manager repository.
+
+# Prepare Dataset
+The code supports interface with the [Criteo Kaggle Display Advertising Challenge Dataset](https://ailab.criteo.com/ressources/).
+   - Please do the following to prepare the dataset for use with DLRM code:
+     - First, specify the raw data file (train.txt) as downloaded with
+     - This is then pre-processed (categorize, concat across days...) to allow using with dlrm code
+     - The processed data is stored as *.npz file
+datset dir need have train.txt and kaggleAdDisplayChallenge_processed.npz
+
+you can get the checkpoints by running the command
+./bench/dlrm_s_criteo_kaggle.sh [--test-freq=1024]
+
+
+## Inference
+1. `git clone https://github.com/IntelAI/models.git`
+2. `cd models/models_v2/pytorch/dlrm/inference/gpu`
+3. Create virtual environment `venv` and activate it:
+    ```
+    python3 -m venv venv
+    . ./venv/bin/activate
+    ```
+4. Run setup.sh
+    ```
+    ./setup.sh
+    ```
+5. Install the latest GPU versions of [torch, torchvision and intel_extension_for_pytorch](https://intel.github.io/intel-extension-for-pytorch/index.html#installation):
+  ```
+  python -m pip install torch==<torch_version> torchvision==<torchvvision_version> intel-extension-for-pytorch==<ipex_version> --extra-index-url https://pytorch-extension.intel.com/release-whl-aitools/
+  ```
+6. Set environment variables for Intel® oneAPI Base Toolkit: 
+    Default installation location `{ONEAPI_ROOT}` is `/opt/intel/oneapi` for root account, `${HOME}/intel/oneapi` for other accounts
+    ```bash
+    source {ONEAPI_ROOT}/compiler/latest/env/vars.sh
+    source {ONEAPI_ROOT}/mkl/latest/env/vars.sh
+    source {ONEAPI_ROOT}/tbb/latest/env/vars.sh
+    source {ONEAPI_ROOT}/mpi/latest/env/vars.sh
+    source {ONEAPI_ROOT}/ccl/latest/env/vars.sh
+    ```
+7. Setup required environment paramaters
+
+| **Parameter**                |                                  **export command**                                  |
+|:---------------------------:|:------------------------------------------------------------------------------------:|
+| **MULTI_TILE**               | `export MULTI_TILE=False` (False)                                             |
+| **PLATFORM**                 | `export PLATFORM=Flex` (Flex)                                                 |
+| **DATASET_DIR**              |                               `export DATASET_DIR=`                                  |
+| **CKPT_DIR**                 |                               `export CKPT_DIR=`                                     |
+| **BATCH_SIZE** (optional)    |                               `export BATCH_SIZE=32768`                                |
+| **PRECISION** (optional)     |                               `export PRECISION=fp16` (fp16 and fp32 for Flex)                               |
+| **OUTPUT_DIR** (optional)    |                               `export OUTPUT_DIR=$PWD`                               |
+|**NUM_ITERATIONS** (optional) |                               `export NUM_ITERATIONS=20`                             |
+8. Run `run_model.sh`
+
+> [!NOTE]
+> Refer to [CONTAINER.md](CONTAINER.md) for DLRM-v1 inference instructions using docker containers.
+## Output
+
+Single-tile output will typically look like:
+
+```
+accuracy 76.215 %, best 76.215 %
+dlrm_inf latency:  0.11193203926086426  s
+dlrm_inf avg time:  0.007462135950724284  s, ant the time count is : 15
+dlrm_inf throughput:  4391235.996821996  samples/s
+```
+
+
+Final results of the inference run can be found in `results.yaml` file.
+```
+results:
+ - key: throughput
+   value: 4391236.0
+   unit: inst/s
+ - key: latency
+   value: 0.007462135950724283
+   unit: s
+ - key: accuracy
+   value: 76.215
+   unit: accuracy
+```
diff --git a/models_v2/pytorch/dlrm/inference/gpu/bench/dlrm_s_benchmark.sh b/models_v2/pytorch/dlrm/inference/gpu/bench/dlrm_s_benchmark.sh
new file mode 100755
index 000000000..612f439de
--- /dev/null
+++ b/models_v2/pytorch/dlrm/inference/gpu/bench/dlrm_s_benchmark.sh
@@ -0,0 +1,147 @@
+#!/bin/bash
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+#check if extra argument is passed to the test
+if [[ $# == 1 ]]; then
+    dlrm_extra_option=$1
+else
+    dlrm_extra_option=""
+fi
+#echo $dlrm_extra_option
+
+cpu=1
+gpu=1
+pt=1
+c2=1
+
+ncores=28 #12 #6
+nsockets="0"
+
+ngpus="1 2 4 8"
+
+numa_cmd="numactl --physcpubind=0-$((ncores-1)) -m $nsockets" #run on one socket, without HT
+dlrm_pt_bin="python dlrm_s_pytorch.py"
+dlrm_c2_bin="python dlrm_s_caffe2.py"
+
+data=random #synthetic
+print_freq=100
+rand_seed=727
+
+c2_net="async_scheduling"
+
+#Model param
+mb_size=2048 #1024 #512 #256
+nbatches=1000 #500 #100
+bot_mlp="512-512-64"
+top_mlp="1024-1024-1024-1"
+emb_size=64
+nindices=100
+emb="1000000-1000000-1000000-1000000-1000000-1000000-1000000-1000000"
+interaction="dot"
+tnworkers=0
+tmb_size=16384
+
+#_args="--mini-batch-size="${mb_size}\
+_args=" --num-batches="${nbatches}\
+" --data-generation="${data}\
+" --arch-mlp-bot="${bot_mlp}\
+" --arch-mlp-top="${top_mlp}\
+" --arch-sparse-feature-size="${emb_size}\
+" --arch-embedding-size="${emb}\
+" --num-indices-per-lookup="${nindices}\
+" --arch-interaction-op="${interaction}\
+" --numpy-rand-seed="${rand_seed}\
+" --print-freq="${print_freq}\
+" --print-time"\
+" --enable-profiling "
+
+c2_args=" --caffe2-net-type="${c2_net}
+
+
+# CPU Benchmarking
+if [ $cpu = 1 ]; then
+  echo "--------------------------------------------"
+  echo "CPU Benchmarking - running on $ncores cores"
+  echo "--------------------------------------------"
+  if [ $pt = 1 ]; then
+    outf="model1_CPU_PT_$ncores.log"
+    outp="dlrm_s_pytorch.prof"
+    echo "-------------------------------"
+    echo "Running PT (log file: $outf)"
+    echo "-------------------------------"
+    cmd="$numa_cmd $dlrm_pt_bin --mini-batch-size=$mb_size --test-mini-batch-size=$tmb_size --test-num-workers=$tnworkers $_args $dlrm_extra_option > $outf"
+    echo $cmd
+    eval $cmd
+    min=$(grep "iteration" $outf | awk 'BEGIN{best=999999} {if (best > $7) best=$7} END{print best}')
+    echo "Min time per iteration = $min"
+    # move profiling file(s)
+    mv $outp ${outf//".log"/".prof"}
+    mv ${outp//".prof"/".json"} ${outf//".log"/".json"}
+
+  fi
+  if [ $c2 = 1 ]; then
+    outf="model1_CPU_C2_$ncores.log"
+    outp="dlrm_s_caffe2.prof"
+    echo "-------------------------------"
+    echo "Running C2 (log file: $outf)"
+    echo "-------------------------------"
+    cmd="$numa_cmd $dlrm_c2_bin --mini-batch-size=$mb_size $_args $c2_args $dlrm_extra_option 1> $outf 2> $outp"
+    echo $cmd
+    eval $cmd
+    min=$(grep "iteration" $outf | awk 'BEGIN{best=999999} {if (best > $7) best=$7} END{print best}')
+    echo "Min time per iteration = $min"
+    # move profiling file (collected from stderr above)
+    mv $outp ${outf//".log"/".prof"}
+  fi
+fi
+
+# GPU Benchmarking
+if [ $gpu = 1 ]; then
+  echo "--------------------------------------------"
+  echo "GPU Benchmarking - running on $ngpus GPUs"
+  echo "--------------------------------------------"
+  for _ng in $ngpus
+  do
+    # weak scaling
+    # _mb_size=$((mb_size*_ng))
+    # strong scaling
+    _mb_size=$((mb_size*1))
+    _gpus=$(seq -s, 0 $((_ng-1)))
+    cuda_arg="CUDA_VISIBLE_DEVICES=$_gpus"
+    echo "-------------------"
+    echo "Using GPUS: "$_gpus
+    echo "-------------------"
+    if [ $pt = 1 ]; then
+      outf="model1_GPU_PT_$_ng.log"
+      outp="dlrm_s_pytorch.prof"
+      echo "-------------------------------"
+      echo "Running PT (log file: $outf)"
+      echo "-------------------------------"
+      cmd="$cuda_arg $dlrm_pt_bin --mini-batch-size=$_mb_size --test-mini-batch-size=$tmb_size --test-num-workers=$tnworkers $_args --use-gpu $dlrm_extra_option > $outf"
+      echo $cmd
+      eval $cmd
+      min=$(grep "iteration" $outf | awk 'BEGIN{best=999999} {if (best > $7) best=$7} END{print best}')
+      echo "Min time per iteration = $min"
+      # move profiling file(s)
+      mv $outp ${outf//".log"/".prof"}
+      mv ${outp//".prof"/".json"} ${outf//".log"/".json"}
+    fi
+    if [ $c2 = 1 ]; then
+      outf="model1_GPU_C2_$_ng.log"
+      outp="dlrm_s_caffe2.prof"
+      echo "-------------------------------"
+      echo "Running C2 (log file: $outf)"
+      echo "-------------------------------"
+      cmd="$cuda_arg $dlrm_c2_bin --mini-batch-size=$_mb_size $_args $c2_args --use-gpu $dlrm_extra_option 1> $outf 2> $outp"
+      echo $cmd
+      eval $cmd
+      min=$(grep "iteration" $outf | awk 'BEGIN{best=999999} {if (best > $7) best=$7} END{print best}')
+      echo "Min time per iteration = $min"
+      # move profiling file (collected from stderr above)
+      mv $outp ${outf//".log"/".prof"}
+    fi
+  done
+fi
diff --git a/models_v2/pytorch/dlrm/inference/gpu/bench/dlrm_s_criteo_kaggle.sh b/models_v2/pytorch/dlrm/inference/gpu/bench/dlrm_s_criteo_kaggle.sh
new file mode 100755
index 000000000..1b85886e0
--- /dev/null
+++ b/models_v2/pytorch/dlrm/inference/gpu/bench/dlrm_s_criteo_kaggle.sh
@@ -0,0 +1,32 @@
+#!/bin/bash
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+#WARNING: must have compiled PyTorch and caffe2
+
+#check if extra argument is passed to the test
+if [[ $# == 1 ]]; then
+    dlrm_extra_option=$1
+else
+    dlrm_extra_option=""
+fi
+#echo $dlrm_extra_option
+
+dlrm_pt_bin="python dlrm_s_pytorch.py"
+dlrm_c2_bin="python dlrm_s_caffe2.py"
+
+echo "run pytorch ..."
+# WARNING: the following parameters will be set based on the data set
+# --arch-embedding-size=... (sparse feature sizes)
+# --arch-mlp-bot=... (the input to the first layer of bottom mlp)
+$dlrm_pt_bin --arch-sparse-feature-size=16 --arch-mlp-bot="13-512-256-64-16" --arch-mlp-top="512-256-1" --data-generation=dataset --data-set=kaggle --raw-data-file=./input/train.txt --processed-data-file=./input/kaggleAdDisplayChallenge_processed.npz --loss-function=bce --round-targets=True --learning-rate=0.1 --mini-batch-size=128 --print-freq=1024 --print-time --test-mini-batch-size=16384 --test-num-workers=16 $dlrm_extra_option 2>&1 | tee run_kaggle_pt.log
+
+echo "run caffe2 ..."
+# WARNING: the following parameters will be set based on the data set
+# --arch-embedding-size=... (sparse feature sizes)
+# --arch-mlp-bot=... (the input to the first layer of bottom mlp)
+$dlrm_c2_bin --arch-sparse-feature-size=16 --arch-mlp-bot="13-512-256-64-16" --arch-mlp-top="512-256-1" --data-generation=dataset --data-set=kaggle --raw-data-file=./input/train.txt --processed-data-file=./input/kaggleAdDisplayChallenge_processed.npz --loss-function=bce --round-targets=True --learning-rate=0.1 --mini-batch-size=128 --print-freq=1024 --print-time $dlrm_extra_option 2>&1 | tee run_kaggle_c2.log
+
+echo "done"
diff --git a/models_v2/pytorch/dlrm/inference/gpu/bench/dlrm_s_criteo_terabyte.sh b/models_v2/pytorch/dlrm/inference/gpu/bench/dlrm_s_criteo_terabyte.sh
new file mode 100755
index 000000000..28cad418d
--- /dev/null
+++ b/models_v2/pytorch/dlrm/inference/gpu/bench/dlrm_s_criteo_terabyte.sh
@@ -0,0 +1,32 @@
+#!/bin/bash
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+#WARNING: must have compiled PyTorch and caffe2
+
+#check if extra argument is passed to the test
+if [[ $# == 1 ]]; then
+    dlrm_extra_option=$1
+else
+    dlrm_extra_option=""
+fi
+#echo $dlrm_extra_option
+
+dlrm_pt_bin="python dlrm_s_pytorch.py"
+dlrm_c2_bin="python dlrm_s_caffe2.py"
+
+echo "run pytorch ..."
+# WARNING: the following parameters will be set based on the data set
+# --arch-embedding-size=... (sparse feature sizes)
+# --arch-mlp-bot=... (the input to the first layer of bottom mlp)
+$dlrm_pt_bin --arch-sparse-feature-size=64 --arch-mlp-bot="13-512-256-64" --arch-mlp-top="512-512-256-1" --max-ind-range=10000000 --data-generation=dataset --data-set=terabyte --raw-data-file=./input/day --processed-data-file=./input/terabyte_processed.npz --loss-function=bce --round-targets=True --learning-rate=0.1 --mini-batch-size=2048 --print-freq=1024 --print-time --test-mini-batch-size=16384 --test-num-workers=16 $dlrm_extra_option 2>&1 | tee run_terabyte_pt.log
+
+echo "run caffe2 ..."
+# WARNING: the following parameters will be set based on the data set
+# --arch-embedding-size=... (sparse feature sizes)
+# --arch-mlp-bot=... (the input to the first layer of bottom mlp)
+$dlrm_c2_bin --arch-sparse-feature-size=64 --arch-mlp-bot="13-512-256-64" --arch-mlp-top="512-512-256-1" --max-ind-range=10000000 --data-generation=dataset --data-set=terabyte --raw-data-file=./input/day --processed-data-file=./input/terabyte_processed.npz --loss-function=bce --round-targets=True --learning-rate=0.1 --mini-batch-size=2048 --print-freq=1024 --print-time $dlrm_extra_option 2>&1 | tee run_terabyte_c2.log
+
+echo "done"
diff --git a/models_v2/pytorch/dlrm/inference/gpu/bench/run_and_time.sh b/models_v2/pytorch/dlrm/inference/gpu/bench/run_and_time.sh
new file mode 100755
index 000000000..3522f7df4
--- /dev/null
+++ b/models_v2/pytorch/dlrm/inference/gpu/bench/run_and_time.sh
@@ -0,0 +1,19 @@
+#!/bin/bash
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+#WARNING: must have compiled PyTorch and caffe2
+
+#check if extra argument is passed to the test
+if [[ $# == 1 ]]; then
+    dlrm_extra_option=$1
+else
+    dlrm_extra_option=""
+fi
+#echo $dlrm_extra_option
+
+python dlrm_s_pytorch.py --arch-sparse-feature-size=128 --arch-mlp-bot="13-512-256-128" --arch-mlp-top="1024-1024-512-256-1" --max-ind-range=40000000 --data-generation=dataset --data-set=terabyte --raw-data-file=./input/day --processed-data-file=./input/terabyte_processed.npz --loss-function=bce --round-targets=True --learning-rate=1.0 --mini-batch-size=2048 --print-freq=2048 --print-time --test-freq=102400 --test-mini-batch-size=16384 --test-num-workers=16 --memory-map --mlperf-logging --mlperf-auc-threshold=0.8025 --mlperf-bin-loader --mlperf-bin-shuffle $dlrm_extra_option 2>&1 | tee run_terabyte_mlperf_pt.log
+
+echo "done"
diff --git a/models_v2/pytorch/dlrm/inference/gpu/data_loader_terabyte.py b/models_v2/pytorch/dlrm/inference/gpu/data_loader_terabyte.py
new file mode 100644
index 000000000..3895e2399
--- /dev/null
+++ b/models_v2/pytorch/dlrm/inference/gpu/data_loader_terabyte.py
@@ -0,0 +1,407 @@
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# @lint-ignore-every LICENSELINT
+
+# Copyright (c) 2019, NVIDIA CORPORATION.  All rights reserved.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+
+from __future__ import absolute_import, division, print_function, unicode_literals
+
+import argparse
+import math
+
+import os
+import time
+
+import numpy as np
+import torch
+from torch.utils.data import Dataset
+from tqdm import tqdm
+
+
+class DataLoader:
+    """
+    DataLoader dedicated for the Criteo Terabyte Click Logs dataset
+    """
+
+    def __init__(
+        self,
+        data_filename,
+        data_directory,
+        days,
+        batch_size,
+        max_ind_range=-1,
+        split="train",
+        drop_last_batch=False,
+    ):
+        self.data_filename = data_filename
+        self.data_directory = data_directory
+        self.days = days
+        self.batch_size = batch_size
+        self.max_ind_range = max_ind_range
+
+        total_file = os.path.join(data_directory, data_filename + "_day_count.npz")
+        with np.load(total_file) as data:
+            total_per_file = data["total_per_file"][np.array(days)]
+
+        self.length = sum(total_per_file)
+        if split == "test" or split == "val":
+            self.length = int(np.ceil(self.length / 2.0))
+        self.split = split
+        self.drop_last_batch = drop_last_batch
+
+    def __iter__(self):
+        return iter(
+            _batch_generator(
+                self.data_filename,
+                self.data_directory,
+                self.days,
+                self.batch_size,
+                self.split,
+                self.drop_last_batch,
+                self.max_ind_range,
+            )
+        )
+
+    def __len__(self):
+        if self.drop_last_batch:
+            return self.length // self.batch_size
+        else:
+            return math.ceil(self.length / self.batch_size)
+
+
+def _transform_features(
+    x_int_batch, x_cat_batch, y_batch, max_ind_range, flag_input_torch_tensor=False
+):
+    if max_ind_range > 0:
+        x_cat_batch = x_cat_batch % max_ind_range
+
+    if flag_input_torch_tensor:
+        x_int_batch = torch.log(x_int_batch.clone().detach().type(torch.float) + 1)
+        x_cat_batch = x_cat_batch.clone().detach().type(torch.long)
+        y_batch = y_batch.clone().detach().type(torch.float32).view(-1, 1)
+    else:
+        x_int_batch = torch.log(torch.tensor(x_int_batch, dtype=torch.float) + 1)
+        x_cat_batch = torch.tensor(x_cat_batch, dtype=torch.long)
+        y_batch = torch.tensor(y_batch, dtype=torch.float32).view(-1, 1)
+
+    batch_size = x_cat_batch.shape[0]
+    feature_count = x_cat_batch.shape[1]
+    lS_o = torch.arange(batch_size).reshape(1, -1).repeat(feature_count, 1)
+
+    return x_int_batch, lS_o, x_cat_batch.t(), y_batch.view(-1, 1)
+
+
+def _batch_generator(
+    data_filename, data_directory, days, batch_size, split, drop_last, max_ind_range
+):
+    previous_file = None
+    for day in days:
+        filepath = os.path.join(
+            data_directory, data_filename + "_{}_reordered.npz".format(day)
+        )
+
+        # print('Loading file: ', filepath)
+        with np.load(filepath) as data:
+            x_int = data["X_int"]
+            x_cat = data["X_cat"]
+            y = data["y"]
+
+        samples_in_file = y.shape[0]
+        batch_start_idx = 0
+        if split == "test" or split == "val":
+            length = int(np.ceil(samples_in_file / 2.0))
+            if split == "test":
+                samples_in_file = length
+            elif split == "val":
+                batch_start_idx = samples_in_file - length
+
+        while batch_start_idx < samples_in_file - batch_size:
+
+            missing_samples = batch_size
+            if previous_file is not None:
+                missing_samples -= previous_file["y"].shape[0]
+
+            current_slice = slice(batch_start_idx, batch_start_idx + missing_samples)
+
+            x_int_batch = x_int[current_slice]
+            x_cat_batch = x_cat[current_slice]
+            y_batch = y[current_slice]
+
+            if previous_file is not None:
+                x_int_batch = np.concatenate(
+                    [previous_file["x_int"], x_int_batch], axis=0
+                )
+                x_cat_batch = np.concatenate(
+                    [previous_file["x_cat"], x_cat_batch], axis=0
+                )
+                y_batch = np.concatenate([previous_file["y"], y_batch], axis=0)
+                previous_file = None
+
+            if x_int_batch.shape[0] != batch_size:
+                raise ValueError("should not happen")
+
+            yield _transform_features(x_int_batch, x_cat_batch, y_batch, max_ind_range)
+
+            batch_start_idx += missing_samples
+        if batch_start_idx != samples_in_file:
+            current_slice = slice(batch_start_idx, samples_in_file)
+            if previous_file is not None:
+                previous_file = {
+                    "x_int": np.concatenate(
+                        [previous_file["x_int"], x_int[current_slice]], axis=0
+                    ),
+                    "x_cat": np.concatenate(
+                        [previous_file["x_cat"], x_cat[current_slice]], axis=0
+                    ),
+                    "y": np.concatenate([previous_file["y"], y[current_slice]], axis=0),
+                }
+            else:
+                previous_file = {
+                    "x_int": x_int[current_slice],
+                    "x_cat": x_cat[current_slice],
+                    "y": y[current_slice],
+                }
+
+    if not drop_last:
+        yield _transform_features(
+            previous_file["x_int"],
+            previous_file["x_cat"],
+            previous_file["y"],
+            max_ind_range,
+        )
+
+
+def _test():
+    generator = _batch_generator(
+        data_filename="day",
+        data_directory="./input",
+        days=range(23),
+        split="train",
+        batch_size=2048,
+        drop_last=True,
+        max_ind_range=-1,
+    )
+    t1 = time.time()
+    for x_int, lS_o, x_cat, y in generator:
+        t2 = time.time()
+        time_diff = t2 - t1
+        t1 = t2
+        print(
+            "time {} x_int.shape: {} lS_o.shape: {} x_cat.shape: {} y.shape: {}".format(
+                time_diff, x_int.shape, lS_o.shape, x_cat.shape, y.shape
+            )
+        )
+
+
+class CriteoBinDataset(Dataset):
+    """Binary version of criteo dataset."""
+
+    def __init__(
+        self,
+        data_file,
+        counts_file,
+        batch_size=1,
+        max_ind_range=-1,
+        bytes_per_feature=4,
+        drop_last=True
+    ):
+        # dataset
+        self.tar_fea = 1  # single target
+        self.den_fea = 13  # 13 dense  features
+        self.spa_fea = 26  # 26 sparse features
+        self.tad_fea = self.tar_fea + self.den_fea
+        self.tot_fea = self.tad_fea + self.spa_fea
+
+        self.batch_size = batch_size
+        self.max_ind_range = max_ind_range
+        self.bytes_per_entry = bytes_per_feature * self.tot_fea * batch_size
+
+        data_file_size = os.path.getsize(data_file)
+        if drop_last:
+            self.num_entries = math.floor(os.path.getsize(data_file) / self.bytes_per_entry)
+        else:
+            self.num_entries = math.ceil(os.path.getsize(data_file) / self.bytes_per_entry)
+
+        bytes_per_sample = bytes_per_feature * self.tot_fea
+        self.num_samples = data_file_size // bytes_per_sample
+
+        print("data file:", data_file, "number of batches:", self.num_entries)
+        self.file = open(data_file, "rb")
+
+        with np.load(counts_file) as data:
+            self.counts = data["counts"]
+
+        # hardcoded for now
+        self.m_den = 13
+
+    def __len__(self):
+        return self.num_entries
+
+    def __getitem__(self, idx):
+        self.file.seek(idx * self.bytes_per_entry, 0)
+        raw_data = self.file.read(self.bytes_per_entry)
+        array = np.frombuffer(raw_data, dtype=np.int32)
+        tensor = torch.from_numpy(array).view((-1, self.tot_fea))
+
+        return _transform_features(
+            x_int_batch=tensor[:, 1:14],
+            x_cat_batch=tensor[:, 14:],
+            y_batch=tensor[:, 0],
+            max_ind_range=self.max_ind_range,
+            flag_input_torch_tensor=True,
+        )
+
+    def __del__(self):
+        self.file.close()
+
+
+def numpy_to_binary(input_files, output_file_path, split="train"):
+    """Convert the data to a binary format to be read with CriteoBinDataset."""
+
+    # WARNING - both categorical and numerical data must fit into int32 for
+    # the following code to work correctly
+
+    with open(output_file_path, "wb") as output_file:
+        if split == "train":
+            for input_file in input_files:
+                print("Processing file: ", input_file)
+
+                np_data = np.load(input_file)
+                np_data = np.concatenate(
+                    [np_data["y"].reshape(-1, 1), np_data["X_int"], np_data["X_cat"]],
+                    axis=1,
+                )
+                np_data = np_data.astype(np.int32)
+
+                output_file.write(np_data.tobytes())
+        else:
+            assert len(input_files) == 1
+            np_data = np.load(input_files[0])
+            np_data = np.concatenate(
+                [np_data["y"].reshape(-1, 1), np_data["X_int"], np_data["X_cat"]],
+                axis=1,
+            )
+            np_data = np_data.astype(np.int32)
+
+            samples_in_file = np_data.shape[0]
+            midpoint = int(np.ceil(samples_in_file / 2.0))
+            if split == "test":
+                begin = 0
+                end = midpoint
+            elif split == "val":
+                begin = midpoint
+                end = samples_in_file
+            else:
+                raise ValueError("Unknown split value: ", split)
+
+            output_file.write(np_data[begin:end].tobytes())
+
+
+def _preprocess(args):
+    train_files = [
+        "{}_{}_reordered.npz".format(args.input_data_prefix, day)
+        for day in range(0, 23)
+    ]
+
+    test_valid_file = args.input_data_prefix + "_23_reordered.npz"
+
+    os.makedirs(args.output_directory, exist_ok=True)
+    for split in ["train", "val", "test"]:
+        print("Running preprocessing for split =", split)
+
+        output_file = os.path.join(args.output_directory, "{}_data.bin".format(split))
+
+        input_files = train_files if split == "train" else [test_valid_file]
+        numpy_to_binary(
+            input_files=input_files, output_file_path=output_file, split=split
+        )
+
+
+def _test_bin():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--output_directory", required=True)
+    parser.add_argument("--input_data_prefix", required=True)
+    parser.add_argument("--split", choices=["train", "test", "val"], required=True)
+    args = parser.parse_args()
+
+    _preprocess(args)
+
+    binary_data_file = os.path.join(
+        args.output_directory, "{}_data.bin".format(args.split)
+    )
+
+    counts_file = os.path.join(args.output_directory, "day_fea_count.npz")
+    dataset_binary = CriteoBinDataset(
+        data_file=binary_data_file,
+        counts_file=counts_file,
+        batch_size=2048,
+    )
+    from dlrm_data_pytorch import (
+        collate_wrapper_criteo_offset as collate_wrapper_criteo,
+        CriteoDataset,
+    )
+
+    binary_loader = torch.utils.data.DataLoader(
+        dataset_binary,
+        batch_size=None,
+        shuffle=False,
+        num_workers=0,
+        collate_fn=None,
+        pin_memory=False,
+        drop_last=False,
+    )
+
+    original_dataset = CriteoDataset(
+        dataset="terabyte",
+        max_ind_range=10 * 1000 * 1000,
+        sub_sample_rate=1,
+        randomize=True,
+        split=args.split,
+        raw_path=args.input_data_prefix,
+        pro_data="dummy_string",
+        memory_map=True,
+    )
+
+    original_loader = torch.utils.data.DataLoader(
+        original_dataset,
+        batch_size=2048,
+        shuffle=False,
+        num_workers=0,
+        collate_fn=collate_wrapper_criteo,
+        pin_memory=False,
+        drop_last=False,
+    )
+
+    assert len(dataset_binary) == len(original_loader)
+    for i, (old_batch, new_batch) in tqdm(
+        enumerate(zip(original_loader, binary_loader)), total=len(dataset_binary)
+    ):
+
+        for j in range(len(new_batch)):
+            if not np.array_equal(old_batch[j], new_batch[j]):
+                raise ValueError("FAILED: Datasets not equal")
+        if i > len(dataset_binary):
+            break
+    print("PASSED")
+
+
+if __name__ == "__main__":
+    _test()
+    _test_bin()
diff --git a/models_v2/pytorch/dlrm/inference/gpu/data_utils.py b/models_v2/pytorch/dlrm/inference/gpu/data_utils.py
new file mode 100644
index 000000000..67c689bb2
--- /dev/null
+++ b/models_v2/pytorch/dlrm/inference/gpu/data_utils.py
@@ -0,0 +1,1324 @@
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+# Description: generate inputs and targets for the DLRM benchmark
+#
+# Utility function(s) to download and pre-process public data sets
+#   - Criteo Kaggle Display Advertising Challenge Dataset
+#     https://labs.criteo.com/2014/02/kaggle-display-advertising-challenge-dataset
+#   - Criteo Terabyte Dataset
+#     https://labs.criteo.com/2013/12/download-terabyte-click-logs
+#
+# After downloading dataset, run:
+#   getCriteoAdData(
+#       datafile="<path-to-train.txt>",
+#       o_filename=kaggleAdDisplayChallenge_processed.npz,
+#       max_ind_range=-1,
+#       sub_sample_rate=0.0,
+#       days=7,
+#       data_split='train',
+#       randomize='total',
+#       criteo_kaggle=True,
+#       memory_map=False
+#   )
+#   getCriteoAdData(
+#       datafile="<path-to-day_{0,...,23}>",
+#       o_filename=terabyte_processed.npz,
+#       max_ind_range=-1,
+#       sub_sample_rate=0.0,
+#       days=24,
+#       data_split='train',
+#       randomize='total',
+#       criteo_kaggle=False,
+#       memory_map=False
+#   )
+
+from __future__ import absolute_import, division, print_function, unicode_literals
+
+import sys
+from multiprocessing import Manager, Process
+
+# import os
+from os import path
+
+# import io
+# from io import StringIO
+# import collections as coll
+
+import numpy as np
+
+
+def convertUStringToDistinctIntsDict(mat, convertDicts, counts):
+    # Converts matrix of unicode strings into distinct integers.
+    #
+    # Inputs:
+    #     mat (np.array): array of unicode strings to convert
+    #     convertDicts (list): dictionary for each column
+    #     counts (list): number of different categories in each column
+    #
+    # Outputs:
+    #     out (np.array): array of output integers
+    #     convertDicts (list): dictionary for each column
+    #     counts (list): number of different categories in each column
+
+    # check if convertDicts and counts match correct length of mat
+    if len(convertDicts) != mat.shape[1] or len(counts) != mat.shape[1]:
+        print("Length of convertDicts or counts does not match input shape")
+        print("Generating convertDicts and counts...")
+
+        convertDicts = [{} for _ in range(mat.shape[1])]
+        counts = [0 for _ in range(mat.shape[1])]
+
+    # initialize output
+    out = np.zeros(mat.shape)
+
+    for j in range(mat.shape[1]):
+        for i in range(mat.shape[0]):
+            # add to convertDict and increment count
+            if mat[i, j] not in convertDicts[j]:
+                convertDicts[j][mat[i, j]] = counts[j]
+                counts[j] += 1
+            out[i, j] = convertDicts[j][mat[i, j]]
+
+    return out, convertDicts, counts
+
+
+def convertUStringToDistinctIntsUnique(mat, mat_uni, counts):
+    # mat is an array of 0,...,# samples, with each being 26 categorical features
+
+    # check if mat_unique and counts match correct length of mat
+    if len(mat_uni) != mat.shape[1] or len(counts) != mat.shape[1]:
+        print("Length of mat_unique or counts does not match input shape")
+        print("Generating mat_unique and counts...")
+
+        mat_uni = [np.array([]) for _ in range(mat.shape[1])]
+        counts = [0 for _ in range(mat.shape[1])]
+
+    # initialize output
+    out = np.zeros(mat.shape)
+    ind_map = [np.array([]) for _ in range(mat.shape[1])]
+
+    # find out and assign unique ids to features
+    for j in range(mat.shape[1]):
+        m = mat_uni[j].size
+        mat_concat = np.concatenate((mat_uni[j], mat[:, j]))
+        mat_uni[j], ind_map[j] = np.unique(mat_concat, return_inverse=True)
+        out[:, j] = ind_map[j][m:]
+        counts[j] = mat_uni[j].size
+
+    return out, mat_uni, counts
+
+
+def processCriteoAdData(d_path, d_file, npzfile, i, convertDicts, pre_comp_counts):
+    # Process Kaggle Display Advertising Challenge or Terabyte Dataset
+    # by converting unicode strings in X_cat to integers and
+    # converting negative integer values in X_int.
+    #
+    # Loads data in the form "{kaggle|terabyte}_day_i.npz" where i is the day.
+    #
+    # Inputs:
+    #   d_path (str): path for {kaggle|terabyte}_day_i.npz files
+    #   i (int): splits in the dataset (typically 0 to 7 or 0 to 24)
+
+    # process data if not all files exist
+    filename_i = npzfile + "_{0}_processed.npz".format(i)
+
+    if path.exists(filename_i):
+        print("Using existing " + filename_i, end="\n")
+    else:
+        print("Not existing " + filename_i)
+        with np.load(npzfile + "_{0}.npz".format(i)) as data:
+            # categorical features
+            """
+            # Approach 1a: using empty dictionaries
+            X_cat, convertDicts, counts = convertUStringToDistinctIntsDict(
+                data["X_cat"], convertDicts, counts
+            )
+            """
+            """
+            # Approach 1b: using empty np.unique
+            X_cat, convertDicts, counts = convertUStringToDistinctIntsUnique(
+                data["X_cat"], convertDicts, counts
+            )
+            """
+            # Approach 2a: using pre-computed dictionaries
+            X_cat_t = np.zeros(data["X_cat_t"].shape)
+            for j in range(26):
+                for k, x in enumerate(data["X_cat_t"][j, :]):
+                    X_cat_t[j, k] = convertDicts[j][x]
+            # continuous features
+            X_int = data["X_int"]
+            X_int[X_int < 0] = 0
+            # targets
+            y = data["y"]
+
+        np.savez_compressed(
+            filename_i,
+            # X_cat = X_cat,
+            X_cat=np.transpose(X_cat_t),  # transpose of the data
+            X_int=X_int,
+            y=y,
+        )
+        print("Processed " + filename_i, end="\n")
+    # sanity check (applicable only if counts have been pre-computed & are re-computed)
+    # for j in range(26):
+    #    if pre_comp_counts[j] != counts[j]:
+    #        sys.exit("ERROR: Sanity check on counts has failed")
+    # print("\nSanity check on counts passed")
+
+    return
+
+
+def concatCriteoAdData(
+    d_path,
+    d_file,
+    npzfile,
+    trafile,
+    days,
+    data_split,
+    randomize,
+    total_per_file,
+    total_count,
+    memory_map,
+    o_filename,
+):
+    # Concatenates different days and saves the result.
+    #
+    # Inputs:
+    #   days (int): total number of days in the dataset (typically 7 or 24)
+    #   d_path (str): path for {kaggle|terabyte}_day_i.npz files
+    #   o_filename (str): output file name
+    #
+    # Output:
+    #   o_file (str): output file path
+
+    if memory_map:
+        # dataset break up per fea
+        # tar_fea = 1   # single target
+        den_fea = 13  # 13 dense  features
+        spa_fea = 26  # 26 sparse features
+        # tad_fea = tar_fea + den_fea
+        # tot_fea = tad_fea + spa_fea
+        # create offset per file
+        offset_per_file = np.array([0] + [x for x in total_per_file])
+        for i in range(days):
+            offset_per_file[i + 1] += offset_per_file[i]
+
+        """
+        # Approach 1, 2 and 3 use indices, while Approach 4 does not use them
+        # create indices
+        indices = np.arange(total_count)
+        if data_split == "none":
+            if randomize == "total":
+                indices = np.random.permutation(indices)
+        else:
+            indices = np.array_split(indices, offset_per_file[1:-1])
+
+            # randomize train data (per day)
+            if randomize == "day":  # or randomize == "total":
+                for i in range(len(indices) - 1):
+                    indices[i] = np.random.permutation(indices[i])
+                print("Randomized indices per day ...")
+
+            train_indices = np.concatenate(indices[:-1])
+            test_indices = indices[-1]
+
+            # randomize train data (across days)
+            if randomize == "total":
+                train_indices = np.random.permutation(train_indices)
+                print("Randomized indices across days ...")
+
+            indices = np.concatenate((train_indices, test_indices))
+        # no reordering
+        # indices = np.arange(total_count)
+        """
+        """
+        # Approach 1: simple and slow (no grouping is used)
+        # check if data already exists
+        recreate_flag = False
+        for j in range(tot_fea):
+            filename_j = trafile + "_{0}_reordered.npy".format(j)
+            if path.exists(filename_j):
+                print("Using existing " + filename_j)
+            else:
+                recreate_flag = True
+        # load, reorder and concatenate data (memmap all reordered files per feature)
+        if recreate_flag:
+            # init reordered files (.npy appended automatically)
+            z = np.zeros((total_count))
+            for j in range(tot_fea):
+                filename_j = trafile + "_{0}_reordered".format(j)
+                np.save(filename_j, z)
+                print("Creating " + filename_j)
+
+            for i in range(days):
+                filename_i = d_path + npzfile + "_{0}_processed.npz".format(i)
+                with np.load(filename_i) as data:
+                    X_cat_t = np.transpose(data["X_cat"])
+                    X_int_t = np.transpose(data["X_int"])
+                    y = data["y"]
+                size = len(y)
+                # sanity check
+                if total_per_file[i] != size:
+                    sys.exit("ERROR: sanity check on number of samples failed")
+                # setup start and end ranges
+                start = offset_per_file[i]
+                end = offset_per_file[i + 1]
+                # print(filename_i)
+                # print("start=" + str(start) + " end=" + str(end)
+                #     + " diff=" + str(end - start) + "=" + str(total_per_file[i]))
+
+                for j in range(tot_fea):
+                    filename_j = trafile + "_{0}_reordered.npy".format(j)
+                    fj = np.load(filename_j, mmap_mode='r+')
+                    if j < tar_fea:
+                        fj[indices[start:end]] = y
+                    elif tar_fea <= j and j < tad_fea:
+                        fj[indices[start:end]] = X_int_t[j - tar_fea, :]
+                    else:
+                        fj[indices[start:end]] = X_cat_t[j - tad_fea, :]
+                    del fj
+        else:
+            print("Reordered fea files already exist, skipping ...")
+
+        # check if data already exists
+        recreate_flag = False
+        for i in range(days):
+            filename_i = d_path + npzfile + "_{0}_reordered.npz".format(i)
+            if path.exists(filename_i):
+                print("Using existing " + filename_i)
+            else:
+                recreate_flag = True
+        # split reordered data by files (memmap all reordered files per feature)
+        # on the day boundary del the file object and memmap again
+        if recreate_flag:
+            for i in range(days):
+                filename_i = d_path + npzfile + "_{0}_reordered.npz".format(i)
+                size = total_per_file[i]
+                X_int_t = np.zeros((den_fea, size))
+                X_cat_t = np.zeros((spa_fea, size))
+                # setup start and end ranges
+                start = offset_per_file[i]
+                end = offset_per_file[i + 1]
+                print("Creating " + filename_i)
+                # print("start=" + str(start) + " end=" + str(end)
+                #     + " diff=" + str(end - start) + "=" + str(total_per_file[i]))
+
+                for j in range(tot_fea):
+                    filename_j = trafile + "_{0}_reordered.npy".format(j)
+                    fj = np.load(filename_j, mmap_mode='r')
+                    if j < tar_fea:
+                        y = fj[start:end]
+                    elif tar_fea <= j and j < tad_fea:
+                        X_int_t[j - tar_fea, :] = fj[start:end]
+                    else:
+                        X_cat_t[j - tad_fea, :] = fj[start:end]
+                    del fj
+
+                np.savez_compressed(
+                    filename_i,
+                    X_cat=np.transpose(X_cat_t),  # transpose of the data
+                    X_int=np.transpose(X_int_t),  # transpose of the data
+                    y=y,
+                )
+        else:
+            print("Reordered day files already exist, skipping ...")
+        """
+        """
+        # Approach 2: group days
+        # check if data already exists
+        recreate_flag = False
+        for j in range(tot_fea):
+            filename_j = trafile + "_{0}_reordered.npy".format(j)
+            if path.exists(filename_j):
+                print("Using existing " + filename_j)
+            else:
+                recreate_flag = True
+        # load, reorder and concatenate data (memmap all reordered files per feature)
+        if recreate_flag:
+            # init reordered files (.npy appended automatically)
+            z = np.zeros((total_count))
+            for j in range(tot_fea):
+                filename_j = trafile + "_{0}_reordered".format(j)
+                np.save(filename_j, z)
+                print("Creating " + filename_j)
+
+            group_day = 3  # e.g. 8, 4 or 3
+            group_num = days // group_day
+            file_group = [i*group_day for i in range(group_num)] + [days]
+            for ii in range(group_num):
+                # for last may be group_size != group_num, therefore reset it below
+                group_size = file_group[ii + 1] - file_group[ii]
+                X_cat_t = [0]*group_size
+                X_int_t = [0]*group_size
+                y = [0]*group_size
+                start = [0]*group_size
+                end  = [0]*group_size
+                for ig in range(group_size):
+                    i = file_group[ii] + ig
+                    filename_i = d_path + npzfile + "_{0}_processed.npz".format(i)
+                    # setup start and end ranges
+                    start[ig] = offset_per_file[i]
+                    end[ig] = offset_per_file[i + 1]
+                    # print(filename_i)
+                    # load a group of files
+                    with np.load(filename_i) as data:
+                        X_cat_t[ig] = np.transpose(data["X_cat"])
+                        X_int_t[ig] = np.transpose(data["X_int"])
+                        y[ig] = data["y"]
+                    # sanity check
+                    if total_per_file[i] != len(y[ig]):
+                        sys.exit("ERROR: sanity check on number of samples failed")
+                # print("start=" + str(start) + " end=" + str(end)
+                #  + " diff=" + str(end[ig]-start[ig]) + "=" + str(total_per_file[i]))
+
+                for j in range(tot_fea):
+                    filename_j = trafile + "_{0}_reordered.npy".format(j)
+                    fj = np.load(filename_j, mmap_mode='r+')
+                    for ig in range(group_size):
+                        if j < tar_fea:
+                            fj[indices[start[ig]:end[ig]]] = y[ig]
+                        elif tar_fea <= j and j < tad_fea:
+                            fj[indices[start[ig]:end[ig]]] = X_int_t[ig][j - tar_fea, :]
+                        else:
+                            fj[indices[start[ig]:end[ig]]] = X_cat_t[ig][j - tad_fea, :]
+                    del fj
+        else:
+            print("Reordered fea files already exist, skipping ...")
+
+        # check if data already exists
+        recreate_flag = False
+        for i in range(days):
+            filename_i = d_path + npzfile + "_{0}_reordered.npz".format(i)
+            if path.exists(filename_i):
+                print("Using existing " + filename_i)
+            else:
+                recreate_flag = True
+        # split reordered data by files (memmap all reordered files per feature)
+        # on the day boundary del the file object and memmap again
+        if recreate_flag:
+            for ii in range(group_num):
+                # for last may be group_size != group_num, therefore reset it below
+                group_size = file_group[ii + 1] - file_group[ii]
+                X_cat_t= []; X_int_t = []
+                for ig in range(group_size):
+                    i = file_group[ii] + ig
+                    X_int_t.append(np.zeros((den_fea, total_per_file[i])))
+                    X_cat_t.append(np.zeros((spa_fea, total_per_file[i])))
+                y = [0]*group_size
+                start = [0]*group_size
+                end  = [0]*group_size
+
+                for j in range(tot_fea):
+                    filename_j = trafile + "_{0}_reordered.npy".format(j)
+                    fj = np.load(filename_j, mmap_mode='r')
+                    # load a group of files
+                    for ig in range(group_size):
+                        i = file_group[ii] + ig
+                        # setup start and end ranges
+                        start[ig] = offset_per_file[i]
+                        end[ig] = offset_per_file[i + 1]
+                        # load data for the group of files
+                        if j < tar_fea:
+                            y[ig] = fj[start[ig]:end[ig]]
+                        elif tar_fea <= j and j < tad_fea:
+                            X_int_t[ig][j - tar_fea, :] = fj[start[ig]:end[ig]]
+                        else:
+                            X_cat_t[ig][j - tad_fea, :] = fj[start[ig]:end[ig]]
+                    del fj
+
+                for ig in range(group_size):
+                    i = file_group[ii] + ig
+                    filename_i = d_path + npzfile + "_{0}_reordered.npz".format(i)
+                    print("Creating " + filename_i)
+                    np.savez_compressed(
+                        filename_i,
+                        X_cat=np.transpose(X_cat_t[ig]),  # transpose of the data
+                        X_int=np.transpose(X_int_t[ig]),  # transpose of the data
+                        y=y[ig],
+                    )
+        else:
+            print("Reordered day files already exist, skipping ...")
+        """
+        """
+        # Approach 3: group features
+        # check if data already exists
+        group_fea = 5  # e.g. 8, 5 or 4
+        group_num = tot_fea // group_fea
+        if tot_fea % group_fea != 0:  # sanity check
+            sys.exit("ERROR: the group_fea must divided tot_fea evenly.")
+        recreate_flag = False
+        for jn in range(group_num):
+            filename_j = trafile + "_{0}_reordered{1}.npy".format(
+                jn, group_fea
+            )
+            if path.exists(filename_j):
+                print("Using existing " + filename_j)
+            else:
+                recreate_flag = True
+        # load, reorder and concatenate data (memmap all reordered files per feature)
+        if recreate_flag:
+            # init reordered files (.npy appended automatically)
+            z = np.zeros((group_fea, total_count))
+            for jn in range(group_num):
+                filename_j = trafile + "_{0}_reordered{1}".format(
+                    jn, group_fea
+                )
+                np.save(filename_j, z)
+                print("Creating " + filename_j)
+
+            for i in range(days):
+                filename_i = d_path + npzfile + "_{0}_processed.npz".format(i)
+                with np.load(filename_i) as data:
+                    X_cat_t = np.transpose(data["X_cat"])
+                    X_int_t = np.transpose(data["X_int"])
+                    y = data["y"]
+                size = len(y)
+                # sanity check
+                if total_per_file[i] != size:
+                    sys.exit("ERROR: sanity check on number of samples failed")
+                # setup start and end ranges
+                start = offset_per_file[i]
+                end = offset_per_file[i + 1]
+                # print(filename_i)
+                # print("start=" + str(start) + " end=" + str(end)
+                #      + " diff=" + str(end - start) + "=" + str(total_per_file[i]))
+
+                for jn in range(group_num):
+                    filename_j = trafile + "_{0}_reordered{1}.npy".format(
+                        jn, group_fea
+                    )
+                    fj = np.load(filename_j, mmap_mode='r+')
+                    for jg in range(group_fea):
+                        j = jn * group_fea + jg
+                        # print("j=" + str(j) + " jn=" + str(jn) + " jg=" + str(jg))
+                        if j < tar_fea:
+                            fj[jg, indices[start:end]] = y
+                        elif tar_fea <= j and j < tad_fea:
+                            fj[jg, indices[start:end]] = X_int_t[j - tar_fea, :]
+                        else:
+                            fj[jg, indices[start:end]] = X_cat_t[j - tad_fea, :]
+                    del fj
+        else:
+            print("Reordered fea files already exist, skipping ...")
+
+        # check if data already exists
+        recreate_flag = False
+        for i in range(days):
+            filename_i = d_path + npzfile + "_{0}_reordered.npz".format(i)
+            if path.exists(filename_i):
+                print("Using existing" + filename_i)
+            else:
+                recreate_flag = True
+        # split reordered data by files (memmap all reordered files per feature)
+        # on the day boundary del the file object and memmap again
+        if recreate_flag:
+            for i in range(days):
+                filename_i = d_path + npzfile + "_{0}_reordered.npz".format(i)
+                size = total_per_file[i]
+                X_int_t = np.zeros((den_fea, size))
+                X_cat_t = np.zeros((spa_fea, size))
+                # setup start and end ranges
+                start = offset_per_file[i]
+                end = offset_per_file[i + 1]
+                print("Creating " + filename_i)
+                # print("start=" + str(start) + " end=" + str(end)
+                #      + " diff=" + str(end - start) + "=" + str(total_per_file[i]))
+
+                for jn in range(group_num):
+                    filename_j = trafile + "_{0}_reordered{1}.npy".format(
+                        jn, group_fea
+                    )
+                    fj = np.load(filename_j, mmap_mode='r')
+                    for jg in range(group_fea):
+                        j = jn * group_fea + jg
+                        # print("j=" + str(j) + " jn=" + str(jn) + " jg=" + str(jg))
+                        if j < tar_fea:
+                            y = fj[jg, start:end]
+                        elif tar_fea <= j and j < tad_fea:
+                            X_int_t[j - tar_fea, :] = fj[jg, start:end]
+                        else:
+                            X_cat_t[j - tad_fea, :] = fj[jg, start:end]
+                    del fj
+
+                np.savez_compressed(
+                    filename_i,
+                    X_cat=np.transpose(X_cat_t),  # transpose of the data
+                    X_int=np.transpose(X_int_t),  # transpose of the data
+                    y=y,
+                )
+
+        else:
+            print("Reordered day files already exist, skipping ...")
+        """
+
+        # Approach 4: Fisher-Yates-Rao (FYR) shuffle algorithm
+        # 1st pass of FYR shuffle
+        # check if data already exists
+        recreate_flag = False
+        for j in range(days):
+            filename_j_y = npzfile + "_{0}_intermediate_y.npy".format(j)
+            filename_j_d = npzfile + "_{0}_intermediate_d.npy".format(j)
+            filename_j_s = npzfile + "_{0}_intermediate_s.npy".format(j)
+            if (
+                path.exists(filename_j_y)
+                and path.exists(filename_j_d)
+                and path.exists(filename_j_s)
+            ):
+                print(
+                    "Using existing\n"
+                    + filename_j_y
+                    + "\n"
+                    + filename_j_d
+                    + "\n"
+                    + filename_j_s
+                )
+            else:
+                recreate_flag = True
+        # reorder across buckets using sampling
+        if recreate_flag:
+            # init intermediate files (.npy appended automatically)
+            for j in range(days):
+                filename_j_y = npzfile + "_{0}_intermediate_y".format(j)
+                filename_j_d = npzfile + "_{0}_intermediate_d".format(j)
+                filename_j_s = npzfile + "_{0}_intermediate_s".format(j)
+                np.save(filename_j_y, np.zeros((total_per_file[j])))
+                np.save(filename_j_d, np.zeros((total_per_file[j], den_fea)))
+                np.save(filename_j_s, np.zeros((total_per_file[j], spa_fea)))
+            # start processing files
+            total_counter = [0] * days
+            for i in range(days):
+                filename_i = npzfile + "_{0}_processed.npz".format(i)
+                with np.load(filename_i) as data:
+                    X_cat = data["X_cat"]
+                    X_int = data["X_int"]
+                    y = data["y"]
+                size = len(y)
+                # sanity check
+                if total_per_file[i] != size:
+                    sys.exit("ERROR: sanity check on number of samples failed")
+                # debug prints
+                print("Reordering (1st pass) " + filename_i)
+
+                # create buckets using sampling of random ints
+                # from (discrete) uniform distribution
+                buckets = []
+                for _j in range(days):
+                    buckets.append([])
+                counter = [0] * days
+                days_to_sample = days if data_split == "none" else days - 1
+                if randomize == "total":
+                    rand_u = np.random.randint(low=0, high=days_to_sample, size=size)
+                    for k in range(size):
+                        # sample and make sure elements per buckets do not overflow
+                        if data_split == "none" or i < days - 1:
+                            # choose bucket
+                            p = rand_u[k]
+                            # retry of the bucket is full
+                            while total_counter[p] + counter[p] >= total_per_file[p]:
+                                p = np.random.randint(low=0, high=days_to_sample)
+                        else:  # preserve the last day/bucket if needed
+                            p = i
+                        buckets[p].append(k)
+                        counter[p] += 1
+                else:  # randomize is day or none
+                    for k in range(size):
+                        # do not sample, preserve the data in this bucket
+                        p = i
+                        buckets[p].append(k)
+                        counter[p] += 1
+
+                # sanity check
+                if np.sum(counter) != size:
+                    sys.exit("ERROR: sanity check on number of samples failed")
+                # debug prints
+                # print(counter)
+                # print(str(np.sum(counter)) + " = " + str(size))
+                # print([len(x) for x in buckets])
+                # print(total_counter)
+
+                # partially feel the buckets
+                for j in range(days):
+                    filename_j_y = npzfile + "_{0}_intermediate_y.npy".format(j)
+                    filename_j_d = npzfile + "_{0}_intermediate_d.npy".format(j)
+                    filename_j_s = npzfile + "_{0}_intermediate_s.npy".format(j)
+                    start = total_counter[j]
+                    end = total_counter[j] + counter[j]
+                    # target buckets
+                    fj_y = np.load(filename_j_y, mmap_mode="r+")
+                    # print("start=" + str(start) + " end=" + str(end)
+                    #       + " end - start=" + str(end - start) + " "
+                    #       + str(fj_y[start:end].shape) + " "
+                    #       + str(len(buckets[j])))
+                    fj_y[start:end] = y[buckets[j]]
+                    del fj_y
+                    # dense buckets
+                    fj_d = np.load(filename_j_d, mmap_mode="r+")
+                    # print("start=" + str(start) + " end=" + str(end)
+                    #       + " end - start=" + str(end - start) + " "
+                    #       + str(fj_d[start:end, :].shape) + " "
+                    #       + str(len(buckets[j])))
+                    fj_d[start:end, :] = X_int[buckets[j], :]
+                    del fj_d
+                    # sparse buckets
+                    fj_s = np.load(filename_j_s, mmap_mode="r+")
+                    # print("start=" + str(start) + " end=" + str(end)
+                    #       + " end - start=" + str(end - start) + " "
+                    #       + str(fj_s[start:end, :].shape) + " "
+                    #       + str(len(buckets[j])))
+                    fj_s[start:end, :] = X_cat[buckets[j], :]
+                    del fj_s
+                    # update counters for next step
+                    total_counter[j] += counter[j]
+
+        # 2nd pass of FYR shuffle
+        # check if data already exists
+        for j in range(days):
+            filename_j = npzfile + "_{0}_reordered.npz".format(j)
+            if path.exists(filename_j):
+                print("Using existing " + filename_j)
+            else:
+                recreate_flag = True
+        # reorder within buckets
+        if recreate_flag:
+            for j in range(days):
+                filename_j_y = npzfile + "_{0}_intermediate_y.npy".format(j)
+                filename_j_d = npzfile + "_{0}_intermediate_d.npy".format(j)
+                filename_j_s = npzfile + "_{0}_intermediate_s.npy".format(j)
+                fj_y = np.load(filename_j_y)
+                fj_d = np.load(filename_j_d)
+                fj_s = np.load(filename_j_s)
+
+                indices = range(total_per_file[j])
+                if randomize == "day" or randomize == "total":
+                    if data_split == "none" or j < days - 1:
+                        indices = np.random.permutation(range(total_per_file[j]))
+
+                filename_r = npzfile + "_{0}_reordered.npz".format(j)
+                print("Reordering (2nd pass) " + filename_r)
+                np.savez_compressed(
+                    filename_r,
+                    X_cat=fj_s[indices, :],
+                    X_int=fj_d[indices, :],
+                    y=fj_y[indices],
+                )
+
+        """
+        # sanity check (under no reordering norms should be zero)
+        for i in range(days):
+            filename_i_o = npzfile + "_{0}_processed.npz".format(i)
+            print(filename_i_o)
+            with np.load(filename_i_o) as data_original:
+                X_cat_o = data_original["X_cat"]
+                X_int_o = data_original["X_int"]
+                y_o = data_original["y"]
+            filename_i_r = npzfile + "_{0}_reordered.npz".format(i)
+            print(filename_i_r)
+            with np.load(filename_i_r) as data_reordered:
+                X_cat_r = data_reordered["X_cat"]
+                X_int_r = data_reordered["X_int"]
+                y_r = data_reordered["y"]
+            print(np.linalg.norm(y_o - y_r))
+            print(np.linalg.norm(X_int_o - X_int_r))
+            print(np.linalg.norm(X_cat_o - X_cat_r))
+        """
+
+    else:
+        print("Concatenating multiple days into %s.npz file" % str(d_path + o_filename))
+
+        # load and concatenate data
+        for i in range(days):
+            filename_i = npzfile + "_{0}_processed.npz".format(i)
+            with np.load(filename_i) as data:
+                if i == 0:
+                    X_cat = data["X_cat"]
+                    X_int = data["X_int"]
+                    y = data["y"]
+                else:
+                    X_cat = np.concatenate((X_cat, data["X_cat"]))
+                    X_int = np.concatenate((X_int, data["X_int"]))
+                    y = np.concatenate((y, data["y"]))
+            print("Loaded day:", i, "y = 1:", len(y[y == 1]), "y = 0:", len(y[y == 0]))
+
+        with np.load(d_path + d_file + "_fea_count.npz") as data:
+            counts = data["counts"]
+        print("Loaded counts!")
+
+        np.savez_compressed(
+            d_path + o_filename + ".npz",
+            X_cat=X_cat,
+            X_int=X_int,
+            y=y,
+            counts=counts,
+        )
+
+    return d_path + o_filename + ".npz"
+
+
+def transformCriteoAdData(X_cat, X_int, y, days, data_split, randomize, total_per_file):
+    # Transforms Criteo Kaggle or terabyte data by applying log transformation
+    # on dense features and converting everything to appropriate tensors.
+    #
+    # Inputs:
+    #     X_cat (ndarray): array of integers corresponding to preprocessed
+    #                      categorical features
+    #     X_int (ndarray): array of integers corresponding to dense features
+    #     y (ndarray):     array of bool corresponding to labels
+    #     data_split(str): flag for splitting dataset into training/validation/test
+    #                      sets
+    #     randomize (str): determines randomization scheme
+    #         "none": no randomization
+    #         "day": randomizes each day"s data (only works if split = True)
+    #         "total": randomizes total dataset
+    #
+    # Outputs:
+    #     if split:
+    #         X_cat_train (tensor): sparse features for training set
+    #         X_int_train (tensor): dense features for training set
+    #         y_train (tensor): labels for training set
+    #         X_cat_val (tensor): sparse features for validation set
+    #         X_int_val (tensor): dense features for validation set
+    #         y_val (tensor): labels for validation set
+    #         X_cat_test (tensor): sparse features for test set
+    #         X_int_test (tensor): dense features for test set
+    #         y_test (tensor): labels for test set
+    #     else:
+    #         X_cat (tensor): sparse features
+    #         X_int (tensor): dense features
+    #         y (tensor): label
+
+    # define initial set of indices
+    indices = np.arange(len(y))
+
+    # create offset per file
+    offset_per_file = np.array([0] + [x for x in total_per_file])
+    for i in range(days):
+        offset_per_file[i + 1] += offset_per_file[i]
+
+    # split dataset
+    if data_split == "train":
+        indices = np.array_split(indices, offset_per_file[1:-1])
+
+        # randomize train data (per day)
+        if randomize == "day":  # or randomize == "total":
+            for i in range(len(indices) - 1):
+                indices[i] = np.random.permutation(indices[i])
+            print("Randomized indices per day ...")
+
+        train_indices = np.concatenate(indices[:-1])
+        test_indices = indices[-1]
+        test_indices, val_indices = np.array_split(test_indices, 2)
+
+        print("Defined training and testing indices...")
+
+        # randomize train data (across days)
+        if randomize == "total":
+            train_indices = np.random.permutation(train_indices)
+            print("Randomized indices across days ...")
+
+        # indices = np.concatenate((train_indices, test_indices))
+
+        # create training, validation, and test sets
+        X_cat_train = X_cat[train_indices]
+        X_int_train = X_int[train_indices]
+        y_train = y[train_indices]
+
+        X_cat_val = X_cat[val_indices]
+        X_int_val = X_int[val_indices]
+        y_val = y[val_indices]
+
+        X_cat_test = X_cat[test_indices]
+        X_int_test = X_int[test_indices]
+        y_test = y[test_indices]
+
+        print("Split data according to indices...")
+
+        X_cat_train = X_cat_train.astype(np.long)
+        X_int_train = np.log(X_int_train.astype(np.float32) + 1)
+        y_train = y_train.astype(np.float32)
+
+        X_cat_val = X_cat_val.astype(np.long)
+        X_int_val = np.log(X_int_val.astype(np.float32) + 1)
+        y_val = y_val.astype(np.float32)
+
+        X_cat_test = X_cat_test.astype(np.long)
+        X_int_test = np.log(X_int_test.astype(np.float32) + 1)
+        y_test = y_test.astype(np.float32)
+
+        print("Converted to tensors...done!")
+
+        return (
+            X_cat_train,
+            X_int_train,
+            y_train,
+            X_cat_val,
+            X_int_val,
+            y_val,
+            X_cat_test,
+            X_int_test,
+            y_test,
+        )
+
+    else:
+
+        # randomize data
+        if randomize == "total":
+            indices = np.random.permutation(indices)
+            print("Randomized indices...")
+
+        X_cat = X_cat[indices].astype(np.long)
+        X_int = np.log(X_int[indices].astype(np.float32) + 1)
+        y = y[indices].astype(np.float32)
+
+        print("Converted to tensors...done!")
+
+        return (X_cat, X_int, y, [], [], [], [], [], [])
+
+
+def getCriteoAdData(
+    datafile,
+    o_filename,
+    max_ind_range=-1,
+    sub_sample_rate=0.0,
+    days=7,
+    data_split="train",
+    randomize="total",
+    criteo_kaggle=True,
+    memory_map=False,
+    dataset_multiprocessing=False,
+):
+    # Passes through entire dataset and defines dictionaries for categorical
+    # features and determines the number of total categories.
+    #
+    # Inputs:
+    #    datafile : path to downloaded raw data file
+    #    o_filename (str): saves results under o_filename if filename is not ""
+    #
+    # Output:
+    #   o_file (str): output file path
+
+    # split the datafile into path and filename
+    lstr = datafile.split("/")
+    d_path = "/".join(lstr[0:-1]) + "/"
+    d_file = lstr[-1].split(".")[0] if criteo_kaggle else lstr[-1]
+    npzfile = d_path + ((d_file + "_day") if criteo_kaggle else d_file)
+    trafile = d_path + ((d_file + "_fea") if criteo_kaggle else "fea")
+
+    # count number of datapoints in training set
+    total_file = d_path + d_file + "_day_count.npz"
+    if path.exists(total_file):
+        with np.load(total_file) as data:
+            total_per_file = list(data["total_per_file"])
+        total_count = np.sum(total_per_file)
+        print("Skipping counts per file (already exist)")
+    else:
+        total_count = 0
+        total_per_file = []
+        if criteo_kaggle:
+            # WARNING: The raw data consists of a single train.txt file
+            # Each line in the file is a sample, consisting of 13 continuous and
+            # 26 categorical features (an extra space indicates that feature is
+            # missing and will be interpreted as 0).
+            if path.exists(datafile):
+                print("Reading data from path=%s" % (datafile))
+                with open(str(datafile)) as f:
+                    for _ in f:
+                        total_count += 1
+                total_per_file.append(total_count)
+                # reset total per file due to split
+                num_data_per_split, extras = divmod(total_count, days)
+                total_per_file = [num_data_per_split] * days
+                for j in range(extras):
+                    total_per_file[j] += 1
+                # split into days (simplifies code later on)
+                file_id = 0
+                boundary = total_per_file[file_id]
+                nf = open(npzfile + "_" + str(file_id), "w")
+                with open(str(datafile)) as f:
+                    for j, line in enumerate(f):
+                        if j == boundary:
+                            nf.close()
+                            file_id += 1
+                            nf = open(npzfile + "_" + str(file_id), "w")
+                            boundary += total_per_file[file_id]
+                        nf.write(line)
+                nf.close()
+            else:
+                sys.exit(
+                    "ERROR: Criteo Kaggle Display Ad Challenge Dataset path is invalid; please download from https://labs.criteo.com/2014/02/kaggle-display-advertising-challenge-dataset"
+                )
+        else:
+            # WARNING: The raw data consist of day_0.gz,... ,day_23.gz text files
+            # Each line in the file is a sample, consisting of 13 continuous and
+            # 26 categorical features (an extra space indicates that feature is
+            # missing and will be interpreted as 0).
+            for i in range(days):
+                datafile_i = datafile + "_" + str(i)  # + ".gz"
+                if path.exists(str(datafile_i)):
+                    print("Reading data from path=%s" % (str(datafile_i)))
+                    # file day_<number>
+                    total_per_file_count = 0
+                    with open(str(datafile_i)) as f:
+                        for _ in f:
+                            total_per_file_count += 1
+                    total_per_file.append(total_per_file_count)
+                    total_count += total_per_file_count
+                else:
+                    sys.exit(
+                        "ERROR: Criteo Terabyte Dataset path is invalid; please download from https://labs.criteo.com/2013/12/download-terabyte-click-logs"
+                    )
+
+    # process a file worth of data and reinitialize data
+    # note that a file main contain a single or multiple splits
+    def process_one_file(
+        datfile,
+        npzfile,
+        split,
+        num_data_in_split,
+        dataset_multiprocessing,
+        convertDictsDay=None,
+        resultDay=None,
+    ):
+        if dataset_multiprocessing:
+            convertDicts_day = [{} for _ in range(26)]
+
+        with open(str(datfile)) as f:
+            y = np.zeros(num_data_in_split, dtype="i4")  # 4 byte int
+            X_int = np.zeros((num_data_in_split, 13), dtype="i4")  # 4 byte int
+            X_cat = np.zeros((num_data_in_split, 26), dtype="i4")  # 4 byte int
+            if sub_sample_rate == 0.0:
+                rand_u = 1.0
+            else:
+                rand_u = np.random.uniform(low=0.0, high=1.0, size=num_data_in_split)
+
+            i = 0
+            percent = 0
+            for k, line in enumerate(f):
+                # process a line (data point)
+                line = line.split("\t")
+                # set missing values to zero
+                for j in range(len(line)):
+                    if (line[j] == "") or (line[j] == "\n"):
+                        line[j] = "0"
+                # sub-sample data by dropping zero targets, if needed
+                target = np.int32(line[0])
+                if (
+                    target == 0
+                    and (rand_u if sub_sample_rate == 0.0 else rand_u[k])
+                    < sub_sample_rate
+                ):
+                    continue
+
+                y[i] = target
+                X_int[i] = np.array(line[1:14], dtype=np.int32)
+                if max_ind_range > 0:
+                    X_cat[i] = np.array(
+                        list(map(lambda x: int(x, 16) % max_ind_range, line[14:])),
+                        dtype=np.int32,
+                    )
+                else:
+                    X_cat[i] = np.array(
+                        list(map(lambda x: int(x, 16), line[14:])), dtype=np.int32
+                    )
+
+                # count uniques
+                if dataset_multiprocessing:
+                    for j in range(26):
+                        convertDicts_day[j][X_cat[i][j]] = 1
+                    # debug prints
+                    if float(i) / num_data_in_split * 100 > percent + 1:
+                        percent = int(float(i) / num_data_in_split * 100)
+                        print(
+                            "Load %d/%d (%d%%) Split: %d  Label True: %d  Stored: %d"
+                            % (
+                                i,
+                                num_data_in_split,
+                                percent,
+                                split,
+                                target,
+                                y[i],
+                            ),
+                            end="\n",
+                        )
+                else:
+                    for j in range(26):
+                        convertDicts[j][X_cat[i][j]] = 1
+                    # debug prints
+                    print(
+                        "Load %d/%d  Split: %d  Label True: %d  Stored: %d"
+                        % (
+                            i,
+                            num_data_in_split,
+                            split,
+                            target,
+                            y[i],
+                        ),
+                        end="\r",
+                    )
+                i += 1
+
+            # store num_data_in_split samples or extras at the end of file
+            # count uniques
+            # X_cat_t  = np.transpose(X_cat)
+            # for j in range(26):
+            #     for x in X_cat_t[j,:]:
+            #         convertDicts[j][x] = 1
+            # store parsed
+            filename_s = npzfile + "_{0}.npz".format(split)
+            if path.exists(filename_s):
+                print("\nSkip existing " + filename_s)
+            else:
+                np.savez_compressed(
+                    filename_s,
+                    X_int=X_int[0:i, :],
+                    # X_cat=X_cat[0:i, :],
+                    X_cat_t=np.transpose(X_cat[0:i, :]),  # transpose of the data
+                    y=y[0:i],
+                )
+                print("\nSaved " + npzfile + "_{0}.npz!".format(split))
+
+        if dataset_multiprocessing:
+            resultDay[split] = i
+            convertDictsDay[split] = convertDicts_day
+            return
+        else:
+            return i
+
+    # create all splits (reuse existing files if possible)
+    recreate_flag = False
+    convertDicts = [{} for _ in range(26)]
+    # WARNING: to get reproducable sub-sampling results you must reset the seed below
+    # np.random.seed(123)
+    # in this case there is a single split in each day
+    for i in range(days):
+        npzfile_i = npzfile + "_{0}.npz".format(i)
+        npzfile_p = npzfile + "_{0}_processed.npz".format(i)
+        if path.exists(npzfile_i):
+            print("Skip existing " + npzfile_i)
+        elif path.exists(npzfile_p):
+            print("Skip existing " + npzfile_p)
+        else:
+            recreate_flag = True
+
+    if recreate_flag:
+        if dataset_multiprocessing:
+            resultDay = Manager().dict()
+            convertDictsDay = Manager().dict()
+            processes = [
+                Process(
+                    target=process_one_file,
+                    name="process_one_file:%i" % i,
+                    args=(
+                        npzfile + "_{0}".format(i),
+                        npzfile,
+                        i,
+                        total_per_file[i],
+                        dataset_multiprocessing,
+                        convertDictsDay,
+                        resultDay,
+                    ),
+                )
+                for i in range(0, days)
+            ]
+            for process in processes:
+                process.start()
+            for process in processes:
+                process.join()
+            for day in range(days):
+                total_per_file[day] = resultDay[day]
+                print("Constructing convertDicts Split: {}".format(day))
+                convertDicts_tmp = convertDictsDay[day]
+                for i in range(26):
+                    for j in convertDicts_tmp[i]:
+                        convertDicts[i][j] = 1
+        else:
+            for i in range(days):
+                total_per_file[i] = process_one_file(
+                    npzfile + "_{0}".format(i),
+                    npzfile,
+                    i,
+                    total_per_file[i],
+                    dataset_multiprocessing,
+                )
+
+    # report and save total into a file
+    total_count = np.sum(total_per_file)
+    if not path.exists(total_file):
+        np.savez_compressed(total_file, total_per_file=total_per_file)
+    print("Total number of samples:", total_count)
+    print("Divided into days/splits:\n", total_per_file)
+
+    # dictionary files
+    counts = np.zeros(26, dtype=np.int32)
+    if recreate_flag:
+        # create dictionaries
+        for j in range(26):
+            for i, x in enumerate(convertDicts[j]):
+                convertDicts[j][x] = i
+            dict_file_j = d_path + d_file + "_fea_dict_{0}.npz".format(j)
+            if not path.exists(dict_file_j):
+                np.savez_compressed(
+                    dict_file_j, unique=np.array(list(convertDicts[j]), dtype=np.int32)
+                )
+            counts[j] = len(convertDicts[j])
+        # store (uniques and) counts
+        count_file = d_path + d_file + "_fea_count.npz"
+        if not path.exists(count_file):
+            np.savez_compressed(count_file, counts=counts)
+    else:
+        # create dictionaries (from existing files)
+        for j in range(26):
+            with np.load(d_path + d_file + "_fea_dict_{0}.npz".format(j)) as data:
+                unique = data["unique"]
+            for i, x in enumerate(unique):
+                convertDicts[j][x] = i
+        # load (uniques and) counts
+        with np.load(d_path + d_file + "_fea_count.npz") as data:
+            counts = data["counts"]
+
+    # process all splits
+    if dataset_multiprocessing:
+        processes = [
+            Process(
+                target=processCriteoAdData,
+                name="processCriteoAdData:%i" % i,
+                args=(
+                    d_path,
+                    d_file,
+                    npzfile,
+                    i,
+                    convertDicts,
+                    counts,
+                ),
+            )
+            for i in range(0, days)
+        ]
+        for process in processes:
+            process.start()
+        for process in processes:
+            process.join()
+
+    else:
+        for i in range(days):
+            processCriteoAdData(d_path, d_file, npzfile, i, convertDicts, counts)
+
+    o_file = concatCriteoAdData(
+        d_path,
+        d_file,
+        npzfile,
+        trafile,
+        days,
+        data_split,
+        randomize,
+        total_per_file,
+        total_count,
+        memory_map,
+        o_filename,
+    )
+
+    return o_file
+
+
+def loadDataset(
+    dataset,
+    max_ind_range,
+    sub_sample_rate,
+    randomize,
+    data_split,
+    raw_path="",
+    pro_data="",
+    memory_map=False,
+):
+    # dataset
+    if dataset == "kaggle":
+        days = 7
+        o_filename = "kaggleAdDisplayChallenge_processed"
+    elif dataset == "terabyte":
+        days = 24
+        o_filename = "terabyte_processed"
+    else:
+        raise (ValueError("Data set option is not supported"))
+
+    # split the datafile into path and filename
+    lstr = raw_path.split("/")
+    d_path = "/".join(lstr[0:-1]) + "/"
+    d_file = lstr[-1].split(".")[0] if dataset == "kaggle" else lstr[-1]
+    npzfile = (d_file + "_day") if dataset == "kaggle" else d_file
+    # trafile = d_path + ((d_file + "_fea") if dataset == "kaggle" else "fea")
+
+    # check if pre-processed data is available
+    data_ready = True
+    if memory_map:
+        for i in range(days):
+            reo_data = d_path + npzfile + "_{0}_reordered.npz".format(i)
+            if not path.exists(str(reo_data)):
+                data_ready = False
+    else:
+        if not path.exists(str(pro_data)):
+            data_ready = False
+
+    # pre-process data if needed
+    # WARNNING: when memory mapping is used we get a collection of files
+    if data_ready:
+        print("Reading pre-processed data=%s" % (str(pro_data)))
+        file = str(pro_data)
+    else:
+        print("Reading raw data=%s" % (str(raw_path)))
+        file = getCriteoAdData(
+            raw_path,
+            o_filename,
+            max_ind_range,
+            sub_sample_rate,
+            days,
+            data_split,
+            randomize,
+            dataset == "kaggle",
+            memory_map,
+        )
+
+    return file, days
+
+
+if __name__ == "__main__":
+    ### import packages ###
+    import argparse
+
+    ### parse arguments ###
+    parser = argparse.ArgumentParser(description="Preprocess Criteo dataset")
+    # model related parameters
+    parser.add_argument("--max-ind-range", type=int, default=-1)
+    parser.add_argument("--data-sub-sample-rate", type=float, default=0.0)  # in [0, 1]
+    parser.add_argument("--data-randomize", type=str, default="total")  # or day or none
+    parser.add_argument("--memory-map", action="store_true", default=False)
+    parser.add_argument("--data-set", type=str, default="kaggle")  # or terabyte
+    parser.add_argument("--raw-data-file", type=str, default="")
+    parser.add_argument("--processed-data-file", type=str, default="")
+    args = parser.parse_args()
+
+    loadDataset(
+        args.data_set,
+        args.max_ind_range,
+        args.data_sub_sample_rate,
+        args.data_randomize,
+        "train",
+        args.raw_data_file,
+        args.processed_data_file,
+        args.memory_map,
+    )
diff --git a/models_v2/pytorch/dlrm/inference/gpu/dlrm-kaggle.py b/models_v2/pytorch/dlrm/inference/gpu/dlrm-kaggle.py
new file mode 100644
index 000000000..3bd07e916
--- /dev/null
+++ b/models_v2/pytorch/dlrm/inference/gpu/dlrm-kaggle.py
@@ -0,0 +1,35 @@
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Consts for dlrm-kaggle inference."""
+
+ACC = {
+    "type": "total",
+    "pattern": r"accuracy (\d+.\d+) %",
+    "unit": "accuracy",
+}
+
+PERF = {
+    "type": "total",
+    "pattern": r"dlrm_inf throughput:  (\d+.\d+)  samples/s",
+    "inverse": False,
+    "multiply": False,
+    "use_batch_size": False,
+    "unit": "inst/s",
+}
+
+FUNCTIONAL = {
+    "pattern": r"dlrm_inf throughput:  \d+.\d+  samples/s"
+}
diff --git a/models_v2/pytorch/dlrm/inference/gpu/dlrm_data_caffe2.py b/models_v2/pytorch/dlrm/inference/gpu/dlrm_data_caffe2.py
new file mode 100644
index 000000000..95d04fa44
--- /dev/null
+++ b/models_v2/pytorch/dlrm/inference/gpu/dlrm_data_caffe2.py
@@ -0,0 +1,843 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+# Description: generate inputs and targets for the dlrm benchmark
+# The inpts and outputs are generated according to the following three option(s)
+# 1) random distribution
+# 2) synthetic distribution, based on unique accesses and distances between them
+#    i) R. Hassan, A. Harris, N. Topham and A. Efthymiou "Synthetic Trace-Driven
+#    Simulation of Cache Memory", IEEE AINAM'07
+# 3) public data set
+#    i)  Criteo Kaggle Display Advertising Challenge Dataset
+#    https://labs.criteo.com/2014/02/kaggle-display-advertising-challenge-dataset
+#    ii) Criteo Terabyte Dataset
+#    https://labs.criteo.com/2013/12/download-terabyte-click-logs
+
+
+from __future__ import absolute_import, division, print_function, unicode_literals
+
+import bisect
+import collections
+
+# others
+# from os import path
+import sys
+
+import data_utils
+
+# numpy
+import numpy as np
+
+# pytorch
+import torch
+from numpy import random as ra
+from torch.utils.data import Dataset
+
+
+# Kaggle Display Advertising Challenge Dataset
+# dataset (str): name of dataset (Kaggle or Terabyte)
+# randomize (str): determines randomization scheme
+#            'none': no randomization
+#            'day': randomizes each day's data (only works if split = True)
+#            'total': randomizes total dataset
+# split (bool) : to split into train, test, validation data-sets
+
+
+class CriteoDatasetWMemoryMap(Dataset):
+    def __init__(
+        self,
+        dataset,
+        max_ind_range,
+        sub_sample_rate,
+        randomize,
+        split="train",
+        raw_path="",
+        pro_data="",
+    ):
+        # dataset
+        # tar_fea = 1   # single target
+        den_fea = 13  # 13 dense  features
+        # spa_fea = 26  # 26 sparse features
+        # tad_fea = tar_fea + den_fea
+        # tot_fea = tad_fea + spa_fea
+        if dataset == "kaggle":
+            days = 7
+        elif dataset == "terabyte":
+            days = 24
+        else:
+            raise (ValueError("Data set option is not supported"))
+        self.max_ind_range = max_ind_range
+
+        # split the datafile into path and filename
+        lstr = raw_path.split("/")
+        self.d_path = "/".join(lstr[0:-1]) + "/"
+        self.d_file = lstr[-1].split(".")[0] if dataset == "kaggle" else lstr[-1]
+        self.npzfile = self.d_path + (
+            (self.d_file + "_day") if dataset == "kaggle" else self.d_file
+        )
+        self.trafile = self.d_path + (
+            (self.d_file + "_fea") if dataset == "kaggle" else "fea"
+        )
+
+        # get a number of samples per day
+        total_file = self.d_path + self.d_file + "_day_count.npz"
+        with np.load(total_file) as data:
+            total_per_file = data["total_per_file"]
+        # compute offsets per file
+        self.offset_per_file = np.array([0] + list(total_per_file))
+        for i in range(days):
+            self.offset_per_file[i + 1] += self.offset_per_file[i]
+        # print(self.offset_per_file)
+
+        # setup data
+        self.split = split
+        if split == "none" or split == "train":
+            self.day = 0
+            self.max_day_range = days if split == "none" else days - 1
+        elif split == "test" or split == "val":
+            self.day = days - 1
+            num_samples = self.offset_per_file[days] - self.offset_per_file[days - 1]
+            self.test_size = int(np.ceil(num_samples / 2.0))
+            self.val_size = num_samples - self.test_size
+        else:
+            sys.exit("ERROR: dataset split is neither none, nor train or test.")
+
+        # load unique counts
+        with np.load(self.d_path + self.d_file + "_fea_count.npz") as data:
+            self.counts = data["counts"]
+        self.m_den = den_fea  # X_int.shape[1]
+        self.n_emb = len(self.counts)
+        print("Sparse features= %d, Dense features= %d" % (self.n_emb, self.m_den))
+
+        # Load the test data
+        # Only a single day is used for testing
+        if self.split == "test" or self.split == "val":
+            # only a single day is used for testing
+            fi = self.npzfile + "_{0}_reordered.npz".format(self.day)
+            with np.load(fi) as data:
+                self.X_int = data["X_int"]  # continuous  feature
+                self.X_cat = data["X_cat"]  # categorical feature
+                self.y = data["y"]  # target
+
+    def __getitem__(self, index):
+
+        if isinstance(index, slice):
+            return [
+                self[idx]
+                for idx in range(
+                    index.start or 0, index.stop or len(self), index.step or 1
+                )
+            ]
+        if self.split == "none" or self.split == "train":
+            # check if need to swicth to next day and load data
+            if index == self.offset_per_file[self.day]:
+                # print("day_boundary switch", index)
+                self.day_boundary = self.offset_per_file[self.day]
+                fi = self.npzfile + "_{0}_reordered.npz".format(self.day)
+                # print('Loading file: ', fi)
+                with np.load(fi) as data:
+                    self.X_int = data["X_int"]  # continuous  feature
+                    self.X_cat = data["X_cat"]  # categorical feature
+                    self.y = data["y"]  # target
+                self.day = (self.day + 1) % self.max_day_range
+
+            i = index - self.day_boundary
+        elif self.split == "test" or self.split == "val":
+            # only a single day is used for testing
+            i = index + (0 if self.split == "test" else self.test_size)
+        else:
+            sys.exit("ERROR: dataset split is neither none, nor train or test.")
+
+        if self.max_ind_range > 0:
+            return self.X_int[i], self.X_cat[i] % self.max_ind_range, self.y[i]
+        else:
+            return self.X_int[i], self.X_cat[i], self.y[i]
+
+    def _default_preprocess(self, X_int, X_cat, y):
+        X_int = torch.log(torch.tensor(X_int, dtype=torch.float) + 1)
+        if self.max_ind_range > 0:
+            X_cat = torch.tensor(X_cat % self.max_ind_range, dtype=torch.long)
+        else:
+            X_cat = torch.tensor(X_cat, dtype=torch.long)
+        y = torch.tensor(y.astype(np.float32))
+
+        return X_int, X_cat, y
+
+    def __len__(self):
+        if self.split == "none":
+            return self.offset_per_file[-1]
+        elif self.split == "train":
+            return self.offset_per_file[-2]
+        elif self.split == "test":
+            return self.test_size
+        elif self.split == "val":
+            return self.val_size
+        else:
+            sys.exit("ERROR: dataset split is neither none, nor train nor test.")
+
+
+def collate_wrapper_criteo(list_of_tuples):
+    # where each tuple is (X_int, X_cat, y)
+    transposed_data = list(zip(*list_of_tuples))
+    X_int = torch.log(torch.tensor(transposed_data[0], dtype=torch.float) + 1)
+    X_cat = torch.tensor(transposed_data[1], dtype=torch.long)
+    T = torch.tensor(transposed_data[2], dtype=torch.float32).view(-1, 1)
+
+    batchSize = X_cat.shape[0]
+    featureCnt = X_cat.shape[1]
+
+    lS_i = [X_cat[:, i] for i in range(featureCnt)]
+    lS_o = [torch.tensor(range(batchSize)) for _ in range(featureCnt)]
+
+    return X_int, torch.stack(lS_o), torch.stack(lS_i), T
+
+
+# Conversion from offset to length
+def offset_to_length_convertor(lS_o, lS_i):
+    def diff(tensor):
+        return tensor[1:] - tensor[:-1]
+
+    return torch.stack(
+        [
+            diff(torch.cat((S_o, torch.tensor(lS_i[ind].shape))).int())
+            for ind, S_o in enumerate(lS_o)
+        ]
+    )
+
+
+def unpack_batch(b, data_gen, data_set):
+    return b[0], b[1], b[2], b[3], torch.ones(b[3].size())
+
+
+def read_dataset(
+    dataset,
+    max_ind_range,
+    sub_sample_rate,
+    mini_batch_size,
+    num_batches,
+    randomize,
+    split="train",
+    raw_data="",
+    processed_data="",
+    memory_map=False,
+    inference_only=False,
+    test_mini_batch_size=1,
+):
+    # split the datafile into path and filename
+    lstr = raw_data.split("/")
+    d_path = "/".join(lstr[0:-1]) + "/"
+    d_file = lstr[-1].split(".")[0] if dataset == "kaggle" else lstr[-1]
+    # npzfile = d_path + ((d_file + "_day") if dataset == "kaggle" else d_file)
+    # trafile = d_path + ((d_file + "_fea") if dataset == "kaggle" else "fea")
+
+    # load
+    print("Loading %s dataset..." % dataset)
+    nbatches = 0
+    file, days = data_utils.loadDataset(
+        dataset,
+        max_ind_range,
+        sub_sample_rate,
+        randomize,
+        split,
+        raw_data,
+        processed_data,
+        memory_map,
+    )
+
+    if memory_map:
+        # WARNING: at this point the data has been reordered and shuffled across files
+        # e.g. day_<number>_reordered.npz, what remains is simply to read and feed
+        # the data from each file, going in the order of days file-by-file, to the
+        # model during training.
+        train_data = CriteoDatasetWMemoryMap(
+            dataset,
+            max_ind_range,
+            sub_sample_rate,
+            randomize,
+            "train",
+            raw_data,
+            processed_data,
+        )
+
+        test_data = CriteoDatasetWMemoryMap(
+            dataset,
+            max_ind_range,
+            sub_sample_rate,
+            randomize,
+            "test",
+            raw_data,
+            processed_data,
+        )
+
+        train_loader = torch.utils.data.DataLoader(
+            train_data,
+            batch_size=mini_batch_size,
+            shuffle=False,
+            num_workers=0,
+            collate_fn=collate_wrapper_criteo,
+            pin_memory=False,
+            drop_last=False,  # True
+        )
+
+        test_loader = torch.utils.data.DataLoader(
+            test_data,
+            batch_size=test_mini_batch_size,
+            shuffle=False,
+            num_workers=0,
+            collate_fn=collate_wrapper_criteo,
+            pin_memory=False,
+            drop_last=False,  # True
+        )
+
+        return train_data, train_loader, test_data, test_loader
+
+    else:
+        # load and preprocess data
+        with np.load(file) as data:
+            X_int = data["X_int"]
+            X_cat = data["X_cat"]
+            y = data["y"]
+            counts = data["counts"]
+
+        # get a number of samples per day
+        total_file = d_path + d_file + "_day_count.npz"
+        with np.load(total_file) as data:
+            total_per_file = data["total_per_file"]
+
+        # transform
+        (
+            X_cat_train,
+            X_int_train,
+            y_train,
+            X_cat_val,
+            X_int_val,
+            y_val,
+            X_cat_test,
+            X_int_test,
+            y_test,
+        ) = data_utils.transformCriteoAdData(
+            X_cat, X_int, y, days, split, randomize, total_per_file
+        )
+        ln_emb = counts
+        m_den = X_int_train.shape[1]
+        n_emb = len(counts)
+        print("Sparse features = %d, Dense features = %d" % (n_emb, m_den))
+
+        # adjust parameters
+        def assemble_samples(X_cat, X_int, y, max_ind_range, print_message):
+            if max_ind_range > 0:
+                X_cat = X_cat % max_ind_range
+
+            nsamples = len(y)
+            data_size = nsamples
+            # using floor is equivalent to dropping last mini-batch (drop_last = True)
+            nbatches = int(np.floor((data_size * 1.0) / mini_batch_size))
+            print(print_message)
+            if num_batches != 0 and num_batches < nbatches:
+                print(
+                    "Limiting to %d batches of the total % d batches"
+                    % (num_batches, nbatches)
+                )
+                nbatches = num_batches
+            else:
+                print("Total number of batches %d" % nbatches)
+
+            # data main loop
+            lX = []
+            lS_lengths = []
+            lS_indices = []
+            lT = []
+            for j in range(0, nbatches):
+                # number of data points in a batch
+                print("Reading in batch: %d / %d" % (j + 1, nbatches), end="\r")
+                n = min(mini_batch_size, data_size - (j * mini_batch_size))
+                # dense feature
+                idx_start = j * mini_batch_size
+                lX.append((X_int[idx_start : (idx_start + n)]).astype(np.float32))
+                # Targets - outputs
+                lT.append(
+                    (y[idx_start : idx_start + n]).reshape(-1, 1).astype(np.int32)
+                )
+                # sparse feature (sparse indices)
+                lS_emb_indices = []
+                # for each embedding generate a list of n lookups,
+                # where each lookup is composed of multiple sparse indices
+                for size in range(n_emb):
+                    lS_batch_indices = []
+                    for _b in range(n):
+                        # num of sparse indices to be used per embedding, e.g. for
+                        # store lengths and indices
+                        lS_batch_indices += (
+                            (X_cat[idx_start + _b][size].reshape(-1)).astype(np.int32)
+                        ).tolist()
+                    lS_emb_indices.append(lS_batch_indices)
+                lS_indices.append(lS_emb_indices)
+                # Criteo Kaggle data it is 1 because data is categorical
+                lS_lengths.append(
+                    [(list(np.ones(n).astype(np.int32))) for _ in range(n_emb)]
+                )
+            print("\n")
+
+            return nbatches, lX, lS_lengths, lS_indices, lT
+
+        # adjust training data
+        (nbatches, lX, lS_lengths, lS_indices, lT) = assemble_samples(
+            X_cat_train, X_int_train, y_train, max_ind_range, "Training data"
+        )
+
+        # adjust testing data
+        (nbatches_t, lX_t, lS_lengths_t, lS_indices_t, lT_t) = assemble_samples(
+            X_cat_test, X_int_test, y_test, max_ind_range, "Testing data"
+        )
+    # end if memory_map
+
+    return (
+        nbatches,
+        lX,
+        lS_lengths,
+        lS_indices,
+        lT,
+        nbatches_t,
+        lX_t,
+        lS_lengths_t,
+        lS_indices_t,
+        lT_t,
+        ln_emb,
+        m_den,
+    )
+
+
+def generate_random_data(
+    m_den,
+    ln_emb,
+    data_size,
+    num_batches,
+    mini_batch_size,
+    num_indices_per_lookup,
+    num_indices_per_lookup_fixed,
+    num_targets=1,
+    round_targets=False,
+    data_generation="random",
+    trace_file="",
+    enable_padding=False,
+):
+    nbatches = int(np.ceil((data_size * 1.0) / mini_batch_size))
+    if num_batches != 0:
+        nbatches = num_batches
+        data_size = nbatches * mini_batch_size
+    # print("Total number of batches %d" % nbatches)
+
+    # inputs and targets
+    lT = []
+    lX = []
+    lS_lengths = []
+    lS_indices = []
+    for j in range(0, nbatches):
+        # number of data points in a batch
+        n = min(mini_batch_size, data_size - (j * mini_batch_size))
+
+        # generate a batch of dense and sparse features
+        if data_generation == "random":
+            (Xt, lS_emb_lengths, lS_emb_indices) = generate_uniform_input_batch(
+                m_den, ln_emb, n, num_indices_per_lookup, num_indices_per_lookup_fixed
+            )
+        elif data_generation == "synthetic":
+            (Xt, lS_emb_lengths, lS_emb_indices) = generate_synthetic_input_batch(
+                m_den,
+                ln_emb,
+                n,
+                num_indices_per_lookup,
+                num_indices_per_lookup_fixed,
+                trace_file,
+                enable_padding,
+            )
+        else:
+            sys.exit(
+                "ERROR: --data-generation=" + data_generation + " is not supported"
+            )
+        # dense feature
+        lX.append(Xt)
+        # sparse feature (sparse indices)
+        lS_lengths.append(lS_emb_lengths)
+        lS_indices.append(lS_emb_indices)
+
+        # generate a batch of target (probability of a click)
+        P = generate_random_output_batch(n, num_targets, round_targets)
+        lT.append(P)
+
+    return (nbatches, lX, lS_lengths, lS_indices, lT)
+
+
+def generate_random_output_batch(n, num_targets=1, round_targets=False):
+    # target (probability of a click)
+    if round_targets:
+        P = np.round(ra.rand(n, num_targets).astype(np.float32)).astype(np.int32)
+    else:
+        P = ra.rand(n, num_targets).astype(np.float32)
+
+    return P
+
+
+# uniform ditribution (input data)
+def generate_uniform_input_batch(
+    m_den,
+    ln_emb,
+    n,
+    num_indices_per_lookup,
+    num_indices_per_lookup_fixed,
+):
+    # dense feature
+    Xt = ra.rand(n, m_den).astype(np.float32)
+
+    # sparse feature (sparse indices)
+    lS_emb_lengths = []
+    lS_emb_indices = []
+    # for each embedding generate a list of n lookups,
+    # where each lookup is composed of multiple sparse indices
+    for size in ln_emb:
+        lS_batch_lengths = []
+        lS_batch_indices = []
+        for _ in range(n):
+            # num of sparse indices to be used per embedding (between
+            if num_indices_per_lookup_fixed:
+                sparse_group_size = np.int32(num_indices_per_lookup)
+            else:
+                # random between [1,num_indices_per_lookup])
+                r = ra.random(1)
+                sparse_group_size = np.int32(
+                    max(1, np.round(r * min(size, num_indices_per_lookup))[0])
+                )
+            # sparse indices to be used per embedding
+            r = ra.random(sparse_group_size)
+            sparse_group = np.unique(np.round(r * (size - 1)).astype(np.int32))
+            # reset sparse_group_size in case some index duplicates were removed
+            sparse_group_size = np.int32(sparse_group.size)
+            # store lengths and indices
+            lS_batch_lengths += [sparse_group_size]
+            lS_batch_indices += sparse_group.tolist()
+        lS_emb_lengths.append(lS_batch_lengths)
+        lS_emb_indices.append(lS_batch_indices)
+
+    return (Xt, lS_emb_lengths, lS_emb_indices)
+
+
+# synthetic distribution (input data)
+def generate_synthetic_input_batch(
+    m_den,
+    ln_emb,
+    n,
+    num_indices_per_lookup,
+    num_indices_per_lookup_fixed,
+    trace_file,
+    enable_padding=False,
+):
+    # dense feature
+    Xt = ra.rand(n, m_den).astype(np.float32)
+
+    # sparse feature (sparse indices)
+    lS_emb_lengths = []
+    lS_emb_indices = []
+    # for each embedding generate a list of n lookups,
+    # where each lookup is composed of multiple sparse indices
+    for i, size in enumerate(ln_emb):
+        lS_batch_lengths = []
+        lS_batch_indices = []
+        for _ in range(n):
+            # num of sparse indices to be used per embedding (between
+            if num_indices_per_lookup_fixed:
+                sparse_group_size = np.int32(num_indices_per_lookup)
+            else:
+                # random between [1,num_indices_per_lookup])
+                r = ra.random(1)
+                sparse_group_size = np.int32(
+                    max(1, np.round(r * min(size, num_indices_per_lookup))[0])
+                )
+            # sparse indices to be used per embedding
+            file_path = trace_file
+            line_accesses, list_sd, cumm_sd = read_dist_from_file(
+                file_path.replace("j", str(i))
+            )
+            # debug print
+            # print('input')
+            # print(line_accesses); print(list_sd); print(cumm_sd);
+            # print(sparse_group_size)
+            # approach 1: rand
+            # r = trace_generate_rand(
+            #     line_accesses, list_sd, cumm_sd, sparse_group_size, enable_padding
+            # )
+            # approach 2: lru
+            r = trace_generate_lru(
+                line_accesses, list_sd, cumm_sd, sparse_group_size, enable_padding
+            )
+            # WARNING: if the distribution in the file is not consistent with
+            # embedding table dimensions, below mod guards against out of
+            # range access
+            sparse_group = np.unique(r).astype(np.int32)
+            minsg = np.min(sparse_group)
+            maxsg = np.max(sparse_group)
+            if (minsg < 0) or (size <= maxsg):
+                print(
+                    "WARNING: distribution is inconsistent with embedding "
+                    + "table size (using mod to recover and continue)"
+                )
+                sparse_group = np.mod(sparse_group, size).astype(np.int32)
+            # sparse_group = np.unique(np.array(np.mod(r, size-1)).astype(np.int32))
+            # reset sparse_group_size in case some index duplicates were removed
+            sparse_group_size = np.int32(sparse_group.size)
+            # store lengths and indices
+            lS_batch_lengths += [sparse_group_size]
+            lS_batch_indices += sparse_group.tolist()
+        lS_emb_lengths.append(lS_batch_lengths)
+        lS_emb_indices.append(lS_batch_indices)
+
+    return (Xt, lS_emb_lengths, lS_emb_indices)
+
+
+def generate_stack_distance(cumm_val, cumm_dist, max_i, i, enable_padding=False):
+    u = ra.rand(1)
+    if i < max_i:
+        # only generate stack distances up to the number of new references seen so far
+        j = bisect.bisect(cumm_val, i) - 1
+        fi = cumm_dist[j]
+        u *= fi  # shrink distribution support to exclude last values
+    elif enable_padding:
+        # WARNING: disable generation of new references (once all have been seen)
+        fi = cumm_dist[0]
+        u = (1.0 - fi) * u + fi  # remap distribution support to exclude first value
+
+    for (j, f) in enumerate(cumm_dist):
+        if u <= f:
+            return cumm_val[j]
+
+
+# WARNING: global define, must be consistent across all synthetic functions
+cache_line_size = 1
+
+
+def trace_generate_lru(
+    line_accesses, list_sd, cumm_sd, out_trace_len, enable_padding=False
+):
+    max_sd = list_sd[-1]
+    l = len(line_accesses)
+    i = 0
+    ztrace = []
+    for _ in range(out_trace_len):
+        sd = generate_stack_distance(list_sd, cumm_sd, max_sd, i, enable_padding)
+        mem_ref_within_line = 0  # floor(ra.rand(1)*cache_line_size) #0
+        # generate memory reference
+        if sd == 0:  # new reference #
+            line_ref = line_accesses.pop(0)
+            line_accesses.append(line_ref)
+            mem_ref = np.uint64(line_ref * cache_line_size + mem_ref_within_line)
+            i += 1
+        else:  # existing reference #
+            line_ref = line_accesses[l - sd]
+            mem_ref = np.uint64(line_ref * cache_line_size + mem_ref_within_line)
+            line_accesses.pop(l - sd)
+            line_accesses.append(line_ref)
+        # save generated memory reference
+        ztrace.append(mem_ref)
+
+    return ztrace
+
+
+def trace_generate_rand(
+    line_accesses, list_sd, cumm_sd, out_trace_len, enable_padding=False
+):
+    max_sd = list_sd[-1]
+    l = len(line_accesses)  # !!!Unique,
+    i = 0
+    ztrace = []
+    for _ in range(out_trace_len):
+        sd = generate_stack_distance(list_sd, cumm_sd, max_sd, i, enable_padding)
+        mem_ref_within_line = 0  # floor(ra.rand(1)*cache_line_size) #0
+        # generate memory reference
+        if sd == 0:  # new reference #
+            line_ref = line_accesses.pop(0)
+            line_accesses.append(line_ref)
+            mem_ref = np.uint64(line_ref * cache_line_size + mem_ref_within_line)
+            i += 1
+        else:  # existing reference #
+            line_ref = line_accesses[l - sd]
+            mem_ref = np.uint64(line_ref * cache_line_size + mem_ref_within_line)
+        ztrace.append(mem_ref)
+
+    return ztrace
+
+
+def trace_profile(trace, enable_padding=False):
+    # number of elements in the array (assuming 1D)
+    # n = trace.size
+
+    rstack = []  # S
+    stack_distances = []  # SDS
+    line_accesses = []  # L
+    for x in trace:
+        r = np.uint64(x / cache_line_size)
+        l = len(rstack)
+        try:  # found #
+            i = rstack.index(r)
+            # WARNING: I believe below is the correct depth in terms of meaning of the
+            #          algorithm, but that is not what seems to be in the paper alg.
+            #          -1 can be subtracted if we defined the distance between
+            #          consecutive accesses (e.g. r, r) as 0 rather than 1.
+            sd = l - i  # - 1
+            # push r to the end of stack_distances
+            stack_distances.insert(0, sd)
+            # remove r from its position and insert to the top of stack
+            rstack.pop(i)  # rstack.remove(r)
+            rstack.insert(l - 1, r)
+        except ValueError:  # not found #
+            sd = 0  # -1
+            # push r to the end of stack_distances/line_accesses
+            stack_distances.insert(0, sd)
+            line_accesses.insert(0, r)
+            # push r to the top of stack
+            rstack.insert(l, r)
+
+    if enable_padding:
+        # WARNING: notice that as the ratio between the number of samples (l)
+        # and cardinality [c] of a sample increases the probability of
+        # generating a sample gets smaller and smaller because there are
+        # few new samples compared to repeated samples. This means that for a
+        # long trace with relatively small cardinality it will take longer to
+        # generate all new samples and therefore obtain full distribution support
+        # and hence it takes longer for distribution to resemble the original.
+        # Therefore, we may pad the number of new samples to be on par with
+        # average number of samples l/c artificially.
+        l = len(stack_distances)
+        c = max(stack_distances)
+        padding = int(np.ceil(l / c))
+        stack_distances = stack_distances + [0] * padding
+
+    return (rstack, stack_distances, line_accesses)
+
+
+# auxiliary read/write routines
+def read_trace_from_file(file_path):
+    try:
+        with open(file_path) as f:
+            if args.trace_file_binary_type:
+                array = np.fromfile(f, dtype=np.uint64)
+                trace = array.astype(np.uint64).tolist()
+            else:
+                line = f.readline()
+                trace = list(map(lambda x: np.uint64(x), line.split(", ")))
+            return trace
+    except Exception:
+        print("ERROR: no input trace file has been provided")
+
+
+def write_trace_to_file(file_path, trace):
+    try:
+        if args.trace_file_binary_type:
+            with open(file_path, "wb+") as f:
+                np.array(trace).astype(np.uint64).tofile(f)
+        else:
+            with open(file_path, "w+") as f:
+                s = str(trace)
+                f.write(s[1 : len(s) - 1])
+    except Exception:
+        print("ERROR: no output trace file has been provided")
+
+
+def read_dist_from_file(file_path):
+    try:
+        with open(file_path, "r") as f:
+            lines = f.read().splitlines()
+    except Exception:
+        print("Wrong file or file path")
+    # read unique accesses
+    unique_accesses = [int(el) for el in lines[0].split(", ")]
+    # read cumulative distribution (elements are passed as two separate lists)
+    list_sd = [int(el) for el in lines[1].split(", ")]
+    cumm_sd = [float(el) for el in lines[2].split(", ")]
+
+    return unique_accesses, list_sd, cumm_sd
+
+
+def write_dist_to_file(file_path, unique_accesses, list_sd, cumm_sd):
+    try:
+        with open(file_path, "w") as f:
+            # unique_acesses
+            s = str(unique_accesses)
+            f.write(s[1 : len(s) - 1] + "\n")
+            # list_sd
+            s = str(list_sd)
+            f.write(s[1 : len(s) - 1] + "\n")
+            # cumm_sd
+            s = str(cumm_sd)
+            f.write(s[1 : len(s) - 1] + "\n")
+    except Exception:
+        print("Wrong file or file path")
+
+
+if __name__ == "__main__":
+    import argparse
+    import operator
+    import sys
+
+    ### parse arguments ###
+    parser = argparse.ArgumentParser(description="Generate Synthetic Distributions")
+    parser.add_argument("--trace-file", type=str, default="./input/trace.log")
+    parser.add_argument("--trace-file-binary-type", type=bool, default=False)
+    parser.add_argument("--trace-enable-padding", type=bool, default=False)
+    parser.add_argument("--dist-file", type=str, default="./input/dist.log")
+    parser.add_argument(
+        "--synthetic-file", type=str, default="./input/trace_synthetic.log"
+    )
+    parser.add_argument("--numpy-rand-seed", type=int, default=123)
+    parser.add_argument("--print-precision", type=int, default=5)
+    args = parser.parse_args()
+
+    ### some basic setup ###
+    np.random.seed(args.numpy_rand_seed)
+    np.set_printoptions(precision=args.print_precision)
+
+    ### read trace ###
+    trace = read_trace_from_file(args.trace_file)
+    # print(trace)
+
+    ### profile trace ###
+    (_, stack_distances, line_accesses) = trace_profile(
+        trace, args.trace_enable_padding
+    )
+    stack_distances.reverse()
+    line_accesses.reverse()
+    # print(line_accesses)
+    # print(stack_distances)
+
+    ### compute probability distribution ###
+    # count items
+    l = len(stack_distances)
+    dc = sorted(
+        collections.Counter(stack_distances).items(), key=operator.itemgetter(0)
+    )
+
+    # create a distribution
+    list_sd = list(map(lambda tuple_x_k: tuple_x_k[0], dc))  # x = tuple_x_k[0]
+    dist_sd = list(
+        map(lambda tuple_x_k: tuple_x_k[1] / float(l), dc)
+    )  # k = tuple_x_k[1]
+    cumm_sd = []  # np.cumsum(dc).tolist() #prefixsum
+    for i, (_, k) in enumerate(dc):
+        if i == 0:
+            cumm_sd.append(k / float(l))
+        else:
+            # add the 2nd element of the i-th tuple in the dist_sd list
+            cumm_sd.append(cumm_sd[i - 1] + (k / float(l)))
+
+    ### write stack_distance and line_accesses to a file ###
+    write_dist_to_file(args.dist_file, line_accesses, list_sd, cumm_sd)
+
+    ### generate correspondinf synthetic ###
+    # line_accesses, list_sd, cumm_sd = read_dist_from_file(args.dist_file)
+    synthetic_trace = trace_generate_lru(
+        line_accesses, list_sd, cumm_sd, len(trace), args.trace_enable_padding
+    )
+    # synthetic_trace = trace_generate_rand(
+    #     line_accesses, list_sd, cumm_sd, len(trace), args.trace_enable_padding
+    # )
+    write_trace_to_file(args.synthetic_file, synthetic_trace)
diff --git a/models_v2/pytorch/dlrm/inference/gpu/dlrm_data_pytorch.py b/models_v2/pytorch/dlrm/inference/gpu/dlrm_data_pytorch.py
new file mode 100644
index 000000000..45c9c0efa
--- /dev/null
+++ b/models_v2/pytorch/dlrm/inference/gpu/dlrm_data_pytorch.py
@@ -0,0 +1,1303 @@
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+# Description: generate inputs and targets for the dlrm benchmark
+# The inpts and outputs are generated according to the following three option(s)
+# 1) random distribution
+# 2) synthetic distribution, based on unique accesses and distances between them
+#    i) R. Hassan, A. Harris, N. Topham and A. Efthymiou "Synthetic Trace-Driven
+#    Simulation of Cache Memory", IEEE AINAM'07
+# 3) public data set
+#    i)  Criteo Kaggle Display Advertising Challenge Dataset
+#    https://labs.criteo.com/2014/02/kaggle-display-advertising-challenge-dataset
+#    ii) Criteo Terabyte Dataset
+#    https://labs.criteo.com/2013/12/download-terabyte-click-logs
+
+
+from __future__ import absolute_import, division, print_function, unicode_literals
+
+import bisect
+import collections
+import sys
+from collections import deque
+
+# others
+from os import path
+
+import data_loader_terabyte
+
+import data_utils
+import mlperf_logger
+
+# numpy
+import numpy as np
+
+# pytorch
+import torch
+from numpy import random as ra
+from torch.utils.data import Dataset, RandomSampler
+
+
+# Kaggle Display Advertising Challenge Dataset
+# dataset (str): name of dataset (Kaggle or Terabyte)
+# randomize (str): determines randomization scheme
+#            "none": no randomization
+#            "day": randomizes each day"s data (only works if split = True)
+#            "total": randomizes total dataset
+# split (bool) : to split into train, test, validation data-sets
+class CriteoDataset(Dataset):
+    def __init__(
+        self,
+        dataset,
+        max_ind_range,
+        sub_sample_rate,
+        randomize,
+        split="train",
+        raw_path="",
+        pro_data="",
+        memory_map=False,
+        dataset_multiprocessing=False,
+    ):
+        # dataset
+        # tar_fea = 1   # single target
+        den_fea = 13  # 13 dense  features
+        # spa_fea = 26  # 26 sparse features
+        # tad_fea = tar_fea + den_fea
+        # tot_fea = tad_fea + spa_fea
+        if dataset == "kaggle":
+            days = 7
+            out_file = "kaggleAdDisplayChallenge_processed"
+        elif dataset == "terabyte":
+            days = 24
+            out_file = "terabyte_processed"
+        else:
+            raise (ValueError("Data set option is not supported"))
+        self.max_ind_range = max_ind_range
+        self.memory_map = memory_map
+
+        # split the datafile into path and filename
+        lstr = raw_path.split("/")
+        self.d_path = "/".join(lstr[0:-1]) + "/"
+        self.d_file = lstr[-1].split(".")[0] if dataset == "kaggle" else lstr[-1]
+        self.npzfile = self.d_path + (
+            (self.d_file + "_day") if dataset == "kaggle" else self.d_file
+        )
+        self.trafile = self.d_path + (
+            (self.d_file + "_fea") if dataset == "kaggle" else "fea"
+        )
+
+        # check if pre-processed data is available
+        data_ready = True
+        if memory_map:
+            for i in range(days):
+                reo_data = self.npzfile + "_{0}_reordered.npz".format(i)
+                if not path.exists(str(reo_data)):
+                    data_ready = False
+        else:
+            if not path.exists(str(pro_data)):
+                data_ready = False
+
+        # pre-process data if needed
+        # WARNNING: when memory mapping is used we get a collection of files
+        if data_ready:
+            print("Reading pre-processed data=%s" % (str(pro_data)))
+            file = str(pro_data)
+        else:
+            print("Reading raw data=%s" % (str(raw_path)))
+            file = data_utils.getCriteoAdData(
+                raw_path,
+                out_file,
+                max_ind_range,
+                sub_sample_rate,
+                days,
+                split,
+                randomize,
+                dataset == "kaggle",
+                memory_map,
+                dataset_multiprocessing,
+            )
+
+        # get a number of samples per day
+        total_file = self.d_path + self.d_file + "_day_count.npz"
+        with np.load(total_file) as data:
+            total_per_file = data["total_per_file"]
+        # compute offsets per file
+        self.offset_per_file = np.array([0] + [x for x in total_per_file])
+        for i in range(days):
+            self.offset_per_file[i + 1] += self.offset_per_file[i]
+        # print(self.offset_per_file)
+
+        # setup data
+        if memory_map:
+            # setup the training/testing split
+            self.split = split
+            if split == "none" or split == "train":
+                self.day = 0
+                self.max_day_range = days if split == "none" else days - 1
+            elif split == "test" or split == "val":
+                self.day = days - 1
+                num_samples = (
+                    self.offset_per_file[days] - self.offset_per_file[days - 1]
+                )
+                self.test_size = int(np.ceil(num_samples / 2.0))
+                self.val_size = num_samples - self.test_size
+            else:
+                sys.exit("ERROR: dataset split is neither none, nor train or test.")
+
+            """
+            # text
+            print("text")
+            for i in range(days):
+                fi = self.npzfile + "_{0}".format(i)
+                with open(fi) as data:
+                    ttt = 0; nnn = 0
+                    for _j, line in enumerate(data):
+                        ttt +=1
+                        if np.int32(line[0]) > 0:
+                            nnn +=1
+                    print("day=" + str(i) + " total=" + str(ttt) + " non-zeros="
+                          + str(nnn) + " ratio=" +str((nnn * 100.) / ttt) + "%")
+            # processed
+            print("processed")
+            for i in range(days):
+                fi = self.npzfile + "_{0}_processed.npz".format(i)
+                with np.load(fi) as data:
+                    yyy = data["y"]
+                ttt = len(yyy)
+                nnn = np.count_nonzero(yyy)
+                print("day=" + str(i) + " total=" + str(ttt) + " non-zeros="
+                      + str(nnn) + " ratio=" +str((nnn * 100.) / ttt) + "%")
+            # reordered
+            print("reordered")
+            for i in range(days):
+                fi = self.npzfile + "_{0}_reordered.npz".format(i)
+                with np.load(fi) as data:
+                    yyy = data["y"]
+                ttt = len(yyy)
+                nnn = np.count_nonzero(yyy)
+                print("day=" + str(i) + " total=" + str(ttt) + " non-zeros="
+                      + str(nnn) + " ratio=" +str((nnn * 100.) / ttt) + "%")
+            """
+
+            # load unique counts
+            with np.load(self.d_path + self.d_file + "_fea_count.npz") as data:
+                self.counts = data["counts"]
+            self.m_den = den_fea  # X_int.shape[1]
+            self.n_emb = len(self.counts)
+            print("Sparse features= %d, Dense features= %d" % (self.n_emb, self.m_den))
+
+            # Load the test data
+            # Only a single day is used for testing
+            if self.split == "test" or self.split == "val":
+                # only a single day is used for testing
+                fi = self.npzfile + "_{0}_reordered.npz".format(self.day)
+                with np.load(fi) as data:
+                    self.X_int = data["X_int"]  # continuous  feature
+                    self.X_cat = data["X_cat"]  # categorical feature
+                    self.y = data["y"]  # target
+
+        else:
+            # load and preprocess data
+            with np.load(file) as data:
+                X_int = data["X_int"]  # continuous  feature
+                X_cat = data["X_cat"]  # categorical feature
+                y = data["y"]  # target
+                self.counts = data["counts"]
+            self.m_den = X_int.shape[1]  # den_fea
+            self.n_emb = len(self.counts)
+            print("Sparse fea = %d, Dense fea = %d" % (self.n_emb, self.m_den))
+
+            # create reordering
+            indices = np.arange(len(y))
+
+            if split == "none":
+                # randomize all data
+                if randomize == "total":
+                    indices = np.random.permutation(indices)
+                    print("Randomized indices...")
+
+                X_int[indices] = X_int
+                X_cat[indices] = X_cat
+                y[indices] = y
+
+            else:
+                indices = np.array_split(indices, self.offset_per_file[1:-1])
+
+                # randomize train data (per day)
+                if randomize == "day":  # or randomize == "total":
+                    for i in range(len(indices) - 1):
+                        indices[i] = np.random.permutation(indices[i])
+                    print("Randomized indices per day ...")
+
+                train_indices = np.concatenate(indices[:-1])
+                test_indices = indices[-1]
+                test_indices, val_indices = np.array_split(test_indices, 2)
+
+                print("Defined %s indices..." % (split))
+
+                # randomize train data (across days)
+                if randomize == "total":
+                    train_indices = np.random.permutation(train_indices)
+                    print("Randomized indices across days ...")
+
+                # create training, validation, and test sets
+                if split == "train":
+                    self.X_int = [X_int[i] for i in train_indices]
+                    self.X_cat = [X_cat[i] for i in train_indices]
+                    self.y = [y[i] for i in train_indices]
+                elif split == "val":
+                    self.X_int = [X_int[i] for i in val_indices]
+                    self.X_cat = [X_cat[i] for i in val_indices]
+                    self.y = [y[i] for i in val_indices]
+                elif split == "test":
+                    self.X_int = [X_int[i] for i in test_indices]
+                    self.X_cat = [X_cat[i] for i in test_indices]
+                    self.y = [y[i] for i in test_indices]
+
+            print("Split data according to indices...")
+
+    def __getitem__(self, index):
+
+        if isinstance(index, slice):
+            return [
+                self[idx]
+                for idx in range(
+                    index.start or 0, index.stop or len(self), index.step or 1
+                )
+            ]
+
+        if self.memory_map:
+            if self.split == "none" or self.split == "train":
+                # check if need to swicth to next day and load data
+                if index == self.offset_per_file[self.day]:
+                    # print("day_boundary switch", index)
+                    self.day_boundary = self.offset_per_file[self.day]
+                    fi = self.npzfile + "_{0}_reordered.npz".format(self.day)
+                    # print('Loading file: ', fi)
+                    with np.load(fi) as data:
+                        self.X_int = data["X_int"]  # continuous  feature
+                        self.X_cat = data["X_cat"]  # categorical feature
+                        self.y = data["y"]  # target
+                    self.day = (self.day + 1) % self.max_day_range
+
+                i = index - self.day_boundary
+            elif self.split == "test" or self.split == "val":
+                # only a single day is used for testing
+                i = index + (0 if self.split == "test" else self.test_size)
+            else:
+                sys.exit("ERROR: dataset split is neither none, nor train or test.")
+        else:
+            i = index
+
+        if self.max_ind_range > 0:
+            return self.X_int[i], self.X_cat[i] % self.max_ind_range, self.y[i]
+        else:
+            return self.X_int[i], self.X_cat[i], self.y[i]
+
+    def _default_preprocess(self, X_int, X_cat, y):
+        X_int = torch.log(torch.tensor(X_int, dtype=torch.float) + 1)
+        if self.max_ind_range > 0:
+            X_cat = torch.tensor(X_cat % self.max_ind_range, dtype=torch.long)
+        else:
+            X_cat = torch.tensor(X_cat, dtype=torch.long)
+        y = torch.tensor(y.astype(np.float32))
+
+        return X_int, X_cat, y
+
+    def __len__(self):
+        if self.memory_map:
+            if self.split == "none":
+                return self.offset_per_file[-1]
+            elif self.split == "train":
+                return self.offset_per_file[-2]
+            elif self.split == "test":
+                return self.test_size
+            elif self.split == "val":
+                return self.val_size
+            else:
+                sys.exit("ERROR: dataset split is neither none, nor train nor test.")
+        else:
+            return len(self.y)
+
+
+def collate_wrapper_criteo_offset(list_of_tuples):
+    # where each tuple is (X_int, X_cat, y)
+    transposed_data = list(zip(*list_of_tuples))
+    X_int = torch.log(torch.tensor(transposed_data[0], dtype=torch.float) + 1)
+    X_cat = torch.tensor(transposed_data[1], dtype=torch.long)
+    T = torch.tensor(transposed_data[2], dtype=torch.float32).view(-1, 1)
+
+    batchSize = X_cat.shape[0]
+    featureCnt = X_cat.shape[1]
+
+    lS_i = [X_cat[:, i] for i in range(featureCnt)]
+    lS_o = [torch.tensor(range(batchSize)) for _ in range(featureCnt)]
+
+    return X_int, torch.stack(lS_o), torch.stack(lS_i), T
+
+
+def ensure_dataset_preprocessed(args, d_path):
+    _ = CriteoDataset(
+        args.data_set,
+        args.max_ind_range,
+        args.data_sub_sample_rate,
+        args.data_randomize,
+        "train",
+        args.raw_data_file,
+        args.processed_data_file,
+        args.memory_map,
+        args.dataset_multiprocessing,
+    )
+
+    _ = CriteoDataset(
+        args.data_set,
+        args.max_ind_range,
+        args.data_sub_sample_rate,
+        args.data_randomize,
+        "test",
+        args.raw_data_file,
+        args.processed_data_file,
+        args.memory_map,
+        args.dataset_multiprocessing,
+    )
+
+    for split in ["train", "val", "test"]:
+        print("Running preprocessing for split =", split)
+
+        train_files = [
+            "{}_{}_reordered.npz".format(args.raw_data_file, day)
+            for day in range(0, 23)
+        ]
+
+        test_valid_file = args.raw_data_file + "_23_reordered.npz"
+
+        output_file = d_path + "_{}.bin".format(split)
+
+        input_files = train_files if split == "train" else [test_valid_file]
+        data_loader_terabyte.numpy_to_binary(
+            input_files=input_files, output_file_path=output_file, split=split
+        )
+
+
+# Conversion from offset to length
+def offset_to_length_converter(lS_o, lS_i):
+    def diff(tensor):
+        return tensor[1:] - tensor[:-1]
+
+    return torch.stack(
+        [
+            diff(torch.cat((S_o, torch.tensor(lS_i[ind].shape))).int())
+            for ind, S_o in enumerate(lS_o)
+        ]
+    )
+
+
+def collate_wrapper_criteo_length(list_of_tuples):
+    # where each tuple is (X_int, X_cat, y)
+    transposed_data = list(zip(*list_of_tuples))
+    X_int = torch.log(torch.tensor(transposed_data[0], dtype=torch.float) + 1)
+    X_cat = torch.tensor(transposed_data[1], dtype=torch.long)
+    T = torch.tensor(transposed_data[2], dtype=torch.float32).view(-1, 1)
+
+    batchSize = X_cat.shape[0]
+    featureCnt = X_cat.shape[1]
+
+    lS_i = torch.stack([X_cat[:, i] for i in range(featureCnt)])
+    lS_o = torch.stack([torch.tensor(range(batchSize)) for _ in range(featureCnt)])
+
+    lS_l = offset_to_length_converter(lS_o, lS_i)
+
+    return X_int, lS_l, lS_i, T
+
+
+def make_criteo_data_and_loaders(args, offset_to_length_converter=False):
+    if args.mlperf_logging and args.memory_map and args.data_set == "terabyte":
+        # more efficient for larger batches
+        data_directory = path.dirname(args.raw_data_file)
+
+        if args.mlperf_bin_loader:
+            lstr = args.processed_data_file.split("/")
+            d_path = "/".join(lstr[0:-1]) + "/" + lstr[-1].split(".")[0]
+            train_file = d_path + "_train.bin"
+            test_file = d_path + "_test.bin"
+            # val_file = d_path + "_val.bin"
+            counts_file = args.raw_data_file + "_fea_count.npz"
+
+            if any(not path.exists(p) for p in [train_file, test_file, counts_file]):
+                ensure_dataset_preprocessed(args, d_path)
+
+            train_data = data_loader_terabyte.CriteoBinDataset(
+                data_file=train_file,
+                counts_file=counts_file,
+                batch_size=args.mini_batch_size,
+                max_ind_range=args.max_ind_range,
+                drop_last=True
+            )
+
+            mlperf_logger.log_event(
+                key=mlperf_logger.constants.TRAIN_SAMPLES, value=train_data.num_samples
+            )
+
+            train_loader = torch.utils.data.DataLoader(
+                train_data,
+                batch_size=None,
+                batch_sampler=None,
+                shuffle=False,
+                num_workers=0,
+                collate_fn=None,
+                pin_memory=False,
+                drop_last=False,
+                sampler=RandomSampler(train_data) if args.mlperf_bin_shuffle else None,
+            )
+
+            test_data = data_loader_terabyte.CriteoBinDataset(
+                data_file=test_file,
+                counts_file=counts_file,
+                batch_size=args.test_mini_batch_size,
+                max_ind_range=args.max_ind_range,
+            )
+
+            mlperf_logger.log_event(
+                key=mlperf_logger.constants.EVAL_SAMPLES, value=test_data.num_samples
+            )
+
+            test_loader = torch.utils.data.DataLoader(
+                test_data,
+                batch_size=None,
+                batch_sampler=None,
+                shuffle=False,
+                num_workers=0,
+                collate_fn=None,
+                pin_memory=False,
+                drop_last=False,
+            )
+        else:
+            data_filename = args.raw_data_file.split("/")[-1]
+
+            train_data = CriteoDataset(
+                args.data_set,
+                args.max_ind_range,
+                args.data_sub_sample_rate,
+                args.data_randomize,
+                "train",
+                args.raw_data_file,
+                args.processed_data_file,
+                args.memory_map,
+                args.dataset_multiprocessing,
+            )
+
+            test_data = CriteoDataset(
+                args.data_set,
+                args.max_ind_range,
+                args.data_sub_sample_rate,
+                args.data_randomize,
+                "test",
+                args.raw_data_file,
+                args.processed_data_file,
+                args.memory_map,
+                args.dataset_multiprocessing,
+            )
+
+            train_loader = data_loader_terabyte.DataLoader(
+                data_directory=data_directory,
+                data_filename=data_filename,
+                days=list(range(23)),
+                batch_size=args.mini_batch_size,
+                max_ind_range=args.max_ind_range,
+                split="train",
+            )
+
+            test_loader = data_loader_terabyte.DataLoader(
+                data_directory=data_directory,
+                data_filename=data_filename,
+                days=[23],
+                batch_size=args.test_mini_batch_size,
+                max_ind_range=args.max_ind_range,
+                split="test",
+            )
+    else:
+        train_data = CriteoDataset(
+            args.data_set,
+            args.max_ind_range,
+            args.data_sub_sample_rate,
+            args.data_randomize,
+            "train",
+            args.raw_data_file,
+            args.processed_data_file,
+            args.memory_map,
+            args.dataset_multiprocessing,
+        )
+
+        test_data = CriteoDataset(
+            args.data_set,
+            args.max_ind_range,
+            args.data_sub_sample_rate,
+            args.data_randomize,
+            "test",
+            args.raw_data_file,
+            args.processed_data_file,
+            args.memory_map,
+            args.dataset_multiprocessing,
+        )
+
+        collate_wrapper_criteo = collate_wrapper_criteo_offset
+        if offset_to_length_converter:
+            collate_wrapper_criteo = collate_wrapper_criteo_length
+
+        train_loader = torch.utils.data.DataLoader(
+            train_data,
+            batch_size=args.mini_batch_size,
+            shuffle=False,
+            num_workers=args.num_workers,
+            collate_fn=collate_wrapper_criteo,
+            pin_memory=False,
+            drop_last=False,  # True
+        )
+
+        test_loader = torch.utils.data.DataLoader(
+            test_data,
+            batch_size=args.test_mini_batch_size,
+            shuffle=False,
+            num_workers=args.test_num_workers,
+            collate_fn=collate_wrapper_criteo,
+            pin_memory=False,
+            drop_last=False,  # True
+        )
+
+    return train_data, train_loader, test_data, test_loader
+
+
+# uniform ditribution (input data)
+class RandomDataset(Dataset):
+    def __init__(
+        self,
+        m_den,
+        ln_emb,
+        data_size,
+        num_batches,
+        mini_batch_size,
+        num_indices_per_lookup,
+        num_indices_per_lookup_fixed,
+        num_targets=1,
+        round_targets=False,
+        data_generation="random",
+        trace_file="",
+        enable_padding=False,
+        reset_seed_on_access=False,
+        rand_data_dist="uniform",
+        rand_data_min=1,
+        rand_data_max=1,
+        rand_data_mu=-1,
+        rand_data_sigma=1,
+        rand_seed=0,
+    ):
+        # compute batch size
+        nbatches = int(np.ceil((data_size * 1.0) / mini_batch_size))
+        if num_batches != 0:
+            nbatches = num_batches
+            data_size = nbatches * mini_batch_size
+            # print("Total number of batches %d" % nbatches)
+
+        # save args (recompute data_size if needed)
+        self.m_den = m_den
+        self.ln_emb = ln_emb
+        self.data_size = data_size
+        self.num_batches = nbatches
+        self.mini_batch_size = mini_batch_size
+        self.num_indices_per_lookup = num_indices_per_lookup
+        self.num_indices_per_lookup_fixed = num_indices_per_lookup_fixed
+        self.num_targets = num_targets
+        self.round_targets = round_targets
+        self.data_generation = data_generation
+        self.trace_file = trace_file
+        self.enable_padding = enable_padding
+        self.reset_seed_on_access = reset_seed_on_access
+        self.rand_seed = rand_seed
+        self.rand_data_dist = rand_data_dist
+        self.rand_data_min = rand_data_min
+        self.rand_data_max = rand_data_max
+        self.rand_data_mu = rand_data_mu
+        self.rand_data_sigma = rand_data_sigma
+
+    def reset_numpy_seed(self, numpy_rand_seed):
+        np.random.seed(numpy_rand_seed)
+        # torch.manual_seed(numpy_rand_seed)
+
+    def __getitem__(self, index):
+
+        if isinstance(index, slice):
+            return [
+                self[idx]
+                for idx in range(
+                    index.start or 0, index.stop or len(self), index.step or 1
+                )
+            ]
+
+        # WARNING: reset seed on access to first element
+        # (e.g. if same random samples needed across epochs)
+        if self.reset_seed_on_access and index == 0:
+            self.reset_numpy_seed(self.rand_seed)
+
+        # number of data points in a batch
+        n = min(self.mini_batch_size, self.data_size - (index * self.mini_batch_size))
+
+        # generate a batch of dense and sparse features
+        if self.data_generation == "random":
+            (X, lS_o, lS_i) = generate_dist_input_batch(
+                self.m_den,
+                self.ln_emb,
+                n,
+                self.num_indices_per_lookup,
+                self.num_indices_per_lookup_fixed,
+                rand_data_dist=self.rand_data_dist,
+                rand_data_min=self.rand_data_min,
+                rand_data_max=self.rand_data_max,
+                rand_data_mu=self.rand_data_mu,
+                rand_data_sigma=self.rand_data_sigma,
+            )
+        elif self.data_generation == "synthetic":
+            (X, lS_o, lS_i) = generate_synthetic_input_batch(
+                self.m_den,
+                self.ln_emb,
+                n,
+                self.num_indices_per_lookup,
+                self.num_indices_per_lookup_fixed,
+                self.trace_file,
+                self.enable_padding,
+            )
+        else:
+            sys.exit(
+                "ERROR: --data-generation=" + self.data_generation + " is not supported"
+            )
+
+        # generate a batch of target (probability of a click)
+        T = generate_random_output_batch(n, self.num_targets, self.round_targets)
+
+        return (X, lS_o, lS_i, T)
+
+    def __len__(self):
+        # WARNING: note that we produce bacthes of outputs in __getitem__
+        # therefore we should use num_batches rather than data_size below
+        return self.num_batches
+
+
+def collate_wrapper_random_offset(list_of_tuples):
+    # where each tuple is (X, lS_o, lS_i, T)
+    (X, lS_o, lS_i, T) = list_of_tuples[0]
+    return (X, torch.stack(lS_o), lS_i, T)
+
+
+def collate_wrapper_random_length(list_of_tuples):
+    # where each tuple is (X, lS_o, lS_i, T)
+    (X, lS_o, lS_i, T) = list_of_tuples[0]
+    return (X, offset_to_length_converter(torch.stack(lS_o), lS_i), lS_i, T)
+
+
+def make_random_data_and_loader(
+    args,
+    ln_emb,
+    m_den,
+    offset_to_length_converter=False,
+):
+
+    train_data = RandomDataset(
+        m_den,
+        ln_emb,
+        args.data_size,
+        args.num_batches,
+        args.mini_batch_size,
+        args.num_indices_per_lookup,
+        args.num_indices_per_lookup_fixed,
+        1,  # num_targets
+        args.round_targets,
+        args.data_generation,
+        args.data_trace_file,
+        args.data_trace_enable_padding,
+        reset_seed_on_access=True,
+        rand_data_dist=args.rand_data_dist,
+        rand_data_min=args.rand_data_min,
+        rand_data_max=args.rand_data_max,
+        rand_data_mu=args.rand_data_mu,
+        rand_data_sigma=args.rand_data_sigma,
+        rand_seed=args.numpy_rand_seed,
+    )  # WARNING: generates a batch of lookups at once
+
+    test_data = RandomDataset(
+        m_den,
+        ln_emb,
+        args.data_size,
+        args.num_batches,
+        args.mini_batch_size,
+        args.num_indices_per_lookup,
+        args.num_indices_per_lookup_fixed,
+        1,  # num_targets
+        args.round_targets,
+        args.data_generation,
+        args.data_trace_file,
+        args.data_trace_enable_padding,
+        reset_seed_on_access=True,
+        rand_data_dist=args.rand_data_dist,
+        rand_data_min=args.rand_data_min,
+        rand_data_max=args.rand_data_max,
+        rand_data_mu=args.rand_data_mu,
+        rand_data_sigma=args.rand_data_sigma,
+        rand_seed=args.numpy_rand_seed,
+    )
+
+    collate_wrapper_random = collate_wrapper_random_offset
+    if offset_to_length_converter:
+        collate_wrapper_random = collate_wrapper_random_length
+
+    train_loader = torch.utils.data.DataLoader(
+        train_data,
+        batch_size=1,
+        shuffle=False,
+        num_workers=args.num_workers,
+        collate_fn=collate_wrapper_random,
+        pin_memory=False,
+        drop_last=False,  # True
+    )
+
+    test_loader = torch.utils.data.DataLoader(
+        test_data,
+        batch_size=1,
+        shuffle=False,
+        num_workers=args.num_workers,
+        collate_fn=collate_wrapper_random,
+        pin_memory=False,
+        drop_last=False,  # True
+    )
+    return train_data, train_loader, test_data, test_loader
+
+
+def generate_random_data(
+    m_den,
+    ln_emb,
+    data_size,
+    num_batches,
+    mini_batch_size,
+    num_indices_per_lookup,
+    num_indices_per_lookup_fixed,
+    num_targets=1,
+    round_targets=False,
+    data_generation="random",
+    trace_file="",
+    enable_padding=False,
+    length=False,  # length for caffe2 version (except dlrm_s_caffe2)
+):
+    nbatches = int(np.ceil((data_size * 1.0) / mini_batch_size))
+    if num_batches != 0:
+        nbatches = num_batches
+        data_size = nbatches * mini_batch_size
+    # print("Total number of batches %d" % nbatches)
+
+    # inputs
+    lT = []
+    lX = []
+    lS_offsets = []
+    lS_indices = []
+    for j in range(0, nbatches):
+        # number of data points in a batch
+        n = min(mini_batch_size, data_size - (j * mini_batch_size))
+
+        # generate a batch of dense and sparse features
+        if data_generation == "random":
+            (Xt, lS_emb_offsets, lS_emb_indices) = generate_uniform_input_batch(
+                m_den,
+                ln_emb,
+                n,
+                num_indices_per_lookup,
+                num_indices_per_lookup_fixed,
+                length,
+            )
+        elif data_generation == "synthetic":
+            (Xt, lS_emb_offsets, lS_emb_indices) = generate_synthetic_input_batch(
+                m_den,
+                ln_emb,
+                n,
+                num_indices_per_lookup,
+                num_indices_per_lookup_fixed,
+                trace_file,
+                enable_padding,
+            )
+        else:
+            sys.exit(
+                "ERROR: --data-generation=" + data_generation + " is not supported"
+            )
+        # dense feature
+        lX.append(Xt)
+        # sparse feature (sparse indices)
+        lS_offsets.append(lS_emb_offsets)
+        lS_indices.append(lS_emb_indices)
+
+        # generate a batch of target (probability of a click)
+        P = generate_random_output_batch(n, num_targets, round_targets)
+        lT.append(P)
+
+    return (nbatches, lX, lS_offsets, lS_indices, lT)
+
+
+def generate_random_output_batch(n, num_targets, round_targets=False):
+    # target (probability of a click)
+    if round_targets:
+        P = np.round(ra.rand(n, num_targets).astype(np.float32)).astype(np.float32)
+    else:
+        P = ra.rand(n, num_targets).astype(np.float32)
+
+    return torch.tensor(P)
+
+
+# uniform ditribution (input data)
+def generate_uniform_input_batch(
+    m_den,
+    ln_emb,
+    n,
+    num_indices_per_lookup,
+    num_indices_per_lookup_fixed,
+    length,
+):
+    # dense feature
+    Xt = torch.tensor(ra.rand(n, m_den).astype(np.float32))
+
+    # sparse feature (sparse indices)
+    lS_emb_offsets = []
+    lS_emb_indices = []
+    # for each embedding generate a list of n lookups,
+    # where each lookup is composed of multiple sparse indices
+    for size in ln_emb:
+        lS_batch_offsets = []
+        lS_batch_indices = []
+        offset = 0
+        for _ in range(n):
+            # num of sparse indices to be used per embedding (between
+            if num_indices_per_lookup_fixed:
+                sparse_group_size = np.int64(num_indices_per_lookup)
+            else:
+                # random between [1,num_indices_per_lookup])
+                r = ra.random(1)
+                sparse_group_size = np.int64(
+                    np.round(max([1.0], r * min(size, num_indices_per_lookup)))
+                )
+            # sparse indices to be used per embedding
+            r = ra.random(sparse_group_size)
+            sparse_group = np.unique(np.round(r * (size - 1)).astype(np.int64))
+            # reset sparse_group_size in case some index duplicates were removed
+            sparse_group_size = np.int32(sparse_group.size)
+            # store lengths and indices
+            if length:  # for caffe2 version
+                lS_batch_offsets += [sparse_group_size]
+            else:
+                lS_batch_offsets += [offset]
+            lS_batch_indices += sparse_group.tolist()
+            # update offset for next iteration
+            offset += sparse_group_size
+        lS_emb_offsets.append(torch.tensor(lS_batch_offsets))
+        lS_emb_indices.append(torch.tensor(lS_batch_indices))
+
+    return (Xt, lS_emb_offsets, lS_emb_indices)
+
+
+# random data from uniform or gaussian ditribution (input data)
+def generate_dist_input_batch(
+    m_den,
+    ln_emb,
+    n,
+    num_indices_per_lookup,
+    num_indices_per_lookup_fixed,
+    rand_data_dist,
+    rand_data_min,
+    rand_data_max,
+    rand_data_mu,
+    rand_data_sigma,
+):
+    # dense feature
+    Xt = torch.tensor(ra.rand(n, m_den).astype(np.float32))
+
+    # sparse feature (sparse indices)
+    lS_emb_offsets = []
+    lS_emb_indices = []
+    # for each embedding generate a list of n lookups,
+    # where each lookup is composed of multiple sparse indices
+    for size in ln_emb:
+        lS_batch_offsets = []
+        lS_batch_indices = []
+        offset = 0
+        for _ in range(n):
+            # num of sparse indices to be used per embedding (between
+            if num_indices_per_lookup_fixed:
+                sparse_group_size = np.int64(num_indices_per_lookup)
+            else:
+                # random between [1,num_indices_per_lookup])
+                r = ra.random(1)
+                sparse_group_size = np.int64(
+                    np.round(max([1.0], r * min(size, num_indices_per_lookup)))
+                )
+            # sparse indices to be used per embedding
+            if rand_data_dist == "gaussian":
+                if rand_data_mu == -1:
+                    rand_data_mu = (rand_data_max + rand_data_min) / 2.0
+                r = ra.normal(rand_data_mu, rand_data_sigma, sparse_group_size)
+                sparse_group = np.clip(r, rand_data_min, rand_data_max)
+                sparse_group = np.unique(sparse_group).astype(np.int64)
+            elif rand_data_dist == "uniform":
+                r = ra.random(sparse_group_size)
+                sparse_group = np.unique(np.round(r * (size - 1)).astype(np.int64))
+            else:
+                raise (
+                    rand_data_dist,
+                    "distribution is not supported. \
+                     please select uniform or gaussian",
+                )
+
+            # reset sparse_group_size in case some index duplicates were removed
+            sparse_group_size = np.int64(sparse_group.size)
+            # store lengths and indices
+            lS_batch_offsets += [offset]
+            lS_batch_indices += sparse_group.tolist()
+            # update offset for next iteration
+            offset += sparse_group_size
+        lS_emb_offsets.append(torch.tensor(lS_batch_offsets))
+        lS_emb_indices.append(torch.tensor(lS_batch_indices))
+
+    return (Xt, lS_emb_offsets, lS_emb_indices)
+
+
+# synthetic distribution (input data)
+def generate_synthetic_input_batch(
+    m_den,
+    ln_emb,
+    n,
+    num_indices_per_lookup,
+    num_indices_per_lookup_fixed,
+    trace_file,
+    enable_padding=False,
+):
+    # dense feature
+    Xt = torch.tensor(ra.rand(n, m_den).astype(np.float32))
+
+    # sparse feature (sparse indices)
+    lS_emb_offsets = []
+    lS_emb_indices = []
+    # for each embedding generate a list of n lookups,
+    # where each lookup is composed of multiple sparse indices
+    for i, size in enumerate(ln_emb):
+        lS_batch_offsets = []
+        lS_batch_indices = []
+        offset = 0
+        for _ in range(n):
+            # num of sparse indices to be used per embedding (between
+            if num_indices_per_lookup_fixed:
+                sparse_group_size = np.int64(num_indices_per_lookup)
+            else:
+                # random between [1,num_indices_per_lookup])
+                r = ra.random(1)
+                sparse_group_size = np.int64(
+                    max(1, np.round(r * min(size, num_indices_per_lookup))[0])
+                )
+            # sparse indices to be used per embedding
+            file_path = trace_file
+            line_accesses, list_sd, cumm_sd = read_dist_from_file(
+                file_path.replace("j", str(i))
+            )
+            # debug prints
+            # print("input")
+            # print(line_accesses); print(list_sd); print(cumm_sd);
+            # print(sparse_group_size)
+            # approach 1: rand
+            # r = trace_generate_rand(
+            #     line_accesses, list_sd, cumm_sd, sparse_group_size, enable_padding
+            # )
+            # approach 2: lru
+            r = trace_generate_lru(
+                line_accesses, list_sd, cumm_sd, sparse_group_size, enable_padding
+            )
+            # WARNING: if the distribution in the file is not consistent
+            # with embedding table dimensions, below mod guards against out
+            # of range access
+            sparse_group = np.unique(r).astype(np.int64)
+            minsg = np.min(sparse_group)
+            maxsg = np.max(sparse_group)
+            if (minsg < 0) or (size <= maxsg):
+                print(
+                    "WARNING: distribution is inconsistent with embedding "
+                    + "table size (using mod to recover and continue)"
+                )
+                sparse_group = np.mod(sparse_group, size).astype(np.int64)
+            # sparse_group = np.unique(np.array(np.mod(r, size-1)).astype(np.int64))
+            # reset sparse_group_size in case some index duplicates were removed
+            sparse_group_size = np.int64(sparse_group.size)
+            # store lengths and indices
+            lS_batch_offsets += [offset]
+            lS_batch_indices += sparse_group.tolist()
+            # update offset for next iteration
+            offset += sparse_group_size
+        lS_emb_offsets.append(torch.tensor(lS_batch_offsets))
+        lS_emb_indices.append(torch.tensor(lS_batch_indices))
+
+    return (Xt, lS_emb_offsets, lS_emb_indices)
+
+
+def generate_stack_distance(cumm_val, cumm_dist, max_i, i, enable_padding=False):
+    u = ra.rand(1)
+    if i < max_i:
+        # only generate stack distances up to the number of new references seen so far
+        j = bisect.bisect(cumm_val, i) - 1
+        fi = cumm_dist[j]
+        u *= fi  # shrink distribution support to exclude last values
+    elif enable_padding:
+        # WARNING: disable generation of new references (once all have been seen)
+        fi = cumm_dist[0]
+        u = (1.0 - fi) * u + fi  # remap distribution support to exclude first value
+
+    for (j, f) in enumerate(cumm_dist):
+        if u <= f:
+            return cumm_val[j]
+
+
+# WARNING: global define, must be consistent across all synthetic functions
+cache_line_size = 1
+
+
+def trace_generate_lru(
+    line_accesses, list_sd, cumm_sd, out_trace_len, enable_padding=False
+):
+    max_sd = list_sd[-1]
+    l = len(line_accesses)
+    i = 0
+    ztrace = deque()
+    for _ in range(out_trace_len):
+        sd = generate_stack_distance(list_sd, cumm_sd, max_sd, i, enable_padding)
+        mem_ref_within_line = 0  # floor(ra.rand(1)*cache_line_size) #0
+
+        # generate memory reference
+        if sd == 0:  # new reference #
+            line_ref = line_accesses[0]
+            del line_accesses[0]
+            line_accesses.append(line_ref)
+            mem_ref = np.uint64(line_ref * cache_line_size + mem_ref_within_line)
+            i += 1
+        else:  # existing reference #
+            line_ref = line_accesses[l - sd]
+            mem_ref = np.uint64(line_ref * cache_line_size + mem_ref_within_line)
+            del line_accesses[l - sd]
+            line_accesses.append(line_ref)
+        # save generated memory reference
+        ztrace.append(mem_ref)
+
+    return ztrace
+
+
+def trace_generate_rand(
+    line_accesses, list_sd, cumm_sd, out_trace_len, enable_padding=False
+):
+    max_sd = list_sd[-1]
+    l = len(line_accesses)  # !!!Unique,
+    i = 0
+    ztrace = []
+    for _ in range(out_trace_len):
+        sd = generate_stack_distance(list_sd, cumm_sd, max_sd, i, enable_padding)
+        mem_ref_within_line = 0  # floor(ra.rand(1)*cache_line_size) #0
+        # generate memory reference
+        if sd == 0:  # new reference #
+            line_ref = line_accesses.pop(0)
+            line_accesses.append(line_ref)
+            mem_ref = np.uint64(line_ref * cache_line_size + mem_ref_within_line)
+            i += 1
+        else:  # existing reference #
+            line_ref = line_accesses[l - sd]
+            mem_ref = np.uint64(line_ref * cache_line_size + mem_ref_within_line)
+        ztrace.append(mem_ref)
+
+    return ztrace
+
+
+def trace_profile(trace, enable_padding=False):
+    # number of elements in the array (assuming 1D)
+    # n = trace.size
+
+    rstack = deque()  # S
+    stack_distances = deque()  # SDS
+    line_accesses = deque()  # L
+    for x in trace:
+        r = np.uint64(x / cache_line_size)
+        l = len(rstack)
+        try:  # found #
+            i = rstack.index(r)
+            # WARNING: I believe below is the correct depth in terms of meaning of the
+            #          algorithm, but that is not what seems to be in the paper alg.
+            #          -1 can be subtracted if we defined the distance between
+            #          consecutive accesses (e.g. r, r) as 0 rather than 1.
+            sd = l - i  # - 1
+            # push r to the end of stack_distances
+            stack_distances.appendleft(sd)
+            # remove r from its position and insert to the top of stack
+            del rstack[i]  # rstack.remove(r)
+            rstack.append(r)
+        except ValueError:  # not found #
+            sd = 0  # -1
+            # push r to the end of stack_distances/line_accesses
+            stack_distances.appendleft(sd)
+            line_accesses.appendleft(r)
+            # push r to the top of stack
+            rstack.append(r)
+
+    if enable_padding:
+        # WARNING: notice that as the ratio between the number of samples (l)
+        # and cardinality [c] of a sample increases the probability of
+        # generating a sample gets smaller and smaller because there are
+        # few new samples compared to repeated samples. This means that for a
+        # long trace with relatively small cardinality it will take longer to
+        # generate all new samples and therefore obtain full distribution support
+        # and hence it takes longer for distribution to resemble the original.
+        # Therefore, we may pad the number of new samples to be on par with
+        # average number of samples l/c artificially.
+        l = len(stack_distances)
+        c = max(stack_distances)
+        padding = int(np.ceil(l / c))
+        stack_distances = stack_distances + [0] * padding
+
+    return (rstack, stack_distances, line_accesses)
+
+
+# auxiliary read/write routines
+def read_trace_from_file(file_path):
+    try:
+        with open(file_path) as f:
+            if args.trace_file_binary_type:
+                array = np.fromfile(f, dtype=np.uint64)
+                trace = array.astype(np.uint64).tolist()
+            else:
+                line = f.readline()
+                trace = list(map(lambda x: np.uint64(x), line.split(", ")))
+            return trace
+    except Exception:
+        print(f"ERROR: trace file '{file_path}' is not available.")
+
+
+def write_trace_to_file(file_path, trace):
+    try:
+        if args.trace_file_binary_type:
+            with open(file_path, "wb+") as f:
+                np.array(trace).astype(np.uint64).tofile(f)
+        else:
+            with open(file_path, "w+") as f:
+                s = str(list(trace))
+                f.write(s[1 : len(s) - 1])
+    except Exception:
+        print("ERROR: no output trace file has been provided")
+
+
+def read_dist_from_file(file_path):
+    try:
+        with open(file_path, "r") as f:
+            lines = f.read().splitlines()
+    except Exception:
+        print("{file_path} Wrong file or file path")
+    # read unique accesses
+    unique_accesses = [int(el) for el in lines[0].split(", ")]
+    # read cumulative distribution (elements are passed as two separate lists)
+    list_sd = [int(el) for el in lines[1].split(", ")]
+    cumm_sd = [float(el) for el in lines[2].split(", ")]
+
+    return unique_accesses, list_sd, cumm_sd
+
+
+def write_dist_to_file(file_path, unique_accesses, list_sd, cumm_sd):
+    try:
+        with open(file_path, "w") as f:
+            # unique_acesses
+            s = str(list(unique_accesses))
+            f.write(s[1 : len(s) - 1] + "\n")
+            # list_sd
+            s = str(list_sd)
+            f.write(s[1 : len(s) - 1] + "\n")
+            # cumm_sd
+            s = str(list(cumm_sd))
+            f.write(s[1 : len(s) - 1] + "\n")
+    except Exception:
+        print("Wrong file or file path")
+
+
+if __name__ == "__main__":
+    import argparse
+    import operator
+
+    ### parse arguments ###
+    parser = argparse.ArgumentParser(description="Generate Synthetic Distributions")
+    parser.add_argument("--trace-file", type=str, default="./input/trace.log")
+    parser.add_argument("--trace-file-binary-type", type=bool, default=False)
+    parser.add_argument("--trace-enable-padding", type=bool, default=False)
+    parser.add_argument("--dist-file", type=str, default="./input/dist.log")
+    parser.add_argument(
+        "--synthetic-file", type=str, default="./input/trace_synthetic.log"
+    )
+    parser.add_argument("--numpy-rand-seed", type=int, default=123)
+    parser.add_argument("--print-precision", type=int, default=5)
+    args = parser.parse_args()
+
+    ### some basic setup ###
+    np.random.seed(args.numpy_rand_seed)
+    np.set_printoptions(precision=args.print_precision)
+
+    ### read trace ###
+    trace = read_trace_from_file(args.trace_file)
+    # print(trace)
+
+    ### profile trace ###
+    (_, stack_distances, line_accesses) = trace_profile(
+        trace, args.trace_enable_padding
+    )
+    stack_distances.reverse()
+    line_accesses.reverse()
+    # print(line_accesses)
+    # print(stack_distances)
+
+    ### compute probability distribution ###
+    # count items
+    l = len(stack_distances)
+    dc = sorted(
+        collections.Counter(stack_distances).items(), key=operator.itemgetter(0)
+    )
+
+    # create a distribution
+    list_sd = list(map(lambda tuple_x_k: tuple_x_k[0], dc))  # x = tuple_x_k[0]
+    dist_sd = list(
+        map(lambda tuple_x_k: tuple_x_k[1] / float(l), dc)
+    )  # k = tuple_x_k[1]
+    cumm_sd = deque()  # np.cumsum(dc).tolist() #prefixsum
+    for i, (_, k) in enumerate(dc):
+        if i == 0:
+            cumm_sd.append(k / float(l))
+        else:
+            # add the 2nd element of the i-th tuple in the dist_sd list
+            cumm_sd.append(cumm_sd[i - 1] + (k / float(l)))
+
+    ### write stack_distance and line_accesses to a file ###
+    write_dist_to_file(args.dist_file, line_accesses, list_sd, cumm_sd)
+
+    ### generate corresponding synthetic ###
+    # line_accesses, list_sd, cumm_sd = read_dist_from_file(args.dist_file)
+    synthetic_trace = trace_generate_lru(
+        line_accesses, list_sd, cumm_sd, len(trace), args.trace_enable_padding
+    )
+    # synthetic_trace = trace_generate_rand(
+    #     line_accesses, list_sd, cumm_sd, len(trace), args.trace_enable_padding
+    # )
+    write_trace_to_file(args.synthetic_file, synthetic_trace)
diff --git a/models_v2/pytorch/dlrm/inference/gpu/dlrm_s_caffe2.py b/models_v2/pytorch/dlrm/inference/gpu/dlrm_s_caffe2.py
new file mode 100644
index 000000000..51c48d491
--- /dev/null
+++ b/models_v2/pytorch/dlrm/inference/gpu/dlrm_s_caffe2.py
@@ -0,0 +1,1817 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+# Description: an implementation of a deep learning recommendation model (DLRM)
+# The model input consists of dense and sparse features. The former is a vector
+# of floating point values. The latter is a list of sparse indices into
+# embedding tables, which consist of vectors of floating point values.
+# The selected vectors are passed to mlp networks denoted by triangles,
+# in some cases the vectors are interacted through operators (Ops).
+#
+# output:
+#                         vector of values
+# model:                        |
+#                              /\
+#                             /__\
+#                               |
+#       _____________________> Op  <___________________
+#     /                         |                      \
+#    /\                        /\                      /\
+#   /__\                      /__\           ...      /__\
+#    |                          |                       |
+#    |                         Op                      Op
+#    |                    ____/__\_____           ____/__\____
+#    |                   |_Emb_|____|__|    ...  |_Emb_|__|___|
+# input:
+# [ dense features ]     [sparse indices] , ..., [sparse indices]
+#
+# More precise definition of model layers:
+# 1) fully connected layers of an mlp
+# z = f(y)
+# y = Wx + b
+#
+# 2) embedding lookup (for a list of sparse indices p=[p1,...,pk])
+# z = Op(e1,...,ek)
+# obtain vectors e1=E[:,p1], ..., ek=E[:,pk]
+#
+# 3) Operator Op can be one of the following
+# Sum(e1,...,ek) = e1 + ... + ek
+# Dot(e1,...,ek) = [e1'e1, ..., e1'ek, ..., ek'e1, ..., ek'ek]
+# Cat(e1,...,ek) = [e1', ..., ek']'
+# where ' denotes transpose operation
+#
+# References:
+# [1] Maxim Naumov, Dheevatsa Mudigere, Hao-Jun Michael Shi, Jianyu Huang,
+# Narayanan Sundaram, Jongsoo Park, Xiaodong Wang, Udit Gupta, Carole-Jean Wu,
+# Alisson G. Azzolini, Dmytro Dzhulgakov, Andrey Mallevich, Ilia Cherniavskii,
+# Yinghai Lu, Raghuraman Krishnamoorthi, Ansha Yu, Volodymyr Kondratenko,
+# Stephanie Pereira, Xianjie Chen, Wenlin Chen, Vijay Rao, Bill Jia, Liang Xiong,
+# Misha Smelyanskiy, "Deep Learning Recommendation Model for Personalization and
+# Recommendation Systems", CoRR, arXiv:1906.00091, 2019
+
+from __future__ import absolute_import, division, print_function, unicode_literals
+
+import copy
+
+import functools
+
+# others
+import operator
+import time
+
+# onnx
+# The onnx import causes deprecation warnings every time workers
+# are spawned during testing. So, we filter out those warnings.
+import warnings
+
+# data generation
+import dlrm_data_pytorch as dp
+
+# numpy
+import numpy as np
+import sklearn.metrics
+
+with warnings.catch_warnings():
+    warnings.filterwarnings("ignore", category=DeprecationWarning)
+    try:
+        import caffe2.python.onnx.frontend
+        import onnx
+    except ImportError as error:
+        print("Unable to import onnx or caffe2.python.onnx.frontend ", error)
+
+# from caffe2.python import data_parallel_model
+
+# caffe2
+from caffe2.proto import caffe2_pb2
+from caffe2.python import brew, core, dyndep, model_helper, net_drawer, workspace
+
+"""
+# auxiliary routine used to split input on the mini-bacth dimension
+def where_to_split(mini_batch_size, ndevices, _add_leftover=False):
+    n = (mini_batch_size + ndevices - 1) // ndevices  # ceiling
+    l = mini_batch_size - n * (ndevices - 1)  # leftover
+    s = [n] * (ndevices - 1)
+    if _add_leftover:
+        ls += [l if l > 0 else n]
+    return ls
+"""
+
+
+### define dlrm in Caffe2 ###
+class DLRM_Net(object):
+    def FeedBlobWrapper(self, tag, val, add_prefix=True, split=False, device_id=-1):
+        if self.ndevices > 1 and add_prefix:
+            if split:
+                # split across devices
+                mini_batch_size = val.shape[0]
+                # approach 1: np and caffe2 operators assume the mini-batch size is
+                # divisible exactly by the number of available devices
+                if mini_batch_size % self.ndevices != 0:
+                    sys.exit(
+                        "ERROR: caffe2 net assumes that the mini_batch_size "
+                        + str(mini_batch_size)
+                        + " is evenly divisible by the number of available devices"
+                        + str(self.ndevices)
+                    )
+                vals = np.split(val, self.ndevices, axis=0)
+                """
+                # approach 2: np and caffe2 operators do not assume exact divisibility
+                if args.mini_batch_size != mini_batch_size:
+                    sys.exit("ERROR: caffe2 net was prepared for mini-batch size "
+                             + str(args.mini_batch_size)
+                             + " which is different from current mini-batch size "
+                             + str(mini_batch_size) + " being passed to it. "
+                             + "This is common for the last mini-batch, when "
+                             + "mini-batch size does not evenly divided the number of "
+                             + "elements in the data set.")
+                ls = where_to_split(mini_batch_size, self.ndevices)
+                vals = np.split(val, ls, axis=0)
+                """
+                # feed to multiple devices
+                for d in range(self.ndevices):
+                    tag_on_device = "gpu_" + str(d) + "/" + tag
+                    _d = core.DeviceOption(workspace.GpuDeviceType, d)
+                    workspace.FeedBlob(tag_on_device, vals[d], device_option=_d)
+            else:
+                # feed to multiple devices
+                for d in range(self.ndevices):
+                    tag_on_device = "gpu_" + str(d) + "/" + tag
+                    _d = core.DeviceOption(workspace.GpuDeviceType, d)
+                    workspace.FeedBlob(tag_on_device, val, device_option=_d)
+        else:
+            # feed to a single device (named or not)
+            if device_id >= 0:
+                _d = core.DeviceOption(workspace.GpuDeviceType, device_id)
+                workspace.FeedBlob(tag, val, device_option=_d)
+            else:
+                workspace.FeedBlob(tag, val)
+
+    def FetchBlobWrapper(self, tag, add_prefix=True, reduce_across=None, device_id=-1):
+        if self.ndevices > 1 and add_prefix:
+            # fetch from multiple devices
+            vals = []
+            for d in range(self.ndevices):
+                if tag.__class__ == list:
+                    tag_on_device = tag[d]
+                else:
+                    tag_on_device = "gpu_" + str(0) + "/" + tag
+                val = workspace.FetchBlob(tag_on_device)
+                vals.append(val)
+            # reduce across devices
+            if reduce_across == "add":
+                return functools.reduce(operator.add, vals)
+            elif reduce_across == "concat":
+                return np.concatenate(vals)
+            else:
+                return vals
+        else:
+            # fetch from a single device (named or not)
+            if device_id >= 0:
+                tag_on_device = "gpu_" + str(device_id) + "/" + tag
+                return workspace.FetchBlob(tag_on_device)
+            else:
+                return workspace.FetchBlob(tag)
+
+    def AddLayerWrapper(
+        self, layer, inp_blobs, out_blobs, add_prefix=True, reset_grad=False, **kwargs
+    ):
+        # auxiliary routine to adjust tags
+        def adjust_tag(blobs, on_device):
+            if blobs.__class__ == str:
+                _blobs = on_device + blobs
+            elif blobs.__class__ == list:
+                _blobs = list(map(lambda tag: on_device + tag, blobs))
+            else:  # blobs.__class__ == model_helper.ModelHelper or something else
+                _blobs = blobs
+            return _blobs
+
+        if self.ndevices > 1 and add_prefix:
+            # add layer on multiple devices
+            ll = []
+            for d in range(self.ndevices):
+                # add prefix on_device
+                on_device = "gpu_" + str(d) + "/"
+                _inp_blobs = adjust_tag(inp_blobs, on_device)
+                _out_blobs = adjust_tag(out_blobs, on_device)
+                # WARNING: reset_grad option was exlusively designed for WeightedSum
+                #         with inp_blobs=[w, tag_one, "", lr], where "" will be replaced
+                if reset_grad:
+                    w_grad = self.gradientMap[_inp_blobs[0]]
+                    _inp_blobs[2] = w_grad
+                # add layer to the model
+                with core.DeviceScope(core.DeviceOption(workspace.GpuDeviceType, d)):
+                    if kwargs:
+                        new_layer = layer(_inp_blobs, _out_blobs, **kwargs)
+                    else:
+                        new_layer = layer(_inp_blobs, _out_blobs)
+                ll.append(new_layer)
+            return ll
+        else:
+            # add layer on a single device
+            # WARNING: reset_grad option was exlusively designed for WeightedSum
+            #          with inp_blobs=[w, tag_one, "", lr], where "" will be replaced
+            if reset_grad:
+                w_grad = self.gradientMap[inp_blobs[0]]
+                inp_blobs[2] = w_grad
+            # add layer to the model
+            if kwargs:
+                new_layer = layer(inp_blobs, out_blobs, **kwargs)
+            else:
+                new_layer = layer(inp_blobs, out_blobs)
+            return new_layer
+
+    def create_mlp(self, ln, sigmoid_layer, model, tag):
+        (tag_layer, tag_in, tag_out) = tag
+
+        # build MLP layer by layer
+        layers = []
+        weights = []
+        for i in range(1, ln.size):
+            n = ln[i - 1]
+            m = ln[i]
+
+            # create tags
+            tag_fc_w = tag_layer + ":::" + "fc" + str(i) + "_w"
+            tag_fc_b = tag_layer + ":::" + "fc" + str(i) + "_b"
+            tag_fc_y = tag_layer + ":::" + "fc" + str(i) + "_y"
+            tag_fc_z = tag_layer + ":::" + "fc" + str(i) + "_z"
+            if i == ln.size - 1:
+                tag_fc_z = tag_out
+            weights.append(tag_fc_w)
+            weights.append(tag_fc_b)
+
+            # initialize the weights
+            # approach 1: custom Xavier input, output or two-sided fill
+            mean = 0.0  # std_dev = np.sqrt(variance)
+            std_dev = np.sqrt(2 / (m + n))  # np.sqrt(1 / m) # np.sqrt(1 / n)
+            W = np.random.normal(mean, std_dev, size=(m, n)).astype(np.float32)
+            std_dev = np.sqrt(1 / m)  # np.sqrt(2 / (m + 1))
+            b = np.random.normal(mean, std_dev, size=m).astype(np.float32)
+            self.FeedBlobWrapper(tag_fc_w, W)
+            self.FeedBlobWrapper(tag_fc_b, b)
+            # approach 2: caffe2 xavier
+            # W = self.AddLayerWrapper(
+            #     model.param_init_net.XavierFill,
+            #     [],
+            #     tag_fc_w,
+            #     shape=[m, n]
+            # )
+            # b = self.AddLayerWrapper(
+            #     model.param_init_net.ConstantFill,
+            #     [],
+            #     tag_fc_b,
+            #     shape=[m]
+            # )
+
+            # initialize the MLP's momentum for the Adagrad optimizer
+            if self.emb_optimizer in ["adagrad", "rwsadagrad"]:
+                # momentum of the weights
+                self.FeedBlobWrapper(
+                    "momentum_mlp_{}_{}".format(tag_layer, 2 * i - 1),
+                    np.full((m, n), 0, dtype=np.float32),
+                )
+                # momentum of the biases
+                self.FeedBlobWrapper(
+                    "momentum_mlp_{}_{}".format(tag_layer, 2 * i),
+                    np.full((m), 0, dtype=np.float32),
+                )
+
+            # save the blob shapes for latter (only needed if onnx is requested)
+            if self.save_onnx:
+                self.onnx_tsd[tag_fc_w] = (onnx.TensorProto.FLOAT, W.shape)
+                self.onnx_tsd[tag_fc_b] = (onnx.TensorProto.FLOAT, b.shape)
+
+            # approach 1: construct fully connected operator using model.net
+            fc = self.AddLayerWrapper(
+                model.net.FC, [tag_in, tag_fc_w, tag_fc_b], tag_fc_y
+            )
+            # approach 2: construct fully connected operator using brew
+            # https://github.com/caffe2/tutorials/blob/master/MNIST.ipynb
+            # fc = brew.fc(model, layer, tag_fc_w, dim_in=m, dim_out=n)
+            layers.append(fc)
+
+            if i == sigmoid_layer:
+                # approach 1: construct sigmoid operator using model.net
+                layer = self.AddLayerWrapper(model.net.Sigmoid, tag_fc_y, tag_fc_z)
+                # approach 2: using brew (which currently does not support sigmoid)
+                # tag_sigm = tag_layer + ":::" + "sigmoid" + str(i)
+                # layer = brew.sigmoid(model,fc,tag_sigmoid)
+            else:
+                # approach 1: construct relu operator using model.net
+                layer = self.AddLayerWrapper(model.net.Relu, tag_fc_y, tag_fc_z)
+                # approach 2: using brew
+                # tag_relu = tag_layer + ":::" + "relu" + str(i)
+                # layer = brew.relu(model,fc,tag_relu)
+            tag_in = tag_fc_z
+            layers.append(layer)
+
+        # WARNING: the dependency between layers is implicit in the tags,
+        # so only the last layer is added to the layers list. It will
+        # later be used for interactions.
+        return layers, weights
+
+    def create_emb(self, m, ln, model, tag):
+        (tag_layer, tag_in, tag_out) = tag
+        emb_l = []
+        weights_l = []
+        vw_l = []
+        for i in range(0, ln.size):
+            n = ln[i]
+
+            # select device
+            if self.ndevices > 1:
+                d = i % self.ndevices
+            else:
+                d = -1
+
+            # create tags
+            on_device = "" if self.ndevices <= 1 else "gpu_" + str(d) + "/"
+            len_s = on_device + tag_layer + ":::" + "sls" + str(i) + "_l"
+            ind_s = on_device + tag_layer + ":::" + "sls" + str(i) + "_i"
+            tbl_s = on_device + tag_layer + ":::" + "sls" + str(i) + "_w"
+            sum_s = on_device + tag_layer + ":::" + "sls" + str(i) + "_z"
+            weights_l.append(tbl_s)
+
+            # initialize the weights
+            # approach 1a: custom
+            W = np.random.uniform(
+                low=-np.sqrt(1 / n), high=np.sqrt(1 / n), size=(n, m)
+            ).astype(np.float32)
+            # approach 1b: numpy rand
+            # W = ra.rand(n, m).astype(np.float32)
+            self.FeedBlobWrapper(tbl_s, W, False, device_id=d)
+            # approach 2: caffe2 xavier
+            # with core.DeviceScope(core.DeviceOption(workspace.GpuDeviceType, d)):
+            #     W = model.param_init_net.XavierFill([], tbl_s, shape=[n, m])
+            # save the blob shapes for latter (only needed if onnx is requested)
+
+            # initialize the embedding's momentum for the Adagrad optimizer
+            if self.emb_optimizer == "adagrad":
+                self.FeedBlobWrapper(
+                    "momentum_emb_{}".format(i),
+                    np.full((n, m), 0),
+                    add_prefix=False,
+                    device_id=d,
+                )
+            elif self.emb_optimizer == "rwsadagrad":
+                self.FeedBlobWrapper(
+                    "momentum_emb_{}".format(i),
+                    np.full((n), 0),
+                    add_prefix=False,
+                    device_id=d,
+                )
+
+            if self.save_onnx:
+                self.onnx_tsd[tbl_s] = (onnx.TensorProto.FLOAT, W.shape)
+
+            # create operator
+            if self.weighted_pooling is not None:
+                vw_s = on_device + tag_layer + ":::" + "sls" + str(i) + "_v"
+                psw_s = on_device + tag_layer + ":::" + "sls" + str(i) + "_s"
+                VW = np.ones(n).astype(np.float32)
+                self.FeedBlobWrapper(vw_s, VW, False, device_id=d)
+                if self.weighted_pooling == "learned":
+                    vw_l.append(vw_s)
+                    grad_on_weights = True
+                else:
+                    grad_on_weights = False
+                if self.save_onnx:
+                    self.onnx_tsd[vw_s] = (onnx.TensorProto.FLOAT, VW.shape)
+                if self.ndevices <= 1:
+                    PSW = model.net.Gather([vw_s, ind_s], [psw_s])
+                    EE = model.net.SparseLengthsWeightedSum(
+                        [tbl_s, PSW, ind_s, len_s],
+                        [sum_s],
+                        grad_on_weights=grad_on_weights,
+                    )
+                else:
+                    with core.DeviceScope(
+                        core.DeviceOption(workspace.GpuDeviceType, d)
+                    ):
+                        PSW = model.net.Gather([vw_s, ind_s], [psw_s])
+                        EE = model.net.SparseLengthsWeightedSum(
+                            [tbl_s, PSW, ind_s, len_s],
+                            [sum_s],
+                            grad_on_weights=grad_on_weights,
+                        )
+            else:
+                if self.ndevices <= 1:
+                    EE = model.net.SparseLengthsSum([tbl_s, ind_s, len_s], [sum_s])
+                else:
+                    with core.DeviceScope(
+                        core.DeviceOption(workspace.GpuDeviceType, d)
+                    ):
+                        EE = model.net.SparseLengthsSum([tbl_s, ind_s, len_s], [sum_s])
+            emb_l.append(EE)
+
+        return emb_l, weights_l, vw_l
+
+    def create_interactions(self, x, ly, model, tag):
+        (tag_dense_in, tag_sparse_in, tag_int_out) = tag
+
+        if self.arch_interaction_op == "dot":
+            # concatenate dense and sparse features
+            tag_int_out_info = tag_int_out + "_info"
+            T, T_info = model.net.Concat(
+                x + ly,
+                [tag_int_out + "_cat_axis0", tag_int_out_info + "_cat_axis0"],
+                axis=1,
+                add_axis=1,
+            )
+            # perform a dot product
+            Z = model.net.BatchMatMul([T, T], tag_int_out + "_matmul", trans_b=1)
+            # append dense feature with the interactions (into a row vector)
+            # approach 1: all
+            # Zflat = model.net.Flatten(Z, tag_int_out + "_flatten", axis=1)
+            # approach 2: unique
+            Zflat_all = model.net.Flatten(Z, tag_int_out + "_flatten_all", axis=1)
+            Zflat = model.net.BatchGather(
+                [Zflat_all, tag_int_out + "_tril_indices"], tag_int_out + "_flatten"
+            )
+            R, R_info = model.net.Concat(
+                x + [Zflat], [tag_int_out, tag_int_out_info], axis=1
+            )
+        elif self.arch_interaction_op == "cat":
+            # concatenation features (into a row vector)
+            tag_int_out_info = tag_int_out + "_info"
+            R, R_info = model.net.Concat(
+                x + ly, [tag_int_out, tag_int_out_info], axis=1
+            )
+        else:
+            sys.exit(
+                "ERROR: --arch-interaction-op="
+                + self.arch_interaction_op
+                + " is not supported"
+            )
+
+        return R
+
+    def create_sequential_forward_ops(self):
+        # embeddings
+        tag = (self.temb, self.tsin, self.tsout)
+        self.emb_l, self.emb_w, self.emb_vw = self.create_emb(
+            self.m_spa, self.ln_emb, self.model, tag
+        )
+        # bottom mlp
+        tag = (self.tbot, self.tdin, self.tdout)
+        self.bot_l, self.bot_w = self.create_mlp(
+            self.ln_bot, self.sigmoid_bot, self.model, tag
+        )
+        # interactions
+        tag = (self.tdout, self.tsout, self.tint)
+        Z = self.create_interactions([self.bot_l[-1]], self.emb_l, self.model, tag)
+
+        # top mlp
+        tag = (self.ttop, Z, self.tout)
+        self.top_l, self.top_w = self.create_mlp(
+            self.ln_top, self.sigmoid_top, self.model, tag
+        )
+        # debug prints
+        # print(self.emb_l)
+        # print(self.bot_l)
+        # print(self.top_l)
+
+        # setup the last output variable
+        self.last_output = self.top_l[-1]
+
+    def create_parallel_forward_ops(self):
+        # distribute embeddings (model parallelism)
+        tag = (self.temb, self.tsin, self.tsout)
+        self.emb_l, self.emb_w, self.emb_vw = self.create_emb(
+            self.m_spa, self.ln_emb, self.model, tag
+        )
+        # replicate mlp (data parallelism)
+        tag = (self.tbot, self.tdin, self.tdout)
+        self.bot_l, self.bot_w = self.create_mlp(
+            self.ln_bot, self.sigmoid_bot, self.model, tag
+        )
+
+        # add communication (butterfly shuffle)
+        t_list = []
+        for i, emb_output in enumerate(self.emb_l):
+            # split input
+            src_d = i % self.ndevices
+            lo = [emb_output + "_split_" + str(d) for d in range(self.ndevices)]
+            # approach 1: np and caffe2 operators assume the mini-batch size is
+            # divisible exactly by the number of available devices
+            with core.DeviceScope(core.DeviceOption(workspace.GpuDeviceType, src_d)):
+                self.model.net.Split(emb_output, lo, axis=0)
+            """
+            # approach 2: np and caffe2 operators do not assume exact divisibility
+            ls = where_to_split(args.mini_batch_size, self.ndevices, _add_leftover=True)
+            with core.DeviceScope(core.DeviceOption(workspace.GpuDeviceType, src_d)):
+                emb_output_split = self.model.net.Split(
+                    emb_output, lo, split=lp, axis=0
+                )
+            """
+            # scatter
+            y = []
+            for dst_d in range(len(lo)):
+                src_blob = lo[dst_d]
+                dst_blob = str(src_blob).replace(
+                    "gpu_" + str(src_d), "gpu_" + str(dst_d), 1
+                )
+                if src_blob != dst_blob:
+                    with core.DeviceScope(
+                        core.DeviceOption(workspace.GpuDeviceType, dst_d)
+                    ):
+                        blob = self.model.Copy(src_blob, dst_blob)
+                else:
+                    blob = dst_blob
+                y.append(blob)
+            t_list.append(y)
+        # adjust lists to be ordered per device
+        x = list(map(lambda x: list(x), zip(*self.bot_l)))
+        ly = list(map(lambda y: list(y), zip(*t_list)))
+
+        # interactions
+        for d in range(self.ndevices):
+            on_device = "gpu_" + str(d) + "/"
+            tag = (
+                on_device + self.tdout,
+                on_device + self.tsout,
+                on_device + self.tint,
+            )
+            with core.DeviceScope(core.DeviceOption(workspace.GpuDeviceType, d)):
+                self.create_interactions([x[d][-1]], ly[d], self.model, tag)
+
+        # replicate mlp (data parallelism)
+        tag = (self.ttop, self.tint, self.tout)
+        self.top_l, self.top_w = self.create_mlp(
+            self.ln_top, self.sigmoid_top, self.model, tag
+        )
+
+        # debug prints
+        # print(self.model.net.Proto(),end='\n')
+        # sys.exit("ERROR: debugging")
+
+        # setup the last output variable
+        self.last_output = self.top_l[-1]
+
+    def __init__(
+        self,
+        m_spa,
+        ln_emb,
+        ln_bot,
+        ln_top,
+        arch_interaction_op,
+        arch_interaction_itself=False,
+        sigmoid_bot=-1,
+        sigmoid_top=-1,
+        save_onnx=False,
+        model=None,
+        test_net=None,
+        tag=None,
+        ndevices=-1,
+        forward_ops=True,
+        enable_prof=False,
+        weighted_pooling=None,
+        emb_optimizer="sgd",
+    ):
+        super(DLRM_Net, self).__init__()
+
+        # init model
+        if model is None:
+            global_init_opt = ["caffe2", "--caffe2_log_level=0"]
+            if enable_prof:
+                global_init_opt += [
+                    "--logtostderr=0",
+                    "--log_dir=$HOME",
+                    "--caffe2_logging_print_net_summary=1",
+                ]
+            workspace.GlobalInit(global_init_opt)
+            self.set_tags()
+            self.model = model_helper.ModelHelper(name="DLRM", init_params=True)
+            self.test_net = None
+        else:
+            # WARNING: assume that workspace and tags have been initialized elsewhere
+            self.set_tags(
+                tag[0],
+                tag[1],
+                tag[2],
+                tag[3],
+                tag[4],
+                tag[5],
+                tag[6],
+                tag[7],
+                tag[8],
+                tag[9],
+            )
+            self.model = model
+            self.test_net = test_net
+
+        # save arguments
+        self.m_spa = m_spa
+        self.ln_emb = ln_emb
+        self.ln_bot = ln_bot
+        self.ln_top = ln_top
+        self.arch_interaction_op = arch_interaction_op
+        self.arch_interaction_itself = arch_interaction_itself
+        self.sigmoid_bot = sigmoid_bot
+        self.sigmoid_top = sigmoid_top
+        self.save_onnx = save_onnx
+        self.ndevices = ndevices
+        self.emb_optimizer = emb_optimizer
+        if weighted_pooling is not None and weighted_pooling != "fixed":
+            self.weighted_pooling = "learned"
+        else:
+            self.weighted_pooling = weighted_pooling
+        # onnx types and shapes dictionary
+        if self.save_onnx:
+            self.onnx_tsd = {}
+        # create forward operators
+        if forward_ops:
+            if self.ndevices <= 1:
+                return self.create_sequential_forward_ops()
+            else:
+                return self.create_parallel_forward_ops()
+
+    def set_tags(
+        self,
+        _tag_layer_top_mlp="top",
+        _tag_layer_bot_mlp="bot",
+        _tag_layer_embedding="emb",
+        _tag_feature_dense_in="dense_in",
+        _tag_feature_dense_out="dense_out",
+        _tag_feature_sparse_in="sparse_in",
+        _tag_feature_sparse_out="sparse_out",
+        _tag_interaction="interaction",
+        _tag_dense_output="prob_click",
+        _tag_dense_target="target",
+    ):
+        # layer tags
+        self.ttop = _tag_layer_top_mlp
+        self.tbot = _tag_layer_bot_mlp
+        self.temb = _tag_layer_embedding
+        # dense feature tags
+        self.tdin = _tag_feature_dense_in
+        self.tdout = _tag_feature_dense_out
+        # sparse feature tags
+        self.tsin = _tag_feature_sparse_in
+        self.tsout = _tag_feature_sparse_out
+        # output and target tags
+        self.tint = _tag_interaction
+        self.ttar = _tag_dense_target
+        self.tout = _tag_dense_output
+
+    def parameters(self):
+        return self.model
+
+    def get_loss(self):
+        return self.FetchBlobWrapper(self.loss, reduce_across="add")
+
+    def get_output(self):
+        return self.FetchBlobWrapper(self.last_output, reduce_across="concat")
+
+    def create(self, X, S_lengths, S_indices, T):
+        self.create_input(X, S_lengths, S_indices, T)
+        self.create_model(X, S_lengths, S_indices, T)
+
+    def create_input(self, X, S_lengths, S_indices, T):
+        # feed input data to blobs
+        self.FeedBlobWrapper(self.tdin, X, split=True)
+        # save the blob shapes for latter (only needed if onnx is requested)
+        if self.save_onnx:
+            self.onnx_tsd[self.tdin] = (onnx.TensorProto.FLOAT, X.shape)
+
+        for i in range(len(self.emb_l)):
+            # select device
+            if self.ndevices > 1:
+                d = i % self.ndevices
+            else:
+                d = -1
+            # create tags
+            on_device = "" if self.ndevices <= 1 else "gpu_" + str(d) + "/"
+            len_s = on_device + self.temb + ":::" + "sls" + str(i) + "_l"
+            ind_s = on_device + self.temb + ":::" + "sls" + str(i) + "_i"
+            self.FeedBlobWrapper(len_s, np.array(S_lengths[i]), False, device_id=d)
+            self.FeedBlobWrapper(ind_s, np.array(S_indices[i]), False, device_id=d)
+            # save the blob shapes for latter (only needed if onnx is requested)
+            if self.save_onnx:
+                lshape = (len(S_lengths[i]),)  # =args.mini_batch_size
+                ishape = (len(S_indices[i]),)
+                self.onnx_tsd[len_s] = (onnx.TensorProto.INT32, lshape)
+                self.onnx_tsd[ind_s] = (onnx.TensorProto.INT32, ishape)
+
+        # feed target data to blobs
+        if T is not None:
+            zeros_fp32 = np.zeros(T.shape).astype(np.float32)
+            self.FeedBlobWrapper(self.ttar, zeros_fp32, split=True)
+            # save the blob shapes for latter (only needed if onnx is requested)
+            if self.save_onnx:
+                self.onnx_tsd[self.ttar] = (onnx.TensorProto.FLOAT, T.shape)
+
+    def create_model(self, X, S_lengths, S_indices, T):
+        # setup tril indices for the interactions
+        offset = 1 if self.arch_interaction_itself else 0
+        num_fea = len(self.emb_l) + 1
+        tril_indices = np.array(
+            [j + i * num_fea for i in range(num_fea) for j in range(i + offset)]
+        )
+        self.FeedBlobWrapper(self.tint + "_tril_indices", tril_indices)
+
+        # create compute graph
+        if T is not None:
+            # WARNING: RunNetOnce call is needed only if we use brew and ConstantFill.
+            # We could use direct calls to self.model functions above to avoid it
+            workspace.RunNetOnce(self.model.param_init_net)
+            workspace.CreateNet(self.model.net)
+            if self.test_net is not None:
+                workspace.CreateNet(self.test_net)
+
+    def run(self, X, S_lengths, S_indices, T, test_net=False, enable_prof=False):
+        # feed input data to blobs
+        # dense features
+        self.FeedBlobWrapper(self.tdin, X, split=True)
+        # sparse features
+        for i in range(len(self.emb_l)):
+            # select device
+            if self.ndevices > 1:
+                d = i % self.ndevices
+            else:
+                d = -1
+            # create tags
+            on_device = "" if self.ndevices <= 1 else "gpu_" + str(d) + "/"
+            len_s = on_device + self.temb + ":::" + "sls" + str(i) + "_l"
+            ind_s = on_device + self.temb + ":::" + "sls" + str(i) + "_i"
+            self.FeedBlobWrapper(len_s, np.array(S_lengths[i]), False, device_id=d)
+            self.FeedBlobWrapper(ind_s, np.array(S_indices[i]), False, device_id=d)
+
+        # feed target data to blobs if needed
+        if T is not None:
+            self.FeedBlobWrapper(self.ttar, T, split=True)
+            # execute compute graph
+            if test_net:
+                workspace.RunNet(self.test_net)
+            else:
+                if enable_prof:
+                    workspace.C.benchmark_net(self.model.net.Name(), 0, 1, True)
+                else:
+                    workspace.RunNet(self.model.net)
+        # debug prints
+        # print("intermediate")
+        # print(self.FetchBlobWrapper(self.bot_l[-1]))
+        # for tag_emb in self.emb_l:
+        #     print(self.FetchBlobWrapper(tag_emb))
+        # print(self.FetchBlobWrapper(self.tint))
+
+    def MSEloss(self, scale=1.0):
+        # add MSEloss to the model
+        self.AddLayerWrapper(self.model.SquaredL2Distance, [self.tout, self.ttar], "sd")
+        self.AddLayerWrapper(self.model.Scale, "sd", "sd2", scale=2.0 * scale)
+        # WARNING: "loss" is a special tag and should not be changed
+        self.loss = self.AddLayerWrapper(self.model.AveragedLoss, "sd2", "loss")
+
+    def BCEloss(self, scale=1.0, threshold=0.0):
+        # add BCEloss to the mode
+        if 0.0 < threshold and threshold < 1.0:
+            self.AddLayerWrapper(
+                self.model.Clip,
+                self.tout,
+                "tout_c",
+                min=threshold,
+                max=(1.0 - threshold),
+            )
+            self.AddLayerWrapper(self.model.MakeTwoClass, "tout_c", "tout_2c")
+        else:
+            self.AddLayerWrapper(self.model.MakeTwoClass, self.tout, "tout_2c")
+        self.AddLayerWrapper(self.model.LabelCrossEntropy, ["tout_2c", self.ttar], "sd")
+        # WARNING: "loss" is a special tag and should not be changed
+        if scale == 1.0:
+            self.loss = self.AddLayerWrapper(self.model.AveragedLoss, "sd", "loss")
+        else:
+            self.AddLayerWrapper(self.model.Scale, "sd", "sd2", scale=scale)
+            self.loss = self.AddLayerWrapper(self.model.AveragedLoss, "sd2", "loss")
+
+    def sgd_optimizer(
+        self, learning_rate, T=None, _gradientMap=None, sync_dense_params=True
+    ):
+        # create one, it and lr tags (or use them if already present)
+        if T is not None:
+            (tag_one, tag_it, tag_lr) = T
+        else:
+            (tag_one, tag_it, tag_lr) = ("const_one", "optim_it", "optim_lr")
+
+            # approach 1: feed values directly
+            # self.FeedBlobWrapper(tag_one, np.ones(1).astype(np.float32))
+            # self.FeedBlobWrapper(tag_it, np.zeros(1).astype(np.int64))
+            # it = self.AddLayerWrapper(self.model.Iter, tag_it, tag_it)
+            # lr = self.AddLayerWrapper(self.model.LearningRate, tag_it, tag_lr,
+            #                           base_lr=-1 * learning_rate, policy="fixed")
+            # approach 2: use brew
+            self.AddLayerWrapper(
+                self.model.param_init_net.ConstantFill,
+                [],
+                tag_one,
+                shape=[1],
+                value=1.0,
+            )
+            self.AddLayerWrapper(brew.iter, self.model, tag_it)
+            self.AddLayerWrapper(
+                self.model.LearningRate,
+                tag_it,
+                tag_lr,
+                base_lr=-1 * learning_rate,
+                policy="fixed",
+            )
+            # save the blob shapes for latter (only needed if onnx is requested)
+            if self.save_onnx:
+                self.onnx_tsd[tag_one] = (onnx.TensorProto.FLOAT, (1,))
+                self.onnx_tsd[tag_it] = (onnx.TensorProto.INT64, (1,))
+
+        # create gradient maps (or use them if already present)
+        if _gradientMap is not None:
+            self.gradientMap = _gradientMap
+        else:
+            if self.loss.__class__ == list:
+                self.gradientMap = self.model.AddGradientOperators(self.loss)
+            else:
+                self.gradientMap = self.model.AddGradientOperators([self.loss])
+
+        # update weights
+        # approach 1: builtin function
+        # optimizer.build_sgd(self.model, base_learning_rate=learning_rate)
+        # approach 2: custom code
+        # top MLP weight and bias
+        for w in self.top_w:
+            # allreduce across devices if needed
+            if sync_dense_params and self.ndevices > 1:
+                grad_blobs = [
+                    self.gradientMap["gpu_{}/".format(d) + w]
+                    for d in range(self.ndevices)
+                ]
+                self.model.NCCLAllreduce(grad_blobs, grad_blobs)
+            # update weights
+            self.AddLayerWrapper(
+                self.model.WeightedSum, [w, tag_one, "", tag_lr], w, reset_grad=True
+            )
+        # bottom MLP weight and bias
+        for w in self.bot_w:
+            # allreduce across devices if needed
+            if sync_dense_params and self.ndevices > 1:
+                grad_blobs = [
+                    self.gradientMap["gpu_{}/".format(d) + w]
+                    for d in range(self.ndevices)
+                ]
+                self.model.NCCLAllreduce(grad_blobs, grad_blobs)
+            # update weights
+            self.AddLayerWrapper(
+                self.model.WeightedSum, [w, tag_one, "", tag_lr], w, reset_grad=True
+            )
+        # update embeddings
+        for i, w in enumerate(self.emb_w):
+            # select device
+            if self.ndevices > 1:
+                d = i % self.ndevices
+            # create tags
+            on_device = "" if self.ndevices <= 1 else "gpu_" + str(d) + "/"
+            _tag_one = on_device + tag_one
+            _tag_lr = on_device + tag_lr
+            # pickup gradient
+            w_grad = self.gradientMap[w]
+            # update weights
+            if self.ndevices > 1:
+                with core.DeviceScope(core.DeviceOption(workspace.GpuDeviceType, d)):
+                    self.model.ScatterWeightedSum(
+                        [w, _tag_one, w_grad.indices, w_grad.values, _tag_lr], w
+                    )
+            else:
+                self.model.ScatterWeightedSum(
+                    [w, _tag_one, w_grad.indices, w_grad.values, _tag_lr], w
+                )
+
+        # update per sample weights
+        if self.weighted_pooling == "learned":
+            for i, w in enumerate(self.emb_vw):
+                # select device
+                if self.ndevices > 1:
+                    d = i % self.ndevices
+                # create tags
+                on_device = "" if self.ndevices <= 1 else "gpu_" + str(d) + "/"
+                _tag_one = on_device + tag_one
+                _tag_lr = on_device + tag_lr
+                # pickup gradient
+                w_grad = self.gradientMap[w]
+                # update weights
+                if self.ndevices > 1:
+                    with core.DeviceScope(
+                        core.DeviceOption(workspace.GpuDeviceType, d)
+                    ):
+                        self.model.ScatterWeightedSum(
+                            [w, _tag_one, w_grad.indices, w_grad.values, _tag_lr], w
+                        )
+                else:
+                    self.model.ScatterWeightedSum(
+                        [w, _tag_one, w_grad.indices, w_grad.values, _tag_lr], w
+                    )
+
+    def adagrad_optimizer(
+        self,
+        learning_rate,
+        T=None,
+        _gradientMap=None,
+        sync_dense_params=True,
+        epsilon=1e-10,
+        decay_=0.0,
+        weight_decay_=0.0,
+    ):
+        # create one, it and lr tags (or use them if already present)
+        if T is not None:
+            (tag_one, tag_it, tag_lr) = T
+        else:
+            (tag_one, tag_it, tag_lr) = ("const_one", "optim_it", "optim_lr")
+
+            # approach 1: feed values directly
+            # self.FeedBlobWrapper(tag_one, np.ones(1).astype(np.float32))
+            # self.FeedBlobWrapper(tag_it, np.zeros(1).astype(np.int64))
+            # it = self.AddLayerWrapper(self.model.Iter, tag_it, tag_it)
+            # lr = self.AddLayerWrapper(self.model.LearningRate, tag_it, tag_lr,
+            #                           base_lr=-1 * learning_rate, policy="fixed")
+            # approach 2: use brew
+            self.AddLayerWrapper(
+                self.model.param_init_net.ConstantFill,
+                [],
+                tag_one,
+                shape=[1],
+                value=1.0,
+            )
+            self.AddLayerWrapper(brew.iter, self.model, tag_it)
+            self.AddLayerWrapper(
+                self.model.LearningRate,
+                tag_it,
+                tag_lr,
+                base_lr=-1 * learning_rate,
+                policy="fixed",
+            )
+            # save the blob shapes for latter (only needed if onnx is requested)
+            if self.save_onnx:
+                self.onnx_tsd[tag_one] = (onnx.TensorProto.FLOAT, (1,))
+                self.onnx_tsd[tag_it] = (onnx.TensorProto.INT64, (1,))
+
+        # create gradient maps (or use them if already present)
+        if _gradientMap is not None:
+            self.gradientMap = _gradientMap
+        else:
+            if self.loss.__class__ == list:
+                self.gradientMap = self.model.AddGradientOperators(self.loss)
+            else:
+                self.gradientMap = self.model.AddGradientOperators([self.loss])
+
+        # update weights
+        # approach 1: builtin function
+        # optimizer.build_sgd(self.model, base_learning_rate=learning_rate)
+        # approach 2: custom code
+        # top MLP weight and bias
+        for i, w in enumerate(self.top_w):
+            # allreduce across devices if needed
+            if sync_dense_params and self.ndevices > 1:
+                grad_blobs = [
+                    self.gradientMap["gpu_{}/".format(d) + w]
+                    for d in range(self.ndevices)
+                ]
+                self.model.NCCLAllreduce(grad_blobs, grad_blobs)
+            # update weights
+            self.model.Adagrad(
+                [w, "momentum_mlp_top_{}".format(i + 1), self.gradientMap[w], tag_lr],
+                [w, "momentum_mlp_top_{}".format(i + 1)],
+                epsilon=epsilon,
+                decay_=decay_,
+                weight_decay_=weight_decay_,
+            )
+
+        # bottom MLP weight and bias
+        for i, w in enumerate(self.bot_w):
+            # allreduce across devices if needed
+            if sync_dense_params and self.ndevices > 1:
+                grad_blobs = [
+                    self.gradientMap["gpu_{}/".format(d) + w]
+                    for d in range(self.ndevices)
+                ]
+                self.model.NCCLAllreduce(grad_blobs, grad_blobs)
+            # update weights
+            self.model.Adagrad(
+                [w, "momentum_mlp_bot_{}".format(i + 1), self.gradientMap[w], tag_lr],
+                [w, "momentum_mlp_bot_{}".format(i + 1)],
+                epsilon=epsilon,
+                decay_=decay_,
+                weight_decay_=weight_decay_,
+            )
+
+        # update embeddings
+        for i, w in enumerate(self.emb_w):
+            # select device
+            if self.ndevices > 1:
+                d = i % self.ndevices
+            # create tags
+            on_device = "" if self.ndevices <= 1 else "gpu_" + str(d) + "/"
+            _tag_one = on_device + tag_one
+            _tag_lr = on_device + tag_lr
+            # pickup gradient
+            w_grad = self.gradientMap[w]
+            # update weights
+            def add_optimizer():
+                self.model.Unique(
+                    w_grad.indices,
+                    ["unique_w_grad_indices", "remapping_w_grad_indices"],
+                )
+                self.model.UnsortedSegmentSum(
+                    [w_grad.values, "remapping_w_grad_indices"], "unique_w_grad_values"
+                )
+
+                if self.emb_optimizer == "adagrad":
+                    self.model.SparseAdagrad(
+                        [
+                            w,
+                            "momentum_emb_{}".format(i),
+                            "unique_w_grad_indices",
+                            "unique_w_grad_values",
+                            _tag_lr,
+                        ],
+                        [w, "momentum_emb_{}".format(i)],
+                        epsilon=epsilon,
+                        decay_=decay_,
+                        weight_decay_=weight_decay_,
+                    )
+
+                elif self.emb_optimizer == "rwsadagrad":
+                    self.model.RowWiseSparseAdagrad(
+                        [
+                            w,
+                            "momentum_emb_{}".format(i),
+                            "unique_w_grad_indices",
+                            "unique_w_grad_values",
+                            _tag_lr,
+                        ],
+                        [w, "momentum_emb_{}".format(i)],
+                        epsilon=epsilon,
+                        decay_=decay_,
+                        weight_decay_=weight_decay_,
+                    )
+
+            if self.ndevices > 1:
+                with core.DeviceScope(core.DeviceOption(workspace.GpuDeviceType, d)):
+                    add_optimizer()
+            else:
+                add_optimizer()
+
+        # update per sample weights
+        if self.weighted_pooling == "learned":
+            for i, w in enumerate(self.emb_vw):
+                # select device
+                if self.ndevices > 1:
+                    d = i % self.ndevices
+                # create tags
+                on_device = "" if self.ndevices <= 1 else "gpu_" + str(d) + "/"
+                _tag_one = on_device + tag_one
+                _tag_lr = on_device + tag_lr
+                # pickup gradient
+                w_grad = self.gradientMap[w]
+                # update weights
+                if self.ndevices > 1:
+                    with core.DeviceScope(
+                        core.DeviceOption(workspace.GpuDeviceType, d)
+                    ):
+                        self.model.ScatterWeightedSum(
+                            [w, _tag_one, w_grad.indices, w_grad.values, _tag_lr], w
+                        )
+                else:
+                    self.model.ScatterWeightedSum(
+                        [w, _tag_one, w_grad.indices, w_grad.values, _tag_lr], w
+                    )
+
+    def print_all(self):
+        # approach 1: all
+        print(workspace.Blobs(), end="\n")
+        for _, l in enumerate(workspace.Blobs()):
+            print(l)
+            print(self.FetchBlobWrapper(l))
+        # approach 2: only summary
+        # for param in self.model.params:
+        #    self.model.Summarize(param, [], to_file=1)
+        #    self.model.Summarize(self.model.param_to_grad[param], [], to_file=1)
+
+    def print_weights(self):
+        for _, l in enumerate(self.emb_w):
+            # print(l)
+            print(self.FetchBlobWrapper(l, False))
+        if self.weighted_pooling == "learned":
+            for _, l in enumerate(self.emb_vw):
+                # print(l)
+                print(self.FetchBlobWrapper(l, False))
+        for _, l in enumerate(self.bot_w):
+            # print(l)
+            if self.ndevices > 1:
+                print(self.FetchBlobWrapper(l, False, device_id=0))
+            else:
+                print(self.FetchBlobWrapper(l))
+        for _, l in enumerate(self.top_w):
+            # print(l)
+            if self.ndevices > 1:
+                print(self.FetchBlobWrapper(l, False, device_id=0))
+            else:
+                print(self.FetchBlobWrapper(l))
+
+    def print_activations(self):
+        for _, l in enumerate(self.emb_l):
+            print(l)
+            print(self.FetchBlobWrapper(l, False))
+        for _, l in enumerate(self.bot_l):
+            print(l)
+            print(self.FetchBlobWrapper(l))
+        print(self.tint)
+        print(self.FetchBlobWrapper(self.tint))
+        for _, l in enumerate(self.top_l):
+            print(l)
+            print(self.FetchBlobWrapper(l))
+
+
+def define_metrics():
+    metrics = {
+        "loss": lambda y_true, y_score: sklearn.metrics.log_loss(
+            y_true=y_true, y_pred=y_score, labels=[0, 1]
+        ),
+        "recall": lambda y_true, y_score: sklearn.metrics.recall_score(
+            y_true=y_true, y_pred=np.round(y_score)
+        ),
+        "precision": lambda y_true, y_score: sklearn.metrics.precision_score(
+            y_true=y_true, y_pred=np.round(y_score)
+        ),
+        "f1": lambda y_true, y_score: sklearn.metrics.f1_score(
+            y_true=y_true, y_pred=np.round(y_score)
+        ),
+        "ap": sklearn.metrics.average_precision_score,
+        "roc_auc": sklearn.metrics.roc_auc_score,
+        "accuracy": lambda y_true, y_score: sklearn.metrics.accuracy_score(
+            y_true=y_true, y_pred=np.round(y_score)
+        ),
+        # 'pre_curve' : sklearn.metrics.precision_recall_curve,
+        # 'roc_curve' :  sklearn.metrics.roc_curve,
+    }
+    return metrics
+
+
+def calculate_metrics(targets, scores):
+    scores = np.concatenate(scores, axis=0)
+    targets = np.concatenate(targets, axis=0)
+
+    metrics = define_metrics()
+
+    # print("Compute time for validation metric : ", end="")
+    # first_it = True
+    validation_results = {}
+    for metric_name, metric_function in metrics.items():
+        # if first_it:
+        #     first_it = False
+        # else:
+        #     print(", ", end="")
+        # metric_compute_start = time_wrap(False)
+        try:
+            validation_results[metric_name] = metric_function(targets, scores)
+        except Exception as error:
+            validation_results[metric_name] = -1
+            print("{} in calculating {}".format(error, metric_name))
+        # metric_compute_end = time_wrap(False)
+        # met_time = metric_compute_end - metric_compute_start
+        # print("{} {:.4f}".format(metric_name, 1000 * (met_time)),
+        #      end="")
+    # print(" ms")
+    return validation_results
+
+
+if __name__ == "__main__":
+    import argparse
+
+    ### import packages ###
+    import sys
+
+    ### parse arguments ###
+    parser = argparse.ArgumentParser(
+        description="Train Deep Learning Recommendation Model (DLRM)"
+    )
+    # model related parameters
+    parser.add_argument("--arch-sparse-feature-size", type=int, default=2)
+    parser.add_argument("--arch-embedding-size", type=str, default="4-3-2")
+    parser.add_argument("--arch-mlp-bot", type=str, default="4-3-2")
+    parser.add_argument("--arch-mlp-top", type=str, default="4-2-1")
+    parser.add_argument("--arch-interaction-op", type=str, default="dot")
+    parser.add_argument("--arch-interaction-itself", action="store_true", default=False)
+    # activations and loss
+    parser.add_argument("--activation-function", type=str, default="relu")
+    parser.add_argument("--loss-function", type=str, default="mse")  # or bce
+    parser.add_argument("--loss-threshold", type=float, default=0.0)  # 1.0e-7
+    parser.add_argument("--round-targets", type=bool, default=False)
+    parser.add_argument("--weighted-pooling", type=str, default=None)
+    # data
+    parser.add_argument("--data-size", type=int, default=1)
+    parser.add_argument("--num-batches", type=int, default=0)
+    parser.add_argument(
+        "--data-generation", type=str, default="random"
+    )  # or synthetic or dataset
+    parser.add_argument(
+        "--rand-data-dist", type=str, default="uniform"
+    )  # uniform or gaussian
+    parser.add_argument("--rand-data-min", type=float, default=0)
+    parser.add_argument("--rand-data-max", type=float, default=1)
+    parser.add_argument("--rand-data-mu", type=float, default=-1)
+    parser.add_argument("--rand-data-sigma", type=float, default=1)
+    parser.add_argument("--data-trace-file", type=str, default="./input/dist_emb_j.log")
+    parser.add_argument("--data-set", type=str, default="kaggle")  # or terabyte
+    parser.add_argument("--raw-data-file", type=str, default="")
+    parser.add_argument("--processed-data-file", type=str, default="")
+    parser.add_argument("--data-randomize", type=str, default="total")  # or day or none
+    parser.add_argument("--data-trace-enable-padding", type=bool, default=False)
+    parser.add_argument("--max-ind-range", type=int, default=-1)
+    parser.add_argument("--data-sub-sample-rate", type=float, default=0.0)  # in [0, 1]
+    parser.add_argument("--num-indices-per-lookup", type=int, default=10)
+    parser.add_argument("--num-indices-per-lookup-fixed", type=bool, default=False)
+    parser.add_argument("--num-workers", type=int, default=0)
+    parser.add_argument("--memory-map", action="store_true", default=False)
+    # training
+    parser.add_argument("--mini-batch-size", type=int, default=1)
+    parser.add_argument("--nepochs", type=int, default=1)
+    parser.add_argument("--learning-rate", type=float, default=0.01)
+    parser.add_argument("--print-precision", type=int, default=5)
+    parser.add_argument("--numpy-rand-seed", type=int, default=123)
+    parser.add_argument("--sync-dense-params", type=bool, default=True)
+    parser.add_argument("--caffe2-net-type", type=str, default="")
+    parser.add_argument(
+        "--optimizer",
+        type=str,
+        default="sgd",
+        help="""This is the optimizer for embedding tables.""",
+    )
+    parser.add_argument(
+        "--dataset-multiprocessing",
+        action="store_true",
+        default=False,
+        help="The Kaggle dataset can be multiprocessed in an environment \
+                        with more than 7 CPU cores and more than 20 GB of memory. \n \
+                        The Terabyte dataset can be multiprocessed in an environment \
+                        with more than 24 CPU cores and at least 1 TB of memory.",
+    )
+    # inference
+    parser.add_argument("--inference-only", action="store_true", default=False)
+    # onnx (or protobuf with shapes)
+    parser.add_argument("--save-onnx", action="store_true", default=False)
+    parser.add_argument("--save-proto-types-shapes", action="store_true", default=False)
+    # gpu
+    parser.add_argument("--use-gpu", action="store_true", default=False)
+    # debugging and profiling
+    parser.add_argument("--print-freq", type=int, default=1)
+    parser.add_argument("--test-freq", type=int, default=-1)
+    parser.add_argument("--test-mini-batch-size", type=int, default=-1)
+    parser.add_argument("--test-num-workers", type=int, default=-1)
+    parser.add_argument("--print-time", action="store_true", default=False)
+    parser.add_argument("--debug-mode", action="store_true", default=False)
+    parser.add_argument("--enable-profiling", action="store_true", default=False)
+    parser.add_argument("--plot-compute-graph", action="store_true", default=False)
+    # mlperf logging (disables other output and stops early)
+    parser.add_argument("--mlperf-logging", action="store_true", default=False)
+    # stop at target accuracy Kaggle 0.789, Terabyte (sub-sampled=0.875) 0.8107
+    parser.add_argument("--mlperf-acc-threshold", type=float, default=0.0)
+    # stop at target AUC Terabyte (no subsampling) 0.8025
+    parser.add_argument("--mlperf-auc-threshold", type=float, default=0.0)
+    args = parser.parse_args()
+
+    if args.dataset_multiprocessing:
+        assert float(sys.version[:3]) > 3.7, (
+            "The dataset_multiprocessing "
+            + "flag is susceptible to a bug in Python 3.7 and under. "
+            + "https://github.com/facebookresearch/dlrm/issues/172"
+        )
+
+    ### some basic setup ###
+    # WARNING: to obtain exactly the same initialization for
+    # the weights we need to start from the same random seed.
+    np.random.seed(args.numpy_rand_seed)
+
+    np.set_printoptions(precision=args.print_precision)
+    if args.test_mini_batch_size < 0:
+        # if the parameter is not set, use the training batch size
+        args.test_mini_batch_size = args.mini_batch_size
+    if args.test_num_workers < 0:
+        # if the parameter is not set, use the same parameter for training
+        args.test_num_workers = args.num_workers
+
+    use_gpu = args.use_gpu
+    if use_gpu:
+        device_opt = core.DeviceOption(workspace.GpuDeviceType, 0)
+        ngpus = workspace.NumGpuDevices()  # 1
+        print("Using {} GPU(s)...".format(ngpus))
+    else:
+        device_opt = core.DeviceOption(caffe2_pb2.CPU)
+        print("Using CPU...")
+
+    ### prepare training data ###
+    ln_bot = np.fromstring(args.arch_mlp_bot, dtype=int, sep="-")
+    if args.data_generation == "dataset":
+        if args.num_workers > 0 or args.test_num_workers > 0:
+            print(
+                "WARNING: non default --num-workers or --test-num-workers options"
+                + " are not supported and will be ignored"
+            )
+        if args.mini_batch_size != args.test_mini_batch_size:
+            print(
+                "WARNING: non default ----test-mini-batch-size option"
+                + " is not supported and will be ignored"
+            )
+
+        # input and target from dataset
+
+        train_data, train_ld, test_data, test_ld = dp.make_criteo_data_and_loaders(
+            args,
+            offset_to_length_converter=True,
+        )
+
+        nbatches = args.num_batches if args.num_batches > 0 else len(train_ld)
+
+        nbatches_test = len(test_ld)
+
+        ln_emb = train_data.counts
+        m_den = train_data.m_den
+
+        # enforce maximum limit on number of vectors per embedding
+        if args.max_ind_range > 0:
+            ln_emb = np.array(
+                list(
+                    map(
+                        lambda x: x if x < args.max_ind_range else args.max_ind_range,
+                        ln_emb,
+                    )
+                )
+            )
+        ln_bot[0] = m_den
+
+    else:
+        if args.num_workers > 0 or args.test_num_workers > 0:
+            print(
+                "WARNING: non default --num-workers or --test-num-workers options"
+                + " are not supported and will be ignored"
+            )
+        if args.mini_batch_size != args.test_mini_batch_size:
+            print(
+                "WARNING: non default ----test-mini-batch-size option"
+                + " is not supported and will be ignored"
+            )
+
+        # input and target at random
+        ln_emb = np.fromstring(args.arch_embedding_size, dtype=int, sep="-")
+        m_den = ln_bot[0]
+        train_data, train_ld, test_data, test_ld = dp.make_random_data_and_loader(
+            args,
+            ln_emb,
+            m_den,
+            offset_to_length_converter=True,
+        )
+        nbatches = args.num_batches if args.num_batches > 0 else len(train_ld)
+        nbatches_test = len(test_ld)
+        # table_feature_map = {idx : idx for idx in range(len(ln_emb))}
+
+    ### parse command line arguments ###
+    m_spa = args.arch_sparse_feature_size
+    ln_emb = np.asarray(ln_emb)
+    num_fea = ln_emb.size + 1  # num sparse + num dense features
+    m_den_out = ln_bot[ln_bot.size - 1]
+    if args.arch_interaction_op == "dot":
+        # approach 1: all
+        # num_int = num_fea * num_fea + m_den_out
+        # approach 2: unique
+        if args.arch_interaction_itself:
+            num_int = (num_fea * (num_fea + 1)) // 2 + m_den_out
+        else:
+            num_int = (num_fea * (num_fea - 1)) // 2 + m_den_out
+    elif args.arch_interaction_op == "cat":
+        num_int = num_fea * m_den_out
+    else:
+        sys.exit(
+            "ERROR: --arch-interaction-op="
+            + args.arch_interaction_op
+            + " is not supported"
+        )
+    arch_mlp_top_adjusted = str(num_int) + "-" + args.arch_mlp_top
+    ln_top = np.fromstring(arch_mlp_top_adjusted, dtype=int, sep="-")
+    # sanity check: feature sizes and mlp dimensions must match
+    if m_den != ln_bot[0]:
+        sys.exit(
+            "ERROR: arch-dense-feature-size "
+            + str(m_den)
+            + " does not match first dim of bottom mlp "
+            + str(ln_bot[0])
+        )
+    if m_spa != m_den_out:
+        sys.exit(
+            "ERROR: arch-sparse-feature-size "
+            + str(m_spa)
+            + " does not match last dim of bottom mlp "
+            + str(m_den_out)
+        )
+    if num_int != ln_top[0]:
+        sys.exit(
+            "ERROR: # of feature interactions "
+            + str(num_int)
+            + " does not match first dim of top mlp "
+            + str(ln_top[0])
+        )
+
+    # test prints (model arch)
+    if args.debug_mode:
+        print("model arch:")
+        print(
+            "mlp top arch "
+            + str(ln_top.size - 1)
+            + " layers, with input to output dimensions:"
+        )
+        print(ln_top)
+
+        print("# of interactions")
+        print(num_int)
+        print(
+            "mlp bot arch "
+            + str(ln_bot.size - 1)
+            + " layers, with input to output dimensions:"
+        )
+        print(ln_bot)
+        print("# of features (sparse and dense)")
+        print(num_fea)
+        print("dense feature size")
+        print(m_den)
+        print("sparse feature size")
+        print(m_spa)
+        print(
+            "# of embeddings (= # of sparse features) "
+            + str(ln_emb.size)
+            + ", with dimensions "
+            + str(m_spa)
+            + "x:"
+        )
+        print(ln_emb)
+
+        print("data (inputs and targets):")
+        for j, inputBatch in enumerate(train_ld):
+            lX_j, lS_l_j, lS_i_j, lT_j = inputBatch
+            print("mini-batch: %d" % j)
+            print(lX_j)
+            print(lS_l_j)
+            print(lS_i_j)
+            print(lT_j)
+
+    ### construct the neural network specified above ###
+    # WARNING: to obtain exactly the same initialization for
+    # the weights we need to start from the same random seed.
+    # np.random.seed(args.numpy_rand_seed)
+    ndevices = min(ngpus, args.mini_batch_size, num_fea - 1) if use_gpu else -1
+    flag_types_shapes = args.save_onnx or args.save_proto_types_shapes
+    flag_forward_ops = not (use_gpu and ndevices > 1)
+    with core.DeviceScope(device_opt):
+        dlrm = DLRM_Net(
+            m_spa,
+            ln_emb,
+            ln_bot,
+            ln_top,
+            args.arch_interaction_op,
+            arch_interaction_itself=args.arch_interaction_itself,
+            sigmoid_bot=-1,
+            sigmoid_top=ln_top.size - 1,
+            save_onnx=flag_types_shapes,
+            ndevices=ndevices,
+            # forward_ops = flag_forward_ops
+            enable_prof=args.enable_profiling,
+            weighted_pooling=args.weighted_pooling,
+            emb_optimizer=args.optimizer,
+        )
+    # load nccl if using multiple devices
+    if args.sync_dense_params and ndevices > 1:
+        dyndep.InitOpsLibrary("//caffe2/caffe2/contrib/nccl:nccl_ops")
+    # set the net type for better performance (dag, async_scheduling, etc)
+    if args.caffe2_net_type:
+        dlrm.parameters().net.Proto().type = args.caffe2_net_type
+    # plot compute graph
+    if args.plot_compute_graph:
+        graph = net_drawer.GetPydotGraph(
+            dlrm.parameters().net, "dlrm_s_caffe2_graph", "BT"
+        )
+        graph.write_pdf(graph.get_name() + ".pdf")
+    # test prints
+    if args.debug_mode:
+        print("initial parameters (weights and bias):")
+        dlrm.print_weights()
+
+    # add training loss if needed
+    if not args.inference_only:
+        with core.DeviceScope(device_opt):
+            # specify the loss function
+            nd = 1.0 if dlrm.ndevices <= 1 else 1.0 / dlrm.ndevices  # 1
+            if args.loss_function == "mse":
+                dlrm.MSEloss(scale=nd)
+            elif args.loss_function == "bce":
+                dlrm.BCEloss(scale=nd, threshold=args.loss_threshold)
+            else:
+                sys.exit(
+                    "ERROR: --loss-function=" + args.loss_function + " is not supported"
+                )
+
+            # define test net (as train net without gradients)
+            dlrm.test_net = core.Net(copy.deepcopy(dlrm.model.net.Proto()))
+
+            # specify the optimizer algorithm
+            if args.optimizer == "sgd":
+                dlrm.sgd_optimizer(
+                    args.learning_rate, sync_dense_params=args.sync_dense_params
+                )
+            elif args.optimizer in ["adagrad", "rwsadagrad"]:
+                dlrm.adagrad_optimizer(
+                    args.learning_rate, sync_dense_params=args.sync_dense_params
+                )
+            else:
+                sys.exit(
+                    """ERROR: Select an optimizer for
+                                embedding tables : 'sgd', 'adagrad',
+                                or 'rwsadagrad' """
+                )
+
+    # init/create
+    X, lS_l, lS_i, T = next(
+        iter(train_ld)
+    )  # does not affect the enumerate(train_ld) in the main loop
+    dlrm.create(X, lS_l, lS_i, T.int())
+
+    ### main loop ###
+    best_gA_test = 0
+    best_auc_test = 0
+    total_time = 0
+    total_loss = 0
+    total_accu = 0
+    total_iter = 0
+    total_samp = 0
+    k = 0
+
+    print("time/loss/accuracy (if enabled):")
+    while k < args.nepochs:
+        j = 0
+        for j, inputBatch in enumerate(train_ld):
+            # forward and backward pass, where the latter runs only
+            # when gradients and loss have been added to the net
+            time1 = time.time()
+            lX_j, lS_l_j, lS_i_j, lT_j = inputBatch
+            lT_j = lT_j.int() if args.loss_function == "bce" else lT_j
+            dlrm.run(lX_j, lS_l_j, lS_i_j, lT_j)
+
+            time2 = time.time()
+            total_time += time2 - time1
+
+            # compte loss and accuracy
+            Z = dlrm.get_output()  # numpy array
+            T = lT_j.numpy()
+            """
+            # debug prints
+            print("output and loss")
+            print(Z)
+            print(dlrm.get_loss())
+            """
+            mbs = T.shape[0]  # = args.mini_batch_size except maybe for last
+            A = np.sum((np.round(Z, 0) == T).astype(np.uint8))
+            total_accu += 0 if args.inference_only else A
+            total_loss += 0 if args.inference_only else dlrm.get_loss() * mbs
+            total_iter += 1
+            total_samp += mbs
+
+            # print time, loss and accuracy
+            should_print = ((j + 1) % args.print_freq == 0) or (j + 1 == nbatches)
+            should_test = (
+                (args.test_freq > 0)
+                and (args.data_generation in ["dataset", "random"])
+                and (((j + 1) % args.test_freq == 0) or (j + 1 == nbatches))
+            )
+            if should_print or should_test:
+                gT = 1000.0 * total_time / total_iter if args.print_time else -1
+                total_time = 0
+
+                gA = total_accu / total_samp
+                total_accu = 0
+
+                gL = total_loss / total_samp
+                total_loss = 0
+
+                str_run_type = "inference" if args.inference_only else "training"
+                print(
+                    "Finished {} it {}/{} of epoch {}, {:.2f} ms/it,".format(
+                        str_run_type, j + 1, nbatches, k, gT
+                    )
+                    + " loss {:.6f}".format(gL)
+                )
+                total_iter = 0
+                total_samp = 0
+                # debug prints
+                # print(Z)
+                # print(T)
+
+                # testing
+                if should_test and not args.inference_only:
+                    # don't measure training iter time in a test iteration
+                    if args.mlperf_logging:
+                        previous_iteration_time = None
+
+                    test_accu = 0
+                    test_loss = 0
+                    test_samp = 0
+
+                    if args.mlperf_logging:
+                        scores = []
+                        targets = []
+
+                    for i, testBatch in enumerate(test_ld):
+                        # early exit if nbatches was set by the user and was exceeded
+                        if nbatches > 0 and i >= nbatches:
+                            break
+
+                        # forward pass
+
+                        lX_test_i, lS_l_test_i, lS_i_test_i, lT_test_i = testBatch
+                        lT_test_i = (
+                            lT_test_i.int()
+                            if args.loss_function == "bce"
+                            else lT_test_i
+                        )
+                        dlrm.run(
+                            lX_test_i,
+                            lS_l_test_i,
+                            lS_i_test_i,
+                            lT_test_i,
+                            test_net=True,
+                        )
+
+                        Z_test = dlrm.get_output()
+                        T_test = lT_test_i.numpy()
+
+                        if args.mlperf_logging:
+                            scores.append(Z_test)
+                            targets.append(T_test)
+                        else:
+                            # compte loss and accuracy
+                            L_test = dlrm.get_loss()
+                            mbs_test = T_test.shape[0]  # = mini_batch_size except last
+                            A_test = np.sum(
+                                (np.round(Z_test, 0) == T_test).astype(np.uint8)
+                            )
+                            test_accu += A_test
+                            test_loss += L_test * mbs_test
+                            test_samp += mbs_test
+
+                    # compute metrics (after test loop has finished)
+                    if args.mlperf_logging:
+                        validation_results = calculate_metrics(targets, scores)
+                        gA_test = validation_results["accuracy"]
+                        gL_test = validation_results["loss"]
+                    else:
+                        gA_test = test_accu / test_samp
+                        gL_test = test_loss / test_samp
+
+                    # print metrics
+                    is_best = gA_test > best_gA_test
+                    if is_best:
+                        best_gA_test = gA_test
+
+                    if args.mlperf_logging:
+                        is_best = validation_results["roc_auc"] > best_auc_test
+                        if is_best:
+                            best_auc_test = validation_results["roc_auc"]
+
+                        print(
+                            "Testing at - {}/{} of epoch {},".format(j + 1, nbatches, k)
+                            + " loss {:.6f}, recall {:.4f}, precision {:.4f},".format(
+                                validation_results["loss"],
+                                validation_results["recall"],
+                                validation_results["precision"],
+                            )
+                            + " f1 {:.4f}, ap {:.4f},".format(
+                                validation_results["f1"],
+                                validation_results["ap"],
+                            )
+                            + " auc {:.4f}, best auc {:.4f},".format(
+                                validation_results["roc_auc"], best_auc_test
+                            )
+                            + " accuracy {:3.3f} %, best accuracy {:3.3f} %".format(
+                                validation_results["accuracy"] * 100, best_gA_test * 100
+                            )
+                        )
+                    else:
+                        print(
+                            "Testing at - {}/{} of epoch {},".format(j + 1, nbatches, 0)
+                            + " loss {:.6f}, accuracy {:3.3f} %, best {:3.3f} %".format(
+                                gL_test, gA_test * 100, best_gA_test * 100
+                            )
+                        )
+
+                    # check thresholds
+                    if (
+                        args.mlperf_logging
+                        and (args.mlperf_acc_threshold > 0)
+                        and (best_gA_test > args.mlperf_acc_threshold)
+                    ):
+                        print(
+                            "MLPerf testing accuracy threshold "
+                            + str(args.mlperf_acc_threshold)
+                            + " reached, stop training"
+                        )
+                        break
+
+                    if (
+                        args.mlperf_logging
+                        and (args.mlperf_auc_threshold > 0)
+                        and (best_auc_test > args.mlperf_auc_threshold)
+                    ):
+                        print(
+                            "MLPerf testing auc threshold "
+                            + str(args.mlperf_auc_threshold)
+                            + " reached, stop training"
+                        )
+                        break
+
+            j += 1  # nbatches
+        k += 1  # nepochs
+
+    # test prints
+    if not args.inference_only and args.debug_mode:
+        print("updated parameters (weights and bias):")
+        dlrm.print_weights()
+
+    # build onnx model from caffe2
+    if args.save_onnx:
+        pnet = dlrm.parameters().net.Proto()
+        inet = dlrm.parameters().param_init_net.Proto()
+        value_info = dlrm.onnx_tsd  # None
+        # debug prints
+        # print(value_info)
+
+        # WARNING: Why Caffe2 to ONNX net transformation currently does not work?
+        # 1. ONNX does not support SparseLengthsSum operator directly. A workaround
+        # could be for the Caffe2 ONNX frontend to indirectly map this operator to
+        # Gather and ReducedSum ONNX operators, following the PyTorch approach.
+        c2f = caffe2.python.onnx.frontend.Caffe2Frontend()
+        dlrm_caffe2_onnx = c2f.caffe2_net_to_onnx_model(pnet, inet, value_info)
+        # check the onnx model
+        onnx.checker.check_model(dlrm_caffe2_onnx)
+
+        # save model to a file
+        with open("dlrm_s_caffe2.onnx", "w+") as dlrm_caffe2_onnx_file:
+            dlrm_caffe2_onnx_file.write(str(dlrm_caffe2_onnx))
+
+    # build protobuf with types and shapes
+    if args.save_proto_types_shapes:
+        # add types and shapes to protobuf
+        __TYPE_MAPPING = {
+            onnx.TensorProto.FLOAT: caffe2_pb2.TensorProto.FLOAT,
+            onnx.TensorProto.UINT8: caffe2_pb2.TensorProto.UINT8,
+            onnx.TensorProto.INT8: caffe2_pb2.TensorProto.INT8,
+            onnx.TensorProto.UINT16: caffe2_pb2.TensorProto.UINT16,
+            onnx.TensorProto.INT16: caffe2_pb2.TensorProto.INT16,
+            onnx.TensorProto.INT32: caffe2_pb2.TensorProto.INT32,
+            onnx.TensorProto.INT64: caffe2_pb2.TensorProto.INT64,
+            onnx.TensorProto.STRING: caffe2_pb2.TensorProto.STRING,
+            onnx.TensorProto.BOOL: caffe2_pb2.TensorProto.BOOL,
+            onnx.TensorProto.FLOAT16: caffe2_pb2.TensorProto.FLOAT16,
+            onnx.TensorProto.DOUBLE: caffe2_pb2.TensorProto.DOUBLE,
+        }
+
+        pnet = dlrm.parameters().net.Proto()
+        arg = pnet.arg.add()
+        arg.name = "input_shape_info"
+        for i in pnet.external_input:
+            if i in dlrm.onnx_tsd:
+                onnx_dtype, shape = dlrm.onnx_tsd[i]
+                t = arg.tensors.add()
+                t.name = i
+                t.data_type = __TYPE_MAPPING[onnx_dtype]
+                t.dims.extend(shape)
+            else:
+                print("Warning: we don't have shape/type info for input: {}".format(i))
+        # debug print
+        # print(pnet)
+
+        # export the protobuf with types and shapes
+        with open("dlrm_s_caffe2.proto", "w+") as dlrm_s_proto_file:
+            dlrm_s_proto_file.write(str(pnet))
+
+        """
+        # export the protobuf with types and shapes as well as weights
+        # see https://github.com/pytorch/pytorch/issues/9533
+        #save
+        net = dlrm.parameters().net
+        params = dlrm.parameters().params
+        init_net, predict_net = mobile_exporter.Export(workspace, net, params)
+        with open("dlrm_s_caffe2.predict", "wb") as dlrm_s_predict_file:
+            dlrm_s_predict_file.write(predict_net.SerializeToString())
+        with open("dlrm_s_caffe2.init", "wb") as dlrm_s_init_file:
+            dlrm_s_init_file.write(init_net.SerializeToString())
+        #load
+        net_def = caffe2_pb2.NetDef()
+        init_def= caffe2_pb2.NetDef()
+        with open("dlrm_s_caffe2.predict", "rb") as dlrm_s_predict_file:
+            net_def.ParseFromString(dlrm_s_predict_file.read())
+            print(net_def)
+        with open("dlrm_s_caffe2.init", "rb") as dlrm_s_init_file:
+            init_def.ParseFromString(dlrm_s_init_file.read())
+            print(init_def)
+        """
diff --git a/models_v2/pytorch/dlrm/inference/gpu/dlrm_s_pytorch.py b/models_v2/pytorch/dlrm/inference/gpu/dlrm_s_pytorch.py
new file mode 100644
index 000000000..139bbf79c
--- /dev/null
+++ b/models_v2/pytorch/dlrm/inference/gpu/dlrm_s_pytorch.py
@@ -0,0 +1,2331 @@
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+# Description: an implementation of a deep learning recommendation model (DLRM)
+# The model input consists of dense and sparse features. The former is a vector
+# of floating point values. The latter is a list of sparse indices into
+# embedding tables, which consist of vectors of floating point values.
+# The selected vectors are passed to mlp networks denoted by triangles,
+# in some cases the vectors are interacted through operators (Ops).
+#
+# output:
+#                         vector of values
+# model:                        |
+#                              /\
+#                             /__\
+#                               |
+#       _____________________> Op  <___________________
+#     /                         |                      \
+#    /\                        /\                      /\
+#   /__\                      /__\           ...      /__\
+#    |                          |                       |
+#    |                         Op                      Op
+#    |                    ____/__\_____           ____/__\____
+#    |                   |_Emb_|____|__|    ...  |_Emb_|__|___|
+# input:
+# [ dense features ]     [sparse indices] , ..., [sparse indices]
+#
+# More precise definition of model layers:
+# 1) fully connected layers of an mlp
+# z = f(y)
+# y = Wx + b
+#
+# 2) embedding lookup (for a list of sparse indices p=[p1,...,pk])
+# z = Op(e1,...,ek)
+# obtain vectors e1=E[:,p1], ..., ek=E[:,pk]
+#
+# 3) Operator Op can be one of the following
+# Sum(e1,...,ek) = e1 + ... + ek
+# Dot(e1,...,ek) = [e1'e1, ..., e1'ek, ..., ek'e1, ..., ek'ek]
+# Cat(e1,...,ek) = [e1', ..., ek']'
+# where ' denotes transpose operation
+#
+# References:
+# [1] Maxim Naumov, Dheevatsa Mudigere, Hao-Jun Michael Shi, Jianyu Huang,
+# Narayanan Sundaram, Jongsoo Park, Xiaodong Wang, Udit Gupta, Carole-Jean Wu,
+# Alisson G. Azzolini, Dmytro Dzhulgakov, Andrey Mallevich, Ilia Cherniavskii,
+# Yinghai Lu, Raghuraman Krishnamoorthi, Ansha Yu, Volodymyr Kondratenko,
+# Stephanie Pereira, Xianjie Chen, Wenlin Chen, Vijay Rao, Bill Jia, Liang Xiong,
+# Misha Smelyanskiy, "Deep Learning Recommendation Model for Personalization and
+# Recommendation Systems", CoRR, arXiv:1906.00091, 2019
+
+from __future__ import absolute_import, division, print_function, unicode_literals
+
+import argparse
+
+# miscellaneous
+import builtins
+import datetime
+import json
+import sys
+import time
+import os
+
+# onnx
+# The onnx import causes deprecation warnings every time workers
+# are spawned during testing. So, we filter out those warnings.
+import warnings
+
+# data generation
+import dlrm_data_pytorch as dp
+
+# For distributed run
+import extend_distributed as ext_dist
+import mlperf_logger
+
+# numpy
+import numpy as np
+import optim.rwsadagrad as RowWiseSparseAdagrad
+import sklearn.metrics
+
+# pytorch
+import torch
+import torch.nn as nn
+from torch._ops import ops
+from torch.autograd.profiler import record_function
+from torch.nn.parallel.parallel_apply import parallel_apply
+from torch.nn.parallel.replicate import replicate
+from torch.nn.parallel.scatter_gather import gather, scatter
+from torch.nn.parameter import Parameter
+from torch.optim.lr_scheduler import _LRScheduler
+from torch.utils.tensorboard import SummaryWriter
+
+# mixed-dimension trick
+from tricks.md_embedding_bag import md_solver, PrEmbeddingBag
+
+# quotient-remainder trick
+from tricks.qr_embedding_bag import QREmbeddingBag
+
+with warnings.catch_warnings():
+    warnings.filterwarnings("ignore", category=DeprecationWarning)
+    try:
+        import onnx
+    except ImportError as error:
+        print("Unable to import onnx. ", error)
+
+# from torchviz import make_dot
+# import torch.nn.functional as Functional
+# from torch.nn.parameter import Parameter
+
+exc = getattr(builtins, "IOError", "FileNotFoundError")
+
+
+def time_wrap(use_gpu, use_xpu):
+    if use_gpu:
+        torch.cuda.synchronize()
+    if use_xpu:
+        torch.xpu.synchronize()
+    return time.time()
+
+
+def dlrm_wrap(X, lS_o, lS_i, use_gpu, use_xpu, device, args, amp, amp_dtype, ndevices=1):
+    with record_function("DLRM forward"):
+        if use_gpu:  # .cuda()
+            # lS_i can be either a list of tensors or a stacked tensor.
+            # Handle each case below:
+            if ndevices == 1:
+                lS_i = (
+                    [S_i.to(device) for S_i in lS_i]
+                    if isinstance(lS_i, list)
+                    else lS_i.to(device)
+                )
+                lS_o = (
+                    [S_o.to(device) for S_o in lS_o]
+                    if isinstance(lS_o, list)
+                    else lS_o.to(device)
+                )
+        if use_xpu:  # .xpu()
+            # lS_i can be either a list of tensors or a stacked tensor.
+            # Handle each case below:
+            if ndevices == 1:
+                lS_i = (
+                    [S_i.to(device) for S_i in lS_i]
+                    if isinstance(lS_i, list)
+                    else lS_i.to(device)
+                )
+                lS_o = (
+                    [S_o.to(device) for S_o in lS_o]
+                    if isinstance(lS_o, list)
+                    else lS_o.to(device)
+                )
+        if args.dynamo:
+            with torch.xpu.amp.autocast(enabled=amp, dtype=amp_dtype):
+                res = dlrm(X.to(device), lS_o, lS_i)
+        else:
+            res = dlrm(X.to(device), lS_o, lS_i)
+        return res
+
+def dlrm_wrap_dynamo(X, lS_o, lS_i, use_gpu, use_xpu, device, args, amp, amp_dtype, ndevices=1):
+    with record_function("DLRM forward"):
+        if ndevices == 1:
+            lS_i = (
+                [S_i.to(device) for S_i in lS_i]
+                if isinstance(lS_i, list)
+                else lS_i.to(device)
+            )
+            lS_o = (
+                [S_o.to(device) for S_o in lS_o]
+                if isinstance(lS_o, list)
+                else lS_o.to(device)
+            )
+        with torch.xpu.amp.autocast(enabled=amp, dtype=amp_dtype):
+            res = dlrm(X.to(device), lS_o, lS_i)
+        return res
+
+def loss_fn_wrap(Z, T, device):
+    with record_function("DLRM loss compute"):
+        if args.loss_function == "mse" or args.loss_function == "bce":
+            return dlrm.loss_fn(Z, T.to(device))
+        elif args.loss_function == "wbce":
+            loss_ws_ = dlrm.loss_ws[T.data.view(-1).long()].view_as(T).to(device)
+            loss_fn_ = dlrm.loss_fn(Z, T.to(device))
+            loss_sc_ = loss_ws_ * loss_fn_
+            return loss_sc_.mean()
+
+
+# The following function is a wrapper to avoid checking this multiple times in th
+# loop below.
+def unpack_batch(b):
+    # Experiment with unweighted samples
+    return b[0], b[1], b[2], b[3], torch.ones(b[3].size()), None
+
+
+class LRPolicyScheduler(_LRScheduler):
+    def __init__(self, optimizer, num_warmup_steps, decay_start_step, num_decay_steps):
+        self.num_warmup_steps = num_warmup_steps
+        self.decay_start_step = decay_start_step
+        self.decay_end_step = decay_start_step + num_decay_steps
+        self.num_decay_steps = num_decay_steps
+
+        if self.decay_start_step < self.num_warmup_steps:
+            sys.exit("Learning rate warmup must finish before the decay starts")
+
+        super(LRPolicyScheduler, self).__init__(optimizer)
+
+    def get_lr(self):
+        step_count = self._step_count
+        if step_count < self.num_warmup_steps:
+            # warmup
+            scale = 1.0 - (self.num_warmup_steps - step_count) / self.num_warmup_steps
+            lr = [base_lr * scale for base_lr in self.base_lrs]
+            self.last_lr = lr
+        elif self.decay_start_step <= step_count and step_count < self.decay_end_step:
+            # decay
+            decayed_steps = step_count - self.decay_start_step
+            scale = ((self.num_decay_steps - decayed_steps) / self.num_decay_steps) ** 2
+            min_lr = 0.0000001
+            lr = [max(min_lr, base_lr * scale) for base_lr in self.base_lrs]
+            self.last_lr = lr
+        else:
+            if self.num_decay_steps > 0:
+                # freeze at last, either because we're after decay
+                # or because we're between warmup and decay
+                lr = self.last_lr
+            else:
+                # do not adjust
+                lr = self.base_lrs
+        return lr
+
+
+### define dlrm in PyTorch ###
+class DLRM_Net(nn.Module):
+    def create_mlp(self, ln, sigmoid_layer):
+        # build MLP layer by layer
+        layers = nn.ModuleList()
+        for i in range(0, ln.size - 1):
+            n = ln[i]
+            m = ln[i + 1]
+
+            # construct fully connected operator
+            LL = nn.Linear(int(n), int(m), bias=True)
+
+            # initialize the weights
+            # with torch.no_grad():
+            # custom Xavier input, output or two-sided fill
+            mean = 0.0  # std_dev = np.sqrt(variance)
+            std_dev = np.sqrt(2 / (m + n))  # np.sqrt(1 / m) # np.sqrt(1 / n)
+            W = np.random.normal(mean, std_dev, size=(m, n)).astype(np.float32)
+            std_dev = np.sqrt(1 / m)  # np.sqrt(2 / (m + 1))
+            bt = np.random.normal(mean, std_dev, size=m).astype(np.float32)
+            # approach 1
+            LL.weight.data = torch.tensor(W, requires_grad=True)
+            LL.bias.data = torch.tensor(bt, requires_grad=True)
+            # approach 2
+            # LL.weight.data.copy_(torch.tensor(W))
+            # LL.bias.data.copy_(torch.tensor(bt))
+            # approach 3
+            # LL.weight = Parameter(torch.tensor(W),requires_grad=True)
+            # LL.bias = Parameter(torch.tensor(bt),requires_grad=True)
+            layers.append(LL)
+
+            # construct sigmoid or relu operator
+            if i == sigmoid_layer:
+                layers.append(nn.Sigmoid())
+            else:
+                layers.append(nn.ReLU())
+
+        # approach 1: use ModuleList
+        # return layers
+        # approach 2: use Sequential container to wrap all layers
+        return torch.nn.Sequential(*layers)
+
+    ########################diff with cpu
+    def create_emb(self, m, ln, weighted_pooling=None):
+        emb_l = nn.ModuleList()
+        v_W_l = []
+        for i in range(0, ln.size):
+            if ext_dist.my_size > 1:
+                if i not in self.local_emb_indices:
+                    continue
+            n = ln[i]
+
+            # construct embedding operator
+            if self.qr_flag and n > self.qr_threshold:
+                EE = QREmbeddingBag(
+                    n,
+                    m,
+                    self.qr_collisions,
+                    operation=self.qr_operation,
+                    mode="sum",
+                    sparse=True,
+                )
+            elif self.md_flag and n > self.md_threshold:
+                base = max(m)
+                _m = m[i] if n > self.md_threshold else base
+                EE = PrEmbeddingBag(n, _m, base)
+                # use np initialization as below for consistency...
+                W = np.random.uniform(
+                    low=-np.sqrt(1 / n), high=np.sqrt(1 / n), size=(n, _m)
+                ).astype(np.float32)
+                EE.embs.weight.data = torch.tensor(W, requires_grad=True)
+            else:
+                _m = m[i] if self.md_flag else m
+                EE = nn.EmbeddingBag(n, _m, mode="sum", sparse=True)
+                # initialize embeddings
+                # nn.init.uniform_(EE.weight, a=-np.sqrt(1 / n), b=np.sqrt(1 / n))
+                W = np.random.uniform(
+                    low=-np.sqrt(1 / n), high=np.sqrt(1 / n), size=(n, _m)
+                ).astype(np.float32)
+                # approach 1
+                EE.weight.data = torch.tensor(W, requires_grad=True)
+                # approach 2
+                # EE.weight.data.copy_(torch.tensor(W))
+                # approach 3
+                # EE.weight = Parameter(torch.tensor(W),requires_grad=True)
+            if weighted_pooling is None:
+                v_W_l.append(None)
+            else:
+                v_W_l.append(torch.ones(n, dtype=torch.float32))
+            emb_l.append(EE)
+        return emb_l, v_W_l
+
+    ###########diff no ipex
+    def __init__(
+        self,
+        m_spa=None,
+        ln_emb=None,
+        ln_bot=None,
+        ln_top=None,
+        arch_interaction_op=None,
+        arch_interaction_itself=False,
+        sigmoid_bot=-1,
+        sigmoid_top=-1,
+        sync_dense_params=True,
+        loss_threshold=0.0,
+        ndevices=-1,
+        use_device="cpu",
+        qr_flag=False,
+        qr_operation="mult",
+        qr_collisions=0,
+        qr_threshold=200,
+        md_flag=False,
+        md_threshold=200,
+        weighted_pooling=None,
+        loss_function="bce",
+    ):
+        super(DLRM_Net, self).__init__()
+
+        if (
+            (m_spa is not None)
+            and (ln_emb is not None)
+            and (ln_bot is not None)
+            and (ln_top is not None)
+            and (arch_interaction_op is not None)
+        ):
+
+            # save arguments
+            self.ndevices = ndevices
+            self.use_device = use_device
+            self.output_d = 0
+            self.parallel_model_batch_size = -1
+            self.parallel_model_is_not_prepared = True
+            self.arch_interaction_op = arch_interaction_op
+            self.arch_interaction_itself = arch_interaction_itself
+            self.sync_dense_params = sync_dense_params
+            self.loss_threshold = loss_threshold
+            self.loss_function = loss_function
+            if weighted_pooling is not None and weighted_pooling != "fixed":
+                self.weighted_pooling = "learned"
+            else:
+                self.weighted_pooling = weighted_pooling
+            # create variables for QR embedding if applicable
+            self.qr_flag = qr_flag
+            if self.qr_flag:
+                self.qr_collisions = qr_collisions
+                self.qr_operation = qr_operation
+                self.qr_threshold = qr_threshold
+            # create variables for MD embedding if applicable
+            self.md_flag = md_flag
+            if self.md_flag:
+                self.md_threshold = md_threshold
+
+            # If running distributed, get local slice of embedding tables
+            if ext_dist.my_size > 1:
+                n_emb = len(ln_emb)
+                if n_emb < ext_dist.my_size:
+                    sys.exit(
+                        "only (%d) sparse features for (%d) devices, table partitions will fail"
+                        % (n_emb, ext_dist.my_size)
+                    )
+                self.n_global_emb = n_emb
+                self.n_local_emb, self.n_emb_per_rank = ext_dist.get_split_lengths(
+                    n_emb
+                )
+                self.local_emb_slice = ext_dist.get_my_slice(n_emb)
+                self.local_emb_indices = list(range(n_emb))[self.local_emb_slice]
+                print("self.n_global_emb:", self.n_global_emb)
+                print("self.n_local_emb:", self.n_local_emb)
+                print("self.n_emb_per_rank:", self.n_emb_per_rank)
+                print("self.local_emb_slice:", self.local_emb_slice)
+                print("self.local_emb_indices:", self.local_emb_indices)
+
+            # create operators
+            if ndevices <= 1:
+                self.emb_l, w_list = self.create_emb(m_spa, ln_emb, weighted_pooling)
+                if self.weighted_pooling == "learned":
+                    self.v_W_l = nn.ParameterList()
+                    for w in w_list:
+                        self.v_W_l.append(Parameter(w))
+                else:
+                    self.v_W_l = w_list
+            self.bot_l = self.create_mlp(ln_bot, sigmoid_bot)
+            self.top_l = self.create_mlp(ln_top, sigmoid_top)
+
+            # quantization
+            self.quantize_emb = False
+            self.emb_l_q = []
+            self.quantize_bits = 32
+
+            # specify the loss function
+            if self.loss_function == "mse":
+                self.loss_fn = torch.nn.MSELoss(reduction="mean")
+            elif self.loss_function == "bce":
+                self.loss_fn = torch.nn.BCELoss(reduction="mean")
+            elif self.loss_function == "wbce":
+                self.loss_ws = torch.tensor(
+                    np.fromstring(args.loss_weights, dtype=float, sep="-")
+                )
+                self.loss_fn = torch.nn.BCELoss(reduction="none")
+            else:
+                sys.exit(
+                    "ERROR: --loss-function=" + self.loss_function + " is not supported"
+                )
+
+    def apply_mlp(self, x, layers):
+        # approach 1: use ModuleList
+        # for layer in layers:
+        #     x = layer(x)
+        # return x
+        # approach 2: use Sequential container to wrap all layers
+        return layers(x)
+
+    ###################diff cpu emb_merge
+    ###################delete lS_o
+    def apply_emb(self, lS_o, lS_i, emb_l, v_W_l):
+        # WARNING: notice that we are processing the batch at once. We implicitly
+        # assume that the data is laid out such that:
+        # 1. each embedding is indexed with a group of sparse indices,
+        #   corresponding to a single lookup
+        # 2. for each embedding the lookups are further organized into a batch
+        # 3. for a list of embedding tables there is a list of batched lookups
+
+        ly = []
+        for k, sparse_index_group_batch in enumerate(lS_i):
+            sparse_offset_group_batch = lS_o[k]
+
+            # embedding lookup
+            # We are using EmbeddingBag, which implicitly uses sum operator.
+            # The embeddings are represented as tall matrices, with sum
+            # happening vertically across 0 axis, resulting in a row vector
+            # E = emb_l[k]
+
+            if v_W_l[k] is not None:
+                per_sample_weights = v_W_l[k].gather(0, sparse_index_group_batch)
+            else:
+                per_sample_weights = None
+
+            if self.quantize_emb:
+                s1 = self.emb_l_q[k].element_size() * self.emb_l_q[k].nelement()
+                s2 = self.emb_l_q[k].element_size() * self.emb_l_q[k].nelement()
+                print("quantized emb sizes:", s1, s2)
+
+                if self.quantize_bits == 4:
+                    QV = ops.quantized.embedding_bag_4bit_rowwise_offsets(
+                        self.emb_l_q[k],
+                        sparse_index_group_batch,
+                        sparse_offset_group_batch,
+                        per_sample_weights=per_sample_weights,
+                    )
+                elif self.quantize_bits == 8:
+                    QV = ops.quantized.embedding_bag_byte_rowwise_offsets(
+                        self.emb_l_q[k],
+                        sparse_index_group_batch,
+                        sparse_offset_group_batch,
+                        per_sample_weights=per_sample_weights,
+                    )
+
+                ly.append(QV)
+            else:
+                E = emb_l[k]
+                V = E(
+                    sparse_index_group_batch,
+                    sparse_offset_group_batch,
+                    per_sample_weights=per_sample_weights,
+                )
+
+                ly.append(V)
+
+        # print(ly)
+        return ly
+
+    #  using quantizing functions from caffe2/aten/src/ATen/native/quantized/cpu
+    def quantize_embedding(self, bits):
+
+        n = len(self.emb_l)
+        self.emb_l_q = [None] * n
+        for k in range(n):
+            if bits == 4:
+                self.emb_l_q[k] = ops.quantized.embedding_bag_4bit_prepack(
+                    self.emb_l[k].weight
+                )
+            elif bits == 8:
+                self.emb_l_q[k] = ops.quantized.embedding_bag_byte_prepack(
+                    self.emb_l[k].weight
+                )
+            else:
+                return
+        self.emb_l = None
+        self.quantize_emb = True
+        self.quantize_bits = bits
+
+    def interact_features(self, x, ly):
+
+        if args.fp16:
+            ly = [t.half() if t.dtype != torch.float16 else t for t in ly]
+            x = x.half() if x.dtype != torch.float16 else x
+        elif args.bf16:
+            ly = [t.bfloat16() if t.dtype != torch.bfloat16 else t for t in ly]
+            x = x.bfloat16() if x.dtype != torch.bfloat16 else x
+        if self.arch_interaction_op == "dot":
+            # concatenate dense and sparse features
+            (batch_size, d) = x.shape
+            T = torch.cat([x] + ly, dim=1).view((batch_size, -1, d))
+            # perform a dot product
+            Z = torch.bmm(T, torch.transpose(T, 1, 2))
+            # append dense feature with the interactions (into a row vector)
+            # approach 1: all
+            # Zflat = Z.view((batch_size, -1))
+            # approach 2: unique
+            _, ni, nj = Z.shape
+            # approach 1: tril_indices
+            # offset = 0 if self.arch_interaction_itself else -1
+            # li, lj = torch.tril_indices(ni, nj, offset=offset)
+            # approach 2: custom
+            offset = 1 if self.arch_interaction_itself else 0
+            li = torch.tensor([i for i in range(ni) for j in range(i + offset)])
+            lj = torch.tensor([j for i in range(nj) for j in range(i + offset)])
+            Zflat = Z[:, li, lj]
+            # concatenate dense features and interactions
+            R = torch.cat([x] + [Zflat], dim=1)
+        elif self.arch_interaction_op == "cat":
+            # concatenation features (into a row vector)
+            R = torch.cat([x] + ly, dim=1)
+        else:
+            sys.exit(
+                "ERROR: --arch-interaction-op="
+                + self.arch_interaction_op
+                + " is not supported"
+            )
+
+        return R
+
+    def forward(self, dense_x, lS_o, lS_i):
+        if ext_dist.my_size > 1:
+            # multi-node multi-device run
+            return self.distributed_forward(dense_x, lS_o, lS_i)
+        elif self.ndevices <= 1:
+            # single device run
+            return self.sequential_forward(dense_x, lS_o, lS_i)
+        else:
+            # single-node multi-device run
+            return self.parallel_forward(dense_x, lS_o, lS_i)
+
+    def distributed_forward(self, dense_x, lS_o, lS_i):
+        batch_size = dense_x.size()[0]
+        # WARNING: # of ranks must be <= batch size in distributed_forward call
+        if batch_size < ext_dist.my_size:
+            sys.exit(
+                "ERROR: batch_size (%d) must be larger than number of ranks (%d)"
+                % (batch_size, ext_dist.my_size)
+            )
+        if batch_size % ext_dist.my_size != 0:
+            sys.exit(
+                "ERROR: batch_size %d can not split across %d ranks evenly"
+                % (batch_size, ext_dist.my_size)
+            )
+
+        dense_x = dense_x[ext_dist.get_my_slice(batch_size)]
+        lS_o = lS_o[self.local_emb_slice]
+        lS_i = lS_i[self.local_emb_slice]
+
+        if (len(self.emb_l) != len(lS_o)) or (len(self.emb_l) != len(lS_i)):
+            sys.exit(
+                "ERROR: corrupted model input detected in distributed_forward call"
+            )
+
+        # embeddings
+        with record_function("DLRM embedding forward"):
+            ly = self.apply_emb(lS_o, lS_i, self.emb_l, self.v_W_l)
+
+        # WARNING: Note that at this point we have the result of the embedding lookup
+        # for the entire batch on each rank. We would like to obtain partial results
+        # corresponding to all embedding lookups, but part of the batch on each rank.
+        # Therefore, matching the distribution of output of bottom mlp, so that both
+        # could be used for subsequent interactions on each device.
+        if len(self.emb_l) != len(ly):
+            sys.exit("ERROR: corrupted intermediate result in distributed_forward call")
+
+        a2a_req = ext_dist.alltoall(ly, self.n_emb_per_rank)
+
+        with record_function("DLRM bottom nlp forward"):
+            x = self.apply_mlp(dense_x, self.bot_l)
+
+        ly = a2a_req.wait()
+        ly = list(ly)
+
+        # interactions
+        with record_function("DLRM interaction forward"):
+            z = self.interact_features(x, ly)
+
+        # top mlp
+        with record_function("DLRM top nlp forward"):
+            p = self.apply_mlp(z, self.top_l)
+
+        # clamp output if needed
+        if 0.0 < self.loss_threshold and self.loss_threshold < 1.0:
+            z = torch.clamp(p, min=self.loss_threshold, max=(1.0 - self.loss_threshold))
+        else:
+            z = p
+
+        return z
+
+    def sequential_forward(self, dense_x, lS_o, lS_i):
+        # process dense features (using bottom mlp), resulting in a row vector
+        x = self.apply_mlp(dense_x, self.bot_l)
+        # debug prints
+        # print("intermediate")
+        # print(x.detach().cpu().numpy())
+
+        # process sparse features(using embeddings), resulting in a list of row vectors
+        ly = self.apply_emb(lS_o, lS_i, self.emb_l, self.v_W_l)
+        # for y in ly:
+        #     print(y.detach().cpu().numpy())
+
+        # interact features (dense and sparse)
+        z = self.interact_features(x, ly)
+        # print(z.detach().cpu().numpy())
+
+        # obtain probability of a click (using top mlp)
+        p = self.apply_mlp(z, self.top_l)
+
+        # clamp output if needed
+        if 0.0 < self.loss_threshold and self.loss_threshold < 1.0:
+            z = torch.clamp(p, min=self.loss_threshold, max=(1.0 - self.loss_threshold))
+        else:
+            z = p
+
+        return z
+
+    def parallel_forward(self, dense_x, lS_o, lS_i):
+        ### prepare model (overwrite) ###
+        # WARNING: # of devices must be >= batch size in parallel_forward call
+        batch_size = dense_x.size()[0]
+        ndevices = min(self.ndevices, batch_size, len(self.emb_l))
+        if not (self.use_device=="xpu"):
+            device_ids = range(ndevices)
+        else:
+            device_ids = []
+            for i in range(ndevices):
+                device_ids.append(torch.device("xpu", i))
+        # WARNING: must redistribute the model if mini-batch size changes(this is common
+        # for last mini-batch, when # of elements in the dataset/batch size is not even
+        if self.parallel_model_batch_size != batch_size:
+            self.parallel_model_is_not_prepared = True
+
+        if self.parallel_model_is_not_prepared or self.sync_dense_params:
+            # replicate mlp (data parallelism)
+            self.bot_l_replicas = replicate(self.bot_l, device_ids)
+            self.top_l_replicas = replicate(self.top_l, device_ids)
+            self.parallel_model_batch_size = batch_size
+
+        if self.parallel_model_is_not_prepared:
+            # distribute embeddings (model parallelism)
+            t_list = []
+            w_list = []
+            for k, emb in enumerate(self.emb_l):
+                d = torch.device("cuda:" + str(k % ndevices))
+                t_list.append(emb.to(d))
+                if self.weighted_pooling == "learned":
+                    w_list.append(Parameter(self.v_W_l[k].to(d)))
+                elif self.weighted_pooling == "fixed":
+                    w_list.append(self.v_W_l[k].to(d))
+                else:
+                    w_list.append(None)
+            self.emb_l = nn.ModuleList(t_list)
+            if self.weighted_pooling == "learned":
+                self.v_W_l = nn.ParameterList(w_list)
+            else:
+                self.v_W_l = w_list
+            self.parallel_model_is_not_prepared = False
+
+        ### prepare input (overwrite) ###
+        # scatter dense features (data parallelism)
+        # print(dense_x.device)
+        dense_x = scatter(dense_x, device_ids, dim=0)
+        # distribute sparse features (model parallelism)
+        if (len(self.emb_l) != len(lS_o)) or (len(self.emb_l) != len(lS_i)):
+            sys.exit("ERROR: corrupted model input detected in parallel_forward call")
+
+        t_list = []
+        i_list = []
+        for k, _ in enumerate(self.emb_l):
+            d = torch.device("cuda:" + str(k % ndevices))
+            t_list.append(lS_o[k].to(d))
+            i_list.append(lS_i[k].to(d))
+        lS_o = t_list
+        lS_i = i_list
+
+        ### compute results in parallel ###
+        # bottom mlp
+        # WARNING: Note that the self.bot_l is a list of bottom mlp modules
+        # that have been replicated across devices, while dense_x is a tuple of dense
+        # inputs that has been scattered across devices on the first (batch) dimension.
+        # The output is a list of tensors scattered across devices according to the
+        # distribution of dense_x.
+        x = parallel_apply(self.bot_l_replicas, dense_x, None, device_ids)
+        # debug prints
+        # print(x)
+
+        # embeddings
+        ly = self.apply_emb(lS_o, lS_i, self.emb_l, self.v_W_l)
+        # debug prints
+        # print(ly)
+
+        # butterfly shuffle (implemented inefficiently for now)
+        # WARNING: Note that at this point we have the result of the embedding lookup
+        # for the entire batch on each device. We would like to obtain partial results
+        # corresponding to all embedding lookups, but part of the batch on each device.
+        # Therefore, matching the distribution of output of bottom mlp, so that both
+        # could be used for subsequent interactions on each device.
+        if len(self.emb_l) != len(ly):
+            sys.exit("ERROR: corrupted intermediate result in parallel_forward call")
+
+        t_list = []
+        for k, _ in enumerate(self.emb_l):
+            d = torch.device("cuda:" + str(k % ndevices))
+            y = scatter(ly[k], device_ids, dim=0)
+            t_list.append(y)
+        # adjust the list to be ordered per device
+        ly = list(map(lambda y: list(y), zip(*t_list)))
+        # debug prints
+        # print(ly)
+
+        # interactions
+        z = []
+        for k in range(ndevices):
+            zk = self.interact_features(x[k], ly[k])
+            z.append(zk)
+        # debug prints
+        # print(z)
+
+        # top mlp
+        # WARNING: Note that the self.top_l is a list of top mlp modules that
+        # have been replicated across devices, while z is a list of interaction results
+        # that by construction are scattered across devices on the first (batch) dim.
+        # The output is a list of tensors scattered across devices according to the
+        # distribution of z.
+        p = parallel_apply(self.top_l_replicas, z, None, device_ids)
+
+        ### gather the distributed results ###
+        p0 = gather(p, self.output_d, dim=0)
+
+        # clamp output if needed
+        if 0.0 < self.loss_threshold and self.loss_threshold < 1.0:
+            z0 = torch.clamp(
+                p0, min=self.loss_threshold, max=(1.0 - self.loss_threshold)
+            )
+        else:
+            z0 = p0
+
+        return z0
+
+
+def dash_separated_ints(value):
+    vals = value.split("-")
+    for val in vals:
+        try:
+            int(val)
+        except ValueError:
+            raise argparse.ArgumentTypeError(
+                "%s is not a valid dash separated list of ints" % value
+            )
+
+    return value
+
+
+def dash_separated_floats(value):
+    vals = value.split("-")
+    for val in vals:
+        try:
+            float(val)
+        except ValueError:
+            raise argparse.ArgumentTypeError(
+                "%s is not a valid dash separated list of floats" % value
+            )
+
+    return value
+
+
+def inference(
+    args,
+    dlrm_model,
+    best_acc_test,
+    best_auc_test,
+    test_ld,
+    device,
+    use_gpu,
+    use_xpu,
+    amp_dtype,
+    log_iter=-1,
+    amp=False,
+    inference_only=False,
+    profile=False,
+):
+    test_accu = 0
+    test_samp = 0
+
+    if args.mlperf_logging:
+        scores = []
+        targets = []
+
+    if not inference_only:
+        profile=False
+    time_count = 0
+    batch = 0
+    total_time = 0
+    if inference_only:
+        with torch.no_grad():
+            for i, testBatch in enumerate(test_ld):
+                # early exit if nbatches was set by the user and was exceeded
+                if nbatches > 0 and i >= nbatches:
+                    break
+
+                X_test, lS_o_test, lS_i_test, T_test, W_test, CBPP_test = unpack_batch(
+                    testBatch
+                )
+                if not amp:
+                    X_test = X_test.to(amp_dtype)
+                    T_test = T_test.to(amp_dtype)
+
+                if i == 0:
+                    batch = X_test.size(0)
+                    if ext_dist.my_size <= 1:
+                        if ndevices == 1:
+                            lS_i = (
+                                [S_i.to(device) for S_i in lS_o_test]
+                                if isinstance(lS_o_test, list)
+                                else lS_o_test.to(device)
+                            )
+                            lS_o = (
+                                [S_o.to(device) for S_o in lS_o_test]
+                                if isinstance(lS_o_test, list)
+                                else lS_o_test.to(device)
+                            )
+                        global dlrm
+                        if use_xpu:
+                            if args.dynamo:
+                                with torch.no_grad():
+                                    dynamo_inf = torch.compile(dlrm_wrap_dynamo, backend="inductor", options={"freezing":True})
+                            else:
+                                with torch.xpu.amp.autocast(enabled=amp, dtype=amp_dtype):
+                                    dlrm = torch.jit.trace(dlrm, (X_test.to(device), lS_o, lS_i), check_trace=False)
+                                    dlrm = torch.jit.freeze(dlrm)
+                        elif use_gpu:
+                            with torch.cuda.amp.autocast(enabled=amp, dtype=amp_dtype):
+                                dlrm = torch.jit.trace(dlrm, (X_test.to(device), lS_o, lS_i), check_trace=False)
+                                dlrm = torch.jit.freeze(dlrm)
+                        else:
+                            pass
+                # Skip the batch if batch size not multiple of total ranks
+                if ext_dist.my_size > 1 and X_test.size(0) % ext_dist.my_size != 0:
+                    print("Warning: Skiping the batch %d with size %d" % (i, X_test.size(0)))
+                    continue
+
+                if profile:
+                    if use_xpu:
+                        sort_key = "self_xpu_time_total"
+                    elif use_cuda:
+                        sort_key = "self_cuda_time_total"
+                    else:
+                        sort_key = "self_cpu_time_total"
+                with (torch.autograd.profiler_legacy.profile(profile, use_xpu=True) if use_xpu else torch.profiler.profile(activities=[torch.profiler.ProfilerActivity.CUDA])) as prof1:
+                    # forward pass
+
+                    if use_xpu and (not (ext_dist.my_size > 1)):
+                        try:
+                            import memory_check
+                            memory_check.display_mem("xpu:0")
+                        except:
+                            pass
+                    t1 = time.time()
+                    if ext_dist.my_size > 1 and use_xpu:
+                        with torch.xpu.amp.autocast(enabled=amp, dtype=amp_dtype):
+                            Z_test = dlrm_wrap(
+                                X_test,
+                                lS_o_test,
+                                lS_i_test,
+                                use_gpu,
+                                use_xpu,
+                                device,
+                                args,
+                                amp,
+                                amp_dtype,
+                                ndevices=ndevices
+                            )
+                    elif (ext_dist.my_size > 1 and use_gpu):
+                        with torch.cuda.amp.autocast(enabled=amp, dtype=amp_dtype):
+                            Z_test = dlrm_wrap(
+                                X_test,
+                                lS_o_test,
+                                lS_i_test,
+                                use_gpu,
+                                use_xpu,
+                                device,
+                                args,
+                                amp,
+                                amp_dtype,
+                                ndevices=ndevices,
+                            )
+                    else:
+                        if args.dynamo:
+                            Z_test = dynamo_inf(X_test,
+                                        lS_o_test,
+                                        lS_i_test,
+                                        use_gpu,
+                                        use_xpu,
+                                        device,
+                                        args,
+                                        amp,
+                                        amp_dtype,
+                                        ndevices=ndevices)
+                        else:
+                            Z_test = dlrm_wrap(
+                                X_test,
+                                lS_o_test,
+                                lS_i_test,
+                                use_gpu,
+                                use_xpu,
+                                device,
+                                args,
+                                amp,
+                                amp_dtype,
+                                ndevices=ndevices
+                            )
+                    ### gather the distributed results on each rank ###
+                    # For some reason it requires explicit sync before all_gather call if
+                    # tensor is on GPU memory
+                    if Z_test.is_cuda:
+                        torch.cuda.synchronize()
+                    if Z_test.is_xpu:
+                        torch.xpu.synchronize()
+                    (_, batch_split_lengths) = ext_dist.get_split_lengths(X_test.size(0))
+                    if ext_dist.my_size > 1:
+                        Z_test = ext_dist.all_gather(Z_test, batch_split_lengths)
+
+                    if args.mlperf_logging:
+                        if amp_dtype == torch.bfloat16:
+                            S_test = Z_test.detach().cpu().float().numpy()  # numpy array
+                            T_test = T_test.detach().cpu().float().numpy()  # numpy array
+                        else:
+                            S_test = Z_test.detach().cpu().numpy()  # numpy array
+                            T_test = T_test.detach().cpu().numpy()  # numpy array
+                        scores.append(S_test)
+                        targets.append(T_test)
+                    else:
+                        with record_function("DLRM accuracy compute"):
+                            # compute loss and accuracy
+                            if amp_dtype == torch.bfloat16:
+                                S_test = Z_test.detach().cpu().float().numpy()  # numpy array
+                                T_test = T_test.detach().cpu().float().numpy()  # numpy array
+                            else:
+                                S_test = Z_test.detach().cpu().numpy()  # numpy array
+                                T_test = T_test.detach().cpu().numpy()  # numpy array
+
+                            mbs_test = T_test.shape[0]  # = mini_batch_size except last
+                            A_test = np.sum((np.round(S_test, 0) == T_test).astype(np.uint8))
+
+                            test_accu += A_test
+                            test_samp += mbs_test
+                    if i >= 5:
+                        total_time += time.time() - t1
+                        time_count += 1
+
+                if profile and i == 15:
+                    if use_xpu:
+                        rank_index = ""
+                        if ext_dist.my_size > 1:
+                            rank_index = "rank_" + str(ext_dist.my_local_rank) + "_"
+                        path ='./report/dlrm_inference_' + rank_index + 'profiling'
+                        torch.save(prof1.key_averages().table(sort_by="self_xpu_time_total", row_limit=100000), \
+                            path + '.pt')
+                        prof1.export_chrome_trace(path + '.json')
+                        torch.save(prof1.table(sort_by="id", row_limit=100000), path + '_detail.pt')
+                    else:
+                        print(prof.key_averages().table(sort_by=sort_key, row_limit=100000))
+    else:
+        for i, testBatch in enumerate(test_ld):
+            # early exit if nbatches was set by the user and was exceeded
+            if nbatches > 0 and i >= nbatches:
+                break
+
+            X_test, lS_o_test, lS_i_test, T_test, W_test, CBPP_test = unpack_batch(
+                testBatch
+            )
+            if not amp:
+                X_test = X_test.to(amp_dtype)
+                T_test = T_test.to(amp_dtype)
+
+            # Skip the batch if batch size not multiple of total ranks
+            if ext_dist.my_size > 1 and X_test.size(0) % ext_dist.my_size != 0:
+                print("Warning: Skiping the batch %d with size %d" % (i, X_test.size(0)))
+                continue
+
+            # forward pass
+            with torch.xpu.amp.autocast(enabled=amp, dtype=amp_dtype):
+                Z_test = dlrm_wrap(
+                    X_test,
+                    lS_o_test,
+                    lS_i_test,
+                    use_gpu,
+                    use_xpu,
+                    device,
+                    ndevices=ndevices,
+                )
+            ### gather the distributed results on each rank ###
+            # For some reason it requires explicit sync before all_gather call if
+            # tensor is on GPU memory
+            if Z_test.is_cuda:
+                torch.cuda.synchronize()
+            if Z_test.is_xpu:
+                torch.xpu.synchronize()
+            (_, batch_split_lengths) = ext_dist.get_split_lengths(X_test.size(0))
+            if ext_dist.my_size > 1:
+                Z_test = ext_dist.all_gather(Z_test, batch_split_lengths)
+
+            if args.mlperf_logging:
+                if amp_dtype == torch.bfloat16:
+                    S_test = Z_test.detach().cpu().float().numpy()  # numpy array
+                    T_test = T_test.detach().cpu().float().numpy()  # numpy array
+                else:
+                    S_test = Z_test.detach().cpu().numpy()  # numpy array
+                    T_test = T_test.detach().cpu().numpy()  # numpy array
+                scores.append(S_test)
+                targets.append(T_test)
+            else:
+                with record_function("DLRM accuracy compute"):
+                    # compute loss and accuracy
+                    if amp_dtype == torch.bfloat16:
+                        S_test = Z_test.detach().cpu().float().numpy()  # numpy array
+                        T_test = T_test.detach().cpu().float().numpy()  # numpy array
+                    else:
+                        S_test = Z_test.detach().cpu().numpy()  # numpy array
+                        T_test = T_test.detach().cpu().numpy()  # numpy array
+
+                    mbs_test = T_test.shape[0]  # = mini_batch_size except last
+                    A_test = np.sum((np.round(S_test, 0) == T_test).astype(np.uint8))
+
+                    test_accu += A_test
+                    test_samp += mbs_test
+
+    if args.mlperf_logging:
+        with record_function("DLRM mlperf sklearn metrics compute"):
+            scores = np.concatenate(scores, axis=0)
+            targets = np.concatenate(targets, axis=0)
+
+            metrics = {
+                "recall": lambda y_true, y_score: sklearn.metrics.recall_score(
+                    y_true=y_true, y_pred=np.round(y_score)
+                ),
+                "precision": lambda y_true, y_score: sklearn.metrics.precision_score(
+                    y_true=y_true, y_pred=np.round(y_score)
+                ),
+                "f1": lambda y_true, y_score: sklearn.metrics.f1_score(
+                    y_true=y_true, y_pred=np.round(y_score)
+                ),
+                "ap": sklearn.metrics.average_precision_score,
+                "roc_auc": sklearn.metrics.roc_auc_score,
+                "accuracy": lambda y_true, y_score: sklearn.metrics.accuracy_score(
+                    y_true=y_true, y_pred=np.round(y_score)
+                ),
+            }
+
+        validation_results = {}
+        for metric_name, metric_function in metrics.items():
+            validation_results[metric_name] = metric_function(targets, scores)
+            writer.add_scalar(
+                "mlperf-metrics-test/" + metric_name,
+                validation_results[metric_name],
+                log_iter,
+            )
+        acc_test = validation_results["accuracy"]
+    else:
+        acc_test = test_accu / test_samp
+        writer.add_scalar("Test/Acc", acc_test, log_iter)
+
+    model_metrics_dict = {
+        "nepochs": args.nepochs,
+        "nbatches": nbatches,
+        "nbatches_test": nbatches_test,
+        "state_dict": dlrm_model.state_dict(),
+        "test_acc": acc_test,
+    }
+
+    if args.mlperf_logging:
+        is_best = validation_results["roc_auc"] > best_auc_test
+        if is_best:
+            best_auc_test = validation_results["roc_auc"]
+            model_metrics_dict["test_auc"] = best_auc_test
+        print(
+            "recall {:.4f}, precision {:.4f},".format(
+                validation_results["recall"],
+                validation_results["precision"],
+            )
+            + " f1 {:.4f}, ap {:.4f},".format(
+                validation_results["f1"], validation_results["ap"]
+            )
+            + " auc {:.4f}, best auc {:.4f},".format(
+                validation_results["roc_auc"], best_auc_test
+            )
+            + " accuracy {:3.3f} %, best accuracy {:3.3f} %".format(
+                validation_results["accuracy"] * 100, best_acc_test * 100
+            ),
+            flush=True,
+        )
+    else:
+        is_best = acc_test > best_acc_test
+        if is_best:
+            best_acc_test = acc_test
+        print(
+            " accuracy {:3.3f} %, best {:3.3f} %".format(
+                acc_test * 100, best_acc_test * 100
+            ),
+            flush=True,
+        )
+    if inference_only:
+        if total_time == 0:
+            print("The total_time is zero, please check args.")
+        else:
+            print('dlrm_inf latency: ', total_time, ' s')
+            print('dlrm_inf avg time: ', total_time/time_count, ' s, ant the time count is :', time_count)
+            print('dlrm_inf throughput: ', batch * time_count/total_time, ' samples/s')
+    return model_metrics_dict, is_best
+
+
+def run():
+    ### parse arguments ###
+    parser = argparse.ArgumentParser(
+        description="Train Deep Learning Recommendation Model (DLRM)"
+    )
+    # model related parameters
+    parser.add_argument("--arch-sparse-feature-size", type=int, default=2)
+    parser.add_argument(
+        "--arch-embedding-size", type=dash_separated_ints, default="4-3-2"
+    )
+    # j will be replaced with the table number
+    parser.add_argument("--arch-mlp-bot", type=dash_separated_ints, default="4-3-2")
+    parser.add_argument("--arch-mlp-top", type=dash_separated_ints, default="4-2-1")
+    parser.add_argument(
+        "--arch-interaction-op", type=str, choices=["dot", "cat"], default="dot"
+    )
+    parser.add_argument("--arch-interaction-itself", action="store_true", default=False)
+    parser.add_argument("--weighted-pooling", type=str, default=None)
+    # embedding table options
+    parser.add_argument("--md-flag", action="store_true", default=False)
+    parser.add_argument("--md-threshold", type=int, default=200)
+    parser.add_argument("--md-temperature", type=float, default=0.3)
+    parser.add_argument("--md-round-dims", action="store_true", default=False)
+    parser.add_argument("--qr-flag", action="store_true", default=False)
+    parser.add_argument("--qr-threshold", type=int, default=200)
+    parser.add_argument("--qr-operation", type=str, default="mult")
+    parser.add_argument("--qr-collisions", type=int, default=4)
+    # activations and loss
+    parser.add_argument("--activation-function", type=str, default="relu")
+    parser.add_argument("--loss-function", type=str, default="mse")  # or bce or wbce
+    parser.add_argument(
+        "--loss-weights", type=dash_separated_floats, default="1.0-1.0"
+    )  # for wbce
+    parser.add_argument("--loss-threshold", type=float, default=0.0)  # 1.0e-7
+    parser.add_argument("--round-targets", type=bool, default=False)
+    # data
+    parser.add_argument("--data-size", type=int, default=1)
+    parser.add_argument("--num-batches", type=int, default=0)
+    parser.add_argument(
+        "--data-generation", type=str, default="random"
+    )  # synthetic or dataset
+    parser.add_argument(
+        "--rand-data-dist", type=str, default="uniform"
+    )  # uniform or gaussian
+    parser.add_argument("--rand-data-min", type=float, default=0)
+    parser.add_argument("--rand-data-max", type=float, default=1)
+    parser.add_argument("--rand-data-mu", type=float, default=-1)
+    parser.add_argument("--rand-data-sigma", type=float, default=1)
+    parser.add_argument("--data-trace-file", type=str, default="./input/dist_emb_j.log")
+    parser.add_argument("--data-set", type=str, default="kaggle")  # or terabyte
+    parser.add_argument("--raw-data-file", type=str, default="")
+    parser.add_argument("--processed-data-file", type=str, default="")
+    parser.add_argument("--data-randomize", type=str, default="total")  # or day or none
+    parser.add_argument("--data-trace-enable-padding", type=bool, default=False)
+    parser.add_argument("--max-ind-range", type=int, default=-1)
+    parser.add_argument("--data-sub-sample-rate", type=float, default=0.0)  # in [0, 1]
+    parser.add_argument("--num-indices-per-lookup", type=int, default=10)
+    parser.add_argument("--num-indices-per-lookup-fixed", type=bool, default=False)
+    parser.add_argument("--num-workers", type=int, default=0)
+    parser.add_argument("--memory-map", action="store_true", default=False)
+    # training
+    parser.add_argument("--mini-batch-size", type=int, default=1)
+    parser.add_argument("--nepochs", type=int, default=1)
+    parser.add_argument("--learning-rate", type=float, default=0.01)
+    parser.add_argument("--print-precision", type=int, default=5)
+    parser.add_argument("--numpy-rand-seed", type=int, default=123)
+    parser.add_argument("--sync-dense-params", type=bool, default=True)
+    parser.add_argument("--optimizer", type=str, default="sgd")
+    parser.add_argument(
+        "--dataset-multiprocessing",
+        action="store_true",
+        default=False,
+        help="The Kaggle dataset can be multiprocessed in an environment \
+                        with more than 7 CPU cores and more than 20 GB of memory. \n \
+                        The Terabyte dataset can be multiprocessed in an environment \
+                        with more than 24 CPU cores and at least 1 TB of memory.",
+    )
+    # inference
+    parser.add_argument("--inference-only", action="store_true", default=False)
+    # quantize
+    parser.add_argument("--quantize-mlp-with-bit", type=int, default=32)
+    parser.add_argument("--quantize-emb-with-bit", type=int, default=32)
+    parser.add_argument("--bf16", action="store_true", default=False)
+    parser.add_argument("--fp16", action="store_true", default=False)
+    parser.add_argument("--amp", action="store_true", default=False)
+    # onnx
+    parser.add_argument("--save-onnx", action="store_true", default=False)
+    # gpu
+    parser.add_argument("--use-gpu", action="store_true", default=False)
+    parser.add_argument("--use-xpu", action="store_true", default=False)
+    parser.add_argument("--parallel", action="store_true", default=False)
+    # distributed
+    parser.add_argument("--local_rank", type=int, default=-1)
+    parser.add_argument("--dist-backend", type=str, default="")
+    # debugging and profiling
+    parser.add_argument("--print-freq", type=int, default=1)
+    parser.add_argument("--test-freq", type=int, default=-1)
+    parser.add_argument("--test-mini-batch-size", type=int, default=-1)
+    parser.add_argument("--test-num-workers", type=int, default=-1)
+    parser.add_argument("--print-time", action="store_true", default=False)
+    parser.add_argument("--print-wall-time", action="store_true", default=False)
+    parser.add_argument("--debug-mode", action="store_true", default=False)
+    parser.add_argument("--enable-profiling", action="store_true", default=False)
+    parser.add_argument("--plot-compute-graph", action="store_true", default=False)
+    parser.add_argument("--tensor-board-filename", type=str, default="run_kaggle_pt")
+    # store/load model
+    parser.add_argument("--save-model", type=str, default="")
+    parser.add_argument("--load-model", type=str, default="")
+    # mlperf logging (disables other output and stops early)
+    parser.add_argument("--mlperf-logging", action="store_true", default=False)
+    # stop at target accuracy Kaggle 0.789, Terabyte (sub-sampled=0.875) 0.8107
+    parser.add_argument("--mlperf-acc-threshold", type=float, default=0.0)
+    # stop at target AUC Terabyte (no subsampling) 0.8025
+    parser.add_argument("--mlperf-auc-threshold", type=float, default=0.0)
+    parser.add_argument("--mlperf-bin-loader", action="store_true", default=False)
+    parser.add_argument("--mlperf-bin-shuffle", action="store_true", default=False)
+    # mlperf gradient accumulation iterations
+    parser.add_argument("--mlperf-grad-accum-iter", type=int, default=1)
+    # LR policy
+    parser.add_argument("--lr-num-warmup-steps", type=int, default=0)
+    parser.add_argument("--lr-decay-start-step", type=int, default=0)
+    parser.add_argument("--lr-num-decay-steps", type=int, default=0)
+    parser.add_argument("--dynamo", action="store_true", default=False)
+
+    global args
+    global nbatches
+    global nbatches_test
+    global writer
+    args = parser.parse_args()
+    if os.getenv('PROFILE') == '1': args.enable_profiling=True
+    if args.use_xpu:
+        import intel_extension_for_pytorch
+        try:
+            import oneccl_bindings_for_pytorch
+        except ImportError as e:
+            pass
+
+    if args.dataset_multiprocessing:
+        assert float(sys.version[:3]) > 3.7, (
+            "The dataset_multiprocessing "
+            + "flag is susceptible to a bug in Python 3.7 and under. "
+            + "https://github.com/facebookresearch/dlrm/issues/172"
+        )
+
+    if args.mlperf_logging:
+        mlperf_logger.log_event(key=mlperf_logger.constants.CACHE_CLEAR, value=True)
+        mlperf_logger.log_start(
+            key=mlperf_logger.constants.INIT_START, log_all_ranks=True
+        )
+
+    if args.weighted_pooling is not None:
+        if args.qr_flag:
+            sys.exit("ERROR: quotient remainder with weighted pooling is not supported")
+        if args.md_flag:
+            sys.exit("ERROR: mixed dimensions with weighted pooling is not supported")
+    #note add xpu not support
+    if args.quantize_emb_with_bit in [4, 8]:
+        if args.qr_flag:
+            sys.exit(
+                "ERROR: 4 and 8-bit quantization with quotient remainder is not supported"
+            )
+        if args.md_flag:
+            sys.exit(
+                "ERROR: 4 and 8-bit quantization with mixed dimensions is not supported"
+            )
+        if args.use_gpu:
+            sys.exit("ERROR: 4 and 8-bit quantization on GPU is not supported")
+
+    ### some basic setup ###
+    np.random.seed(args.numpy_rand_seed)
+    np.set_printoptions(precision=args.print_precision)
+    torch.set_printoptions(precision=args.print_precision)
+    torch.manual_seed(args.numpy_rand_seed)
+
+    if args.test_mini_batch_size < 0:
+        # if the parameter is not set, use the training batch size
+        args.test_mini_batch_size = args.mini_batch_size
+    if args.test_num_workers < 0:
+        # if the parameter is not set, use the same parameter for training
+        args.test_num_workers = args.num_workers
+
+    use_gpu = args.use_gpu and torch.cuda.is_available()
+    use_xpu = args.use_xpu and torch.xpu.is_available()
+
+    use_device = "cpu"
+    if use_gpu:
+        use_device = "cuda"
+    elif use_xpu:
+        use_device = "xpu"
+    else:
+        use_device = "cpu"
+
+    if not args.debug_mode:
+        ext_dist.init_distributed(
+            local_rank=args.local_rank, use_device=use_device, backend=args.dist_backend
+        )
+
+    if use_gpu:
+        torch.cuda.manual_seed_all(args.numpy_rand_seed)
+        torch.backends.cudnn.deterministic = True
+        if ext_dist.my_size > 1:
+            ngpus = 1
+            device = torch.device("cuda", ext_dist.my_local_rank)
+        else:
+            ngpus = torch.cuda.device_count()
+            device = torch.device("cuda", 0)
+        print("Using {} GPU(s)...".format(ngpus))
+    elif use_xpu:
+        torch.xpu.manual_seed_all(args.numpy_rand_seed)
+        #torch.backends.cudnn.deterministic = True
+        if ext_dist.my_size > 1:
+            ngpus = 1
+            device = torch.device("xpu", ext_dist.my_local_rank)
+        else:
+            if args.parallel:
+                ngpus = torch.xpu.device_count()
+            else:
+                ngpus = 1
+            device = torch.device("xpu", 0)
+        print("Using {} GPU(s)...".format(ngpus))
+    else:
+        device = torch.device("cpu")
+        print("Using CPU...")
+
+    ### prepare training data ###
+    ln_bot = np.fromstring(args.arch_mlp_bot, dtype=int, sep="-")
+    # input data
+
+    if args.mlperf_logging:
+        mlperf_logger.barrier(use_device)
+        mlperf_logger.log_end(key=mlperf_logger.constants.INIT_STOP)
+        mlperf_logger.barrier(use_device)
+        mlperf_logger.log_start(key=mlperf_logger.constants.RUN_START)
+        mlperf_logger.barrier(use_device)
+
+    if args.data_generation == "dataset":
+        train_data, train_ld, test_data, test_ld = dp.make_criteo_data_and_loaders(args)
+        table_feature_map = {idx: idx for idx in range(len(train_data.counts))}
+        nbatches = args.num_batches if args.num_batches > 0 else len(train_ld)
+        nbatches_test = len(test_ld)
+
+        ln_emb = train_data.counts
+        # enforce maximum limit on number of vectors per embedding
+        if args.max_ind_range > 0:
+            ln_emb = np.array(
+                list(
+                    map(
+                        lambda x: x if x < args.max_ind_range else args.max_ind_range,
+                        ln_emb,
+                    )
+                )
+            )
+        else:
+            ln_emb = np.array(ln_emb)
+        m_den = train_data.m_den
+        ln_bot[0] = m_den
+    else:
+        # input and target at random
+        #ln_emb = np.fromstring(args.arch_embedding_size, dtype=int, sep="-")
+        ln_emb = np.array([2500000, 39043, 17289, 7420, 20263, 3, 7120, 1543, 63, 2500000, 2500000, 403346, 10, 2208, 11938, 155, 4, 976, 14, 2500000, 2500000, 2500000, 585935, 12972, 108, 36])
+        m_den = ln_bot[0]
+        train_data, train_ld, test_data, test_ld = dp.make_random_data_and_loader(
+            args, ln_emb, m_den
+        )
+        nbatches = args.num_batches if args.num_batches > 0 else len(train_ld)
+        nbatches_test = len(test_ld)
+
+    args.ln_emb = ln_emb.tolist()
+    if args.mlperf_logging:
+        print("command line args: ", json.dumps(vars(args)))
+
+    ### parse command line arguments ###
+    m_spa = args.arch_sparse_feature_size
+    ln_emb = np.asarray(ln_emb)
+    num_fea = ln_emb.size + 1  # num sparse + num dense features
+
+    m_den_out = ln_bot[ln_bot.size - 1]
+    if args.arch_interaction_op == "dot":
+        # approach 1: all
+        # num_int = num_fea * num_fea + m_den_out
+        # approach 2: unique
+        if args.arch_interaction_itself:
+            num_int = (num_fea * (num_fea + 1)) // 2 + m_den_out
+        else:
+            num_int = (num_fea * (num_fea - 1)) // 2 + m_den_out
+    elif args.arch_interaction_op == "cat":
+        num_int = num_fea * m_den_out
+    else:
+        sys.exit(
+            "ERROR: --arch-interaction-op="
+            + args.arch_interaction_op
+            + " is not supported"
+        )
+    arch_mlp_top_adjusted = str(num_int) + "-" + args.arch_mlp_top
+    ln_top = np.fromstring(arch_mlp_top_adjusted, dtype=int, sep="-")
+
+    # sanity check: feature sizes and mlp dimensions must match
+    if m_den != ln_bot[0]:
+        sys.exit(
+            "ERROR: arch-dense-feature-size "
+            + str(m_den)
+            + " does not match first dim of bottom mlp "
+            + str(ln_bot[0])
+        )
+    if args.qr_flag:
+        if args.qr_operation == "concat" and 2 * m_spa != m_den_out:
+            sys.exit(
+                "ERROR: 2 arch-sparse-feature-size "
+                + str(2 * m_spa)
+                + " does not match last dim of bottom mlp "
+                + str(m_den_out)
+                + " (note that the last dim of bottom mlp must be 2x the embedding dim)"
+            )
+        if args.qr_operation != "concat" and m_spa != m_den_out:
+            sys.exit(
+                "ERROR: arch-sparse-feature-size "
+                + str(m_spa)
+                + " does not match last dim of bottom mlp "
+                + str(m_den_out)
+            )
+    else:
+        if m_spa != m_den_out:
+            sys.exit(
+                "ERROR: arch-sparse-feature-size "
+                + str(m_spa)
+                + " does not match last dim of bottom mlp "
+                + str(m_den_out)
+            )
+    if num_int != ln_top[0]:
+        sys.exit(
+            "ERROR: # of feature interactions "
+            + str(num_int)
+            + " does not match first dimension of top mlp "
+            + str(ln_top[0])
+        )
+
+    # assign mixed dimensions if applicable
+    if args.md_flag:
+        _m_spa = md_solver(
+            torch.tensor(ln_emb),
+            args.md_temperature,  # alpha
+            d0=m_spa,
+            round_dim=args.md_round_dims,
+        ).tolist()
+        for i in range(len(ln_emb)):
+            if ln_emb[i] <= args.md_threshold:
+                _m_spa[i] = 128
+        m_spa = map(int, _m_spa)
+        m_spa = list(m_spa)
+
+    # test prints (model arch)
+    if args.debug_mode:
+        print("model arch:")
+        print(
+            "mlp top arch "
+            + str(ln_top.size - 1)
+            + " layers, with input to output dimensions:"
+        )
+        print(ln_top)
+        print("# of interactions")
+        print(num_int)
+        print(
+            "mlp bot arch "
+            + str(ln_bot.size - 1)
+            + " layers, with input to output dimensions:"
+        )
+        print(ln_bot)
+        print("# of features (sparse and dense)")
+        print(num_fea)
+        print("dense feature size")
+        print(m_den)
+        print("sparse feature size")
+        print(m_spa)
+        print(
+            "# of embeddings (= # of sparse features) "
+            + str(ln_emb.size)
+            + ", with dimensions "
+            + str(m_spa)
+            + "x:"
+        )
+        print(ln_emb)
+
+        print("data (inputs and targets):")
+        for j, inputBatch in enumerate(train_ld):
+            X, lS_o, lS_i, T, W, CBPP = unpack_batch(inputBatch)
+
+            torch.set_printoptions(precision=4)
+            # early exit if nbatches was set by the user and has been exceeded
+            if nbatches > 0 and j >= nbatches:
+                break
+            print("mini-batch: %d" % j)
+            print(X.detach().cpu())
+            # transform offsets to lengths when printing
+            print(
+                torch.IntTensor(
+                    [
+                        np.diff(
+                            S_o.detach().cpu().tolist() + list(lS_i[i].shape)
+                        ).tolist()
+                        for i, S_o in enumerate(lS_o)
+                    ]
+                )
+            )
+            print([S_i.detach().cpu() for S_i in lS_i])
+            print(T.detach().cpu())
+
+    global ndevices
+    ndevices = min(ngpus, args.mini_batch_size, num_fea - 1) if use_gpu or use_xpu else -1
+
+    ### construct the neural network specified above ###
+    # WARNING: to obtain exactly the same initialization for
+    # the weights we need to start from the same random seed.
+    # np.random.seed(args.numpy_rand_seed)
+    global dlrm
+    dlrm = DLRM_Net(
+        m_spa,
+        ln_emb,
+        ln_bot,
+        ln_top,
+        arch_interaction_op=args.arch_interaction_op,
+        arch_interaction_itself=args.arch_interaction_itself,
+        sigmoid_bot=-1,
+        sigmoid_top=ln_top.size - 2,
+        sync_dense_params=args.sync_dense_params,
+        loss_threshold=args.loss_threshold,
+        ndevices=ndevices,
+        use_device=use_device,
+        qr_flag=args.qr_flag,
+        qr_operation=args.qr_operation,
+        qr_collisions=args.qr_collisions,
+        qr_threshold=args.qr_threshold,
+        md_flag=args.md_flag,
+        md_threshold=args.md_threshold,
+        weighted_pooling=args.weighted_pooling,
+        loss_function=args.loss_function,
+    )
+    print(dlrm)
+    amp_dtype = torch.float32
+    if args.bf16:
+        amp_dtype = torch.bfloat16
+    if args.fp16:
+        amp_dtype = torch.float16
+
+    # test prints
+    if args.debug_mode:
+        print("initial parameters (weights and bias):")
+        for param in dlrm.parameters():
+            print(param.detach().cpu().numpy())
+        # print(dlrm)
+    if use_gpu:
+        # Custom Model-Data Parallel
+        # the mlps are replicated and use data parallelism, while
+        # the embeddings are distributed and use model parallelism
+        dlrm = dlrm.to(device)  # .cuda()
+        if dlrm.ndevices > 1:
+            dlrm.emb_l, dlrm.v_W_l = dlrm.create_emb(
+                m_spa, ln_emb, args.weighted_pooling
+            )
+        else:
+            if dlrm.weighted_pooling == "fixed":
+                for k, w in enumerate(dlrm.v_W_l):
+                    dlrm.v_W_l[k] = w.cuda()
+    elif use_xpu:
+        # Custom Model-Data Parallel
+        # the mlps are replicated and use data parallelism, while
+        # the embeddings are distributed and use model parallelism
+        #if ext_dist.my_size > 1:
+        #    device_ids = "xpu:{}".format(ext_dist.my_local_rank)
+        #    dlrm = dlrm.to(device_ids)  # .xpu()
+        #else:
+        dlrm = dlrm.to(device)  # .xpu()
+
+        if dlrm.ndevices > 1:
+            dlrm.emb_l, dlrm.v_W_l = dlrm.create_emb(
+                m_spa, ln_emb, args.weighted_pooling
+            )
+        else:
+            if dlrm.weighted_pooling == "fixed":
+                for k, w in enumerate(dlrm.v_W_l):
+                    dlrm.v_W_l[k] = w.xpu()
+
+    # distribute data parallel mlps
+    if ext_dist.my_size > 1:
+        if use_gpu:
+            device_ids = [ext_dist.my_local_rank]
+            dlrm.bot_l = ext_dist.DDP(dlrm.bot_l, device_ids=device_ids)
+            dlrm.top_l = ext_dist.DDP(dlrm.top_l, device_ids=device_ids)
+        elif use_xpu:
+            #device_ids = ["xpu:{}".format(ext_dist.my_local_rank)]
+            device_ids = [device]
+            dlrm.bot_l = ext_dist.DDP(dlrm.bot_l, device_ids=device_ids)
+            dlrm.top_l = ext_dist.DDP(dlrm.top_l, device_ids=device_ids)
+        else:
+            dlrm.bot_l = ext_dist.DDP(dlrm.bot_l)
+            dlrm.top_l = ext_dist.DDP(dlrm.top_l)
+
+    if not args.inference_only:
+        #######note add xpu support
+        if use_gpu and args.optimizer in ["rwsadagrad", "adagrad"]:
+            sys.exit("GPU version of Adagrad is not supported by PyTorch.")
+        # specify the optimizer algorithm
+        opts = {
+            "sgd": torch.optim.SGD,
+            "rwsadagrad": RowWiseSparseAdagrad.RWSAdagrad,
+            "adagrad": torch.optim.Adagrad,
+        }
+
+        parameters = (
+            dlrm.parameters()
+            if ext_dist.my_size == 1
+            else [
+                {
+                    "params": [p for emb in dlrm.emb_l for p in emb.parameters()],
+                    "lr": args.learning_rate,
+                },
+                # TODO check this lr setup
+                # bottom mlp has no data parallelism
+                # need to check how do we deal with top mlp
+                {
+                    "params": dlrm.bot_l.parameters(),
+                    "lr": args.learning_rate,
+                },
+                {
+                    "params": dlrm.top_l.parameters(),
+                    "lr": args.learning_rate,
+                },
+            ]
+        )
+        optimizer = opts[args.optimizer](parameters, lr=args.learning_rate)
+        lr_scheduler = LRPolicyScheduler(
+            optimizer,
+            args.lr_num_warmup_steps,
+            args.lr_decay_start_step,
+            args.lr_num_decay_steps,
+        )
+
+    ### main loop ###
+
+    # training or inference
+    best_acc_test = 0
+    best_auc_test = 0
+    skip_upto_epoch = 0
+    skip_upto_batch = 0
+    total_time = 0
+    total_loss = 0
+    total_iter = 0
+    total_samp = 0
+
+    if args.mlperf_logging:
+        mlperf_logger.mlperf_submission_log("dlrm")
+        mlperf_logger.log_event(
+            key=mlperf_logger.constants.SEED, value=args.numpy_rand_seed
+        )
+        mlperf_logger.log_event(
+            key=mlperf_logger.constants.GLOBAL_BATCH_SIZE, value=args.mini_batch_size
+        )
+
+    # Load model is specified
+    if not (args.load_model == ""):
+        print("Loading saved model {}".format(args.load_model))
+        if use_gpu:
+            if dlrm.ndevices > 1:
+                # NOTE: when targeting inference on multiple GPUs,
+                # load the model as is on CPU or GPU, with the move
+                # to multiple GPUs to be done in parallel_forward
+                ld_model = torch.load(args.load_model)
+            else:
+                # NOTE: when targeting inference on single GPU,
+                # note that the call to .to(device) has already happened
+                ld_model = torch.load(
+                    args.load_model,
+                    map_location=torch.device("cuda")
+                    # map_location=lambda storage, loc: storage.cuda(0)
+                )
+        elif use_xpu:
+            if dlrm.ndevices > 1:
+                # NOTE: when targeting inference on multiple GPUs,
+                # load the model as is on CPU or GPU, with the move
+                # to multiple GPUs to be done in parallel_forward
+                ld_model = torch.load(args.load_model)
+            elif (ext_dist.my_size > 1 and args.data_set == "terabyte"):
+                model_path = args.load_model + "dlrm_distributed_checkpoints_rank" + str(ext_dist.my_local_rank) + ".pt"
+                ld_model = torch.load(
+                    model_path,
+                    # map_location=lambda storage, loc: storage.xpu(0)
+                )
+            else:
+                # NOTE: when targeting inference on single GPU,
+                # note that the call to .to(device) has already happened
+                ld_model = torch.load(
+                    args.load_model,
+                    map_location=torch.device("xpu")
+                    # map_location=lambda storage, loc: storage.xpu(0)
+                )
+        else:
+            # when targeting inference on CPU
+            ld_model = torch.load(args.load_model, map_location=torch.device("cpu"))
+        dlrm.load_state_dict(ld_model["state_dict"])
+        ld_j = ld_model["iter"]
+        ld_k = ld_model["epoch"]
+        ld_nepochs = ld_model["nepochs"]
+        ld_nbatches = ld_model["nbatches"]
+        ld_nbatches_test = ld_model["nbatches_test"]
+        ld_train_loss = ld_model["train_loss"]
+        ld_total_loss = ld_model["total_loss"]
+        if args.mlperf_logging:
+            if "test_auc" in ld_model:
+                ld_gAUC_test = ld_model["test_auc"]
+            else:
+                ld_gAUC_test = 1
+        ld_acc_test = ld_model["test_acc"]
+        if not args.inference_only:
+            optimizer.load_state_dict(ld_model["opt_state_dict"])
+            best_acc_test = ld_acc_test
+            total_loss = ld_total_loss
+            skip_upto_epoch = ld_k  # epochs
+            skip_upto_batch = ld_j  # batches
+        else:
+            args.print_freq = ld_nbatches
+            args.test_freq = 0
+
+        print(
+            "Saved at: epoch = {:d}/{:d}, batch = {:d}/{:d}, ntbatch = {:d}".format(
+                ld_k, ld_nepochs, ld_j, ld_nbatches, ld_nbatches_test
+            )
+        )
+        print(
+            "Training state: loss = {:.6f}".format(
+                ld_train_loss,
+            )
+        )
+        if args.mlperf_logging:
+            print(
+                "Testing state: accuracy = {:3.3f} %, auc = {:.3f}".format(
+                    ld_acc_test * 100, ld_gAUC_test
+                )
+            )
+        else:
+            print("Testing state: accuracy = {:3.3f} %".format(ld_acc_test * 100))
+
+    if args.inference_only:
+        # Currently only dynamic quantization with INT8 and FP16 weights are
+        # supported for MLPs and INT4 and INT8 weights for EmbeddingBag
+        # post-training quantization during the inference.
+        # By default we don't do the quantization: quantize_{mlp,emb}_with_bit == 32 (FP32)
+        assert args.quantize_mlp_with_bit in [
+            8,
+            16,
+            32,
+        ], "only support 8/16/32-bit but got {}".format(args.quantize_mlp_with_bit)
+        assert args.quantize_emb_with_bit in [
+            4,
+            8,
+            32,
+        ], "only support 4/8/32-bit but got {}".format(args.quantize_emb_with_bit)
+        if args.quantize_mlp_with_bit != 32:
+            if args.quantize_mlp_with_bit in [8]:
+                quantize_dtype = torch.qint8
+            else:
+                quantize_dtype = torch.float16
+            dlrm = torch.quantization.quantize_dynamic(
+                dlrm, {torch.nn.Linear}, quantize_dtype
+            )
+        if args.quantize_emb_with_bit != 32:
+            dlrm.quantize_embedding(args.quantize_emb_with_bit)
+            # print(dlrm)
+
+    print("time/loss/accuracy (if enabled):")
+
+    if args.mlperf_logging:
+        # LR is logged twice for now because of a compliance checker bug
+        mlperf_logger.log_event(
+            key=mlperf_logger.constants.OPT_BASE_LR, value=args.learning_rate
+        )
+        mlperf_logger.log_event(
+            key=mlperf_logger.constants.OPT_LR_WARMUP_STEPS,
+            value=args.lr_num_warmup_steps,
+        )
+
+        # use logging keys from the official HP table and not from the logging library
+        mlperf_logger.log_event(
+            key="sgd_opt_base_learning_rate", value=args.learning_rate
+        )
+        mlperf_logger.log_event(
+            key="lr_decay_start_steps", value=args.lr_decay_start_step
+        )
+        mlperf_logger.log_event(
+            key="sgd_opt_learning_rate_decay_steps", value=args.lr_num_decay_steps
+        )
+        mlperf_logger.log_event(key="sgd_opt_learning_rate_decay_poly_power", value=2)
+
+    tb_file = "./" + args.tensor_board_filename
+    writer = SummaryWriter(tb_file)
+
+    ext_dist.barrier()
+
+    #######note add xpu support
+    #with torch.autograd.profiler.profile(
+    #    args.enable_profiling, use_cuda=use_gpu, record_shapes=True
+    #) as prof:
+    time_counts = 0
+    batch = 0
+    time_collect = 0
+    if not args.inference_only:
+        k = 0
+        total_time_begin = 0
+        while k < args.nepochs:
+            if args.mlperf_logging:
+                mlperf_logger.barrier(use_device)
+                mlperf_logger.log_start(
+                    key=mlperf_logger.constants.BLOCK_START,
+                    metadata={
+                        mlperf_logger.constants.FIRST_EPOCH_NUM: (k + 1),
+                        mlperf_logger.constants.EPOCH_COUNT: 1,
+                    },
+                )
+                mlperf_logger.barrier(use_device)
+                mlperf_logger.log_start(
+                    key=mlperf_logger.constants.EPOCH_START,
+                    metadata={mlperf_logger.constants.EPOCH_NUM: (k + 1)},
+                )
+
+            if k < skip_upto_epoch:
+                continue
+
+            if args.mlperf_logging:
+                previous_iteration_time = None
+
+            for j, inputBatch in enumerate(train_ld):
+                if j == 0 and args.save_onnx:
+                    X_onnx, lS_o_onnx, lS_i_onnx, _, _, _ = unpack_batch(inputBatch)
+
+                if j < skip_upto_batch:
+                    continue
+
+                X, lS_o, lS_i, T, W, CBPP = unpack_batch(inputBatch)
+                if k == 0 and j == 0:
+                    batch = X.size(0)
+
+                if args.mlperf_logging:
+                    current_time = time_wrap(use_gpu, use_xpu)
+                    if previous_iteration_time:
+                        iteration_time = current_time - previous_iteration_time
+                    else:
+                        iteration_time = 0
+                    previous_iteration_time = current_time
+                else:
+                    t1 = time_wrap(use_gpu, use_xpu)
+
+                # early exit if nbatches was set by the user and has been exceeded
+                if nbatches > 0 and j >= nbatches:
+                    break
+
+                # Skip the batch if batch size not multiple of total ranks
+                if ext_dist.my_size > 1 and X.size(0) % ext_dist.my_size != 0:
+                    print(
+                        "Warning: Skiping the batch %d with size %d"
+                        % (j, X.size(0))
+                    )
+                    continue
+
+                mbs = T.shape[0]  # = args.mini_batch_size except maybe for last
+
+                if (args.enable_profiling):
+                    if use_xpu:
+                        sort_key = "self_xpu_time_total"
+                    elif use_cuda:
+                        sort_key = "self_cuda_time_total"
+                    else:
+                        sort_key = "self_cpu_time_total"
+                with (torch.autograd.profiler_legacy.profile(args.enable_profiling, use_xpu=True) if use_xpu else torch.profiler.profile(activities=[torch.profiler.ProfilerActivity.CUDA])) as prof1:
+                    # forward pass
+                    t1 = time.time()
+                    if not use_xpu:
+                        Z = dlrm_wrap(
+                            X,
+                            lS_o,
+                            lS_i,
+                            use_gpu,
+                            use_xpu,
+                            device,
+                            ndevices=ndevices,
+                        )
+
+                        if ext_dist.my_size > 1:
+                            T = T[ext_dist.get_my_slice(mbs)]
+                            W = W[ext_dist.get_my_slice(mbs)]
+
+                        # loss
+                        E = loss_fn_wrap(Z, T, device)
+                    else:
+                        with torch.xpu.amp.autocast(enabled=True, dtype=amp_dtype):
+                            Z = dlrm_wrap(
+                                X,
+                                lS_o,
+                                lS_i,
+                                use_gpu,
+                                use_xpu,
+                                device,
+                                ndevices=ndevices,
+                            )
+
+                            if ext_dist.my_size > 1:
+                                T = T[ext_dist.get_my_slice(mbs)]
+                                W = W[ext_dist.get_my_slice(mbs)]
+
+                            # loss
+                            E = loss_fn_wrap(Z, T, device)
+
+                    # compute loss and accuracy
+                    L = E.detach().cpu().numpy()  # numpy array
+                    # training accuracy is not disabled
+                    # S = Z.detach().cpu().numpy()  # numpy array
+                    # T = T.detach().cpu().numpy()  # numpy array
+
+                    # # print("res: ", S)
+
+                    # # print("j, train: BCE ", j, L)
+
+                    # mbs = T.shape[0]  # = args.mini_batch_size except maybe for last
+                    # A = np.sum((np.round(S, 0) == T).astype(np.uint8))
+
+                    with record_function("DLRM backward"):
+                        # scaled error gradient propagation
+                        # (where we do not accumulate gradients across mini-batches)
+                        if (
+                            args.mlperf_logging
+                            and (j + 1) % args.mlperf_grad_accum_iter == 0
+                        ) or not args.mlperf_logging:
+                            optimizer.zero_grad(set_to_none=True)
+                        # backward pass
+                        E.backward()
+
+                        # optimizer
+                        if (
+                            args.mlperf_logging
+                            and (j + 1) % args.mlperf_grad_accum_iter == 0
+                        ) or not args.mlperf_logging:
+                            optimizer.step()
+                            lr_scheduler.step()
+
+                    #if args.mlperf_logging:
+                    #    total_time += iteration_time
+                    if(k > 0):
+                        t2 = time_wrap(use_gpu, use_xpu)
+                        total_time += t2 - t1
+                        time_collect += t2 - t1
+                        time_counts += 1
+
+                if args.enable_profiling and k == 4 and j == 10:
+                    if use_xpu:
+                        rank_index = ""
+                        if ext_dist.my_size > 1:
+                            rank_index = "rank_" + str(ext_dist.my_local_rank) + "_"
+                        path ='./report/dlrm_training_' + rank_index + 'profiling'
+                        torch.save(prof1.key_averages().table(sort_by="self_xpu_time_total"), \
+                            path + '.pt')
+                        prof1.export_chrome_trace(path + '.json')
+                        torch.save(prof1.table(sort_by="id", row_limit=100000), path + '_detail.pt')
+                    else:
+                        print(prof.key_averages().table(sort_by=sort_key, row_limit=-1))
+                total_loss += L * mbs
+                total_iter += 1
+                total_samp += mbs
+
+                should_print = ((j + 1) % args.print_freq == 0) or (
+                    j + 1 == nbatches
+                )
+                should_test = (
+                    (args.test_freq > 0)
+                    and (args.data_generation in ["dataset", "random"])
+                    and (((j + 1) % args.test_freq == 0) or (j + 1 == nbatches))
+                )
+
+                # print time, loss and accuracy
+                if should_print or should_test:
+                    gT = 1000.0 * total_time / total_iter if args.print_time else -1
+                    total_time = 0
+
+                    train_loss = total_loss / total_samp
+                    total_loss = 0
+
+                    str_run_type = (
+                        "inference" if args.inference_only else "training"
+                    )
+
+                    wall_time = ""
+                    if args.print_wall_time:
+                        wall_time = " ({})".format(time.strftime("%H:%M"))
+
+                    print(
+                        "Finished {} it {}/{} of epoch {}, {:.2f} ms/it,".format(
+                            str_run_type, j + 1, nbatches, k, gT
+                        )
+                        + " loss {:.6f}".format(train_loss)
+                        + wall_time,
+                        flush=True,
+                    )
+
+                    log_iter = nbatches * k + j + 1
+                    writer.add_scalar("Train/Loss", train_loss, log_iter)
+
+                    total_iter = 0
+                    total_samp = 0
+
+                # testing
+                if should_test:
+                    epoch_num_float = (j + 1) / len(train_ld) + k + 1
+                    if args.mlperf_logging:
+                        mlperf_logger.barrier(use_device)
+                        mlperf_logger.log_start(
+                            key=mlperf_logger.constants.EVAL_START,
+                            metadata={
+                                mlperf_logger.constants.EPOCH_NUM: epoch_num_float
+                            },
+                        )
+
+                    # don't measure training iter time in a test iteration
+                    if args.mlperf_logging:
+                        previous_iteration_time = None
+                    print(
+                        "Testing at - {}/{} of epoch {},".format(j + 1, nbatches, k)
+                    )
+                    model_metrics_dict, is_best = inference(
+                        args,
+                        dlrm,
+                        best_acc_test,
+                        best_auc_test,
+                        test_ld,
+                        device,
+                        use_gpu,
+                        use_xpu,
+                        amp_dtype,
+                        log_iter,
+                        amp=True,
+                        inference_only = args.inference_only,
+                        profile = args.enable_profiling,
+                    )
+
+                    if (
+                        is_best
+                        and not (args.save_model == "")
+                        and not args.inference_only
+                    ):
+                        model_metrics_dict["epoch"] = k
+                        model_metrics_dict["iter"] = j + 1
+                        model_metrics_dict["train_loss"] = train_loss
+                        model_metrics_dict["total_loss"] = total_loss
+                        model_metrics_dict[
+                            "opt_state_dict"
+                        ] = optimizer.state_dict()
+                        print("Saving model to {}".format(args.save_model))
+                        torch.save(model_metrics_dict, args.save_model)
+
+                    if args.mlperf_logging:
+                        mlperf_logger.barrier(use_device)
+                        mlperf_logger.log_end(
+                            key=mlperf_logger.constants.EVAL_STOP,
+                            metadata={
+                                mlperf_logger.constants.EPOCH_NUM: epoch_num_float
+                            },
+                        )
+
+                    # Uncomment the line below to print out the total time with overhead
+                    # print("Total test time for this group: {}" \
+                    # .format(time_wrap(use_gpu) - accum_test_time_begin))
+
+                    if (
+                        args.mlperf_logging
+                        and (args.mlperf_acc_threshold > 0)
+                        and (best_acc_test > args.mlperf_acc_threshold)
+                    ):
+                        print(
+                            "MLPerf testing accuracy threshold "
+                            + str(args.mlperf_acc_threshold)
+                            + " reached, stop training"
+                        )
+                        break
+
+                    if (
+                        args.mlperf_logging
+                        and (args.mlperf_auc_threshold > 0)
+                        and (best_auc_test > args.mlperf_auc_threshold)
+                    ):
+                        print(
+                            "MLPerf testing auc threshold "
+                            + str(args.mlperf_auc_threshold)
+                            + " reached, stop training"
+                        )
+                        if args.mlperf_logging:
+                            mlperf_logger.barrier(use_device)
+                            mlperf_logger.log_end(
+                                key=mlperf_logger.constants.RUN_STOP,
+                                metadata={
+                                    mlperf_logger.constants.STATUS: mlperf_logger.constants.SUCCESS
+                                },
+                            )
+                        break
+
+            if args.mlperf_logging:
+                mlperf_logger.barrier(use_device)
+                mlperf_logger.log_end(
+                    key=mlperf_logger.constants.EPOCH_STOP,
+                    metadata={mlperf_logger.constants.EPOCH_NUM: (k + 1)},
+                )
+                mlperf_logger.barrier(use_device)
+                mlperf_logger.log_end(
+                    key=mlperf_logger.constants.BLOCK_STOP,
+                    metadata={mlperf_logger.constants.FIRST_EPOCH_NUM: (k + 1)},
+                )
+            k += 1  # nepochs
+        if args.mlperf_logging and best_auc_test <= args.mlperf_auc_threshold:
+            mlperf_logger.barrier(use_device)
+            mlperf_logger.log_end(
+                key=mlperf_logger.constants.RUN_STOP,
+                metadata={
+                    mlperf_logger.constants.STATUS: mlperf_logger.constants.ABORTED
+                },
+            )
+        if time_collect == 0:
+            print("The total_time is zero, please check args.")
+        else:
+            print('dlrm_train latency: ', time_collect, ' s')
+            print('dlrm_train avg time: ', time_collect/time_counts, ' s, ant the time counts is :', time_counts)
+            print('dlrm_train throughput: ', batch * time_counts /time_collect, ' samples/s')
+    else:
+        print("Testing for inference only")
+        dlrm.eval()
+        if args.fp16 and (not args.amp):
+            dlrm.half()
+        if use_xpu:
+            dlrm = torch.xpu.optimize(model=dlrm, dtype=amp_dtype)
+        inference(
+            args,
+            dlrm,
+            best_acc_test,
+            best_auc_test,
+            test_ld,
+            device,
+            use_gpu,
+            use_xpu,
+            amp_dtype,
+            amp=args.amp,
+            inference_only = args.inference_only,
+            profile = args.enable_profiling,
+        )
+
+    ## profiling
+    #if args.enable_profiling:
+    #    time_stamp = str(datetime.datetime.now()).replace(" ", "_")
+    #    with open("dlrm_s_pytorch" + time_stamp + "_shape.prof", "w") as prof_f:
+    #        prof_f.write(
+    #            prof.key_averages(group_by_input_shape=True).table(
+    #                sort_by="self_cpu_time_total"
+    #            )
+    #        )
+    #    with open("dlrm_s_pytorch" + time_stamp + "_total.prof", "w") as prof_f:
+    #        prof_f.write(prof.key_averages().table(sort_by="self_cpu_time_total"))
+    #    prof.export_chrome_trace("dlrm_s_pytorch" + time_stamp + ".json")
+    #    # print(prof.key_averages().table(sort_by="cpu_time_total"))
+
+    # plot compute graph
+    if args.plot_compute_graph:
+        sys.exit(
+            "ERROR: Please install pytorchviz package in order to use the"
+            + " visualization. Then, uncomment its import above as well as"
+            + " three lines below and run the code again."
+        )
+        # V = Z.mean() if args.inference_only else E
+        # dot = make_dot(V, params=dict(dlrm.named_parameters()))
+        # dot.render('dlrm_s_pytorch_graph') # write .pdf file
+
+    # test prints
+    if not args.inference_only and args.debug_mode:
+        print("updated parameters (weights and bias):")
+        for param in dlrm.parameters():
+            print(param.detach().cpu().numpy())
+
+    # export the model in onnx
+    if args.save_onnx:
+        """
+        # workaround 1: tensor -> list
+        if torch.is_tensor(lS_i_onnx):
+            lS_i_onnx = [lS_i_onnx[j] for j in range(len(lS_i_onnx))]
+        # workaound 2: list -> tensor
+        lS_i_onnx = torch.stack(lS_i_onnx)
+        """
+        # debug prints
+        # print("inputs", X_onnx, lS_o_onnx, lS_i_onnx)
+        # print("output", dlrm_wrap(X_onnx, lS_o_onnx, lS_i_onnx, use_gpu, use_xpu,device))
+        dlrm_pytorch_onnx_file = "dlrm_s_pytorch.onnx"
+        batch_size = X_onnx.shape[0]
+        print("X_onnx.shape", X_onnx.shape)
+        if torch.is_tensor(lS_o_onnx):
+            print("lS_o_onnx.shape", lS_o_onnx.shape)
+        else:
+            for oo in lS_o_onnx:
+                print("oo.shape", oo.shape)
+        if torch.is_tensor(lS_i_onnx):
+            print("lS_i_onnx.shape", lS_i_onnx.shape)
+        else:
+            for ii in lS_i_onnx:
+                print("ii.shape", ii.shape)
+
+        # name inputs and outputs
+        o_inputs = (
+            ["offsets"]
+            if torch.is_tensor(lS_o_onnx)
+            else ["offsets_" + str(i) for i in range(len(lS_o_onnx))]
+        )
+        i_inputs = (
+            ["indices"]
+            if torch.is_tensor(lS_i_onnx)
+            else ["indices_" + str(i) for i in range(len(lS_i_onnx))]
+        )
+        all_inputs = ["dense_x"] + o_inputs + i_inputs
+        # debug prints
+        print("inputs", all_inputs)
+
+        # create dynamic_axis dictionaries
+        do_inputs = (
+            [{"offsets": {1: "batch_size"}}]
+            if torch.is_tensor(lS_o_onnx)
+            else [
+                {"offsets_" + str(i): {0: "batch_size"}} for i in range(len(lS_o_onnx))
+            ]
+        )
+        di_inputs = (
+            [{"indices": {1: "batch_size"}}]
+            if torch.is_tensor(lS_i_onnx)
+            else [
+                {"indices_" + str(i): {0: "batch_size"}} for i in range(len(lS_i_onnx))
+            ]
+        )
+        dynamic_axes = {"dense_x": {0: "batch_size"}, "pred": {0: "batch_size"}}
+        for do in do_inputs:
+            dynamic_axes.update(do)
+        for di in di_inputs:
+            dynamic_axes.update(di)
+        # debug prints
+        print(dynamic_axes)
+        # export model
+        torch.onnx.export(
+            dlrm,
+            (X_onnx, lS_o_onnx, lS_i_onnx),
+            dlrm_pytorch_onnx_file,
+            verbose=True,
+            opset_version=11,
+            input_names=all_inputs,
+            output_names=["pred"],
+            dynamic_axes=dynamic_axes,
+        )
+        # recover the model back
+        dlrm_pytorch_onnx = onnx.load("dlrm_s_pytorch.onnx")
+        # check the onnx model
+        onnx.checker.check_model(dlrm_pytorch_onnx)
+    total_time_end = time_wrap(use_gpu, use_xpu)
+
+
+if __name__ == "__main__":
+    run()
diff --git a/models_v2/pytorch/dlrm/inference/gpu/extend_distributed.py b/models_v2/pytorch/dlrm/inference/gpu/extend_distributed.py
new file mode 100644
index 000000000..b2dab37cd
--- /dev/null
+++ b/models_v2/pytorch/dlrm/inference/gpu/extend_distributed.py
@@ -0,0 +1,671 @@
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+import builtins
+import os
+import sys
+
+import torch
+import torch.distributed as dist
+from torch.autograd import Function
+from torch.autograd.profiler import record_function
+from torch.nn.parallel import DistributedDataParallel as DDP
+
+try:
+    import intel_extension_for_pytorch
+except ImportError as e:
+    pass
+
+try:
+    if torch.__version__[:6] >= '1.12.0':
+        import oneccl_bindings_for_pytorch
+        torch_ccl = False
+    else:
+        import torch_ccl
+except ImportError as e:
+    # print(e)
+    torch_ccl = False
+
+try:
+    import torch_ucc
+except ImportError as e:
+    torch_ucc = False
+
+
+my_rank = -1
+my_size = -1
+my_local_rank = -1
+my_local_size = -1
+alltoall_supported = False
+a2a_impl = os.environ.get("DLRM_ALLTOALL_IMPL", "")
+
+myreq = None
+
+
+def env2int(env_list, default=-1):
+    for e in env_list:
+        val = int(os.environ.get(e, -1))
+        if val >= 0:
+            return val
+    return default
+
+
+def get_my_slice(n):
+    tmp_size = dist.get_world_size()
+    tmp_rank = dist.get_rank()
+
+    k, m = divmod(n, tmp_size)
+    return slice(
+        tmp_rank * k + min(tmp_rank, m), (tmp_rank + 1) * k + min(tmp_rank + 1, m), 1
+    )
+
+
+def get_split_lengths(n):
+    if my_size >1:
+        tmp_size = dist.get_world_size()
+        tmp_rank = dist.get_rank()
+    else:
+        tmp_size = my_size
+        tmp_rank = my_rank
+
+    k, m = divmod(n, tmp_size)
+    if m == 0:
+        splits = None
+        my_len = k
+    else:
+        splits = [(k + 1) if i < m else k for i in range(tmp_size)]
+        my_len = splits[tmp_rank]
+    return (my_len, splits)
+
+
+def init_distributed(rank=-1, local_rank=-1, size=-1, use_device="cpu", backend=""):
+    global myreq
+    global my_size
+    global my_local_rank
+    global my_local_size
+    global a2a_impl
+    global alltoall_supported
+
+    # guess MPI ranks from env (works for IMPI, OMPI and MVAPICH2)
+    num_mpi_ranks = env2int(
+        ["PMI_SIZE", "OMPI_COMM_WORLD_SIZE", "MV2_COMM_WORLD_SIZE", "WORLD_SIZE"]
+    )
+    if backend == "" and num_mpi_ranks > 1:
+        if torch_ccl or oneccl_bindings_for_pytorch:
+            backend = "ccl"
+        elif use_device=="cuda" and dist.is_nccl_available():
+            backend = "nccl"
+        elif dist.is_mpi_available():
+            backend = "mpi"
+        else:
+            print(
+                "WARNING: MPI multi-process launch detected but PyTorch MPI backend not available."
+            )
+            backend = "gloo"
+
+    if backend != "":
+        # guess Rank and size
+        if rank == -1:
+            rank = env2int(
+                ["PMI_RANK", "OMPI_COMM_WORLD_RANK", "MV2_COMM_WORLD_RANK", "RANK"], 0
+            )
+        if size == -1:
+            size = env2int(
+                [
+                    "PMI_SIZE",
+                    "OMPI_COMM_WORLD_SIZE",
+                    "MV2_COMM_WORLD_SIZE",
+                    "WORLD_SIZE",
+                ],
+                1,
+            )
+        if not os.environ.get("RANK", None) and rank != -1:
+            os.environ["RANK"] = str(rank)
+        if not os.environ.get("WORLD_SIZE", None) and size != -1:
+            os.environ["WORLD_SIZE"] = str(size)
+        if not os.environ.get("MASTER_PORT", None):
+            os.environ["MASTER_PORT"] = "29500"
+        if not os.environ.get("MASTER_ADDR", None):
+            local_size = env2int(
+                [
+                    "MPI_LOCALNRANKS",
+                    "OMPI_COMM_WORLD_LOCAL_SIZE",
+                    "MV2_COMM_WORLD_LOCAL_SIZE",
+                ],
+                1,
+            )
+            if local_size != size and backend != "mpi":
+                print(
+                    "Warning: Looks like distributed multinode run but MASTER_ADDR env not set, using '127.0.0.1' as default"
+                )
+                print(
+                    "If this run hangs, try exporting rank 0's hostname as MASTER_ADDR"
+                )
+            os.environ["MASTER_ADDR"] = "127.0.0.1"
+
+    if size > 1:
+        if local_rank == -1:
+            my_local_rank = env2int(
+                [
+                    "MPI_LOCALRANKID",
+                    "OMPI_COMM_WORLD_LOCAL_RANK",
+                    "MV2_COMM_WORLD_LOCAL_RANK",
+                    "LOCAL_RANK",
+                ],
+                0,
+            )
+        else:
+            my_local_rank = local_rank
+        my_local_size = env2int(
+            [
+                "MPI_LOCALNRANKS",
+                "OMPI_COMM_WORLD_LOCAL_SIZE",
+                "MV2_COMM_WORLD_LOCAL_SIZE",
+            ],
+            1,
+        )
+        if use_device=="cuda":
+            if my_local_size > torch.cuda.device_count():
+                print(
+                    "Not sufficient GPUs available... local_size = %d, ngpus = %d"
+                    % (my_local_size, torch.cuda.device_count())
+                )
+                sys.exit(1)
+            torch.cuda.set_device(my_local_rank)
+        elif use_device=="xpu":
+            if my_local_size > torch.xpu.device_count():
+                print(
+                    "Not sufficient GPUs available... local_size = %d, ngpus = %d"
+                    % (my_local_size, torch.xpu.device_count())
+                )
+                sys.exit(1)
+            torch.xpu.set_device(my_local_rank)
+        dist.init_process_group(backend, rank=rank, world_size=size)
+        this_rank = dist.get_rank()
+        my_size = dist.get_world_size()
+        #my_local_rank = env2int(['MPI_LOCALRANKID', 'OMPI_COMM_WORLD_LOCAL_RANK', 'MV2_COMM_WORLD_LOCAL_RANK'], 0)
+        #my_local_size = env2int(['MPI_LOCALNRANKS', 'OMPI_COMM_WORLD_LOCAL_SIZE', 'MV2_COMM_WORLD_LOCAL_SIZE'], 1)
+        if this_rank == 0:
+            print("Running on %d ranks using %s backend" % (my_size, backend))
+        if hasattr(dist, "all_to_all_single"):
+            try:
+                a = torch.rand([1, 3]).xpu("xpu:{}".format(my_local_rank))
+                dist.broadcast(a, src=0)
+                print("Successfully execute distributed broadcast on the xpu:{}".format(this_rank))
+
+                tensor_list = [torch.zeros(2, dtype=torch.int64).xpu("xpu:{}".format(my_local_rank)) for _ in range(2)]
+                tensor = (torch.arange(2, dtype=torch.int64) + 1 + 2 * rank).xpu("xpu:{}".format(my_local_rank))
+                dist.all_gather(tensor_list, tensor)
+                print("Successfully execute distributed all_gather on the xpu:{}".format(this_rank))
+
+                t = torch.zeros([4])
+                if use_device == "cuda":
+                    t = t.cuda()
+                elif  use_device == "xpu":
+                    t = t.xpu()
+                dist.all_to_all_single(t, t)
+
+                print("Successfully execute distributed all_to_all_single on the xpu:{}".format(this_rank))
+                alltoall_supported = True
+            except RuntimeError as err:
+                print("fail to enable all_to_all_single primitive: %s" % err)
+                exit()
+        if a2a_impl == "alltoall" and alltoall_supported == False:
+            print(
+                "Requested DLRM_ALLTOALL_IMPL=%s but backend %s does not support it, use scatter/gather based alltoall"
+                % (a2a_impl, backend)
+            )
+            a2a_impl = "scatter"
+        if a2a_impl != "":
+            print("Using DLRM_ALLTOALL_IMPL=%s" % a2a_impl)
+    else:
+        this_rank = 0
+        my_size = 1
+        my_local_rank = 0
+        my_local_size = 1
+    print_all(
+        "world size: %d, current rank: %d, local rank: %d"
+        % (my_size, this_rank, my_local_rank)
+    )
+    myreq = Request()
+
+
+class Request(object):
+    def __init__(self):
+        self.req = None
+        self.tensor = None
+        self.WaitFunction = All2All_Scatter_Wait
+
+    def wait(self):
+        ret = self.WaitFunction.apply(*self.tensor)
+        self.req = None
+        self.tensor = None
+        return ret
+
+
+class All2All_ScatterList_Req(Function):
+    @staticmethod
+    def forward(ctx, a2a_info, *inputs):
+        global myreq
+        tmp_size = dist.get_world_size()
+        tmp_rank = dist.get_rank()
+        batch_split_lengths = (
+            a2a_info.global_batch_partition_slices
+            if a2a_info.global_batch_partition_slices
+            else a2a_info.local_batch_num
+        )
+        table_split_lengths = (
+            a2a_info.global_table_wise_parition_slices
+            if a2a_info.global_table_wise_parition_slices
+            else [a2a_info.local_table_num] * tmp_size
+        )
+        gather_list = []
+        req_list = []
+        for i in range(tmp_size):
+            for j in range(table_split_lengths[i]):
+                out_tensor = inputs[0].new_empty(
+                    [a2a_info.local_batch_num, a2a_info.emb_dim]
+                )
+                scatter_list = (
+                    list(inputs[j].split(batch_split_lengths, dim=0))
+                    if i == tmp_rank
+                    else []
+                )
+                req = dist.scatter(out_tensor, scatter_list, src=i, async_op=True)
+                gather_list.append(out_tensor)
+                req_list.append(req)
+        myreq.req = req_list
+        myreq.tensor = tuple(gather_list)
+        myreq.a2a_info = a2a_info
+        return myreq.tensor
+
+    @staticmethod
+    def backward(ctx, *grad_output):
+        global myreq
+        for r in myreq.req:
+            r.wait()
+        myreq.req = None
+        grad_inputs = myreq.tensor
+        myreq.tensor = None
+        return (None, *grad_inputs)
+
+
+class All2All_ScatterList_Wait(Function):
+    @staticmethod
+    def forward(ctx, *output):
+        global myreq
+        ctx.a2a_info = myreq.a2a_info
+        for r in myreq.req:
+            r.wait()
+        myreq.req = None
+        myreq.tensor = None
+        return output
+
+    @staticmethod
+    def backward(ctx, *grad_output):
+        global myreq
+        tmp_size = dist.get_world_size()
+        tmp_rank = dist.get_rank()
+        a2a_info = ctx.a2a_info
+        grad_output = [t.contiguous() for t in grad_output]
+        batch_split_lengths = (
+            a2a_info.global_batch_partition_slices
+            if a2a_info.global_batch_partition_slices
+            else [a2a_info.local_batch_num] * tmp_size
+        )
+        per_rank_table_splits = (
+            a2a_info.global_table_wise_parition_slices
+            if a2a_info.global_table_wise_parition_slices
+            else [a2a_info.local_table_num] * tmp_size
+        )
+        grad_inputs = [
+            grad_output[0].new_empty([ctx.a2a_info.batch_size, ctx.a2a_info.emb_dim])
+            for _ in range(a2a_info.local_table_num)
+        ]
+        req_list = []
+        ind = 0
+        for i in range(tmp_size):
+            for j in range(per_rank_table_splits[i]):
+                gather_list = (
+                    list(grad_inputs[j].split(batch_split_lengths, dim=0))
+                    if i == tmp_rank
+                    else None
+                )
+                req = dist.gather(grad_output[ind], gather_list, dst=i, async_op=True)
+                req_list.append(req)
+                ind += 1
+        myreq.req = req_list
+        myreq.tensor = grad_inputs
+        return tuple(grad_output)
+
+
+class All2All_Scatter_Req(Function):
+    @staticmethod
+    def forward(ctx, a2a_info, *inputs):
+        tmp_size = dist.get_world_size()
+        tmp_rank = dist.get_rank()
+        global myreq
+        batch_split_lengths = (
+            a2a_info.global_batch_partition_slices
+            if a2a_info.global_batch_partition_slices
+            else a2a_info.local_batch_num
+        )
+        table_split_lengths = (
+            a2a_info.global_table_wise_parition_slices
+            if a2a_info.global_table_wise_parition_slices
+            else [a2a_info.local_table_num] * tmp_size
+        )
+        input = torch.cat(inputs, dim=1)
+        scatter_list = list(input.split(batch_split_lengths, dim=0))
+        gather_list = []
+        req_list = []
+        for i in range(tmp_size):
+            out_tensor = input.new_empty(
+                [a2a_info.local_batch_num, table_split_lengths[i] * a2a_info.emb_dim]
+            )
+            req = dist.scatter(
+                out_tensor, scatter_list if i == tmp_rank else [], src=i, async_op=True
+            )
+            gather_list.append(out_tensor)
+            req_list.append(req)
+        myreq.req = req_list
+        myreq.tensor = tuple(gather_list)
+        myreq.a2a_info = a2a_info
+        ctx.a2a_info = a2a_info
+        return myreq.tensor
+
+    @staticmethod
+    def backward(ctx, *grad_output):
+        global myreq
+        for r in myreq.req:
+            r.wait()
+        myreq.req = None
+        grad_input = myreq.tensor
+        grad_inputs = grad_input.split(ctx.a2a_info.emb_dim, dim=1)
+        myreq.tensor = None
+        return (None, *grad_inputs)
+
+
+class All2All_Scatter_Wait(Function):
+    @staticmethod
+    def forward(ctx, *output):
+        global myreq
+        ctx.a2a_info = myreq.a2a_info
+        for r in myreq.req:
+            r.wait()
+        myreq.req = None
+        myreq.tensor = None
+        return output
+
+    @staticmethod
+    def backward(ctx, *grad_output):
+        tmp_size = dist.get_world_size()
+        tmp_rank = dist.get_rank()
+        global myreq
+        assert len(grad_output) == tmp_size
+        scatter_list = [t.contiguous() for t in grad_output]
+        a2a_info = ctx.a2a_info
+        batch_split_lengths = (
+            a2a_info.global_batch_partition_slices
+            if a2a_info.global_batch_partition_slices
+            else a2a_info.local_batch_num
+        )
+        table_split_lengths = (
+            a2a_info.global_table_wise_parition_slices
+            if a2a_info.global_table_wise_parition_slices
+            else [a2a_info.local_table_num] * tmp_size
+        )
+        grad_input = grad_output[0].new_empty(
+            [a2a_info.batch_size, a2a_info.emb_dim * a2a_info.local_table_num]
+        )
+        gather_list = list(grad_input.split(batch_split_lengths, dim=0))
+        req_list = []
+        for i in range(tmp_size):
+            req = dist.gather(
+                scatter_list[i],
+                gather_list if i == tmp_rank else [],
+                dst=i,
+                async_op=True,
+            )
+            req_list.append(req)
+        myreq.req = req_list
+        myreq.tensor = grad_input
+        return grad_output
+
+
+class All2All_Req(Function):
+    @staticmethod
+    def forward(ctx, a2a_info, *inputs):
+        global myreq
+        with record_function("DLRM alltoall_req_fwd_single"):
+            batch_split_lengths = a2a_info.global_batch_partition_slices
+            if batch_split_lengths:
+                batch_split_lengths = [
+                    m * a2a_info.emb_dim * a2a_info.local_table_num
+                    for m in batch_split_lengths
+                ]
+            table_split_lengths = a2a_info.global_table_wise_parition_slices
+            if table_split_lengths:
+                table_split_lengths = [
+                    a2a_info.local_batch_num * e * a2a_info.emb_dim
+                    for e in table_split_lengths
+                ]
+            input = torch.cat(inputs, dim=1).view([-1])
+            output = input.new_empty(
+                [
+                    a2a_info.global_table_num
+                    * a2a_info.local_batch_num
+                    * a2a_info.emb_dim
+                ]
+            )
+            req = dist.all_to_all_single(
+                output, input, table_split_lengths, batch_split_lengths, async_op=True
+            )
+
+            myreq.req = req
+            myreq.tensor = []
+            myreq.tensor.append(output)
+            myreq.tensor = tuple(myreq.tensor)
+            a2a_info.batch_split_lengths = batch_split_lengths
+            a2a_info.table_split_lengths = table_split_lengths
+            myreq.a2a_info = a2a_info
+            ctx.a2a_info = a2a_info
+            return myreq.tensor
+
+    @staticmethod
+    def backward(ctx, *grad_output):
+        global myreq
+        with record_function("DLRM alltoall_req_bwd_single"):
+            a2a_info = ctx.a2a_info
+            myreq.req.wait()
+            myreq.req = None
+            grad_input = myreq.tensor
+            grad_inputs = grad_input.view([a2a_info.batch_size, -1]).split(
+                a2a_info.emb_dim, dim=1
+            )
+            grad_inputs = [gin.contiguous() for gin in grad_inputs]
+            myreq.tensor = None
+            return (None, *grad_inputs)
+
+
+class All2All_Wait(Function):
+    @staticmethod
+    def forward(ctx, *output):
+        global myreq
+        with record_function("DLRM alltoall_wait_fwd_single"):
+            a2a_info = myreq.a2a_info
+            ctx.a2a_info = a2a_info
+            myreq.req.wait()
+            myreq.req = None
+            myreq.tensor = None
+            table_split_lengths = (
+                a2a_info.table_split_lengths
+                if a2a_info.table_split_lengths
+                else a2a_info.local_table_num
+                * a2a_info.local_batch_num
+                * a2a_info.emb_dim
+            )
+            outputs = output[0].split(table_split_lengths)
+            outputs = tuple(
+                [out.view([a2a_info.local_batch_num, -1]) for out in outputs]
+            )
+            return outputs
+
+    @staticmethod
+    def backward(ctx, *grad_outputs):
+        global myreq
+        with record_function("DLRM alltoall_wait_bwd_single"):
+            a2a_info = ctx.a2a_info
+            grad_outputs = [gout.contiguous().view([-1]) for gout in grad_outputs]
+            grad_output = torch.cat(grad_outputs)
+            grad_input = grad_output.new_empty(
+                [a2a_info.batch_size * a2a_info.local_table_num * a2a_info.emb_dim]
+            )
+            req = dist.all_to_all_single(
+                grad_input,
+                grad_output,
+                a2a_info.batch_split_lengths,
+                a2a_info.table_split_lengths,
+                async_op=True,
+            )
+            myreq.req = req
+            myreq.tensor = grad_input
+            return (grad_output,)
+
+
+class AllGather(Function):
+    @staticmethod
+    def forward(ctx, input, global_lengths, dim=0):
+        tmp_size = dist.get_world_size()
+        tmp_rank = dist.get_rank()
+        if not isinstance(global_lengths, (list, tuple)):
+            global_lengths = [global_lengths] * tmp_size
+
+        assert len(global_lengths) == tmp_size
+        assert global_lengths[tmp_rank] == input.size(dim)
+        local_start = sum(global_lengths[:tmp_rank])
+
+        output_size = list(input.size())
+
+        ctx.dim = dim
+        ctx.local_start = local_start
+        ctx.local_length = global_lengths[tmp_rank]
+
+        input = input.contiguous()
+        if dim == 0:
+            out_len = sum(global_lengths)
+            output_size[dim] = out_len
+            output = input.new_empty(output_size)
+            gather_list = list(output.split(global_lengths, dim=0))
+        else:
+            gather_list = [torch.empty_like(input) for _ in range(tmp_size)]
+            gather_list = []
+            for length in global_lengths:
+                output_size[dim] = length
+                gather_list.append(input.new_empty(output_size))
+
+        dist.all_gather(gather_list, input)
+
+        if dim != 0:
+            output = torch.cat(gather_list, dim=dim)
+
+        return output
+
+    @staticmethod
+    def backward(ctx, grad_output):
+        # print("Inside All2AllBackward")
+        dim = ctx.dim
+        start = ctx.local_start
+        length = ctx.local_length
+
+        grad_input = grad_output.narrow(dim, start, length)
+
+        return (grad_input, None, None)
+
+
+class All2AllInfo(object):
+    pass
+
+
+def alltoall(inputs, per_rank_table_splits):
+    tmp_size = dist.get_world_size()
+    global myreq
+    batch_size, emb_dim = inputs[0].size()
+    a2a_info = All2AllInfo()
+    a2a_info.local_table_num = len(inputs)
+    a2a_info.global_table_wise_parition_slices = per_rank_table_splits
+    (
+        a2a_info.local_batch_num,
+        a2a_info.global_batch_partition_slices,
+    ) = get_split_lengths(batch_size)
+    a2a_info.emb_dim = emb_dim
+    a2a_info.batch_size = batch_size
+    a2a_info.global_table_num = (
+        sum(per_rank_table_splits)
+        if per_rank_table_splits
+        else a2a_info.local_table_num * tmp_size
+    )
+
+    if a2a_impl == "" and alltoall_supported or a2a_impl == "alltoall":
+        # print("Using All2All_Req")
+        output = All2All_Req.apply(a2a_info, *inputs)
+        myreq.WaitFunction = All2All_Wait
+    elif a2a_impl == "" or a2a_impl == "scatter":
+        # print("Using All2All_Scatter_Req")
+        output = All2All_Scatter_Req.apply(a2a_info, *inputs)
+        myreq.WaitFunction = All2All_Scatter_Wait
+    elif a2a_impl == "scatter_list":
+        # print("Using All2All_ScatterList_Req")
+        output = All2All_ScatterList_Req.apply(a2a_info, *inputs)
+        myreq.WaitFunction = All2All_ScatterList_Wait
+    else:
+        print(
+            "Unknown value set for DLRM_ALLTOALL_IMPL (%s), "
+            "please use one of [alltoall, scatter, scatter_list]" % a2a_impl
+        )
+    return myreq
+
+
+def all_gather(input, lengths, dim=0):
+    tmp_size = dist.get_world_size()
+    if not lengths:
+        lengths = [input.size(0)] * tmp_size
+    return AllGather.apply(input, lengths, dim)
+
+
+def barrier():
+    if my_size > 1:
+        dist.barrier()
+
+
+# Override builtin print function to print only from rank 0
+orig_print = builtins.print
+#
+#
+#def rank0_print(*args, **kwargs):
+#    if my_rank <= 0 or kwargs.get("print_all", False):
+#        orig_print(*args, **kwargs)
+#
+#
+#builtins.print = rank0_print
+#
+# Allow printing from all rank with explicit print_all
+def print_all(*args, **kwargs):
+    orig_print(*args, **kwargs)
diff --git a/models_v2/pytorch/dlrm/inference/gpu/mlperf_logger.py b/models_v2/pytorch/dlrm/inference/gpu/mlperf_logger.py
new file mode 100644
index 000000000..93887d6fc
--- /dev/null
+++ b/models_v2/pytorch/dlrm/inference/gpu/mlperf_logger.py
@@ -0,0 +1,132 @@
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# @lint-ignore-every LICENSELINT
+# Copyright (c) 2020, NVIDIA CORPORATION.  All rights reserved.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+
+"""
+Utilities for MLPerf logging
+"""
+import os
+
+import torch
+
+try:
+    from mlperf_logging import mllog
+    from mlperf_logging.mllog import constants
+
+    _MLLOGGER = mllog.get_mllogger()
+except ImportError as error:
+    print("Unable to import mlperf_logging, ", error)
+
+
+def log_start(*args, **kwargs):
+    "log with start tag"
+    _log_print(_MLLOGGER.start, *args, **kwargs)
+
+
+def log_end(*args, **kwargs):
+    "log with end tag"
+    _log_print(_MLLOGGER.end, *args, **kwargs)
+
+
+def log_event(*args, **kwargs):
+    "log with event tag"
+    _log_print(_MLLOGGER.event, *args, **kwargs)
+
+
+def _log_print(logger, *args, **kwargs):
+    "makes mlperf logger aware of distributed execution"
+    if "stack_offset" not in kwargs:
+        kwargs["stack_offset"] = 3
+    if "value" not in kwargs:
+        kwargs["value"] = None
+
+    if kwargs.pop("log_all_ranks", False):
+        log = True
+    else:
+        log = get_rank() == 0
+
+    if log:
+        logger(*args, **kwargs)
+
+
+def config_logger(benchmark):
+    "initiates mlperf logger"
+    mllog.config(
+        filename=os.path.join(
+            os.path.dirname(os.path.abspath(__file__)), f"{benchmark}.log"
+        )
+    )
+    _MLLOGGER.logger.propagate = False
+
+
+def barrier(use_device="cpu"):
+    """
+    Works as a temporary distributed barrier, currently pytorch
+    doesn't implement barrier for NCCL backend.
+    Calls all_reduce on dummy tensor and synchronizes with GPU.
+    """
+    if torch.distributed.is_available() and torch.distributed.is_initialized():
+        if use_device == "cuda":
+            torch.distributed.all_reduce(torch.cuda.FloatTensor(1))
+            torch.cuda.synchronize()
+        elif use_device == "xpu":
+            torch.distributed.all_reduce(torch.xpu.FloatTensor(1))
+            torch.xpu.synchronize()
+        else:
+            pass
+
+
+def get_rank():
+    """
+    Gets distributed rank or returns zero if distributed is not initialized.
+    """
+    if torch.distributed.is_available() and torch.distributed.is_initialized():
+        rank = torch.distributed.get_rank()
+    else:
+        rank = 0
+    return rank
+
+
+def mlperf_submission_log(benchmark):
+    """
+    Logs information needed for MLPerf submission
+    """
+
+    config_logger(benchmark)
+
+    log_event(
+        key=constants.SUBMISSION_BENCHMARK,
+        value=benchmark,
+    )
+
+    log_event(key=constants.SUBMISSION_ORG, value="reference_implementation")
+
+    log_event(key=constants.SUBMISSION_DIVISION, value="closed")
+
+    log_event(key=constants.SUBMISSION_STATUS, value="onprem")
+
+    log_event(key=constants.SUBMISSION_PLATFORM, value="reference_implementation")
+
+    log_event(key=constants.SUBMISSION_ENTRY, value="reference_implementation")
+
+    log_event(key=constants.SUBMISSION_POC_NAME, value="reference_implementation")
+
+    log_event(key=constants.SUBMISSION_POC_EMAIL, value="reference_implementation")
diff --git a/models_v2/pytorch/dlrm/inference/gpu/optim/rwsadagrad.py b/models_v2/pytorch/dlrm/inference/gpu/optim/rwsadagrad.py
new file mode 100644
index 000000000..e474e9fbb
--- /dev/null
+++ b/models_v2/pytorch/dlrm/inference/gpu/optim/rwsadagrad.py
@@ -0,0 +1,152 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+
+import torch
+from torch.optim import Optimizer
+
+
+class RWSAdagrad(Optimizer):
+    """Implements Row Wise Sparse Adagrad algorithm.
+
+    Arguments:
+        params (iterable): iterable of parameters to optimize or dicts defining
+            parameter groups
+        lr (float, optional): learning rate (default: 1e-2)
+        lr_decay (float, optional): learning rate decay (default: 0)
+        weight_decay (float, optional): weight decay (L2 penalty) (default: 0)
+        eps (float, optional): term added to the denominator to improve
+            numerical stability (default: 1e-10)
+
+    """
+
+    def __init__(
+        self,
+        params,
+        lr=1e-2,
+        lr_decay=0.0,
+        weight_decay=0.0,
+        initial_accumulator_value=0.0,
+        eps=1e-10,
+    ):
+        if not 0.0 <= lr:
+            raise ValueError("Invalid learning rate: {}".format(lr))
+        if not 0.0 <= lr_decay:
+            raise ValueError("Invalid lr_decay value: {}".format(lr_decay))
+        if not 0.0 <= weight_decay:
+            raise ValueError("Invalid weight_decay value: {}".format(weight_decay))
+        if not 0.0 <= initial_accumulator_value:
+            raise ValueError(
+                "Invalid initial_accumulator_value value: {}".format(
+                    initial_accumulator_value
+                )
+            )
+        if not 0.0 <= eps:
+            raise ValueError("Invalid epsilon value: {}".format(eps))
+
+        self.defaults = dict(
+            lr=lr,
+            lr_decay=lr_decay,
+            eps=eps,
+            weight_decay=weight_decay,
+            initial_accumulator_value=initial_accumulator_value,
+        )
+        super(RWSAdagrad, self).__init__(params, self.defaults)
+
+        self.momentum_initialized = False
+
+        for group in self.param_groups:
+            for p in group["params"]:
+                self.state[p]["step"] = 0
+
+    def share_memory(self):
+        for group in self.param_groups:
+            for p in group["params"]:
+                state = self.state[p]
+                if p.grad.data.is_sparse:
+                    state["momentum"].share_memory_()
+                else:
+                    state["sum"].share_memory_()
+
+    def step(self, closure=None):
+        """Performs a single optimization step.
+
+        Arguments:
+            closure (callable, optional): A closure that reevaluates the model
+                and returns the loss.
+        """
+        loss = None
+        if closure is not None:
+            loss = closure()
+
+        for group in self.param_groups:
+            for p in group["params"]:
+                if p.grad is None:
+                    continue
+
+                if not self.momentum_initialized:
+                    if p.grad.data.is_sparse:
+                        self.state[p]["momentum"] = torch.full(
+                            [p.data.shape[0]],
+                            self.defaults["initial_accumulator_value"],
+                            dtype=torch.float32,
+                        )
+                    else:
+                        self.state[p]["sum"] = torch.full_like(
+                            p.data,
+                            self.defaults["initial_accumulator_value"],
+                            dtype=torch.float32,
+                        )
+
+                grad = p.grad
+                state = self.state[p]
+
+                state["step"] += 1
+
+                if group["weight_decay"] != 0:
+                    if p.grad.data.is_sparse:
+                        raise RuntimeError(
+                            "weight_decay option is not compatible with sparse gradients"
+                        )
+                    grad = grad.add(group["weight_decay"], p.data)
+
+                clr = group["lr"] / (1.0 + (state["step"] - 1.0) * group["lr_decay"])
+
+                if grad.is_sparse:
+                    grad = (
+                        grad.coalesce()
+                    )  # the update is non-linear so indices must be unique
+                    grad_indices = grad._indices()
+                    grad_values = grad._values()
+                    size = grad.size()
+
+                    def make_sparse(values, row_wise):
+                        constructor = grad.new
+                        matrix_size = [size[0]] if row_wise else size
+                        return constructor(grad_indices, values, matrix_size)
+
+                    if grad_values.numel() > 0:
+                        momentum_update = make_sparse(
+                            grad_values.pow(2).mean(dim=1), True
+                        )
+                        state["momentum"].add_(momentum_update)  # update momentum
+                        std = state["momentum"].sparse_mask(momentum_update.coalesce())
+                        std_values = std._values().sqrt_().add_(group["eps"])
+                        p.data.add_(
+                            make_sparse(
+                                grad_values / std_values.view(std_values.size()[0], 1),
+                                False,
+                            ),
+                            alpha=-clr,
+                        )
+
+                else:
+                    state["sum"].addcmul_(grad, grad, value=1.0)
+                    std = state["sum"].sqrt().add_(group["eps"])
+                    p.data.addcdiv_(grad, std, value=-clr)
+
+        self.momentum_initialized = True
+
+        return loss
diff --git a/models_v2/pytorch/dlrm/inference/gpu/requirements.txt b/models_v2/pytorch/dlrm/inference/gpu/requirements.txt
new file mode 100644
index 000000000..d5b8649a4
--- /dev/null
+++ b/models_v2/pytorch/dlrm/inference/gpu/requirements.txt
@@ -0,0 +1,8 @@
+future
+numpy
+onnx
+pydot
+scikit-learn
+tqdm
+torchrec-nightly
+torchx-nightly
diff --git a/models_v2/pytorch/dlrm/inference/gpu/run_model.sh b/models_v2/pytorch/dlrm/inference/gpu/run_model.sh
new file mode 100755
index 000000000..594f13bbb
--- /dev/null
+++ b/models_v2/pytorch/dlrm/inference/gpu/run_model.sh
@@ -0,0 +1,129 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+
+#!/bin/bash
+
+# Create an array of input directories that are expected and then verify that they exist
+declare -A input_envs
+input_envs[DATASET_DIR]=${DATASET_DIR}
+input_envs[CKPT_DIR]=${CKPT_DIR}
+input_envs[MULTI_TILE]=${MULTI_TILE}
+input_envs[PLATFORM]=${PLATFORM}
+input_envs[OUTPUT_DIR]=${OUTPUT_DIR}
+
+for i in "${!input_envs[@]}"; do
+  var_name=$i
+  env_param=${input_envs[$i]}
+
+  if [[ -z $env_param ]]; then
+    echo "The required environment variable $var_name is not set" >&2
+    exit 1
+  fi
+done
+
+OUTPUT_DIR=${OUTPUT_DIR:-${PWD}}
+if [[ "${PLATFORM}" == "Max" ]]; then
+    echo "Only support Flex as platform"
+    exit 1
+elif [[ "${PLATFORM}" == "Flex" ]]; then
+    BATCH_SIZE=${BATCH_SIZE:-32768}
+    PRECISION=${PRECISION:-fp16}
+    NUM_ITERATIONS=${NUM_ITERATIONS:-20}
+fi
+
+if [[ "${MULTI_TILE}" == "True" ]]; then
+    echo "Only support Single Tile."
+    exit 1
+fi
+
+if [[ ! -d "${DATASET_DIR}" ]]; then
+    echo "The DATASET_DIR '${DATASET_DIR}' does not exist"
+    exit 1
+fi
+
+if [[ ! -d "${CKPT_DIR}" ]]; then
+    echo "The CKPT_DIR '${CKPT_DIR}' does not exist"
+    exit 1
+fi
+
+echo 'Running with parameters:'
+echo " PLATFORM: ${PLATFORM}"
+echo " OUTPUT_DIR: ${OUTPUT_DIR}"
+echo " DATASET_DIR: ${DATASET_DIR}"
+echo " CKPT_DIR: ${CKPT_DIR}"
+echo " PRECISION: ${PRECISION}"
+echo " BATCH_SIZE: ${BATCH_SIZE}"
+echo " NUM_ITERATIONS: ${NUM_ITERATIONS}"
+echo " MULTI_TILE: ${MULTI_TILE}"
+echo " OUTPUT_DIR: ${OUTPUT_DIR}"
+echo "dlrm ${PRECISION} inference plain MultiTile=${MULTI_TILE} BS=${BATCH_SIZE} Iter=${NUM_ITERATIONS}"
+
+# Create the output directory, if it doesn't already exist
+mkdir -p $OUTPUT_DIR
+
+sum_log_analysis() {
+    if [ -f $2 ]; then
+        rm -f $2
+    fi
+    if diff /dev/null ${1}_t0.log |tail -l | grep '^\\ No newline' > /dev/null;then echo >> ${1}_t0.log; fi
+    if diff /dev/null ${1}_t1.log |tail -l | grep '^\\ No newline' > /dev/null;then echo >> ${1}_t1.log; fi
+    bs=$(cat ${1}_t1.log |grep Batch |awk '{print $3}')
+    echo -e "Batch Size: $bs" >$2
+    cat ${1}"_t0.log" ${1}"_t1.log" |grep "Performance" |awk -v tag=$(cat ${1}"_t0.log" ${1}"_t1.log" |grep "Performance" |awk '{sum+=$2} END {printf "%.4f\n",sum}') '{if ( $2=="None" ) {sum="None";nextfile}else  sum=tag} ;END{print "Sum "$1" "sum " "$3}' >> $2
+    cat ${1}"_t0.log" ${1}"_t1.log" |grep "Performance" |awk -v tag=$(cat ${1}"_t0.log" ${1}"_t1.log" |grep "Performance" |awk 'BEGIN {min=1234567890123} {if ($2 <min) {min=$2}}END {printf "%.4f\n",min}') '{if ( $2=="None" ) {min="None";nextfile}else  min=tag} ;END{print "Min "$1" "min " "$3}' >> $2
+    cat ${1}"_t0.log" ${1}"_t1.log" |grep "Latency" |awk '{if ( $2=="N/A" ){avg="N/A";nextfile}else avg=((sum+=$2/2))};END{print "Avg "$1" "avg " "$3}'  >> $2
+    cat ${1}"_t0.log" ${1}"_t1.log" |grep "Accuracy" |awk -v avg=$(cat ${1}"_t0.log" ${1}"_t1.log" |grep "Accuracy" |awk '{sum+=$3}END{printf "%.4f\n",sum/NR}') '{if ( $3=="None" || $2=="N/A" || $3=="nan" || $3=="N/A"){avg="None";nextfile}else avg=avg};END{print "Avg "$1" "$2 " "avg}' >> $2
+    cat ${1}"_t0.log" ${1}"_t1.log" |grep "Functional" | awk -v fail=$(cat ${1}"_t0.log" ${1}"_t1.log" |grep "Functional" |awk '{for(i=1;i<=NF;++i) if($i=="fail") ++sum}END{print sum}') '{if ( fail >= 1 ) tag="fail ";else tag="pass"};END{print $1" "tag}'  >> $2
+    cat ${1}"_t0.log" ${1}"_t1.log" |grep "Error" |awk '{if(a[$1]){a[$1]=a[$1]";"$2}else{a[$1]=$2}}END{for(i in a)print $1" " a[i]}' >> $2
+}
+
+
+modelname=dlrm-kaggle
+
+if [[ ${MULTI_TILE} == "False" ]]; then
+    rm ${OUTPUT_DIR}/${modelname}${PRECISION}_inf_t0_raw.log
+    python -u ./dlrm_s_pytorch.py --data-generation=dataset --data-set=kaggle --raw-data-file=${DATASET_DIR}/train.txt --processed-data-file=${DATASET_DIR}/kaggleAdDisplayChallenge_processed.npz --loss-function=bce --round-targets=True --mini-batch-size=${BATCH_SIZE} --arch-mlp-bot=13-512-256-64-16 --arch-mlp-top=512-256-1 --arch-sparse-feature-size=16 --max-ind-range=600000000 --print-freq=1024 --print-time --test-freq=6400 --test-mini-batch-size=${BATCH_SIZE} --num-batches=${NUM_ITERATIONS} --nepochs=10 --inference-only --use-xpu --load-model=${CKPT_DIR}/dlrm_kaggle_16.pt --${PRECISION} 2>&1 | tee ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0_raw.log
+    python common/parse_result.py -m $modelname -l ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0_raw.log -b ${BATCH_SIZE}
+    throughput=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0.log | grep Performance | awk -F ' ' '{print $2}')
+    throughput_unit=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0.log | grep Performance | awk -F ' ' '{print $3}')
+    latency=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0.log | grep Latency | awk -F ' ' '{print $2}')
+    acc=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0.log | grep Accuracy | awk -F ' ' '{print $3}')
+    acc_unit=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0.log | grep Accuracy | awk -F ' ' '{print $2}')
+else
+    echo "Only support MULTI_TILE=False"
+    exit 1
+fi
+
+yaml_content=$(cat <<EOF
+results:
+ - key: throughput
+   value: $throughput
+   unit: $throughput_unit
+ - key: latency
+   value: $latency
+   unit: s
+ - key: accuracy
+   value: $acc
+   unit: $acc_unit
+EOF
+)
+
+# Write the content to a YAML file
+echo "$yaml_content" >  ${OUTPUT_DIR}/results.yaml
+echo "YAML file created."
diff --git a/models_v2/pytorch/dlrm/inference/gpu/setup.sh b/models_v2/pytorch/dlrm/inference/gpu/setup.sh
new file mode 100755
index 000000000..cec1f210a
--- /dev/null
+++ b/models_v2/pytorch/dlrm/inference/gpu/setup.sh
@@ -0,0 +1,29 @@
+#!/usr/bin/env bash
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# setup.sh
+#  - install OS pkgs
+#  - should create virtual env & install pip  requirement.txt
+#  - git clones & applying patches
+
+set -e
+apt-get update && apt-get install -y python3-venv protobuf-compiler
+
+pip install -r requirements.txt
+pip install lark-parser scikit-learn
+
+cp -r ../../../../common .
diff --git a/models_v2/pytorch/dlrm/inference/gpu/tricks/md_embedding_bag.py b/models_v2/pytorch/dlrm/inference/gpu/tricks/md_embedding_bag.py
new file mode 100644
index 000000000..796b852fb
--- /dev/null
+++ b/models_v2/pytorch/dlrm/inference/gpu/tricks/md_embedding_bag.py
@@ -0,0 +1,85 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+
+# Mixed-Dimensions Trick
+#
+# Description: Applies mixed dimension trick to embeddings to reduce
+# embedding sizes.
+#
+# References:
+# [1] Antonio Ginart, Maxim Naumov, Dheevatsa Mudigere, Jiyan Yang, James Zou,
+# "Mixed Dimension Embeddings with Application to Memory-Efficient Recommendation
+# Systems", CoRR, arXiv:1909.11810, 2019
+from __future__ import absolute_import, division, print_function, unicode_literals
+
+import torch
+import torch.nn as nn
+
+
+def md_solver(n, alpha, d0=None, B=None, round_dim=True, k=None):
+    """
+    An external facing function call for mixed-dimension assignment
+    with the alpha power temperature heuristic
+    Inputs:
+    n -- (torch.LongTensor) ; Vector of num of rows for each embedding matrix
+    alpha -- (torch.FloatTensor); Scalar, non-negative, controls dim. skew
+    d0 -- (torch.FloatTensor); Scalar, baseline embedding dimension
+    B -- (torch.FloatTensor); Scalar, parameter budget for embedding layer
+    round_dim -- (bool); flag for rounding dims to nearest pow of 2
+    k -- (torch.LongTensor) ; Vector of average number of queries per inference
+    """
+    n, indices = torch.sort(n)
+    k = k[indices] if k is not None else torch.ones(len(n))
+    d = alpha_power_rule(n.type(torch.float) / k, alpha, d0=d0, B=B)
+    if round_dim:
+        d = pow_2_round(d)
+    undo_sort = [0] * len(indices)
+    for i, v in enumerate(indices):
+        undo_sort[v] = i
+    return d[undo_sort]
+
+
+def alpha_power_rule(n, alpha, d0=None, B=None):
+    if d0 is not None:
+        lamb = d0 * (n[0].type(torch.float) ** alpha)
+    elif B is not None:
+        lamb = B / torch.sum(n.type(torch.float) ** (1 - alpha))
+    else:
+        raise ValueError("Must specify either d0 or B")
+    d = torch.ones(len(n)) * lamb * (n.type(torch.float) ** (-alpha))
+    for i in range(len(d)):
+        if i == 0 and d0 is not None:
+            d[i] = d0
+        else:
+            d[i] = 1 if d[i] < 1 else d[i]
+    return torch.round(d).type(torch.long)
+
+
+def pow_2_round(dims):
+    return 2 ** torch.round(torch.log2(dims.type(torch.float)))
+
+
+class PrEmbeddingBag(nn.Module):
+    def __init__(self, num_embeddings, embedding_dim, base_dim):
+        super(PrEmbeddingBag, self).__init__()
+        self.embs = nn.EmbeddingBag(
+            num_embeddings, embedding_dim, mode="sum", sparse=True
+        )
+        torch.nn.init.xavier_uniform_(self.embs.weight)
+        if embedding_dim < base_dim:
+            self.proj = nn.Linear(embedding_dim, base_dim, bias=False)
+            torch.nn.init.xavier_uniform_(self.proj.weight)
+        elif embedding_dim == base_dim:
+            self.proj = nn.Identity()
+        else:
+            raise ValueError(
+                "Embedding dim " + str(embedding_dim) + " > base dim " + str(base_dim)
+            )
+
+    def forward(self, input, offsets=None, per_sample_weights=None):
+        return self.proj(
+            self.embs(input, offsets=offsets, per_sample_weights=per_sample_weights)
+        )
diff --git a/models_v2/pytorch/dlrm/inference/gpu/tricks/qr_embedding_bag.py b/models_v2/pytorch/dlrm/inference/gpu/tricks/qr_embedding_bag.py
new file mode 100644
index 000000000..ff7239bcd
--- /dev/null
+++ b/models_v2/pytorch/dlrm/inference/gpu/tricks/qr_embedding_bag.py
@@ -0,0 +1,230 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+# Quotient-Remainder Trick
+#
+# Description: Applies quotient remainder-trick to embeddings to reduce
+# embedding sizes.
+#
+# References:
+# [1] Hao-Jun Michael Shi, Dheevatsa Mudigere, Maxim Naumov, Jiyan Yang,
+# "Compositional Embeddings Using Complementary Partitions for Memory-Efficient
+# Recommendation Systems", CoRR, arXiv:1909.02107, 2019
+
+
+from __future__ import absolute_import, division, print_function, unicode_literals
+
+import numpy as np
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from torch.nn.parameter import Parameter
+
+
+class QREmbeddingBag(nn.Module):
+    r"""Computes sums or means over two 'bags' of embeddings, one using the quotient
+    of the indices and the other using the remainder of the indices, without
+    instantiating the intermediate embeddings, then performs an operation to combine these.
+
+    For bags of constant length and no :attr:`per_sample_weights`, this class
+
+        * with ``mode="sum"`` is equivalent to :class:`~torch.nn.Embedding` followed by ``torch.sum(dim=0)``,
+        * with ``mode="mean"`` is equivalent to :class:`~torch.nn.Embedding` followed by ``torch.mean(dim=0)``,
+        * with ``mode="max"`` is equivalent to :class:`~torch.nn.Embedding` followed by ``torch.max(dim=0)``.
+
+    However, :class:`~torch.nn.EmbeddingBag` is much more time and memory efficient than using a chain of these
+    operations.
+
+    QREmbeddingBag also supports per-sample weights as an argument to the forward
+    pass. This scales the output of the Embedding before performing a weighted
+    reduction as specified by ``mode``. If :attr:`per_sample_weights`` is passed, the
+    only supported ``mode`` is ``"sum"``, which computes a weighted sum according to
+    :attr:`per_sample_weights`.
+
+    Known Issues:
+    Autograd breaks with multiple GPUs. It breaks only with multiple embeddings.
+
+    Args:
+        num_categories (int): total number of unique categories. The input indices must be in
+                              0, 1, ..., num_categories - 1.
+        embedding_dim (list): list of sizes for each embedding vector in each table. If ``"add"``
+                              or ``"mult"`` operation are used, these embedding dimensions must be
+                              the same. If a single embedding_dim is used, then it will use this
+                              embedding_dim for both embedding tables.
+        num_collisions (int): number of collisions to enforce.
+        operation (string, optional): ``"concat"``, ``"add"``, or ``"mult". Specifies the operation
+                                      to compose embeddings. ``"concat"`` concatenates the embeddings,
+                                      ``"add"`` sums the embeddings, and ``"mult"`` multiplies
+                                      (component-wise) the embeddings.
+                                      Default: ``"mult"``
+        max_norm (float, optional): If given, each embedding vector with norm larger than :attr:`max_norm`
+                                    is renormalized to have norm :attr:`max_norm`.
+        norm_type (float, optional): The p of the p-norm to compute for the :attr:`max_norm` option. Default ``2``.
+        scale_grad_by_freq (boolean, optional): if given, this will scale gradients by the inverse of frequency of
+                                                the words in the mini-batch. Default ``False``.
+                                                Note: this option is not supported when ``mode="max"``.
+        mode (string, optional): ``"sum"``, ``"mean"`` or ``"max"``. Specifies the way to reduce the bag.
+                                 ``"sum"`` computes the weighted sum, taking :attr:`per_sample_weights`
+                                 into consideration. ``"mean"`` computes the average of the values
+                                 in the bag, ``"max"`` computes the max value over each bag.
+                                 Default: ``"mean"``
+        sparse (bool, optional): if ``True``, gradient w.r.t. :attr:`weight` matrix will be a sparse tensor. See
+                                 Notes for more details regarding sparse gradients. Note: this option is not
+                                 supported when ``mode="max"``.
+
+    Attributes:
+        weight (Tensor): the learnable weights of each embedding table is the module of shape
+                         `(num_embeddings, embedding_dim)` initialized using a uniform distribution
+                         with sqrt(1 / num_categories).
+
+    Inputs: :attr:`input` (LongTensor), :attr:`offsets` (LongTensor, optional), and
+        :attr:`per_index_weights` (Tensor, optional)
+
+        - If :attr:`input` is 2D of shape `(B, N)`,
+
+          it will be treated as ``B`` bags (sequences) each of fixed length ``N``, and
+          this will return ``B`` values aggregated in a way depending on the :attr:`mode`.
+          :attr:`offsets` is ignored and required to be ``None`` in this case.
+
+        - If :attr:`input` is 1D of shape `(N)`,
+
+          it will be treated as a concatenation of multiple bags (sequences).
+          :attr:`offsets` is required to be a 1D tensor containing the
+          starting index positions of each bag in :attr:`input`. Therefore,
+          for :attr:`offsets` of shape `(B)`, :attr:`input` will be viewed as
+          having ``B`` bags. Empty bags (i.e., having 0-length) will have
+          returned vectors filled by zeros.
+
+        per_sample_weights (Tensor, optional): a tensor of float / double weights, or None
+            to indicate all weights should be taken to be ``1``. If specified, :attr:`per_sample_weights`
+            must have exactly the same shape as input and is treated as having the same
+            :attr:`offsets`, if those are not ``None``. Only supported for ``mode='sum'``.
+
+
+    Output shape: `(B, embedding_dim)`
+
+    """
+    __constants__ = [
+        "num_categories",
+        "embedding_dim",
+        "num_collisions",
+        "operation",
+        "max_norm",
+        "norm_type",
+        "scale_grad_by_freq",
+        "mode",
+        "sparse",
+    ]
+
+    def __init__(
+        self,
+        num_categories,
+        embedding_dim,
+        num_collisions,
+        operation="mult",
+        max_norm=None,
+        norm_type=2.0,
+        scale_grad_by_freq=False,
+        mode="mean",
+        sparse=False,
+        _weight=None,
+    ):
+        super(QREmbeddingBag, self).__init__()
+
+        assert operation in ["concat", "mult", "add"], "Not valid operation!"
+
+        self.num_categories = num_categories
+        if isinstance(embedding_dim, int) or len(embedding_dim) == 1:
+            self.embedding_dim = [embedding_dim, embedding_dim]
+        else:
+            self.embedding_dim = embedding_dim
+        self.num_collisions = num_collisions
+        self.operation = operation
+        self.max_norm = max_norm
+        self.norm_type = norm_type
+        self.scale_grad_by_freq = scale_grad_by_freq
+
+        if self.operation == "add" or self.operation == "mult":
+            assert (
+                self.embedding_dim[0] == self.embedding_dim[1]
+            ), "Embedding dimensions do not match!"
+
+        self.num_embeddings = [
+            int(np.ceil(num_categories / num_collisions)),
+            num_collisions,
+        ]
+
+        if _weight is None:
+            self.weight_q = Parameter(
+                torch.Tensor(self.num_embeddings[0], self.embedding_dim[0])
+            )
+            self.weight_r = Parameter(
+                torch.Tensor(self.num_embeddings[1], self.embedding_dim[1])
+            )
+            self.reset_parameters()
+        else:
+            assert list(_weight[0].shape) == [
+                self.num_embeddings[0],
+                self.embedding_dim[0],
+            ], "Shape of weight for quotient table does not match num_embeddings and embedding_dim"
+            assert list(_weight[1].shape) == [
+                self.num_embeddings[1],
+                self.embedding_dim[1],
+            ], "Shape of weight for remainder table does not match num_embeddings and embedding_dim"
+            self.weight_q = Parameter(_weight[0])
+            self.weight_r = Parameter(_weight[1])
+        self.mode = mode
+        self.sparse = sparse
+
+    def reset_parameters(self):
+        nn.init.uniform_(self.weight_q, np.sqrt(1 / self.num_categories))
+        nn.init.uniform_(self.weight_r, np.sqrt(1 / self.num_categories))
+
+    def forward(self, input, offsets=None, per_sample_weights=None):
+        input_q = (input / self.num_collisions).long()
+        input_r = torch.remainder(input, self.num_collisions).long()
+
+        embed_q = F.embedding_bag(
+            input_q,
+            self.weight_q,
+            offsets,
+            self.max_norm,
+            self.norm_type,
+            self.scale_grad_by_freq,
+            self.mode,
+            self.sparse,
+            per_sample_weights,
+        )
+        embed_r = F.embedding_bag(
+            input_r,
+            self.weight_r,
+            offsets,
+            self.max_norm,
+            self.norm_type,
+            self.scale_grad_by_freq,
+            self.mode,
+            self.sparse,
+            per_sample_weights,
+        )
+
+        if self.operation == "concat":
+            embed = torch.cat((embed_q, embed_r), dim=1)
+        elif self.operation == "add":
+            embed = embed_q + embed_r
+        elif self.operation == "mult":
+            embed = embed_q * embed_r
+
+        return embed
+
+    def extra_repr(self):
+        s = "{num_embeddings}, {embedding_dim}"
+        if self.max_norm is not None:
+            s += ", max_norm={max_norm}"
+        if self.norm_type != 2:
+            s += ", norm_type={norm_type}"
+        if self.scale_grad_by_freq is not False:
+            s += ", scale_grad_by_freq={scale_grad_by_freq}"
+        s += ", mode={mode}"
+        return s.format(**self.__dict__)
diff --git a/models_v2/pytorch/resnet50v1_5/inference/gpu/CONTAINER_MAX.md b/models_v2/pytorch/resnet50v1_5/inference/gpu/CONTAINER_MAX.md
new file mode 100644
index 000000000..e4b8c707c
--- /dev/null
+++ b/models_v2/pytorch/resnet50v1_5/inference/gpu/CONTAINER_MAX.md
@@ -0,0 +1,99 @@
+# Running ResNet50_v1.5 Inference on Intel® Data Center GPU Max Series using Intel® Extension for PyTorch*
+
+## Description 
+
+This document has instructions for running ResNet50 V1.5 Inference using Intel® Extension for PyTorch on Intel®Max Series GPU. 
+
+## Requirements
+| Item | Detail |
+| ------ | ------- |
+| Host machine  | [Intel® Data Center GPU Max Series](https://ark.intel.com/content/www/us/en/ark/products/series/232874/intel-data-center-gpu-max-series.html)  |
+| Drivers | GPU-compatible drivers need to be installed: Download [Driver](https://dgpu-docs.intel.com/driver/installation.html) |
+| Software | Docker* |
+
+## Quick Start Scripts
+
+| Script name | Description |
+|-------------|-------------|
+| `run_model.sh` | Runs ResNet50 V1.5 INT8,FP16,BF16,FP32 and TF32 inference on single tile and two tiles|
+
+## Datasets
+
+The [ImageNet](http://www.image-net.org/) validation dataset is used.
+
+Download and extract the ImageNet2012 dataset from http://www.image-net.org/, then move validation images to labeled subfolders, using [the valprep.sh shell script](https://raw.githubusercontent.com/soumith/imagenetloader.torch/master/valprep.sh)
+
+After running the data prep script, your folder structure should look something like this:
+
+```
+imagenet
+└── val
+    ├── ILSVRC2012_img_val.tar
+    ├── n01440764
+    │   ├── ILSVRC2012_val_00000293.JPEG
+    │   ├── ILSVRC2012_val_00002138.JPEG
+    │   ├── ILSVRC2012_val_00003014.JPEG
+    │   ├── ILSVRC2012_val_00006697.JPEG
+    │   └── ...
+    └── ...
+```
+The folder that contains the `val` directory should be set as the
+`DATASET_DIR`
+(for example: `export DATASET_DIR=/home/<user>/imagenet`).
+
+> [!NOTE]
+> The `run_model.sh` can also run inference on dummy data. In this case, `DATASET_DIR` does not have to be set or volume mounted to the container. 
+
+### Docker pull command:
+
+```
+docker pull intel/image-recognition:pytorch-max-gpu-resnet50v1-5-inference
+```
+The ResNet50 v1.5 inference container includes scripts, model and libraries needed to run INT8,FP16,BF16,FP32 and TF32 inference. To run the `run_model.sh` quickstart script using this container, you'll need to set the environment variable and provide volume mounts for the ImageNet dataset if real dataset is required. Otherwise, the script uses dummy data. You will need to provide an output directory where log files will be written. 
+
+```bash
+#Optional
+export PRECISION=<provide either INT8,BF16,FP16,FP32 or TF32, otherwise (default: INT8)>
+export BATCH_SIZE=<provide batch size, otherwise (default: 1024)>
+export NUM_ITERATIONS=<provide number of iterations,otherwise (default: 500)>
+export DATASET_DIR=<path to ImageNet dataset>
+
+#Required
+export OUTPUT_DIR=<path to output logs directory>
+export PLATFORM=Max
+export MULTI_TILE=<provide True of False to enable/disable multi-tile inference>
+
+IMAGE_NAME=intel/image-recognition:pytorch-max-gpu-resnet50v1-5-inference
+DOCKER_ARGS="--rm -it"
+SCRIPT=run_model.sh
+
+docker run \
+  --privileged \
+  --device=/dev/dri \
+  --ipc=host \
+  --env BATCH_SIZE=${BATCH_SIZE} \
+  --env NUM_ITERATIONS=${NUM_ITERATIONS} \
+  --env DATASET_DIR=${DATASET_DIR} \
+  --env PRECISION=${PRECISION} \
+  --env PLATFORM=${PLATFORM} \
+  --env OUTPUT_DIR=${OUTPUT_DIR} \
+  --env MULTI_TILE=${MULTI_TILE} \
+  --env http_proxy=${http_proxy} \
+  --env https_proxy=${https_proxy} \
+  --env no_proxy=${no_proxy} \
+  --volume ${DATASET_DIR}:${DATASET_DIR} \
+  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
+  ${DOCKER_ARGS} \
+  $IMAGE_NAME \
+  /bin/bash $SCRIPT
+```
+## Documentation and Sources
+
+[GitHub* Repository](https://github.com/IntelAI/models/tree/master/docker/max-gpu)
+
+## Support
+Support for Intel® Extension for PyTorch* is found via the [Intel® AI Analytics Toolkit.](https://www.intel.com/content/www/us/en/developer/tools/oneapi/ai-analytics-toolkit.html#gs.qbretz) Additionally, the Intel® Extension for PyTorch* team tracks both bugs and enhancement requests using [GitHub issues](https://github.com/intel/intel-extension-for-pytorch/issues). Before submitting a suggestion or bug report, please search the GitHub issues to see if your issue has already been reported.
+
+## License Agreement
+
+LEGAL NOTICE: By accessing, downloading or using this software and any required dependent software (the “Software Package”), you agree to the terms and conditions of the software license agreements for the Software Package, which may also include notices, disclaimers, or license terms for third party software included with the Software Package. Please refer to the [license file](https://github.com/IntelAI/models/tree/master/third_party) for additional details.
diff --git a/models_v2/pytorch/resnet50v1_5/inference/gpu/README.md b/models_v2/pytorch/resnet50v1_5/inference/gpu/README.md
new file mode 100644
index 000000000..c3f919a79
--- /dev/null
+++ b/models_v2/pytorch/resnet50v1_5/inference/gpu/README.md
@@ -0,0 +1,96 @@
+# ResNet50v1.5 Inference
+
+ResNet50v1.5 Inference best known configurations with Intel® Extension for PyTorch.
+
+## Model Information
+
+| **Use Case** | **Framework** | **Model Repo** | **Branch/Commit/Tag** | **Optional Patch** |
+|:---:| :---: |:--------------:|:---------------------:|:------------------:|
+|  Inference   |    Pytorch    |       -        |           -           |         -          |
+
+# Pre-Requisite
+* Host has one of the following GPUs:
+  * **Arc Series** - [Intel® Arc™ A-Series Graphics](https://ark.intel.com/content/www/us/en/ark/products/series/227957/intel-arc-a-series-graphics.html)
+  * **Flex Series** - [Intel® Data Center GPU Flex Series](https://ark.intel.com/content/www/us/en/ark/products/series/230021/intel-data-center-gpu-flex-series.html)
+  * **Max Series** - [Intel® Data Center GPU Max Series](https://ark.intel.com/content/www/us/en/ark/products/series/232874/intel-data-center-gpu-max-series.html)
+* Host has installed latest Intel® Data Center GPU Max & Flex Series Drivers https://dgpu-docs.intel.com/driver/installation.html
+* The following Intel® oneAPI Base Toolkit components are required:
+  - Intel® oneAPI DPC++ Compiler (Placeholder DPCPPROOT as its installation path)
+  - Intel® oneAPI Math Kernel Library (oneMKL) (Placeholder MKLROOT as its installation path)
+  - Intel® oneAPI MPI Library
+  - Intel® oneAPI TBB Library
+
+  Follow instructions at [Intel® oneAPI Base Toolkit Download page](https://www.intel.com/content/www/us/en/developer/tools/oneapi/base-toolkit-download.html?operatingsystem=linux) to setup the package manager repository.
+
+# Prepare Dataset
+## Dataset: imagenet
+Default is dummy dataset.
+ImageNet is recommended, the download link is https://image-net.org/challenges/LSVRC/2012/2012-downloads.php.
+
+## Inference
+1. `git clone https://github.com/IntelAI/models.git`
+2. `cd models/models_v2/pytorch/resnet50v1_5/inference/gpu`
+3. Create virtual environment `venv` and activate it:
+    ```
+    python3 -m venv venv
+    . ./venv/bin/activate
+    ```
+4. Run setup.sh
+    ```
+    ./setup.sh
+    ```
+5. Install the latest GPU versions of [torch, torchvision and intel_extension_for_pytorch](https://intel.github.io/intel-extension-for-pytorch/index.html#installation):
+    ```
+    python -m pip install torch==<torch_version> torchvision==<torchvvision_version> intel-extension-for-pytorch==<ipex_version> --extra-index-url https://pytorch-extension.intel.com/release-whl-aitools/
+    ```
+6. Set environment variables for Intel® oneAPI Base Toolkit: 
+    Default installation location `{ONEAPI_ROOT}` is `/opt/intel/oneapi` for root account, `${HOME}/intel/oneapi` for other accounts
+    ```bash
+    source {ONEAPI_ROOT}/compiler/latest/env/vars.sh
+    source {ONEAPI_ROOT}/mkl/latest/env/vars.sh
+    source {ONEAPI_ROOT}/tbb/latest/env/vars.sh
+    source {ONEAPI_ROOT}/mpi/latest/env/vars.sh
+    source {ONEAPI_ROOT}/ccl/latest/env/vars.sh
+7. Setup required environment paramaters
+
+| **Parameter**                |                                  **export command**                                  |
+|:---------------------------:|:------------------------------------------------------------------------------------:|
+| **MULTI_TILE**               | `export MULTI_TILE=True` (True or False)                                             |
+| **PLATFORM**                 | `export PLATFORM=PVC` (Max or Flex or ARC)                                                 |
+| **BATCH_SIZE** (optional)    |                               `export BATCH_SIZE=1024`                                |
+| **PRECISION** (optional)     |`export PRECISION=INT8` (INT8,FP32, FP16 for all platform, BF16 and TF32 only for Max)  |
+| **OUTPUT_DIR** (optional)    |                               `export OUTPUT_DIR=$PWD`                               |
+|**NUM_ITERATIONS** (optional) |                               `export NUM_ITERATIONS=500`                             |
+| **DATASET_DIR** (optional)   |                               `export DATASET_DIR=--dummy`                           |
+8. Run `run_model.sh`
+
+## Output
+
+Single-tile output will typically look like:
+
+```
+Test: [500/500] Time  0.039 ( 0.042)    Loss 8.4575e+00 (8.4625e+00)    Acc@1   0.20 (  0.10)   Acc@5   0.59 (  0.50)
+Quantization Evalution performance: batch size:1024, throughput:26373.51 image/sec, Acc@1:0.10, Acc@5:0.50
+```
+
+Multi-tile output will typically looks like:
+```
+Test: [500/500] Time  0.040 ( 0.044)    Loss 8.4575e+00 (8.4625e+00)    Acc@1   0.20 (  0.10)   Acc@5   0.59 (  0.50)
+Quantization Evalution performance: batch size:1024, throughput:25780.13 image/sec, Acc@1:0.10, Acc@5:0.50
+Test: [500/500] Time  0.039 ( 0.044)    Loss 8.4575e+00 (8.4625e+00)    Acc@1   0.20 (  0.10)   Acc@5   0.59 (  0.50)
+Quantization Evalution performance: batch size:1024, throughput:26216.49 image/sec, Acc@1:0.10, Acc@5:0.50
+```
+
+Final results of the inference run can be found in `results.yaml` file.
+```
+results:
+ - key: throughput
+   value: 26373.51
+   unit: fps
+ - key: latency
+   value: 0.0388268379900893
+   unit: s
+ - key: accuracy
+   value: 0.100
+   unit: top1
+```
diff --git a/models_v2/pytorch/resnet50v1_5/inference/gpu/main.py b/models_v2/pytorch/resnet50v1_5/inference/gpu/main.py
new file mode 100644
index 000000000..f9673d561
--- /dev/null
+++ b/models_v2/pytorch/resnet50v1_5/inference/gpu/main.py
@@ -0,0 +1,1366 @@
+#
+# ****************************************************************************
+# Copyright 2019-2022 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ****************************************************************************
+
+# ****************************************************************************
+# BSD 3-Clause License
+# # This model is originally from the PyTorch Examples repo
+# (https://github.com/pytorch/examples/blob/master/imagenet/main.py)
+# ****************************************************************************
+
+# ****************************************************************************
+# Copyright (c) 2017,
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are met:
+#
+# * Redistributions of source code must retain the above copyright notice, this
+#   list of conditions and the following disclaimer.
+#
+# * Redistributions in binary form must reproduce the above copyright notice,
+#   this list of conditions and the following disclaimer in the documentation
+#   and/or other materials provided with the distribution.
+#
+# * Neither the name of the copyright holder nor the names of its
+#   contributors may be used to endorse or promote products derived from
+#   this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+# ****************************************************************************
+
+import argparse
+import os
+import sys
+import random
+import shutil
+import time
+import warnings
+from enum import Enum
+
+import torch
+import torch.nn as nn
+import torch.nn.parallel
+import torch.backends.cudnn as cudnn
+import torch.distributed as dist
+import torch.optim
+from torch.optim.lr_scheduler import StepLR
+import torch.multiprocessing as mp
+import torch.utils.data
+import torch.utils.data.distributed
+import torchvision.transforms as transforms
+import torchvision.datasets as datasets
+import torchvision.models as models
+from torch.utils.data import Subset
+import math
+
+torch._C._jit_set_profiling_mode(False)
+torch._C._jit_set_profiling_executor(False)
+
+model_names = sorted(name for name in models.__dict__
+    if name.islower() and not name.startswith("__")
+    and callable(models.__dict__[name]))
+
+# for convergence
+converged = False
+final_top1_acc = 0.0
+final_top5_acc = 0.0
+
+cwd = os.path.dirname(os.path.abspath(__file__))
+hub = os.path.expanduser("~/.cache/torch/intel")
+if not os.path.exists(hub):
+    os.makedirs(hub, exist_ok=True)
+
+parser = argparse.ArgumentParser(description='PyTorch ImageNet Training')
+parser.add_argument('data', metavar='DIR', nargs='?', default='imagenet',
+                    help='path to dataset (default: imagenet)')
+parser.add_argument('-a', '--arch', metavar='ARCH', default='resnet18',
+                    choices=model_names,
+                    help='model architecture: ' +
+                        ' | '.join(model_names) +
+                        ' (default: resnet18)')
+parser.add_argument('-j', '--workers', default=4, type=int, metavar='N',
+                    help='number of data loading workers (default: 4)')
+parser.add_argument('--epochs', default=90, type=int, metavar='N',
+                    help='number of total epochs to run')
+parser.add_argument('--eval-start-epoch', default=0, type=int, metavar='N',
+                    help='epoch start to run validation')
+parser.add_argument('--start-epoch', default=0, type=int, metavar='N',
+                    help='manual epoch number (useful on restarts)')
+parser.add_argument('-b', '--batch-size', default=256, type=int,
+                    metavar='N',
+                    help='mini-batch size (default: 256), this is the total '
+                         'batch size of all GPUs on the current node when '
+                         'using Data Parallel or Distributed Data Parallel')
+parser.add_argument('--lr', '--learning-rate', default=0.1, type=float,
+                    metavar='LR', help='initial learning rate', dest='lr')
+parser.add_argument('--momentum', default=0.9, type=float, metavar='M',
+                    help='momentum')
+parser.add_argument('--wd', '--weight-decay', default=1e-4, type=float,
+                    metavar='W', help='weight decay (default: 1e-4)',
+                    dest='weight_decay')
+parser.add_argument('-p', '--print-freq', default=1, type=int,
+                    metavar='N', help='print frequency (default: 10)')
+parser.add_argument('--resume', default='', type=str, metavar='PATH',
+                    help='path to latest checkpoint (default: none)')
+parser.add_argument('-e', '--evaluate', dest='evaluate', action='store_true',
+                    help='evaluate model on validation set')
+parser.add_argument('--pretrained', dest='pretrained', action='store_true',
+                    help='use pre-trained model')
+parser.add_argument('--world-size', default=-1, type=int,
+                    help='number of nodes for distributed training')
+parser.add_argument('--rank', default=-1, type=int,
+                    help='node rank for distributed training')
+parser.add_argument('--dist-url', default='127.0.0.1', type=str,
+                    help='url used to set up distributed training')
+parser.add_argument('--dist-port', default='29500', type=str,
+                    help='url port used to set up distributed training')
+parser.add_argument('--dist-backend', default='ccl', type=str,
+                    help='distributed backend, default is torch-ccl')
+parser.add_argument('--seed', default=None, type=int,
+                    help='seed for initializing training. ')
+parser.add_argument('--gpu', default=None, type=int,
+                    help='GPU id to use.')
+parser.add_argument('--xpu', default=None, type=int,
+                    help='XPU id to use.')
+parser.add_argument('--tf32', default=0, type=int, help='Datatype used: TF32')
+parser.add_argument('--bf32', default=0, type=int, help='Datatype used: BF32')
+parser.add_argument('--fp16', default=0, type=int, help='Datatype used: FP16')
+parser.add_argument('--bf16', default=0, type=int, help='Datatype used: BF16')
+parser.add_argument('--int8', default=0, type=int, help='Use int8 quantization to do inference')
+parser.add_argument('--disable-broadcast-buffers', action='store_true', help='disable syncing buffers')
+parser.add_argument('--bucket-cap', default=25, type=int, help='controls the bucket size in MegaBytes')
+parser.add_argument('--large-first-bucket', action="store_true", help='Configure a large capacity of the first bucket in DDP for allreduce')
+parser.add_argument("--use-gradient-as-bucket-view", action='store_true', help="Turn ON gradient_as_bucket_view optimization in DDP")
+parser.add_argument('--jit-cache', type=str, default=str(hub), help="path to save/load jit model")
+parser.add_argument('--jit-trace', action='store_true',
+                    help='enable PyTorch jit trace graph mode')
+parser.add_argument('--dynamo', action='store_true', help='Use torch.compile to optimize model, inductor backend default')
+parser.add_argument('--calib-iters', default=8, type=int,
+                    help='iteration number for calibration')
+parser.add_argument('--calib-bs', default=32, type=int,
+                    metavar='N', help='mini-batch size for calibration')
+parser.add_argument('--perchannel-weight', default=False,
+                    help='do calibration with weight per channel quantization')
+# TODO: when fix non blocking H2D perf regression, we will remove here option
+parser.add_argument('--non-blocking', default=False, action='store_true',
+                    help='non blocking H2D for input and target, now for int8, default False')
+parser.add_argument('--channels-last', action='store_true', help='enable channels last')
+parser.add_argument('--num-iterations', default=0, type=int)
+parser.add_argument('--converge', default=None, action='store_true',
+                    help='Run convergence and use Tensorboard to visualize the training metrics')
+parser.add_argument('--step-size', default=30, type=int, help='LR decay step size')
+parser.add_argument('--step-gamma', default=0.1, type=float, help='set the step gamma')
+parser.add_argument('--last-step-boundary', default=80, type=int, help='last epoch to decay the LR')
+parser.add_argument('--warm-up-epoch', default=0, type=int, help='warm up epochs number for convergence')
+parser.add_argument('--decay-epochs', default=33, type=int, metavar='N',
+                    help='number of decay epochs to run for lars')
+parser.add_argument('--lars', default=False, action='store_true', help='use lars for training')
+parser.add_argument('--lars-eta', default=0.0, type=float, help='set the lars epsilon')
+parser.add_argument('--skip-checkpoint', default=False, action='store_true', help='skip checkpoint saving')
+parser.add_argument('--skip-tensorboard', default=False, action='store_true', help='skip tensorboard')
+parser.add_argument('--label-smoothing', default=0.0, type=float)
+parser.add_argument('--dummy', action="store_true", help='use dummy data for '
+                    'benchmark training or val')
+parser.add_argument('--lr-scheduler', default='step', type=str,
+                    help='choose lr scheduler, default step, can choose pow')
+parser.add_argument('--power-factor', default=1.0, type=float,
+                    help='power factor for lr decay policy')
+parser.add_argument('--eval-period', default=1, type=int, help='period for doing online evaluation')
+parser.add_argument('--eval-offset', default=0, type=int, help='offset for doing online evaluation')
+parser.add_argument('--sota-target', default=75.9, type=float, help='set the lars epsilon')
+parser.add_argument('--multiprocessing-distributed', action='store_true',
+                    help='Use multi-processing distributed training to launch '
+                         'N processes per node, which has N GPUs. This is the '
+                         'fastest way to use PyTorch for either single node or '
+                         'multi node data parallel training')
+parser.add_argument('--benchmark', default=0, type=int, help='for int8 benchmark '
+                    'performance, move H2D out of E2E time')
+parser.add_argument("--save", help='Path to save entile model, save infernce mode, training is not available')
+parser.add_argument("--load", help='Path to load entile inference model')
+parser.add_argument("--asymm", action='store_true', help='Asymmetric quantization')
+parser.add_argument("--uint8", action='store_true', help="Use u8 quantization")
+parser.add_argument('--end-lr', type=float, default=1e-4,
+                    help='the end learning rate')
+
+# used for record best accrucy after validation
+best_acc1 = 0.0
+tensorboard_data = {'epoch': 0,
+                    'train': {'loss': 0.0, 'top1': 0.0, 'top5': 0.0},
+                    'eval': {'loss': 0.0, 'top1': 0.0, 'top5': 0.0}}
+global_lr = 0.0
+global_num_iter = 0
+
+def main():
+    args = parser.parse_args()
+    if args.converge:
+        print('[info] ------------------ converge arguments ------------------')
+        print('running model:  ', args.arch)
+        print('workers:        ', args.workers)
+        print('running bf16:   ', args.bf16)
+        print('total epochs:   ', args.epochs)
+        print('warm up epoch:  ', args.warm_up_epoch)
+        print('eval epoch:     ', args.eval_start_epoch)
+        print('batch size:     ', args.batch_size)
+        print('initial lr:     ', args.lr)
+        print('lr scheduler:   ', args.lr_scheduler)
+        if args.lr_scheduler == 'step':
+            print('lr step size:   ', args.step_size)
+            print('lr step gamma:  ', args.step_gamma)
+            print('lr step boundary:', args.last_step_boundary)
+        elif args.lr_scheduler == 'pow':
+            print('lr pow factor:  ', args.power_factor)
+        else:
+            pass
+        if args.lars:
+            print('using lars:     ', 'True')
+            print('choose lars eta:', args.lars_eta)
+            print("decay epochs:", args.decay_epochs)
+        else:
+            print('using sgd:      ', 'True')
+        print('label smoothing:', args.label_smoothing)
+        print('momentum:       ', args.momentum)
+        print('weight decay:   ', args.weight_decay)
+        print('seed:           ', args.seed)
+        print('eval period:    ', args.eval_period)
+        print('eval offset:    ', args.eval_offset)
+        print('sota target:    ', args.sota_target)
+        print('skip ckpt:      ', args.skip_checkpoint)
+        print('disable broadcast: ', args.disable_broadcast_buffers)
+        print('large 1st bucket: ', args.large_first_bucket)
+        print('use grad as bucket view: ', args.use_gradient_as_bucket_view)
+        print('[info] --------------------------------------------------------')
+
+    if args.xpu is not None and args.gpu is not None:
+        print('You need to choose running on NV GPU or XPU.')
+        sys.exit()
+
+    if args.gpu is not None and not torch.cuda.is_available():
+        print('Make sure cuda is enabled in torch.')
+        sys.exit()
+
+    if args.xpu is not None:
+        import intel_extension_for_pytorch as ipex
+
+    # only for training
+    if not args.evaluate:
+        if args.tf32:
+            print('doing TF32 training')
+            torch.xpu.set_fp32_math_mode(torch.xpu.FP32MathMode.TF32)
+        elif args.bf32:
+            args.bf16 = 1
+            print('doing BF32 training')
+            torch.xpu.set_fp32_math_mode(torch.xpu.FP32MathMode.BF32)
+        else:
+            torch.xpu.set_fp32_math_mode(torch.xpu.FP32MathMode.FP32)
+
+    if args.dist_backend == 'ccl':
+        try:
+            import oneccl_bindings_for_pytorch
+        except ImportError:
+            print("oneccl_bindings_for_pytorch not available!")
+
+    if args.int8 and (not args.evaluate or args.xpu is None):
+        print('For int8 quantization, it is only used in XPU inference, '
+              'you need to pass -e and --xpu [dev_id] in your command')
+        sys.exit()
+
+    if args.converge is not None:
+        from torch.utils.tensorboard import SummaryWriter
+        global writer
+        writer = SummaryWriter(log_dir='./tensorboard_log')
+        if args.num_iterations is not None:
+            warnings.warn('Tensorboard is displaying at epoch unit.')
+
+    if args.seed is not None:
+        print('Setting the seed: ', args.seed)
+        random.seed(args.seed)
+        torch.manual_seed(args.seed)
+        if args.xpu is not None:
+            torch.xpu.manual_seed(args.seed)
+        if torch.cuda.is_available():
+            cudnn.deterministic = True
+            warnings.warn('You have chosen to seed training. '
+                        'This will turn on the CUDNN deterministic setting, '
+                        'which can slow down your training considerably! '
+                        'You may see unexpected behavior when restarting '
+                        'from checkpoints.')
+
+    if args.gpu is not None:
+        warnings.warn('You have chosen a specific GPU. This will completely '
+                      'disable data parallelism.')
+
+    if args.world_size == -1:
+        mpi_world_size = int(os.environ.get('PMI_SIZE', -1))
+
+        if mpi_world_size > 0:
+            os.environ['MASTER_ADDR'] = args.dist_url #'127.0.0.1'
+            os.environ['MASTER_PORT'] = args.dist_port #'29500'
+            os.environ['RANK'] = os.environ.get('PMI_RANK', -1)
+            os.environ['WORLD_SIZE'] = os.environ.get('PMI_SIZE', -1)
+            args.rank = int(os.environ.get('PMI_RANK', -1))
+        args.world_size = int(os.environ.get("WORLD_SIZE", -1))
+    else: # mpich set
+        if 'PMIX_RANK' in os.environ.keys(): # mpich set
+            os.environ['MASTER_ADDR'] = args.dist_url #'127.0.0.1'
+            os.environ['MASTER_PORT'] = args.dist_port #'29500'
+            os.environ['RANK'] = os.environ.get('PMIX_RANK')
+            os.environ['WORLD_SIZE'] = str(args.world_size)
+            args.rank = int(os.environ.get('PMIX_RANK', -1))
+
+    args.distributed = args.world_size > 1 or args.multiprocessing_distributed
+
+    # 1 XPU card has 2 tile, and both are regarded as isolated devices/nodes
+    ngpus_per_node = 1
+    if args.multiprocessing_distributed:
+        # Since we have ngpus_per_node processes per node, the total world_size
+        # needs to be adjusted accordingly
+        args.world_size = ngpus_per_node * args.world_size
+        # Use torch.multiprocessing.spawn to launch distributed processes: the
+        # main_worker process function
+        mp.spawn(main_worker, nprocs=ngpus_per_node, args=(ngpus_per_node, args))
+    else:
+        # Simply call main_worker function
+        main_worker(ngpus_per_node, args)
+
+def jit_calib(model, val_loader_calib, args):
+    print('doing int8 jit calibration')
+    jit_model_file = os.path.join(args.jit_cache, "rn50_jit_model_int8.pt")
+    if os.path.isfile(jit_model_file):
+        print("=> load jit model from {}".format(jit_model_file))
+        modelJit = torch.load(jit_model_file)
+        print("=> load jit model ... done")
+    else:
+        from torch.jit._recursive import wrap_cpp_module
+        from torch.quantization.quantize_jit import (
+            convert_jit,
+            prepare_jit,
+        )
+        modelJit = torch.jit.script(model)
+        modelJit = wrap_cpp_module(torch._C._jit_pass_fold_convbn(modelJit._c))
+
+        qscheme = torch.per_tensor_affine if args.asymm else torch.per_tensor_symmetric
+        dtype = torch.quint8 if args.uint8 else torch.qint8
+        with torch.inference_mode():
+            if args.perchannel_weight:
+                qconfig = torch.quantization.QConfig(
+                    activation=torch.quantization.observer.MinMaxObserver.with_args(
+                        qscheme=qscheme,
+                        reduce_range=False,
+                        dtype=dtype
+                    ),
+                    weight=torch.quantization.default_per_channel_weight_observer
+                )
+            else:
+                qconfig = torch.quantization.QConfig(
+                    activation=torch.quantization.observer.MinMaxObserver.with_args(
+                        qscheme=qscheme,
+                        reduce_range=False,
+                        dtype=dtype
+                    ),
+                    weight=torch.quantization.default_weight_observer
+                )
+            modelJit = prepare_jit(modelJit, {'': qconfig}, True)
+
+            for i, (input, target) in enumerate(val_loader_calib):
+                calib = input.to(args.xpu)
+                modelJit(calib)
+
+                if i == args.calib_iters - 1:
+                    break
+            modelJit = convert_jit(modelJit, True)
+
+    return modelJit
+
+def main_worker(ngpus_per_node, args):
+    global best_acc1
+
+    if args.distributed:
+        if args.rank == -1:
+            args.rank = int(os.environ["RANK"])
+        if args.multiprocessing_distributed:
+            # For multiprocessing distributed training, rank needs to be the
+            # global rank among all the processes
+            args.rank = args.rank * ngpus_per_node + args.gpu
+        init_method = 'tcp://' + args.dist_url + ':' + args.dist_port
+        dist.init_process_group(backend=args.dist_backend, init_method=init_method,
+                                world_size=args.world_size, rank=args.rank)
+
+        if args.gpu is not None:
+            args.gpu = args.rank
+        elif args.xpu is not None:
+            if 'MPI_LOCALRANKID' in os.environ.keys():
+                local_rank = os.environ['MPI_LOCALRANKID']
+            elif 'OMPI_COMM_WORLD_LOCAL_RANK' in os.environ.keys():
+                local_rank = os.environ['OMPI_COMM_WORLD_LOCAL_RANK']
+            else: # mpich set
+                local_rank = os.environ['PALS_LOCAL_RANKID']
+            args.xpu = local_rank
+            print('world_size:{}, rank:{}, local_rank:{}'.format(args.world_size, args.rank, local_rank))
+
+    if args.gpu is not None:
+        print("Use GPU: {}".format(args.gpu))
+        args.gpu = "cuda:{}".format(args.gpu)
+    elif args.xpu is not None:
+        print("Use XPU: {}".format(args.xpu))
+        args.xpu = "xpu:{}".format(args.xpu)
+    else:
+        print("Use CPU")
+
+    # define loss function (criterion)
+    criterion = nn.CrossEntropyLoss(label_smoothing=args.label_smoothing)
+    if args.label_smoothing:
+        print('using label smoothing: ', args.label_smoothing)
+
+    # create model
+    if args.load:
+        if os.path.isfile(args.load):
+            load_path = args.load
+            if args.jit_trace:
+                model = torch.jit.load(load_path)
+            elif args.evaluate and args.int8:
+                model = torch.jit.load(load_path)
+            else:
+                model = torch.load(load_path)
+                optimizer = torch.optim.SGD(model.parameters(), lr=args.lr,
+                                momentum=args.momentum, weight_decay=args.weight_decay)
+                scheduler = StepLR(optimizer, step_size=args.step_size, gamma=0.1)
+        else:
+            print("=> no saved model found at '{}'".format(args.load))
+            sys.exit(1)
+    else:
+        if args.pretrained:
+            print("=> using pre-trained model '{}'".format(args.arch))
+            model = models.__dict__[args.arch](pretrained=True)
+        else:
+            print("=> creating model '{}'".format(args.arch))
+            model = models.__dict__[args.arch]()
+
+        # channels last
+        # TODO: this will be default memory format in future
+        if args.channels_last:
+            print('model is converted to channels last')
+            model = model.to(memory_format=torch.channels_last)
+
+        if args.distributed:
+            # For multiprocessing distributed, DistributedDataParallel constructor
+            # should always set the single device scope, otherwise,
+            # DistributedDataParallel will use all available devices.
+            if args.gpu is not None:
+                torch.cuda.set_device(args.gpu)
+                model.cuda(args.gpu)
+                # When using a single GPU per process and per
+                # DistributedDataParallel, we need to divide the batch size
+                # ourselves based on the total number of GPUs of the current node.
+                args.batch_size = int(args.batch_size / ngpus_per_node)
+                args.workers = int((args.workers + ngpus_per_node - 1) / ngpus_per_node)
+                model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[args.gpu])
+            elif args.xpu is not None:
+                torch.xpu.set_device(args.xpu)
+                model.xpu(args.xpu)
+        elif args.gpu is not None:
+            torch.cuda.set_device(args.gpu)
+            model = model.cuda(args.gpu)
+            print('model to cuda')
+        elif args.xpu is not None:
+            torch.xpu.set_device(args.xpu)
+            model = model.xpu(args.xpu)
+            print('model to xpu')
+        else:
+            # do training or inference on CPU
+            pass
+
+        # define optimizer, and learning rate scheduler
+        optimizer = torch.optim.SGD(model.parameters(), lr=args.lr,
+                                    momentum=args.momentum, weight_decay=args.weight_decay)
+
+        if args.lars:
+            eeta = args.lars_eta
+            print('using lars, eeta = ', eeta)
+            optimizer = torch.xpu.optim.Lars(model.parameters(), lr=args.lr,
+                                             momentum=args.momentum, weight_decay=args.weight_decay,
+                                             eeta=eeta)
+
+        # TODO: when change the oob of auto channels last function, here need change
+        using_block_layout = os.environ.get("IPEX_XPU_ONEDNN_LAYOUT", "OFF").upper() in ["1", "Y", "ON", "YES", "TRUE"]
+        if using_block_layout:
+            import intel_extension_for_pytorch as ipex
+            ipex.disable_auto_channels_last()
+            print('using block layout and disable the auto channels last')
+
+        # torch.xpu.optimize is only for device xpu and no jit script
+        if args.xpu is not None:
+            if args.evaluate:
+                if not args.int8:
+                    print('doing torch xpu optimize for inference')
+                    model.eval()
+                    dtype = torch.float16 if args.fp16 else torch.float32
+                    dtype = torch.bfloat16 if args.bf16 else dtype
+                    model = torch.xpu.optimize(model=model, dtype=dtype, level="O1")
+            else:
+                model.train()
+                print('doing torch xpu optimize for training')
+                model, optimizer = torch.xpu.optimize(model=model, optimizer=optimizer, level="O1",
+                                                      dtype=torch.bfloat16 if args.bf16 else torch.float32)
+
+        if args.distributed:
+            if args.xpu is not None:
+                # When using a single GPU per process and per
+                # DistributedDataParallel, we need to divide the batch size
+                # ourselves based on the total number of GPUs we have
+                args.batch_size = int(args.batch_size / ngpus_per_node)
+                args.workers = int(args.workers / ngpus_per_node)
+
+                if args.large_first_bucket:
+                    # set the first bucket with maximal size to cover all parameters for allreduce
+                    dist._DEFAULT_FIRST_BUCKET_BYTES = sys.maxsize
+                model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[args.xpu], broadcast_buffers=False if args.disable_broadcast_buffers else True, bucket_cap_mb=args.bucket_cap, gradient_as_bucket_view=args.use_gradient_as_bucket_view)
+
+        """Sets the learning rate to the initial LR decayed by 10 every configured epochs"""
+        scheduler = StepLR(optimizer=optimizer, step_size=args.step_size, gamma=args.step_gamma)
+
+        if not args.evaluate:
+            print('Using StepLR for training, step size ', args.step_size)
+
+        # optionally resume from a checkpoint
+        if args.resume:
+            if os.path.isfile(args.resume):
+                print("=> loading checkpoint '{}'".format(args.resume))
+                if args.gpu is None and args.xpu is None:
+                    # CPU load
+                    checkpoint = torch.load(args.resume)
+                elif args.gpu is not None:
+                    # Map model to be loaded to specified single gpu.
+                    print('loading checkpoint to ', str(args.gpu))
+                    checkpoint = torch.load(args.resume, map_location=args.gpu)
+                elif args.xpu is not None:
+                    # Map model to be loaded to specified single gpu.
+                    print('loading checkpoint to ', str(args.xpu))
+                    checkpoint = torch.load(args.resume, map_location=args.xpu)
+                args.start_epoch = checkpoint['epoch']
+                # keep best_acc1 on cpu for comparing acc after validation
+                best_acc1 = checkpoint['best_acc1']
+                model.load_state_dict(checkpoint['state_dict'])
+                optimizer.load_state_dict(checkpoint['optimizer'])
+                scheduler.load_state_dict(checkpoint['scheduler'])
+            else:
+                print("=> no checkpoint found at '{}'".format(args.resume))
+
+        if args.gpu is not None:
+            cudnn.benchmark = True
+
+    # Data loading code
+    traindir = os.path.join(args.data, 'train')
+    valdir = os.path.join(args.data, 'val')
+    normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],
+                                     std=[0.229, 0.224, 0.225])
+
+    if args.dummy:
+        print("Dummy data is used!")
+        train_dataset = datasets.FakeData(1281167, (3, 224, 224), 1000, transforms.ToTensor())
+        val_dataset_size = args.num_iterations * args.batch_size if (args.dummy and args.num_iterations) else 50000
+        val_dataset = datasets.FakeData(val_dataset_size, (3, 224, 224), 1000, transforms.ToTensor())
+    else:
+        traindir = os.path.join(args.data, 'train')
+        valdir = os.path.join(args.data, 'val')
+        normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],
+                                     std=[0.229, 0.224, 0.225])
+        train_dataset = datasets.ImageFolder(
+            traindir,
+            transforms.Compose([
+                transforms.RandomResizedCrop(224),
+                transforms.RandomHorizontalFlip(),
+                transforms.ToTensor(),
+                normalize,
+            ]))
+        val_dataset = datasets.ImageFolder(
+            valdir,
+            transforms.Compose([
+                transforms.Resize(256),
+                transforms.CenterCrop(224),
+                transforms.ToTensor(),
+                normalize,
+            ]))
+
+    if args.distributed:
+        train_sampler = torch.utils.data.distributed.DistributedSampler(train_dataset)
+        val_sampler = torch.utils.data.distributed.DistributedSampler(val_dataset, shuffle=False, drop_last=True)
+    else:
+        train_sampler = None
+        val_sampler = None
+
+    # [watch out] The pin memory is default enabled on CUDA for now in torch.
+    train_loader = torch.utils.data.DataLoader(
+        train_dataset, batch_size=args.batch_size, shuffle=(train_sampler is None),
+        num_workers=args.workers, pin_memory=True, pin_memory_device="xpu", sampler=train_sampler, drop_last=True)
+
+    val_loader = torch.utils.data.DataLoader(
+        val_dataset, batch_size=args.batch_size, shuffle=False,
+        num_workers=args.workers, pin_memory=True, pin_memory_device="xpu", sampler=val_sampler)
+
+    # Profiling
+    profiling = os.environ.get("PROFILE", "OFF").upper() in ["1", "Y", "ON", "YES", "TRUE"]
+
+    use_autocast = False
+    if args.bf16 or args.fp16:
+        print('using autocast')
+        use_autocast = True
+
+    if args.evaluate:
+        if args.int8:
+            # calibration dataloader
+            val_loader_calib = torch.utils.data.DataLoader(
+                val_dataset, batch_size=args.calib_bs, shuffle=False,
+                num_workers=args.workers, pin_memory=True, pin_memory_device="xpu")
+
+            # do calibration and return quant model
+            if args.load:
+                model_calib = model
+            else:
+                model_calib = jit_calib(model, val_loader_calib, args)
+            if args.save:
+                torch.jit.save(model_calib, args.save)
+            val_loader_inf = torch.utils.data.DataLoader(
+                val_dataset, batch_size=args.batch_size, shuffle=False,
+                num_workers=args.workers, pin_memory=True, pin_memory_device="xpu")
+
+            print('doing int8 inference')
+            validate_quantization(val_loader_inf, model_calib, criterion, profiling, args)
+        else:
+            # epoch pass 0
+            validate(val_loader, model, criterion, 0, profiling, use_autocast, args)
+        return
+
+    global_start_time = time.time()
+
+    # warm up for convergence
+    if args.converge and not args.resume and args.warm_up_epoch > 0:
+        warm_up_epoch = args.warm_up_epoch
+        warm_up_portion = args.lr / float(warm_up_epoch)
+        for epoch in range(0, warm_up_epoch):
+            if args.lars == False:
+                optimizer.param_groups[0]['lr'] = (epoch + 1) * warm_up_portion
+            train(train_loader, model, criterion, optimizer, epoch, profiling, use_autocast, args, mode='warming')
+            print('Warmup [', (epoch + 1), '][', warm_up_epoch, '] lr = ', optimizer.param_groups[0]['lr'])
+
+    last_acc = best_acc1
+    for epoch in range(args.start_epoch, args.epochs):
+        epoch_start_time = time.time()
+
+        global global_lr
+        global_lr = optimizer.param_groups[0]['lr']
+
+        if not args.distributed or (args.distributed and args.rank == 0):
+            print('[info] Epoch[', epoch, '] start time = ', time.asctime(time.localtime(epoch_start_time)))
+
+        if args.distributed:
+            train_sampler.set_epoch(epoch)
+
+        if epoch == args.last_step_boundary and args.lr_scheduler == 'step':
+            optimizer.param_groups[0]['lr'] *= args.step_gamma
+
+        if not args.distributed or (args.distributed and args.rank == 0):
+            print('[info] Epoch[', epoch, '] lr = ', optimizer.param_groups[0]['lr'])
+
+        # train for one epoch
+        train(train_loader, model, criterion, optimizer, epoch, profiling, use_autocast, args, mode='training')
+
+        # evaluate on validation set
+        acc1 = last_acc
+        if (epoch >= args.eval_start_epoch) and (epoch % args.eval_period == args.eval_offset):
+            print('epoch: ', epoch, ' is doing evaluation')
+            acc1 = validate(val_loader, model, criterion, epoch, profiling, use_autocast, args)
+            last_acc = acc1
+
+        # update the LR
+        if args.lars == False:
+            scheduler.step()
+
+        # remember best acc@1 and save checkpoint
+        is_best = acc1 > best_acc1
+        best_acc1 = max(acc1, best_acc1)
+
+        if not args.skip_checkpoint and \
+            (not args.multiprocessing_distributed or (args.multiprocessing_distributed and args.rank % ngpus_per_node == 0)):
+            save_checkpoint(state={
+                'epoch': epoch + 1,
+                'arch': args.arch,
+                'state_dict': model.state_dict(),
+                'best_acc1': best_acc1,
+                'optimizer' : optimizer.state_dict(),
+                'scheduler' : scheduler.state_dict()
+            }, is_best=is_best, args=args)
+
+        # show info
+        if not args.distributed or (args.distributed and args.rank == 0):
+            epoch_end_time = time.time()
+            print('[info] Epoch[', epoch, '] end time = ', time.asctime(time.localtime(epoch_end_time)))
+            print('[info] Epoch[', epoch, '] consume time = ', ((epoch_end_time - epoch_start_time) / 3600.0), ' hours')
+
+        if converged:
+            break
+
+    if not args.distributed or (args.distributed and args.rank == 0):
+        global_end_time = time.time()
+        print('[info] Global start time = ', time.asctime(time.localtime(global_start_time)))
+        print('[info] Global end time = ', time.asctime(time.localtime(global_end_time)))
+        print('[info] Global consume time = ', ((global_end_time - global_start_time) / (3600.0)), ' hours')
+        if converged:
+            print('[Successful] Reach convergence, final top1 acc: ', final_top1_acc)
+            print('[Successful] Reach convergence, final top5 acc: ', final_top5_acc)
+        else:
+            print('[Failed] Miss convergence')
+
+    if args.converge and not args.skip_tensorboard:
+        if not args.distributed or (args.distributed and args.rank == 0):
+            writer.close()
+
+def train(train_loader, model, criterion, optimizer, epoch, profiling, use_autocast, args, mode='training'):
+    batch_time = AverageMeter('Time', ':6.3f')
+    data_time = AverageMeter('Data', ':6.3f')
+    losses = AverageMeter('Loss', ':.4e')
+    top1 = AverageMeter('Acc@1', ':6.2f')
+    top5 = AverageMeter('Acc@5', ':6.2f')
+    progress = ProgressMeter(
+        len(train_loader),
+        [batch_time, data_time, losses, top1, top5],
+        prefix="Epoch: [{}]".format(epoch))
+    global global_num_iter
+
+    def one_iter(model, images):
+        with torch.xpu.amp.autocast(enabled=use_autocast, dtype=torch.bfloat16):
+            # compute output
+            output = model(images)
+            loss = criterion(output, target)
+        optimizer.zero_grad(set_to_none=True)
+        loss.backward()
+        if args.lars:
+            # Update LR for Lars
+            MLPerfLRScheduler(optimizer, global_num_iter, len(train_loader), args)
+        optimizer.step()
+        return [output, loss]
+
+    # switch to train mode
+    model.train()
+
+    if args.dynamo:
+        train_opt = torch.compile(one_iter, backend="inductor")
+    else:
+        train_opt = one_iter
+
+    # record time
+    duration_total = 0.0
+    warmup_iter = 5
+
+    # config profiler
+    import contextlib
+    def profiler_setup(profiling=False, *args, **kwargs):
+        if profiling:
+            return torch.profiler.profile(*args, **kwargs)
+        else:
+            return contextlib.nullcontext()
+    activities = [torch.profiler.ProfilerActivity.CPU]
+    if args.xpu is not None:
+        activities.append(torch.profiler.ProfilerActivity.XPU)
+    elif args.gpu is not None:
+        activities.append(torch.profiler.ProfilerActivity.CUDA)
+    # fixed wait iters = 1, active iters = 1, warmup_iters = 3, at_least 5 iters
+    # more iters will be skipped, and repeat will be fixed to 1
+    num_iters = args.num_iterations if args.num_iterations else len(train_loader)
+    skip_iters = max(num_iters - 5, 0)
+    schedule = torch.profiler.schedule(skip_first=skip_iters,
+                                       wait=1, warmup=3, active=1)
+    def trace_handle(prof):
+        profile_name = 'fp32'
+        if args.fp16:
+            profile_name = 'fp16'
+        elif args.bf16:
+            profile_name = 'bf16'
+        if args.distributed:
+            profile_name += '.xpu.' + str(args.rank)
+        if args.xpu is not None:
+            torch.save(prof.key_averages().table(sort_by="self_xpu_time_total"), './profiling.' + profile_name + '.train.pt')
+        elif args.gpu is not None:
+            torch.save(prof.key_averages().table(sort_by="self_cuda_time_total"), './profiling.card.' + str(args.xpu) + '.pt')
+        else:
+            torch.save(prof.key_averages().table(sort_by="self_cpu_time_total"), './profiling.card.' + str(args.xpu) + '.pt')
+
+
+    # start profiler, or none while profiling is false
+    with profiler_setup(profiling, activities=activities, schedule=schedule, on_trace_ready=trace_handle) as prof:
+        data_start = time.time()
+        for i, (images, target) in enumerate(train_loader):
+            global_num_iter +=1
+            # measure data loading time
+            data_time.update(time.time() - data_start)
+
+            if args.channels_last:
+                print('input to channels last')
+                images = images.to(memory_format=torch.channels_last)
+
+
+            if args.xpu is not None:
+                try:
+                    import memory_check
+                    memory_check.display_mem("xpu:0")
+                except:
+                    pass
+                start_time = time.time()
+                images = images.to(args.xpu, non_blocking=True)
+                target = target.to(args.xpu, non_blocking=True)
+
+                [output, loss] = train_opt(model, images)
+                # D2H
+                loss = loss.cpu()
+                output = output.cpu()
+                target = target.cpu()
+
+            else:
+                start_time = time.time()
+                activities = None
+                prof_sort = None
+
+                if args.gpu is not None:
+                    images = images.cuda(args.gpu, non_blocking=True)
+                    target = target.cuda(args.gpu, non_blocking=True)
+
+                # compute output
+                output = model(images)
+                loss = criterion(output, target)
+
+                # compute gradient and do SGD step
+                optimizer.zero_grad()
+                loss.backward()
+                optimizer.step()
+
+                # D2H, as sync
+                if args.gpu is not None:
+                    loss = loss.cpu()
+                    output = output.cpu()
+                    target = target.cpu()
+
+            if profiling:
+                prof.step()
+
+            # measure elapsed time
+            duration_train = time.time() - start_time
+            batch_time.update(duration_train)
+
+            # measure accuracy and record loss
+            acc1, acc5 = accuracy(output, target, topk=(1, 5))
+            losses.update(loss.item(), images.size(0))
+            top1.update(acc1[0], images.size(0))
+            top5.update(acc5[0], images.size(0))
+
+            if i % args.print_freq == 0:
+                progress.display(i + 1)
+
+            # exclude first iteration for calculating througput
+            if i >= warmup_iter:
+                duration_total += duration_train
+            data_start = time.time()
+
+            if i == (args.num_iterations - 1) and args.num_iterations >= warmup_iter:
+                print('Training performance: batch size:%d, throughput:%.2f image/sec'
+                      % (args.batch_size, (args.batch_size / (duration_total / (args.num_iterations - warmup_iter)))))
+                sys.exit(0)
+            elif args.num_iterations == 0 and i == len(train_loader) - 1:
+                print('Training performance: batch size:%d, throughput:%.2f image/sec'
+                      % (args.batch_size, (args.batch_size / (duration_total / (len(train_loader) - warmup_iter)))))
+                if args.converge is None:
+                    sys.exit(0)
+
+    if args.converge and not args.skip_tensorboard and mode == 'training':
+        global tensorboard_data
+        tensorboard_data['epoch'] = epoch
+        tensorboard_data['train']['loss'] = losses.avg
+        tensorboard_data['train']['top1'] = top1.avg
+        tensorboard_data['train']['top5'] = top5.avg
+
+def validate(val_loader, model, criterion, epoch, profiling, use_autocast, args):
+    from torch._inductor import config
+
+    def compile_model(model, val_loader):
+        print("====Before compile model====")
+        compiled_model = torch.compile(model, backend="inductor", options={"freezing": True})
+        return compiled_model
+    def run_validate(loader, model, autocast_dtype, base_progress=0):
+
+        # record time
+        duration_total = 0.0
+        warmup_iter = 5
+
+        # config profiler
+        import contextlib
+        def profiler_setup(profiling=False, *args, **kwargs):
+            if profiling:
+                return torch.profiler.profile(*args, **kwargs)
+            else:
+                return contextlib.nullcontext()
+        activities = [torch.profiler.ProfilerActivity.CPU]
+        if args.xpu is not None:
+            activities.append(torch.profiler.ProfilerActivity.XPU)
+        elif args.gpu is not None:
+            activities.append(torch.profiler.ProfilerActivity.CUDA)
+        # fixed wait iters = 1, active iters = 1, warmup_iters = 3, at_least 5 iters
+        # more iters will be skipped, and repeat will be fixed to 1
+        num_iters = args.num_iterations if args.num_iterations else len(loader)
+        skip_iters = max(num_iters - 5, 0)
+        schedule = torch.profiler.schedule(skip_first=skip_iters,
+                                           wait=1, warmup=3, active=1)
+        def trace_handle(prof):
+            profile_name = 'fp32'
+            if args.fp16:
+                profile_name = 'fp16'
+            elif args.bf16:
+                profile_name = 'bf16'
+            if args.distributed:
+                profile_name += '.xpu.' + str(args.rank)
+            if args.xpu is not None:
+                print(prof.key_averages().table(sort_by="self_xpu_time_total"))
+                torch.save(prof.key_averages().table(sort_by="self_xpu_time_total"), './profiling.' + profile_name + '.inf.pt')
+            elif args.gpu is not None:
+                print(prof.key_averages().table(sort_by="self_cuda_time_total"))
+                torch.save(prof.key_averages().table(sort_by="self_cuda_time_total"), './profiling.card.' + str(args.xpu) + '.pt')
+            else:
+                print(prof.key_averages().table(sort_by="self_cpu_time_total"))
+                torch.save(prof.key_averages().table(sort_by="self_cpu_time_total"), './profiling.card.' + str(args.xpu) + '.pt')
+
+        # start profiler, or none while profiling is false
+        with profiler_setup(profiling, activities=activities, schedule=schedule, on_trace_ready=trace_handle) as prof, torch.no_grad():
+            for i, (images, target) in enumerate(loader):
+                i = base_progress + i
+
+                if args.channels_last:
+                    images = images.to(memory_format=torch.channels_last)
+                    print('images convert to channels last')
+
+
+                if args.xpu:
+                    try:
+                        import memory_check
+                        memory_check.display_mem("xpu:0")
+                    except:
+                        pass
+                    start_time = time.time()
+                    images = images.to(args.xpu, non_blocking=True)
+
+                    if args.jit_trace:
+                        # compute output
+                        output = model(images)
+                    elif args.dynamo:
+                        with torch.xpu.amp.autocast(enabled=use_autocast, dtype=autocast_dtype):
+                            output = model(images)
+                    else:
+                        with torch.xpu.amp.autocast(enabled=use_autocast, dtype=autocast_dtype):
+                            # compute output
+                            output = model(images)
+
+                    # sync for time measurement
+                    if not args.converge:
+                        torch.xpu.synchronize(args.xpu)
+
+                else:
+                    start_time = time.time()
+                    activities = None
+                    prof_sort = None
+                    if args.gpu is not None:
+                        images = images.cuda(args.gpu, non_blocking=True)
+
+                    # compute output
+                    output = model(images)
+
+                    # sync for time measurement
+                    if args.gpu is not None:
+                        torch.cuda.synchronize(args.gpu)
+
+                if profiling:
+                    prof.step()
+
+                # D2H
+                output = output.cpu()
+
+                # measure elapsed time
+                duration_eval = time.time() - start_time
+                batch_time.update(duration_eval)
+
+                loss = criterion(output.float(), target)
+
+                # measure accuracy and record loss
+                acc1, acc5 = accuracy(output.float(), target, topk=(1, 5))
+                losses.update(loss.item(), images.size(0))
+                top1.update(acc1[0], images.size(0))
+                top5.update(acc5[0], images.size(0))
+
+                if i % args.print_freq == 0:
+                    progress.display(i + 1)
+
+                # exclude first iteration for calculating througput
+                if i >= warmup_iter and not (args.num_iterations == 0 and i == len(val_loader) - 1):
+                    duration_total += duration_eval
+
+                if (not args.num_iterations == 0) and (args.num_iterations <= warmup_iter):
+                    print("At least ", warmup_iter, " iteartions required for performance measure")
+                    sys.exit(0)
+
+                if i == (args.num_iterations - 1) and args.num_iterations >= warmup_iter:
+                    print('Evalution performance: batch size:%d, throughput:%.2f image/sec, Acc@1:%.2f, Acc@5:%.2f'
+                        % (args.batch_size, (args.batch_size / (duration_total / (args.num_iterations - warmup_iter))), top1.avg, top5.avg))
+                    sys.exit(0)
+                elif args.num_iterations == 0 and i == len(val_loader) - 1:
+                    if args.converge and args.distributed:
+                        top1.all_reduce()
+                        top5.all_reduce()
+                    print('Evalution performance: batch size:%d, throughput:%.2f image/sec, Acc@1:%.2f, Acc@5:%.2f'
+                        % (args.batch_size, (args.batch_size / (duration_total / (len(val_loader) - warmup_iter))), top1.avg, top5.avg))
+                    if args.converge:
+                        global final_top1_acc
+                        global final_top5_acc
+                        global converged
+                        final_top1_acc = top1.avg
+                        final_top5_acc = top5.avg
+                        if final_top1_acc >= args.sota_target:
+                            converged = True
+                    else:
+                        sys.exit(0)
+
+    batch_time = AverageMeter('Time', ':6.3f', Summary.NONE)
+    losses = AverageMeter('Loss', ':.4e', Summary.NONE)
+    top1 = AverageMeter('Acc@1', ':6.2f', Summary.AVERAGE)
+    top5 = AverageMeter('Acc@5', ':6.2f', Summary.AVERAGE)
+    progress = ProgressMeter(
+        len(val_loader) + (args.distributed and (len(val_loader.sampler) * args.world_size < len(val_loader.dataset))),
+        [batch_time, losses, top1, top5],
+        prefix='Test: ')
+
+    # switch to evaluate mode
+    model.eval()
+
+    autocast_dtype = torch.float32
+    if args.fp16:
+        autocast_dtype = torch.float16
+    elif args.bf16:
+        autocast_dtype = torch.bfloat16
+
+    if args.jit_trace and not args.load:
+        trace_input = torch.randn(args.batch_size, 3, 224, 224).to(args.xpu)
+        print('jit trace')
+        with torch.xpu.amp.autocast(enabled=use_autocast, dtype=autocast_dtype, cache_enabled=False):
+            model = torch.jit.trace(model, trace_input)
+    elif args.dynamo:
+        model = compile_model(model, val_loader)
+
+
+    if args.save:
+        if args.jit_trace:
+            torch.jit.save(model, args.save)
+        else:
+            torch.save(model, args.save)
+    run_validate(val_loader, model, autocast_dtype)
+
+    progress.display_summary()
+
+    if args.converge and not args.skip_tensorboard:
+        global tensorboard_data
+        tensorboard_data['eval']['loss'] = losses.avg
+        tensorboard_data['eval']['top1'] = final_top1_acc
+        tensorboard_data['eval']['top5'] = final_top5_acc
+        draw_tensorboard(args)
+
+    if args.distributed:
+        return final_top1_acc
+    else:
+        return top1.avg
+
+def validate_quantization(val_loader, model, criterion, profiling, args):
+    batch_time = AverageMeter('Time', ':6.3f', Summary.NONE)
+    losses = AverageMeter('Loss', ':.4e', Summary.NONE)
+    top1 = AverageMeter('Acc@1', ':6.2f', Summary.AVERAGE)
+    top5 = AverageMeter('Acc@5', ':6.2f', Summary.AVERAGE)
+    progress = ProgressMeter(
+        len(val_loader) + (args.distributed and (len(val_loader.sampler) * args.world_size < len(val_loader.dataset))),
+        [batch_time, losses, top1, top5],
+        prefix='Test: ')
+
+    # switch to evaluate mode
+    model.eval()
+
+    # record time
+    duration_total = 0.0
+
+    # TODO: will fix non blocking H2D perf regression, then remove here option
+    non_blocking = False
+    if args.non_blocking:
+        non_blocking = True
+
+    # config profiler
+    import contextlib
+    def profiler_setup(profiling=False, *args, **kwargs):
+        if profiling:
+            return torch.profiler.profile(*args, **kwargs)
+        else:
+            return contextlib.nullcontext()
+    activities = [torch.profiler.ProfilerActivity.CPU]
+    if args.xpu is not None:
+        activities.append(torch.profiler.ProfilerActivity.XPU)
+    elif args.gpu is not None:
+        activities.append(torch.profiler.ProfilerActivity.CUDA)
+    # fixed wait iters = 1, active iters = 1, warmup_iters = 3, at_least 5 iters
+    # more iters will be skipped, and repeat will be fixed to 1
+    num_iters = args.num_iterations if args.num_iterations else len(val_loader)
+    skip_iters = max(num_iters - 5, 0)
+    schedule = torch.profiler.schedule(skip_first=skip_iters,
+                                       wait=1, warmup=3, active=1)
+    def trace_handle(prof):
+        if args.xpu is not None:
+            torch.save(prof.key_averages().table(sort_by="self_xpu_time_total"), './profiling.int8.inf.pt')
+
+    # start profiler, or none while profiling is false
+    with profiler_setup(profiling, activities=activities, schedule=schedule, on_trace_ready=trace_handle) as prof, torch.inference_mode():
+        for i, (images, target) in enumerate(val_loader):
+            if args.xpu is not None and args.benchmark == 1:
+                images = images.to(args.xpu, non_blocking = non_blocking)
+
+            try:
+                import memory_check
+                memory_check.display_mem("xpu:0")
+            except:
+                pass
+            start = time.time()
+            if args.xpu is not None and args.benchmark == 0:
+                images = images.to(args.xpu, non_blocking = non_blocking)
+            if args.channels_last:
+                images = images.to(memory_format=torch.channels_last)
+
+            # compute output
+            output = model(images)
+
+            # D2H
+            output = output.to("cpu")
+
+            # sync for time measurement
+            torch.xpu.synchronize(args.xpu)
+
+            # measure elapsed time
+            end = time.time()
+            batch_time.update(end - start)
+            duration_eval = end - start
+
+            loss = criterion(output, target)
+
+            if profiling:
+                prof.step()
+
+            # measure accuracy and record loss
+            acc1, acc5 = accuracy(output, target, topk=(1, 5))
+            losses.update(loss.item(), images.size(0))
+            top1.update(acc1[0], images.size(0))
+            top5.update(acc5[0], images.size(0))
+
+            if i % args.print_freq == 0:
+                progress.display(i + 1)
+
+            # exclude first iteration for calculating througput
+            perf_start_iter = 1
+            if args.benchmark == 1 and args.num_iterations >= 500:
+                perf_start_iter = math.floor(args.num_iterations * 0.7)
+            if i >= perf_start_iter:
+                duration_total += duration_eval
+
+            if i == (args.num_iterations - 1) and args.num_iterations >= 2:
+                print('Quantization Evalution performance: batch size:%d, throughput:%.2f image/sec, Acc@1:%.2f, Acc@5:%.2f'
+                    % (args.batch_size, (args.batch_size / (duration_total / (args.num_iterations - perf_start_iter))), top1.avg, top5.avg))
+                sys.exit(0)
+            elif args.num_iterations == 0 and i == len(val_loader) - 1:
+                print('Quantization Evalution performance: batch size:%d, throughput:%.2f image/sec, Acc@1:%.2f, Acc@5:%.2f'
+                    % (args.batch_size, (args.batch_size / (duration_total / (len(val_loader) - 2))), top1.avg, top5.avg))
+                sys.exit(0)
+
+        progress.display_summary()
+
+    return top1.avg
+
+def save_checkpoint(state, is_best, args, filename='checkpoint.pth.tar'):
+    if args.distributed:
+        print('rank ', str(args.rank), ' saving checkpoint')
+        filename = 'checkpoint.rank.' + str(args.rank) + '.pth.tar'
+    else:
+        print('single tile saving checkpoint')
+    torch.save(state, filename)
+
+    if is_best:
+        shutil.copyfile(filename, 'model_best.pth.tar')
+
+def qnormalize(tensor, mean, std, scl):
+    dtype = tensor.dtype
+    mean = torch.as_tensor(mean, dtype=dtype, device=tensor.device)
+    std = torch.as_tensor(std, dtype=dtype, device=tensor.device)
+    if mean.ndim == 1:
+        mean = mean[:, None, None]
+    if std.ndim == 1:
+        std = std[:, None, None]
+
+    tensor.sub_(mean).div_(std)#.mul_(scale)#tensor.sub_(255 * mean).mul(128/255*(1/(1-0.406)))
+    out = torch.quantize_per_tensor(tensor, scale=scl, zero_point=0, dtype=torch.qint8)
+    return out
+
+def compute_scale(val_loader_com):
+    for i, (input, target) in enumerate(val_loader_com):
+        scale = 1.0 / (128 / torch.max(input))
+        return scale
+
+class QNormalize(object):
+    def __init__(self, mean, std, scale):
+        self.mean = mean
+        self.std = std
+        self.scale = scale
+
+    def __call__(self, tensor):
+        return qnormalize(tensor, self.mean, self.std, self.scale)
+
+class Summary(Enum):
+    NONE = 0
+    AVERAGE = 1
+    SUM = 2
+    COUNT = 3
+
+class AverageMeter(object):
+    """Computes and stores the average and current value"""
+    def __init__(self, name, fmt=':f', summary_type=Summary.AVERAGE):
+        self.name = name
+        self.fmt = fmt
+        self.summary_type = summary_type
+        self.reset()
+
+    def reset(self):
+        self.val = 0
+        self.avg = 0
+        self.sum = 0
+        self.count = 0
+
+    def update(self, val, n=1):
+        self.val = val
+        self.sum += val * n
+        self.count += n
+        self.avg = self.sum / self.count
+
+    def all_reduce(self):
+        total = torch.FloatTensor([self.sum, self.count])
+        dist.all_reduce(total, dist.ReduceOp.SUM, async_op=False)
+        self.sum, self.count = total.tolist()
+        self.avg = self.sum / self.count
+
+    def __str__(self):
+        fmtstr = '{name} {val' + self.fmt + '} ({avg' + self.fmt + '})'
+        return fmtstr.format(**self.__dict__)
+
+    def summary(self):
+        fmtstr = ''
+        if self.summary_type is Summary.NONE:
+            fmtstr = ''
+        elif self.summary_type is Summary.AVERAGE:
+            fmtstr = '{name} {avg:.3f}'
+        elif self.summary_type is Summary.SUM:
+            fmtstr = '{name} {sum:.3f}'
+        elif self.summary_type is Summary.COUNT:
+            fmtstr = '{name} {count:.3f}'
+        else:
+            raise ValueError('invalid summary type %r' % self.summary_type)
+
+        return fmtstr.format(**self.__dict__)
+
+class ProgressMeter(object):
+    def __init__(self, num_batches, meters, prefix=""):
+        self.batch_fmtstr = self._get_batch_fmtstr(num_batches)
+        self.meters = meters
+        self.prefix = prefix
+
+    def display(self, batch):
+        entries = [self.prefix + self.batch_fmtstr.format(batch)]
+        entries += [str(meter) for meter in self.meters]
+        print('\t'.join(entries))
+
+    def display_summary(self):
+        entries = [" *"]
+        entries += [meter.summary() for meter in self.meters]
+        print(' '.join(entries))
+
+    def _get_batch_fmtstr(self, num_batches):
+        num_digits = len(str(num_batches // 1))
+        fmt = '{:' + str(num_digits) + 'd}'
+        return '[' + fmt + '/' + fmt.format(num_batches) + ']'
+
+def accuracy(output, target, topk=(1,)):
+    """Computes the accuracy over the k top predictions for the specified values of k"""
+    with torch.no_grad():
+        maxk = max(topk)
+        batch_size = target.size(0)
+
+        _, pred = output.topk(maxk, 1, True, True)
+        pred = pred.t()
+        pred = pred.cpu()
+        correct = pred.eq(target.view(1, -1).expand_as(pred))
+
+        res = []
+        for k in topk:
+            correct_k = correct[:k].reshape(-1).float().sum(0, keepdim=True)
+            res.append(correct_k.mul_(100.0 / batch_size))
+        return res
+
+def MLPerfLRScheduler(optimizer, step, iteration, args):
+    global global_lr
+    warmup_iter = args.warm_up_epoch * iteration
+    decay_steps = args.decay_epochs * iteration
+    power = 2
+    if step <= warmup_iter:
+       lr_rate = args.lr * (step / warmup_iter)
+    else:
+       lr_step = min((step - warmup_iter), decay_steps)
+       lr_rate = ((args.lr - args.end_lr) * (1-(lr_step/decay_steps)) ** power) + args.end_lr
+    global_lr = lr_rate
+    optimizer.param_groups[0]['lr'] = global_lr
+
+def draw_tensorboard(args):
+    global tensorboard_data
+    global global_lr
+    if not args.distributed or (args.distributed and args.rank == 0):
+        epoch = tensorboard_data['epoch']
+
+        train_loss = tensorboard_data['train']['loss']
+        eval_loss = tensorboard_data['eval']['loss']
+
+        train_top1 = tensorboard_data['train']['top1']
+        eval_top1 = tensorboard_data['eval']['top1']
+
+        train_top5 = tensorboard_data['train']['top5']
+        eval_top5 = tensorboard_data['eval']['top5']
+
+        writer.add_scalars('top1 acc', {'train acc': train_top1, 'eval acc': eval_top1}, epoch)
+        writer.add_scalars('top5 acc', {'train acc': train_top5, 'eval acc': eval_top5}, epoch)
+        writer.add_scalar('learning rate', global_lr, epoch)
+        writer.add_scalars('loss value', {'train loss': train_loss, 'eval loss': eval_loss}, epoch)
+
+if __name__ == '__main__':
+    main()
diff --git a/models_v2/pytorch/resnet50v1_5/inference/gpu/requirements.txt b/models_v2/pytorch/resnet50v1_5/inference/gpu/requirements.txt
new file mode 100644
index 000000000..3868fb16b
--- /dev/null
+++ b/models_v2/pytorch/resnet50v1_5/inference/gpu/requirements.txt
@@ -0,0 +1 @@
+pillow
diff --git a/models_v2/pytorch/resnet50v1_5/inference/gpu/resnet50.py b/models_v2/pytorch/resnet50v1_5/inference/gpu/resnet50.py
new file mode 100644
index 000000000..606bef621
--- /dev/null
+++ b/models_v2/pytorch/resnet50v1_5/inference/gpu/resnet50.py
@@ -0,0 +1,35 @@
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Consts for resnet50 inference."""
+
+ACC = {
+    "type": "total",
+    "pattern": r" * Acc@1:(\d+.\d+)",
+    "unit": "acc@1",
+}
+
+PERF = {
+    "type": "total",
+    "pattern": r"throughput:(\d+.\d+) image/sec",
+    "inverse": False,
+    "multiply": False,
+    "use_batch_size": False,
+    "unit": "fps",
+}
+
+FUNCTIONAL = {
+    "pattern": r"Evalution performance: batch size:\d+, throughput:\d+.\d+ image/sec"
+}
diff --git a/models_v2/pytorch/resnet50v1_5/inference/gpu/run_model.sh b/models_v2/pytorch/resnet50v1_5/inference/gpu/run_model.sh
new file mode 100755
index 000000000..f47c90309
--- /dev/null
+++ b/models_v2/pytorch/resnet50v1_5/inference/gpu/run_model.sh
@@ -0,0 +1,191 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+
+#!/bin/bash
+
+# Create an array of input directories that are expected and then verify that they exist
+declare -A input_envs
+input_envs[MULTI_TILE]=${MULTI_TILE}
+input_envs[PLATFORM]=${PLATFORM}
+
+for i in "${!input_envs[@]}"; do
+  var_name=$i
+  env_param=${input_envs[$i]}
+
+  if [[ -z $env_param ]]; then
+    echo "The required environment variable $var_name is not set" >&2
+    exit 1
+  fi
+done
+
+OUTPUT_DIR=${OUTPUT_DIR:-$PWD}
+
+if [[ "${PLATFORM}" == "Max" ]]; then
+    BATCH_SIZE=${BATCH_SIZE:-1024}
+    PRECISION=${PRECISION:-INT8}
+    NUM_ITERATIONS=${NUM_ITERATIONS:-500}
+elif [[ "${PLATFORM}" == "Flex" ]]; then
+    if [[ "${MULTI_TILE}" == "True" ]]; then
+	echo "Flex not support multitile"
+	exit 1
+    fi
+    BATCH_SIZE=${BATCH_SIZE:-1024}
+    PRECISION=${PRECISION:-INT8}
+    NUM_ITERATIONS=${NUM_ITERATIONS:-500}
+elif [[ "${PLATFORM}" == "ARC" ]]; then
+    if [[ "${MULTI_TILE}" == "True" ]]; then
+        echo "ARC not support multitile"
+        exit 1
+    fi
+    BATCH_SIZE=${BATCH_SIZE:-256}
+    PRECISION=${PRECISION:-INT8}
+    NUM_ITERATIONS=${NUM_ITERATIONS:-500}
+fi
+
+
+
+if [[ -z "${DATASET_DIR}" ]]; then
+  echo "Using Dummy data since environment variable DATASET_DIR has not been set"
+  DATASET_DIR="--dummy"
+else
+  if [[ ! -d "${DATASET_DIR}" ]]; then
+    echo "The DATASET_DIR '${DATASET_DIR}' does not exist"
+    exit 1
+  fi
+fi
+
+# known issue
+if [[ "${MULTI_TILE}" == "True" ]]; then
+    export ZE_FLAT_DEVICE_HIERARCHY=COMPOSITE
+fi
+
+echo 'Running with parameters:'
+echo " PLATFORM: ${PLATFORM}"
+echo " DATASET_PATH: ${DATASET_DIR}"
+echo " OUTPUT_DIR: ${OUTPUT_DIR}"
+echo " PRECISION: ${PRECISION}"
+echo " BATCH_SIZE: ${BATCH_SIZE}"
+echo " NUM_ITERATIONS: ${NUM_ITERATIONS}"
+echo " MULTI_TILE: ${MULTI_TILE}"
+
+if [[ "${PRECISION}" == "INT8" ]]; then
+    flag="--int8 1 "
+elif [[ "${PRECISION}" == "BF16" ]]; then
+    flag="--bf16 1 "
+elif [[ "${PRECISION}" == "FP32" ]]; then
+    flag=""
+elif [[ "${PRECISION}" == "TF32" ]]; then
+    flag="--tf32 1 "
+elif [[ "${PRECISION}" == "FP16" ]]; then
+    flag="--fp16 1 "
+else
+    echo -e "Invalid input! Only BF16 FP32 TF32 FP16 INT8 are supported."
+    exit 1
+fi
+echo "resnet50 ${PRECISION} inference plain MultiTile=${MULTI_TILE} BS=${BATCH_SIZE} Iter=${NUM_ITERATIONS}"
+
+# Create the output directory, if it doesn't already exist
+mkdir -p $OUTPUT_DIR
+
+sum_log_analysis() {
+    if [ -f $2 ]; then
+        rm -f $2
+    fi
+    if diff /dev/null ${1}_t0.log |tail -l | grep '^\\ No newline' > /dev/null;then echo >> ${1}_t0.log; fi
+    if diff /dev/null ${1}_t1.log |tail -l | grep '^\\ No newline' > /dev/null;then echo >> ${1}_t1.log; fi
+    bs=$(cat ${1}_t1.log |grep Batch |awk '{print $3}')
+    echo -e "Batch Size: $bs" >$2
+    cat ${1}"_t0.log" ${1}"_t1.log" |grep "Performance" |awk -v tag=$(cat ${1}"_t0.log" ${1}"_t1.log" |grep "Performance" |awk '{sum+=$2} END {printf "%.4f\n",sum}') '{if ( $2=="None" ) {sum="None";nextfile}else  sum=tag} ;END{print "Sum "$1" "sum " "$3}' >> $2
+    cat ${1}"_t0.log" ${1}"_t1.log" |grep "Performance" |awk -v tag=$(cat ${1}"_t0.log" ${1}"_t1.log" |grep "Performance" |awk 'BEGIN {min=1234567890123} {if ($2 <min) {min=$2}}END {printf "%.4f\n",min}') '{if ( $2=="None" ) {min="None";nextfile}else  min=tag} ;END{print "Min "$1" "min " "$3}' >> $2
+    cat ${1}"_t0.log" ${1}"_t1.log" |grep "Latency" |awk '{if ( $2=="N/A" ){avg="N/A";nextfile}else avg=((sum+=$2/2))};END{print "Avg "$1" "avg " "$3}'  >> $2
+    cat ${1}"_t0.log" ${1}"_t1.log" |grep "Accuracy" |awk -v avg=$(cat ${1}"_t0.log" ${1}"_t1.log" |grep "Accuracy" |awk '{sum+=$3}END{printf "%.4f\n",sum/NR}') '{if ( $3=="None" || $2=="N/A" || $3=="nan" || $3=="N/A"){avg="None";nextfile}else avg=avg};END{print "Avg "$1" "$2 " "avg}' >> $2
+    cat ${1}"_t0.log" ${1}"_t1.log" |grep "Functional" | awk -v fail=$(cat ${1}"_t0.log" ${1}"_t1.log" |grep "Functional" |awk '{for(i=1;i<=NF;++i) if($i=="fail") ++sum}END{print sum}') '{if ( fail >= 1 ) tag="fail ";else tag="pass"};END{print $1" "tag}'  >> $2
+    cat ${1}"_t0.log" ${1}"_t1.log" |grep "Error" |awk '{if(a[$1]){a[$1]=a[$1]";"$2}else{a[$1]=$2}}END{for(i in a)print $1" " a[i]}' >> $2
+}
+
+modelname=resnet50
+if [[ ${MULTI_TILE} == "False" ]]; then
+    rm ${OUTPUT_DIR}/resnet50_${PRECISION}_inf_t0_raw.log 
+    IPEX_XPU_ONEDNN_LAYOUT=1 python -u main.py \
+        -a resnet50 \
+        -b ${BATCH_SIZE} \
+        --xpu 0 \
+        -e \
+        --pretrained \
+        ${flag} \
+        --num-iterations ${NUM_ITERATIONS} \
+        --benchmark 1 \
+        ${DATASET_DIR}  2>&1 | tee ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0_raw.log
+    python common/parse_result.py -m $modelname -l ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0_raw.log -b ${BATCH_SIZE}
+    throughput=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0.log | grep Performance | awk -F ' ' '{print $2}')
+    throughput_unit=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0.log | grep Performance | awk -F ' ' '{print $3}')
+    latency=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0.log | grep Latency | awk -F ' ' '{print $2}')
+    acc=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0.log | grep Accuracy | awk -F ' ' '{print $3}')
+    acc_unit=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0.log | grep Accuracy | awk -F ' ' '{print $2}')
+else
+    rm ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0_raw.log
+    rm ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t1_raw.log
+    ZE_AFFINITY_MASK=0.0 IPEX_XPU_ONEDNN_LAYOUT=1 python -u main.py \
+        -a resnet50 \
+        -b ${BATCH_SIZE} \
+        --xpu 0 \
+        -e \
+        --pretrained \
+        ${flag} \
+        --num-iterations ${NUM_ITERATIONS} \
+        --benchmark 1 \
+        ${DATASET_DIR} 2>&1 | tee ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0_raw.log &
+    ZE_AFFINITY_MASK=0.1 IPEX_XPU_ONEDNN_LAYOUT=1 python -u main.py \
+        -a resnet50 \
+        -b ${BATCH_SIZE} \
+        --xpu 0 \
+        -e \
+        --pretrained \
+        ${flag} \
+        --num-iterations ${NUM_ITERATIONS} \
+        --benchmark 1 \
+        ${DATASET_DIR}  2>&1 | tee ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t1_raw.log
+    wait
+    python common/parse_result.py -m $modelname -l ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0_raw.log -b ${BATCH_SIZE}
+    python common/parse_result.py -m $modelname -l ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t1_raw.log -b ${BATCH_SIZE}
+    sum_log_analysis ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf.log
+    throughput=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf.log | grep "Sum Performance" | awk -F ' ' '{print $3}')
+    throughput_unit=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf.log | grep "Sum Performance" | awk -F ' ' '{print $4}')
+    latency=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf.log | grep Latency | awk -F ' ' '{print $3}')
+    acc=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf.log | grep Accuracy | awk -F ' ' '{print $4}')
+    acc_unit=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf.log | grep Accuracy | awk -F ' ' '{print $3}')
+fi
+
+yaml_content=$(cat <<EOF
+results:
+ - key: throughput
+   value: $throughput
+   unit: $throughput_unit
+ - key: latency
+   value: $latency
+   unit: s
+ - key: accuracy
+   value: $acc
+   unit: $acc_unit
+EOF
+)
+
+# Write the content to a YAML file
+echo "$yaml_content" >  ${OUTPUT_DIR}/results.yaml
+echo "YAML file created."
diff --git a/models_v2/pytorch/resnet50v1_5/inference/gpu/setup.sh b/models_v2/pytorch/resnet50v1_5/inference/gpu/setup.sh
new file mode 100755
index 000000000..df70afcb7
--- /dev/null
+++ b/models_v2/pytorch/resnet50v1_5/inference/gpu/setup.sh
@@ -0,0 +1,28 @@
+#!/usr/bin/env bash
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# setup.sh
+#  - install OS pkgs
+#  - should create virtual env & install pip  requirement.txt
+#  - git clones & applying patches
+
+set -e
+apt-get update && apt-get install -y python3-venv
+
+pip install -r requirements.txt
+
+cp -r ../../../../common .
diff --git a/models_v2/pytorch/resnet50v1_5/training/gpu/CONTAINER.md b/models_v2/pytorch/resnet50v1_5/training/gpu/CONTAINER.md
new file mode 100644
index 000000000..ef1524163
--- /dev/null
+++ b/models_v2/pytorch/resnet50v1_5/training/gpu/CONTAINER.md
@@ -0,0 +1,96 @@
+# Running ResNet50_v1.5 Training on Intel® Data Center GPU Max Series using Intel® Extension for PyTorch*
+
+## Description 
+This document has instructions for running ResNet50 v1.5 training on Intel® Data Center GPU Max Series using Intel® Extension for PyTorch.
+
+## Requirements
+| Item | Detail |
+| ------ | ------- |
+| Host machine  | [Intel® Data Center GPU Max Series](https://ark.intel.com/content/www/us/en/ark/products/series/232874/intel-data-center-gpu-max-series.html)  |
+| Drivers | GPU-compatible drivers need to be installed: Download [Driver](https://dgpu-docs.intel.com/driver/installation.html) |
+| Software | Docker* |
+
+## Quick Start Scripts
+
+| Script name | Description |
+|-------------|-------------|
+| `run_model.sh` | Runs ResNet50 V1.5 BF16,FP32 and TF32 training on single tile and two tiles|
+
+## Datasets
+Download and extract the ImageNet2012 training and validation dataset from [http://www.image-net.org/ (http://www.image-net.org/),then move validation images to labeled subfolders, using
+[the valprep.sh shell script](https://raw.githubusercontent.com/soumith/imagenetloader.torch/master/valprep.sh)
+
+After running the data prep script and extracting the images, your folder structure
+should look something like this:
+```bash
+imagenet
+├── train
+│   ├── n02085620
+│   │   ├── n02085620_10074.JPEG
+│   │   ├── n02085620_10131.JPEG
+│   │   ├── n02085620_10621.JPEG
+│   │   └── ...
+│   └── ...
+└── val
+    ├── n01440764
+    │   ├── ILSVRC2012_val_00000293.JPEG
+    │   ├── ILSVRC2012_val_00002138.JPEG
+    │   ├── ILSVRC2012_val_00003014.JPEG
+    │   └── ...
+    └── ...
+```
+The folder that contains the `val` and `train` directories should be set as the `DATASET_DIR` environment variable before running the quickstart script.
+
+### Docker pull command:
+```bash
+docker pull intel/image-recognition:pytorch-max-gpu-resnet50v1-5-training
+```
+The ResNet50 v1.5 training container includes scripts, model and libraries needed to run BF16,FP32 and TF32 training. To run the `run_model.sh` quickstart script using this container, you'll need to provide volume mounts for the ImageNet dataset. You will need to provide an output directory where log files will be written. 
+
+```bash
+#Optional
+export PRECISION=<provide either BF16,FP32 or TF32, otherwise (default: BF16)>
+export BATCH_SIZE=<provide batch size, otherwise (default: 256)>
+export NUM_ITERATIONS=<provide number of iterations,otherwise (default: 20)>
+
+#Required
+export DATASET_DIR=<path to ImageNet dataset>
+export OUTPUT_DIR=<path to output logs directory>
+export PLATFORM=Max
+export MULTI_TILE=<provide True or False to enable/disable distributed training>
+
+DOCKER_ARGS="--rm --init -it"
+IMAGE_NAME=intel/image-recognition:pytorch-max-gpu-resnet50v1-5-training
+SCRIPT=run_model.sh
+
+docker run \
+  --privileged \
+  --device=/dev/dri \
+  --ipc=host \
+  --env DATASET_DIR=${DATASET_DIR} \
+  --env OUTPUT_DIR=${OUTPUT_DIR} \
+  --env MULTI_TILE=${MULTI_TILE} \
+  --env PLATFORM=${PLATFORM} \
+  --env BATCH_SIZE=${BATCH_SIZE} \
+  --env NUM_ITERATIONS=${NUM_ITERATIONS} \
+  --env PRECISION=${PRECISION} \
+  --env http_proxy=${http_proxy} \
+  --env https_proxy=${https_proxy} \
+  --env no_proxy=${no_proxy} \
+  --volume ${DATASET_DIR}:${DATASET_DIR} \
+  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
+  --volume /dev/dri:/dev/dri \
+  ${DOCKER_ARGS} \
+  $IMAGE_NAME \
+  /bin/bash $SCRIPT
+  ```
+## Documentation and Sources
+
+[GitHub* Repository](https://github.com/IntelAI/models/tree/master/docker/max-gpu)
+
+## Support
+Support for Intel® Extension for PyTorch* is found via the [Intel® AI Analytics Toolkit.](https://www.intel.com/content/www/us/en/developer/tools/oneapi/ai-analytics-toolkit.html#gs.qbretz) Additionally, the Intel® Extension for PyTorch* team tracks both bugs and enhancement requests using [GitHub issues](https://github.com/intel/intel-extension-for-pytorch/issues). Before submitting a suggestion or bug report, please search the GitHub issues to see if your issue has already been reported.
+
+## License Agreement
+
+LEGAL NOTICE: By accessing, downloading or using this software and any required dependent software (the “Software Package”), you agree to the terms and conditions of the software license agreements for the Software Package, which may also include notices, disclaimers, or license terms for third party software included with the Software Package. Please refer to the [license file](https://github.com/IntelAI/models/tree/master/third_party) for additional details.
diff --git a/models_v2/pytorch/resnet50v1_5/training/gpu/README.md b/models_v2/pytorch/resnet50v1_5/training/gpu/README.md
new file mode 100644
index 000000000..03d7f9828
--- /dev/null
+++ b/models_v2/pytorch/resnet50v1_5/training/gpu/README.md
@@ -0,0 +1,95 @@
+# ResNet50v1.5 Training
+
+ResNet50v1.5 Training best known configurations with Intel® Extension for PyTorch.
+
+## Model Information
+
+| **Use Case** | **Framework** | **Model Repo** | **Branch/Commit/Tag** | **Optional Patch** |
+|:---:| :---: |:--------------:|:---------------------:|:------------------:|
+|  Training    |    Pytorch    |       -        |           -           |         -          |
+
+# Pre-Requisite
+* Host has one of the following GPUs:
+  * **Arc Series** - [Intel® Arc™ A-Series Graphics](https://ark.intel.com/content/www/us/en/ark/products/series/227957/intel-arc-a-series-graphics.html)
+  * **Max Series** - [Intel® Data Center GPU Max Series](https://ark.intel.com/content/www/us/en/ark/products/series/232874/intel-data-center-gpu-max-series.html)
+* Host has installed latest Intel® Data Center GPU Max & Flex Series Drivers https://dgpu-docs.intel.com/driver/installation.html
+* The following Intel® oneAPI Base Toolkit components are required:
+  - Intel® oneAPI DPC++ Compiler (Placeholder DPCPPROOT as its installation path)
+  - Intel® oneAPI Math Kernel Library (oneMKL) (Placeholder MKLROOT as its installation path)
+  - Intel® oneAPI MPI Library
+  - Intel® oneAPI TBB Library
+  - Intel® oneAPI CCL Library
+
+  Follow instructions at [Intel® oneAPI Base Toolkit Download page](https://www.intel.com/content/www/us/en/developer/tools/oneapi/base-toolkit-download.html?operatingsystem=linux) to setup the package manager repository.
+
+# Prepare Dataset
+## Dataset: imagenet
+ImageNet is recommended, the download link is https://image-net.org/challenges/LSVRC/2012/2012-downloads.php.
+
+## Training
+1. `git clone https://github.com/IntelAI/models.git`
+2. `cd models/models_v2/pytorch/resnet50v1_5/training/gpu`
+3. Create virtual environment `venv` and activate it:
+    ```
+    python3 -m venv venv
+    . ./venv/bin/activate
+    ```
+4. Run setup.sh
+    ```
+    ./setup.sh
+    ```
+5. Install the latest GPU versions of [torch, torchvision and intel_extension_for_pytorch](https://intel.github.io/intel-extension-for-pytorch/index.html#installation):
+    ```
+    python -m pip install torch==<torch_version> torchvision==<torchvvision_version> intel-extension-for-pytorch==<ipex_version> --extra-index-url https://pytorch-extension.intel.com/release-whl-aitools/
+    ```
+6. Set environment variables for Intel® oneAPI Base Toolkit: 
+    Default installation location `{ONEAPI_ROOT}` is `/opt/intel/oneapi` for root account, `${HOME}/intel/oneapi` for other accounts
+    ```bash
+    source {ONEAPI_ROOT}/compiler/latest/env/vars.sh
+    source {ONEAPI_ROOT}/mkl/latest/env/vars.sh
+    source {ONEAPI_ROOT}/tbb/latest/env/vars.sh
+    source {ONEAPI_ROOT}/mpi/latest/env/vars.sh
+    source {ONEAPI_ROOT}/ccl/latest/env/vars.sh
+7. Setup required environment paramaters
+
+| **Parameter**                |                                  **export command**                                  |
+|:---------------------------:|:------------------------------------------------------------------------------------:|
+| **MULTI_TILE**               | `export MULTI_TILE=False` (True or False)                                            |
+| **PLATFORM**                 | `export PLATFORM=Max` (Max or ARC)                                                 |
+| **DATASET_DIR**              |                               `export DATASET_DIR=`                                  |
+| **BATCH_SIZE** (optional)    |                               `export BATCH_SIZE=256`                                |
+| **PRECISION**  (optional)    | `export PRECISION=BF16` (BF16 or TF32 or FP32 for Max and BF16 or FP32 for ARC )    |
+|**NUM_ITERATIONS** (optional) |                               `export NUM_ITERATIONS=20`                             |
+| **OUTPUT_DIR** (optional)    |                               `export OUTPUT_DIR=$PWD`                               |
+6. Run `run_model.sh`
+
+## Output
+
+Single-tile output will typically look like:
+
+```
+Epoch: [0][ 20/196]     Time  0.148 ( 0.379)    Data  0.000 ( 0.218)    Loss 7.3804e+00 (8.0065e+00)    Acc@1   0.00 (  0.12)    Acc@5   0.39 (  0.45)
+Training performance: batch size:256, throughput:1716.10 image/sec
+```
+
+Multi-tile output will typically looks like:
+```
+[1] Epoch: [0][  20/2503]       Time  0.173 ( 0.855)    Data  0.001 ( 0.072)    Loss 7.6168e+00 (7.5532e+00)     Acc@1   0.39 (  0.08)   Acc@5   0.39 (  0.43)
+[1] Training performance: batch size:256, throughput:1474.26 image/sec
+[0] Epoch: [0][  20/2503]       Time  0.172 ( 0.861)    Data  0.001 ( 0.065)    Loss 7.6672e+00 (7.5610e+00)     Acc@1   0.00 (  0.16)   Acc@5   0.00 (  0.74)
+[0] Training performance: batch size:256, throughput:1473.24 image/sec
+```
+Final results of the training run can be found in `results.yaml` file.
+
+```
+results:
+ - key: throughput
+   value: 1716.1
+   unit: image/s
+ - key: latency
+   value: 0.14917545597575899
+   unit: s
+ - key: accuracy
+   value: 7.380
+   unit: loss
+```
diff --git a/models_v2/pytorch/resnet50v1_5/training/gpu/main.py b/models_v2/pytorch/resnet50v1_5/training/gpu/main.py
new file mode 100644
index 000000000..f9673d561
--- /dev/null
+++ b/models_v2/pytorch/resnet50v1_5/training/gpu/main.py
@@ -0,0 +1,1366 @@
+#
+# ****************************************************************************
+# Copyright 2019-2022 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ****************************************************************************
+
+# ****************************************************************************
+# BSD 3-Clause License
+# # This model is originally from the PyTorch Examples repo
+# (https://github.com/pytorch/examples/blob/master/imagenet/main.py)
+# ****************************************************************************
+
+# ****************************************************************************
+# Copyright (c) 2017,
+# All rights reserved.
+#
+# Redistribution and use in source and binary forms, with or without
+# modification, are permitted provided that the following conditions are met:
+#
+# * Redistributions of source code must retain the above copyright notice, this
+#   list of conditions and the following disclaimer.
+#
+# * Redistributions in binary form must reproduce the above copyright notice,
+#   this list of conditions and the following disclaimer in the documentation
+#   and/or other materials provided with the distribution.
+#
+# * Neither the name of the copyright holder nor the names of its
+#   contributors may be used to endorse or promote products derived from
+#   this software without specific prior written permission.
+#
+# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+# DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+# OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+# ****************************************************************************
+
+import argparse
+import os
+import sys
+import random
+import shutil
+import time
+import warnings
+from enum import Enum
+
+import torch
+import torch.nn as nn
+import torch.nn.parallel
+import torch.backends.cudnn as cudnn
+import torch.distributed as dist
+import torch.optim
+from torch.optim.lr_scheduler import StepLR
+import torch.multiprocessing as mp
+import torch.utils.data
+import torch.utils.data.distributed
+import torchvision.transforms as transforms
+import torchvision.datasets as datasets
+import torchvision.models as models
+from torch.utils.data import Subset
+import math
+
+torch._C._jit_set_profiling_mode(False)
+torch._C._jit_set_profiling_executor(False)
+
+model_names = sorted(name for name in models.__dict__
+    if name.islower() and not name.startswith("__")
+    and callable(models.__dict__[name]))
+
+# for convergence
+converged = False
+final_top1_acc = 0.0
+final_top5_acc = 0.0
+
+cwd = os.path.dirname(os.path.abspath(__file__))
+hub = os.path.expanduser("~/.cache/torch/intel")
+if not os.path.exists(hub):
+    os.makedirs(hub, exist_ok=True)
+
+parser = argparse.ArgumentParser(description='PyTorch ImageNet Training')
+parser.add_argument('data', metavar='DIR', nargs='?', default='imagenet',
+                    help='path to dataset (default: imagenet)')
+parser.add_argument('-a', '--arch', metavar='ARCH', default='resnet18',
+                    choices=model_names,
+                    help='model architecture: ' +
+                        ' | '.join(model_names) +
+                        ' (default: resnet18)')
+parser.add_argument('-j', '--workers', default=4, type=int, metavar='N',
+                    help='number of data loading workers (default: 4)')
+parser.add_argument('--epochs', default=90, type=int, metavar='N',
+                    help='number of total epochs to run')
+parser.add_argument('--eval-start-epoch', default=0, type=int, metavar='N',
+                    help='epoch start to run validation')
+parser.add_argument('--start-epoch', default=0, type=int, metavar='N',
+                    help='manual epoch number (useful on restarts)')
+parser.add_argument('-b', '--batch-size', default=256, type=int,
+                    metavar='N',
+                    help='mini-batch size (default: 256), this is the total '
+                         'batch size of all GPUs on the current node when '
+                         'using Data Parallel or Distributed Data Parallel')
+parser.add_argument('--lr', '--learning-rate', default=0.1, type=float,
+                    metavar='LR', help='initial learning rate', dest='lr')
+parser.add_argument('--momentum', default=0.9, type=float, metavar='M',
+                    help='momentum')
+parser.add_argument('--wd', '--weight-decay', default=1e-4, type=float,
+                    metavar='W', help='weight decay (default: 1e-4)',
+                    dest='weight_decay')
+parser.add_argument('-p', '--print-freq', default=1, type=int,
+                    metavar='N', help='print frequency (default: 10)')
+parser.add_argument('--resume', default='', type=str, metavar='PATH',
+                    help='path to latest checkpoint (default: none)')
+parser.add_argument('-e', '--evaluate', dest='evaluate', action='store_true',
+                    help='evaluate model on validation set')
+parser.add_argument('--pretrained', dest='pretrained', action='store_true',
+                    help='use pre-trained model')
+parser.add_argument('--world-size', default=-1, type=int,
+                    help='number of nodes for distributed training')
+parser.add_argument('--rank', default=-1, type=int,
+                    help='node rank for distributed training')
+parser.add_argument('--dist-url', default='127.0.0.1', type=str,
+                    help='url used to set up distributed training')
+parser.add_argument('--dist-port', default='29500', type=str,
+                    help='url port used to set up distributed training')
+parser.add_argument('--dist-backend', default='ccl', type=str,
+                    help='distributed backend, default is torch-ccl')
+parser.add_argument('--seed', default=None, type=int,
+                    help='seed for initializing training. ')
+parser.add_argument('--gpu', default=None, type=int,
+                    help='GPU id to use.')
+parser.add_argument('--xpu', default=None, type=int,
+                    help='XPU id to use.')
+parser.add_argument('--tf32', default=0, type=int, help='Datatype used: TF32')
+parser.add_argument('--bf32', default=0, type=int, help='Datatype used: BF32')
+parser.add_argument('--fp16', default=0, type=int, help='Datatype used: FP16')
+parser.add_argument('--bf16', default=0, type=int, help='Datatype used: BF16')
+parser.add_argument('--int8', default=0, type=int, help='Use int8 quantization to do inference')
+parser.add_argument('--disable-broadcast-buffers', action='store_true', help='disable syncing buffers')
+parser.add_argument('--bucket-cap', default=25, type=int, help='controls the bucket size in MegaBytes')
+parser.add_argument('--large-first-bucket', action="store_true", help='Configure a large capacity of the first bucket in DDP for allreduce')
+parser.add_argument("--use-gradient-as-bucket-view", action='store_true', help="Turn ON gradient_as_bucket_view optimization in DDP")
+parser.add_argument('--jit-cache', type=str, default=str(hub), help="path to save/load jit model")
+parser.add_argument('--jit-trace', action='store_true',
+                    help='enable PyTorch jit trace graph mode')
+parser.add_argument('--dynamo', action='store_true', help='Use torch.compile to optimize model, inductor backend default')
+parser.add_argument('--calib-iters', default=8, type=int,
+                    help='iteration number for calibration')
+parser.add_argument('--calib-bs', default=32, type=int,
+                    metavar='N', help='mini-batch size for calibration')
+parser.add_argument('--perchannel-weight', default=False,
+                    help='do calibration with weight per channel quantization')
+# TODO: when fix non blocking H2D perf regression, we will remove here option
+parser.add_argument('--non-blocking', default=False, action='store_true',
+                    help='non blocking H2D for input and target, now for int8, default False')
+parser.add_argument('--channels-last', action='store_true', help='enable channels last')
+parser.add_argument('--num-iterations', default=0, type=int)
+parser.add_argument('--converge', default=None, action='store_true',
+                    help='Run convergence and use Tensorboard to visualize the training metrics')
+parser.add_argument('--step-size', default=30, type=int, help='LR decay step size')
+parser.add_argument('--step-gamma', default=0.1, type=float, help='set the step gamma')
+parser.add_argument('--last-step-boundary', default=80, type=int, help='last epoch to decay the LR')
+parser.add_argument('--warm-up-epoch', default=0, type=int, help='warm up epochs number for convergence')
+parser.add_argument('--decay-epochs', default=33, type=int, metavar='N',
+                    help='number of decay epochs to run for lars')
+parser.add_argument('--lars', default=False, action='store_true', help='use lars for training')
+parser.add_argument('--lars-eta', default=0.0, type=float, help='set the lars epsilon')
+parser.add_argument('--skip-checkpoint', default=False, action='store_true', help='skip checkpoint saving')
+parser.add_argument('--skip-tensorboard', default=False, action='store_true', help='skip tensorboard')
+parser.add_argument('--label-smoothing', default=0.0, type=float)
+parser.add_argument('--dummy', action="store_true", help='use dummy data for '
+                    'benchmark training or val')
+parser.add_argument('--lr-scheduler', default='step', type=str,
+                    help='choose lr scheduler, default step, can choose pow')
+parser.add_argument('--power-factor', default=1.0, type=float,
+                    help='power factor for lr decay policy')
+parser.add_argument('--eval-period', default=1, type=int, help='period for doing online evaluation')
+parser.add_argument('--eval-offset', default=0, type=int, help='offset for doing online evaluation')
+parser.add_argument('--sota-target', default=75.9, type=float, help='set the lars epsilon')
+parser.add_argument('--multiprocessing-distributed', action='store_true',
+                    help='Use multi-processing distributed training to launch '
+                         'N processes per node, which has N GPUs. This is the '
+                         'fastest way to use PyTorch for either single node or '
+                         'multi node data parallel training')
+parser.add_argument('--benchmark', default=0, type=int, help='for int8 benchmark '
+                    'performance, move H2D out of E2E time')
+parser.add_argument("--save", help='Path to save entile model, save infernce mode, training is not available')
+parser.add_argument("--load", help='Path to load entile inference model')
+parser.add_argument("--asymm", action='store_true', help='Asymmetric quantization')
+parser.add_argument("--uint8", action='store_true', help="Use u8 quantization")
+parser.add_argument('--end-lr', type=float, default=1e-4,
+                    help='the end learning rate')
+
+# used for record best accrucy after validation
+best_acc1 = 0.0
+tensorboard_data = {'epoch': 0,
+                    'train': {'loss': 0.0, 'top1': 0.0, 'top5': 0.0},
+                    'eval': {'loss': 0.0, 'top1': 0.0, 'top5': 0.0}}
+global_lr = 0.0
+global_num_iter = 0
+
+def main():
+    args = parser.parse_args()
+    if args.converge:
+        print('[info] ------------------ converge arguments ------------------')
+        print('running model:  ', args.arch)
+        print('workers:        ', args.workers)
+        print('running bf16:   ', args.bf16)
+        print('total epochs:   ', args.epochs)
+        print('warm up epoch:  ', args.warm_up_epoch)
+        print('eval epoch:     ', args.eval_start_epoch)
+        print('batch size:     ', args.batch_size)
+        print('initial lr:     ', args.lr)
+        print('lr scheduler:   ', args.lr_scheduler)
+        if args.lr_scheduler == 'step':
+            print('lr step size:   ', args.step_size)
+            print('lr step gamma:  ', args.step_gamma)
+            print('lr step boundary:', args.last_step_boundary)
+        elif args.lr_scheduler == 'pow':
+            print('lr pow factor:  ', args.power_factor)
+        else:
+            pass
+        if args.lars:
+            print('using lars:     ', 'True')
+            print('choose lars eta:', args.lars_eta)
+            print("decay epochs:", args.decay_epochs)
+        else:
+            print('using sgd:      ', 'True')
+        print('label smoothing:', args.label_smoothing)
+        print('momentum:       ', args.momentum)
+        print('weight decay:   ', args.weight_decay)
+        print('seed:           ', args.seed)
+        print('eval period:    ', args.eval_period)
+        print('eval offset:    ', args.eval_offset)
+        print('sota target:    ', args.sota_target)
+        print('skip ckpt:      ', args.skip_checkpoint)
+        print('disable broadcast: ', args.disable_broadcast_buffers)
+        print('large 1st bucket: ', args.large_first_bucket)
+        print('use grad as bucket view: ', args.use_gradient_as_bucket_view)
+        print('[info] --------------------------------------------------------')
+
+    if args.xpu is not None and args.gpu is not None:
+        print('You need to choose running on NV GPU or XPU.')
+        sys.exit()
+
+    if args.gpu is not None and not torch.cuda.is_available():
+        print('Make sure cuda is enabled in torch.')
+        sys.exit()
+
+    if args.xpu is not None:
+        import intel_extension_for_pytorch as ipex
+
+    # only for training
+    if not args.evaluate:
+        if args.tf32:
+            print('doing TF32 training')
+            torch.xpu.set_fp32_math_mode(torch.xpu.FP32MathMode.TF32)
+        elif args.bf32:
+            args.bf16 = 1
+            print('doing BF32 training')
+            torch.xpu.set_fp32_math_mode(torch.xpu.FP32MathMode.BF32)
+        else:
+            torch.xpu.set_fp32_math_mode(torch.xpu.FP32MathMode.FP32)
+
+    if args.dist_backend == 'ccl':
+        try:
+            import oneccl_bindings_for_pytorch
+        except ImportError:
+            print("oneccl_bindings_for_pytorch not available!")
+
+    if args.int8 and (not args.evaluate or args.xpu is None):
+        print('For int8 quantization, it is only used in XPU inference, '
+              'you need to pass -e and --xpu [dev_id] in your command')
+        sys.exit()
+
+    if args.converge is not None:
+        from torch.utils.tensorboard import SummaryWriter
+        global writer
+        writer = SummaryWriter(log_dir='./tensorboard_log')
+        if args.num_iterations is not None:
+            warnings.warn('Tensorboard is displaying at epoch unit.')
+
+    if args.seed is not None:
+        print('Setting the seed: ', args.seed)
+        random.seed(args.seed)
+        torch.manual_seed(args.seed)
+        if args.xpu is not None:
+            torch.xpu.manual_seed(args.seed)
+        if torch.cuda.is_available():
+            cudnn.deterministic = True
+            warnings.warn('You have chosen to seed training. '
+                        'This will turn on the CUDNN deterministic setting, '
+                        'which can slow down your training considerably! '
+                        'You may see unexpected behavior when restarting '
+                        'from checkpoints.')
+
+    if args.gpu is not None:
+        warnings.warn('You have chosen a specific GPU. This will completely '
+                      'disable data parallelism.')
+
+    if args.world_size == -1:
+        mpi_world_size = int(os.environ.get('PMI_SIZE', -1))
+
+        if mpi_world_size > 0:
+            os.environ['MASTER_ADDR'] = args.dist_url #'127.0.0.1'
+            os.environ['MASTER_PORT'] = args.dist_port #'29500'
+            os.environ['RANK'] = os.environ.get('PMI_RANK', -1)
+            os.environ['WORLD_SIZE'] = os.environ.get('PMI_SIZE', -1)
+            args.rank = int(os.environ.get('PMI_RANK', -1))
+        args.world_size = int(os.environ.get("WORLD_SIZE", -1))
+    else: # mpich set
+        if 'PMIX_RANK' in os.environ.keys(): # mpich set
+            os.environ['MASTER_ADDR'] = args.dist_url #'127.0.0.1'
+            os.environ['MASTER_PORT'] = args.dist_port #'29500'
+            os.environ['RANK'] = os.environ.get('PMIX_RANK')
+            os.environ['WORLD_SIZE'] = str(args.world_size)
+            args.rank = int(os.environ.get('PMIX_RANK', -1))
+
+    args.distributed = args.world_size > 1 or args.multiprocessing_distributed
+
+    # 1 XPU card has 2 tile, and both are regarded as isolated devices/nodes
+    ngpus_per_node = 1
+    if args.multiprocessing_distributed:
+        # Since we have ngpus_per_node processes per node, the total world_size
+        # needs to be adjusted accordingly
+        args.world_size = ngpus_per_node * args.world_size
+        # Use torch.multiprocessing.spawn to launch distributed processes: the
+        # main_worker process function
+        mp.spawn(main_worker, nprocs=ngpus_per_node, args=(ngpus_per_node, args))
+    else:
+        # Simply call main_worker function
+        main_worker(ngpus_per_node, args)
+
+def jit_calib(model, val_loader_calib, args):
+    print('doing int8 jit calibration')
+    jit_model_file = os.path.join(args.jit_cache, "rn50_jit_model_int8.pt")
+    if os.path.isfile(jit_model_file):
+        print("=> load jit model from {}".format(jit_model_file))
+        modelJit = torch.load(jit_model_file)
+        print("=> load jit model ... done")
+    else:
+        from torch.jit._recursive import wrap_cpp_module
+        from torch.quantization.quantize_jit import (
+            convert_jit,
+            prepare_jit,
+        )
+        modelJit = torch.jit.script(model)
+        modelJit = wrap_cpp_module(torch._C._jit_pass_fold_convbn(modelJit._c))
+
+        qscheme = torch.per_tensor_affine if args.asymm else torch.per_tensor_symmetric
+        dtype = torch.quint8 if args.uint8 else torch.qint8
+        with torch.inference_mode():
+            if args.perchannel_weight:
+                qconfig = torch.quantization.QConfig(
+                    activation=torch.quantization.observer.MinMaxObserver.with_args(
+                        qscheme=qscheme,
+                        reduce_range=False,
+                        dtype=dtype
+                    ),
+                    weight=torch.quantization.default_per_channel_weight_observer
+                )
+            else:
+                qconfig = torch.quantization.QConfig(
+                    activation=torch.quantization.observer.MinMaxObserver.with_args(
+                        qscheme=qscheme,
+                        reduce_range=False,
+                        dtype=dtype
+                    ),
+                    weight=torch.quantization.default_weight_observer
+                )
+            modelJit = prepare_jit(modelJit, {'': qconfig}, True)
+
+            for i, (input, target) in enumerate(val_loader_calib):
+                calib = input.to(args.xpu)
+                modelJit(calib)
+
+                if i == args.calib_iters - 1:
+                    break
+            modelJit = convert_jit(modelJit, True)
+
+    return modelJit
+
+def main_worker(ngpus_per_node, args):
+    global best_acc1
+
+    if args.distributed:
+        if args.rank == -1:
+            args.rank = int(os.environ["RANK"])
+        if args.multiprocessing_distributed:
+            # For multiprocessing distributed training, rank needs to be the
+            # global rank among all the processes
+            args.rank = args.rank * ngpus_per_node + args.gpu
+        init_method = 'tcp://' + args.dist_url + ':' + args.dist_port
+        dist.init_process_group(backend=args.dist_backend, init_method=init_method,
+                                world_size=args.world_size, rank=args.rank)
+
+        if args.gpu is not None:
+            args.gpu = args.rank
+        elif args.xpu is not None:
+            if 'MPI_LOCALRANKID' in os.environ.keys():
+                local_rank = os.environ['MPI_LOCALRANKID']
+            elif 'OMPI_COMM_WORLD_LOCAL_RANK' in os.environ.keys():
+                local_rank = os.environ['OMPI_COMM_WORLD_LOCAL_RANK']
+            else: # mpich set
+                local_rank = os.environ['PALS_LOCAL_RANKID']
+            args.xpu = local_rank
+            print('world_size:{}, rank:{}, local_rank:{}'.format(args.world_size, args.rank, local_rank))
+
+    if args.gpu is not None:
+        print("Use GPU: {}".format(args.gpu))
+        args.gpu = "cuda:{}".format(args.gpu)
+    elif args.xpu is not None:
+        print("Use XPU: {}".format(args.xpu))
+        args.xpu = "xpu:{}".format(args.xpu)
+    else:
+        print("Use CPU")
+
+    # define loss function (criterion)
+    criterion = nn.CrossEntropyLoss(label_smoothing=args.label_smoothing)
+    if args.label_smoothing:
+        print('using label smoothing: ', args.label_smoothing)
+
+    # create model
+    if args.load:
+        if os.path.isfile(args.load):
+            load_path = args.load
+            if args.jit_trace:
+                model = torch.jit.load(load_path)
+            elif args.evaluate and args.int8:
+                model = torch.jit.load(load_path)
+            else:
+                model = torch.load(load_path)
+                optimizer = torch.optim.SGD(model.parameters(), lr=args.lr,
+                                momentum=args.momentum, weight_decay=args.weight_decay)
+                scheduler = StepLR(optimizer, step_size=args.step_size, gamma=0.1)
+        else:
+            print("=> no saved model found at '{}'".format(args.load))
+            sys.exit(1)
+    else:
+        if args.pretrained:
+            print("=> using pre-trained model '{}'".format(args.arch))
+            model = models.__dict__[args.arch](pretrained=True)
+        else:
+            print("=> creating model '{}'".format(args.arch))
+            model = models.__dict__[args.arch]()
+
+        # channels last
+        # TODO: this will be default memory format in future
+        if args.channels_last:
+            print('model is converted to channels last')
+            model = model.to(memory_format=torch.channels_last)
+
+        if args.distributed:
+            # For multiprocessing distributed, DistributedDataParallel constructor
+            # should always set the single device scope, otherwise,
+            # DistributedDataParallel will use all available devices.
+            if args.gpu is not None:
+                torch.cuda.set_device(args.gpu)
+                model.cuda(args.gpu)
+                # When using a single GPU per process and per
+                # DistributedDataParallel, we need to divide the batch size
+                # ourselves based on the total number of GPUs of the current node.
+                args.batch_size = int(args.batch_size / ngpus_per_node)
+                args.workers = int((args.workers + ngpus_per_node - 1) / ngpus_per_node)
+                model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[args.gpu])
+            elif args.xpu is not None:
+                torch.xpu.set_device(args.xpu)
+                model.xpu(args.xpu)
+        elif args.gpu is not None:
+            torch.cuda.set_device(args.gpu)
+            model = model.cuda(args.gpu)
+            print('model to cuda')
+        elif args.xpu is not None:
+            torch.xpu.set_device(args.xpu)
+            model = model.xpu(args.xpu)
+            print('model to xpu')
+        else:
+            # do training or inference on CPU
+            pass
+
+        # define optimizer, and learning rate scheduler
+        optimizer = torch.optim.SGD(model.parameters(), lr=args.lr,
+                                    momentum=args.momentum, weight_decay=args.weight_decay)
+
+        if args.lars:
+            eeta = args.lars_eta
+            print('using lars, eeta = ', eeta)
+            optimizer = torch.xpu.optim.Lars(model.parameters(), lr=args.lr,
+                                             momentum=args.momentum, weight_decay=args.weight_decay,
+                                             eeta=eeta)
+
+        # TODO: when change the oob of auto channels last function, here need change
+        using_block_layout = os.environ.get("IPEX_XPU_ONEDNN_LAYOUT", "OFF").upper() in ["1", "Y", "ON", "YES", "TRUE"]
+        if using_block_layout:
+            import intel_extension_for_pytorch as ipex
+            ipex.disable_auto_channels_last()
+            print('using block layout and disable the auto channels last')
+
+        # torch.xpu.optimize is only for device xpu and no jit script
+        if args.xpu is not None:
+            if args.evaluate:
+                if not args.int8:
+                    print('doing torch xpu optimize for inference')
+                    model.eval()
+                    dtype = torch.float16 if args.fp16 else torch.float32
+                    dtype = torch.bfloat16 if args.bf16 else dtype
+                    model = torch.xpu.optimize(model=model, dtype=dtype, level="O1")
+            else:
+                model.train()
+                print('doing torch xpu optimize for training')
+                model, optimizer = torch.xpu.optimize(model=model, optimizer=optimizer, level="O1",
+                                                      dtype=torch.bfloat16 if args.bf16 else torch.float32)
+
+        if args.distributed:
+            if args.xpu is not None:
+                # When using a single GPU per process and per
+                # DistributedDataParallel, we need to divide the batch size
+                # ourselves based on the total number of GPUs we have
+                args.batch_size = int(args.batch_size / ngpus_per_node)
+                args.workers = int(args.workers / ngpus_per_node)
+
+                if args.large_first_bucket:
+                    # set the first bucket with maximal size to cover all parameters for allreduce
+                    dist._DEFAULT_FIRST_BUCKET_BYTES = sys.maxsize
+                model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[args.xpu], broadcast_buffers=False if args.disable_broadcast_buffers else True, bucket_cap_mb=args.bucket_cap, gradient_as_bucket_view=args.use_gradient_as_bucket_view)
+
+        """Sets the learning rate to the initial LR decayed by 10 every configured epochs"""
+        scheduler = StepLR(optimizer=optimizer, step_size=args.step_size, gamma=args.step_gamma)
+
+        if not args.evaluate:
+            print('Using StepLR for training, step size ', args.step_size)
+
+        # optionally resume from a checkpoint
+        if args.resume:
+            if os.path.isfile(args.resume):
+                print("=> loading checkpoint '{}'".format(args.resume))
+                if args.gpu is None and args.xpu is None:
+                    # CPU load
+                    checkpoint = torch.load(args.resume)
+                elif args.gpu is not None:
+                    # Map model to be loaded to specified single gpu.
+                    print('loading checkpoint to ', str(args.gpu))
+                    checkpoint = torch.load(args.resume, map_location=args.gpu)
+                elif args.xpu is not None:
+                    # Map model to be loaded to specified single gpu.
+                    print('loading checkpoint to ', str(args.xpu))
+                    checkpoint = torch.load(args.resume, map_location=args.xpu)
+                args.start_epoch = checkpoint['epoch']
+                # keep best_acc1 on cpu for comparing acc after validation
+                best_acc1 = checkpoint['best_acc1']
+                model.load_state_dict(checkpoint['state_dict'])
+                optimizer.load_state_dict(checkpoint['optimizer'])
+                scheduler.load_state_dict(checkpoint['scheduler'])
+            else:
+                print("=> no checkpoint found at '{}'".format(args.resume))
+
+        if args.gpu is not None:
+            cudnn.benchmark = True
+
+    # Data loading code
+    traindir = os.path.join(args.data, 'train')
+    valdir = os.path.join(args.data, 'val')
+    normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],
+                                     std=[0.229, 0.224, 0.225])
+
+    if args.dummy:
+        print("Dummy data is used!")
+        train_dataset = datasets.FakeData(1281167, (3, 224, 224), 1000, transforms.ToTensor())
+        val_dataset_size = args.num_iterations * args.batch_size if (args.dummy and args.num_iterations) else 50000
+        val_dataset = datasets.FakeData(val_dataset_size, (3, 224, 224), 1000, transforms.ToTensor())
+    else:
+        traindir = os.path.join(args.data, 'train')
+        valdir = os.path.join(args.data, 'val')
+        normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],
+                                     std=[0.229, 0.224, 0.225])
+        train_dataset = datasets.ImageFolder(
+            traindir,
+            transforms.Compose([
+                transforms.RandomResizedCrop(224),
+                transforms.RandomHorizontalFlip(),
+                transforms.ToTensor(),
+                normalize,
+            ]))
+        val_dataset = datasets.ImageFolder(
+            valdir,
+            transforms.Compose([
+                transforms.Resize(256),
+                transforms.CenterCrop(224),
+                transforms.ToTensor(),
+                normalize,
+            ]))
+
+    if args.distributed:
+        train_sampler = torch.utils.data.distributed.DistributedSampler(train_dataset)
+        val_sampler = torch.utils.data.distributed.DistributedSampler(val_dataset, shuffle=False, drop_last=True)
+    else:
+        train_sampler = None
+        val_sampler = None
+
+    # [watch out] The pin memory is default enabled on CUDA for now in torch.
+    train_loader = torch.utils.data.DataLoader(
+        train_dataset, batch_size=args.batch_size, shuffle=(train_sampler is None),
+        num_workers=args.workers, pin_memory=True, pin_memory_device="xpu", sampler=train_sampler, drop_last=True)
+
+    val_loader = torch.utils.data.DataLoader(
+        val_dataset, batch_size=args.batch_size, shuffle=False,
+        num_workers=args.workers, pin_memory=True, pin_memory_device="xpu", sampler=val_sampler)
+
+    # Profiling
+    profiling = os.environ.get("PROFILE", "OFF").upper() in ["1", "Y", "ON", "YES", "TRUE"]
+
+    use_autocast = False
+    if args.bf16 or args.fp16:
+        print('using autocast')
+        use_autocast = True
+
+    if args.evaluate:
+        if args.int8:
+            # calibration dataloader
+            val_loader_calib = torch.utils.data.DataLoader(
+                val_dataset, batch_size=args.calib_bs, shuffle=False,
+                num_workers=args.workers, pin_memory=True, pin_memory_device="xpu")
+
+            # do calibration and return quant model
+            if args.load:
+                model_calib = model
+            else:
+                model_calib = jit_calib(model, val_loader_calib, args)
+            if args.save:
+                torch.jit.save(model_calib, args.save)
+            val_loader_inf = torch.utils.data.DataLoader(
+                val_dataset, batch_size=args.batch_size, shuffle=False,
+                num_workers=args.workers, pin_memory=True, pin_memory_device="xpu")
+
+            print('doing int8 inference')
+            validate_quantization(val_loader_inf, model_calib, criterion, profiling, args)
+        else:
+            # epoch pass 0
+            validate(val_loader, model, criterion, 0, profiling, use_autocast, args)
+        return
+
+    global_start_time = time.time()
+
+    # warm up for convergence
+    if args.converge and not args.resume and args.warm_up_epoch > 0:
+        warm_up_epoch = args.warm_up_epoch
+        warm_up_portion = args.lr / float(warm_up_epoch)
+        for epoch in range(0, warm_up_epoch):
+            if args.lars == False:
+                optimizer.param_groups[0]['lr'] = (epoch + 1) * warm_up_portion
+            train(train_loader, model, criterion, optimizer, epoch, profiling, use_autocast, args, mode='warming')
+            print('Warmup [', (epoch + 1), '][', warm_up_epoch, '] lr = ', optimizer.param_groups[0]['lr'])
+
+    last_acc = best_acc1
+    for epoch in range(args.start_epoch, args.epochs):
+        epoch_start_time = time.time()
+
+        global global_lr
+        global_lr = optimizer.param_groups[0]['lr']
+
+        if not args.distributed or (args.distributed and args.rank == 0):
+            print('[info] Epoch[', epoch, '] start time = ', time.asctime(time.localtime(epoch_start_time)))
+
+        if args.distributed:
+            train_sampler.set_epoch(epoch)
+
+        if epoch == args.last_step_boundary and args.lr_scheduler == 'step':
+            optimizer.param_groups[0]['lr'] *= args.step_gamma
+
+        if not args.distributed or (args.distributed and args.rank == 0):
+            print('[info] Epoch[', epoch, '] lr = ', optimizer.param_groups[0]['lr'])
+
+        # train for one epoch
+        train(train_loader, model, criterion, optimizer, epoch, profiling, use_autocast, args, mode='training')
+
+        # evaluate on validation set
+        acc1 = last_acc
+        if (epoch >= args.eval_start_epoch) and (epoch % args.eval_period == args.eval_offset):
+            print('epoch: ', epoch, ' is doing evaluation')
+            acc1 = validate(val_loader, model, criterion, epoch, profiling, use_autocast, args)
+            last_acc = acc1
+
+        # update the LR
+        if args.lars == False:
+            scheduler.step()
+
+        # remember best acc@1 and save checkpoint
+        is_best = acc1 > best_acc1
+        best_acc1 = max(acc1, best_acc1)
+
+        if not args.skip_checkpoint and \
+            (not args.multiprocessing_distributed or (args.multiprocessing_distributed and args.rank % ngpus_per_node == 0)):
+            save_checkpoint(state={
+                'epoch': epoch + 1,
+                'arch': args.arch,
+                'state_dict': model.state_dict(),
+                'best_acc1': best_acc1,
+                'optimizer' : optimizer.state_dict(),
+                'scheduler' : scheduler.state_dict()
+            }, is_best=is_best, args=args)
+
+        # show info
+        if not args.distributed or (args.distributed and args.rank == 0):
+            epoch_end_time = time.time()
+            print('[info] Epoch[', epoch, '] end time = ', time.asctime(time.localtime(epoch_end_time)))
+            print('[info] Epoch[', epoch, '] consume time = ', ((epoch_end_time - epoch_start_time) / 3600.0), ' hours')
+
+        if converged:
+            break
+
+    if not args.distributed or (args.distributed and args.rank == 0):
+        global_end_time = time.time()
+        print('[info] Global start time = ', time.asctime(time.localtime(global_start_time)))
+        print('[info] Global end time = ', time.asctime(time.localtime(global_end_time)))
+        print('[info] Global consume time = ', ((global_end_time - global_start_time) / (3600.0)), ' hours')
+        if converged:
+            print('[Successful] Reach convergence, final top1 acc: ', final_top1_acc)
+            print('[Successful] Reach convergence, final top5 acc: ', final_top5_acc)
+        else:
+            print('[Failed] Miss convergence')
+
+    if args.converge and not args.skip_tensorboard:
+        if not args.distributed or (args.distributed and args.rank == 0):
+            writer.close()
+
+def train(train_loader, model, criterion, optimizer, epoch, profiling, use_autocast, args, mode='training'):
+    batch_time = AverageMeter('Time', ':6.3f')
+    data_time = AverageMeter('Data', ':6.3f')
+    losses = AverageMeter('Loss', ':.4e')
+    top1 = AverageMeter('Acc@1', ':6.2f')
+    top5 = AverageMeter('Acc@5', ':6.2f')
+    progress = ProgressMeter(
+        len(train_loader),
+        [batch_time, data_time, losses, top1, top5],
+        prefix="Epoch: [{}]".format(epoch))
+    global global_num_iter
+
+    def one_iter(model, images):
+        with torch.xpu.amp.autocast(enabled=use_autocast, dtype=torch.bfloat16):
+            # compute output
+            output = model(images)
+            loss = criterion(output, target)
+        optimizer.zero_grad(set_to_none=True)
+        loss.backward()
+        if args.lars:
+            # Update LR for Lars
+            MLPerfLRScheduler(optimizer, global_num_iter, len(train_loader), args)
+        optimizer.step()
+        return [output, loss]
+
+    # switch to train mode
+    model.train()
+
+    if args.dynamo:
+        train_opt = torch.compile(one_iter, backend="inductor")
+    else:
+        train_opt = one_iter
+
+    # record time
+    duration_total = 0.0
+    warmup_iter = 5
+
+    # config profiler
+    import contextlib
+    def profiler_setup(profiling=False, *args, **kwargs):
+        if profiling:
+            return torch.profiler.profile(*args, **kwargs)
+        else:
+            return contextlib.nullcontext()
+    activities = [torch.profiler.ProfilerActivity.CPU]
+    if args.xpu is not None:
+        activities.append(torch.profiler.ProfilerActivity.XPU)
+    elif args.gpu is not None:
+        activities.append(torch.profiler.ProfilerActivity.CUDA)
+    # fixed wait iters = 1, active iters = 1, warmup_iters = 3, at_least 5 iters
+    # more iters will be skipped, and repeat will be fixed to 1
+    num_iters = args.num_iterations if args.num_iterations else len(train_loader)
+    skip_iters = max(num_iters - 5, 0)
+    schedule = torch.profiler.schedule(skip_first=skip_iters,
+                                       wait=1, warmup=3, active=1)
+    def trace_handle(prof):
+        profile_name = 'fp32'
+        if args.fp16:
+            profile_name = 'fp16'
+        elif args.bf16:
+            profile_name = 'bf16'
+        if args.distributed:
+            profile_name += '.xpu.' + str(args.rank)
+        if args.xpu is not None:
+            torch.save(prof.key_averages().table(sort_by="self_xpu_time_total"), './profiling.' + profile_name + '.train.pt')
+        elif args.gpu is not None:
+            torch.save(prof.key_averages().table(sort_by="self_cuda_time_total"), './profiling.card.' + str(args.xpu) + '.pt')
+        else:
+            torch.save(prof.key_averages().table(sort_by="self_cpu_time_total"), './profiling.card.' + str(args.xpu) + '.pt')
+
+
+    # start profiler, or none while profiling is false
+    with profiler_setup(profiling, activities=activities, schedule=schedule, on_trace_ready=trace_handle) as prof:
+        data_start = time.time()
+        for i, (images, target) in enumerate(train_loader):
+            global_num_iter +=1
+            # measure data loading time
+            data_time.update(time.time() - data_start)
+
+            if args.channels_last:
+                print('input to channels last')
+                images = images.to(memory_format=torch.channels_last)
+
+
+            if args.xpu is not None:
+                try:
+                    import memory_check
+                    memory_check.display_mem("xpu:0")
+                except:
+                    pass
+                start_time = time.time()
+                images = images.to(args.xpu, non_blocking=True)
+                target = target.to(args.xpu, non_blocking=True)
+
+                [output, loss] = train_opt(model, images)
+                # D2H
+                loss = loss.cpu()
+                output = output.cpu()
+                target = target.cpu()
+
+            else:
+                start_time = time.time()
+                activities = None
+                prof_sort = None
+
+                if args.gpu is not None:
+                    images = images.cuda(args.gpu, non_blocking=True)
+                    target = target.cuda(args.gpu, non_blocking=True)
+
+                # compute output
+                output = model(images)
+                loss = criterion(output, target)
+
+                # compute gradient and do SGD step
+                optimizer.zero_grad()
+                loss.backward()
+                optimizer.step()
+
+                # D2H, as sync
+                if args.gpu is not None:
+                    loss = loss.cpu()
+                    output = output.cpu()
+                    target = target.cpu()
+
+            if profiling:
+                prof.step()
+
+            # measure elapsed time
+            duration_train = time.time() - start_time
+            batch_time.update(duration_train)
+
+            # measure accuracy and record loss
+            acc1, acc5 = accuracy(output, target, topk=(1, 5))
+            losses.update(loss.item(), images.size(0))
+            top1.update(acc1[0], images.size(0))
+            top5.update(acc5[0], images.size(0))
+
+            if i % args.print_freq == 0:
+                progress.display(i + 1)
+
+            # exclude first iteration for calculating througput
+            if i >= warmup_iter:
+                duration_total += duration_train
+            data_start = time.time()
+
+            if i == (args.num_iterations - 1) and args.num_iterations >= warmup_iter:
+                print('Training performance: batch size:%d, throughput:%.2f image/sec'
+                      % (args.batch_size, (args.batch_size / (duration_total / (args.num_iterations - warmup_iter)))))
+                sys.exit(0)
+            elif args.num_iterations == 0 and i == len(train_loader) - 1:
+                print('Training performance: batch size:%d, throughput:%.2f image/sec'
+                      % (args.batch_size, (args.batch_size / (duration_total / (len(train_loader) - warmup_iter)))))
+                if args.converge is None:
+                    sys.exit(0)
+
+    if args.converge and not args.skip_tensorboard and mode == 'training':
+        global tensorboard_data
+        tensorboard_data['epoch'] = epoch
+        tensorboard_data['train']['loss'] = losses.avg
+        tensorboard_data['train']['top1'] = top1.avg
+        tensorboard_data['train']['top5'] = top5.avg
+
+def validate(val_loader, model, criterion, epoch, profiling, use_autocast, args):
+    from torch._inductor import config
+
+    def compile_model(model, val_loader):
+        print("====Before compile model====")
+        compiled_model = torch.compile(model, backend="inductor", options={"freezing": True})
+        return compiled_model
+    def run_validate(loader, model, autocast_dtype, base_progress=0):
+
+        # record time
+        duration_total = 0.0
+        warmup_iter = 5
+
+        # config profiler
+        import contextlib
+        def profiler_setup(profiling=False, *args, **kwargs):
+            if profiling:
+                return torch.profiler.profile(*args, **kwargs)
+            else:
+                return contextlib.nullcontext()
+        activities = [torch.profiler.ProfilerActivity.CPU]
+        if args.xpu is not None:
+            activities.append(torch.profiler.ProfilerActivity.XPU)
+        elif args.gpu is not None:
+            activities.append(torch.profiler.ProfilerActivity.CUDA)
+        # fixed wait iters = 1, active iters = 1, warmup_iters = 3, at_least 5 iters
+        # more iters will be skipped, and repeat will be fixed to 1
+        num_iters = args.num_iterations if args.num_iterations else len(loader)
+        skip_iters = max(num_iters - 5, 0)
+        schedule = torch.profiler.schedule(skip_first=skip_iters,
+                                           wait=1, warmup=3, active=1)
+        def trace_handle(prof):
+            profile_name = 'fp32'
+            if args.fp16:
+                profile_name = 'fp16'
+            elif args.bf16:
+                profile_name = 'bf16'
+            if args.distributed:
+                profile_name += '.xpu.' + str(args.rank)
+            if args.xpu is not None:
+                print(prof.key_averages().table(sort_by="self_xpu_time_total"))
+                torch.save(prof.key_averages().table(sort_by="self_xpu_time_total"), './profiling.' + profile_name + '.inf.pt')
+            elif args.gpu is not None:
+                print(prof.key_averages().table(sort_by="self_cuda_time_total"))
+                torch.save(prof.key_averages().table(sort_by="self_cuda_time_total"), './profiling.card.' + str(args.xpu) + '.pt')
+            else:
+                print(prof.key_averages().table(sort_by="self_cpu_time_total"))
+                torch.save(prof.key_averages().table(sort_by="self_cpu_time_total"), './profiling.card.' + str(args.xpu) + '.pt')
+
+        # start profiler, or none while profiling is false
+        with profiler_setup(profiling, activities=activities, schedule=schedule, on_trace_ready=trace_handle) as prof, torch.no_grad():
+            for i, (images, target) in enumerate(loader):
+                i = base_progress + i
+
+                if args.channels_last:
+                    images = images.to(memory_format=torch.channels_last)
+                    print('images convert to channels last')
+
+
+                if args.xpu:
+                    try:
+                        import memory_check
+                        memory_check.display_mem("xpu:0")
+                    except:
+                        pass
+                    start_time = time.time()
+                    images = images.to(args.xpu, non_blocking=True)
+
+                    if args.jit_trace:
+                        # compute output
+                        output = model(images)
+                    elif args.dynamo:
+                        with torch.xpu.amp.autocast(enabled=use_autocast, dtype=autocast_dtype):
+                            output = model(images)
+                    else:
+                        with torch.xpu.amp.autocast(enabled=use_autocast, dtype=autocast_dtype):
+                            # compute output
+                            output = model(images)
+
+                    # sync for time measurement
+                    if not args.converge:
+                        torch.xpu.synchronize(args.xpu)
+
+                else:
+                    start_time = time.time()
+                    activities = None
+                    prof_sort = None
+                    if args.gpu is not None:
+                        images = images.cuda(args.gpu, non_blocking=True)
+
+                    # compute output
+                    output = model(images)
+
+                    # sync for time measurement
+                    if args.gpu is not None:
+                        torch.cuda.synchronize(args.gpu)
+
+                if profiling:
+                    prof.step()
+
+                # D2H
+                output = output.cpu()
+
+                # measure elapsed time
+                duration_eval = time.time() - start_time
+                batch_time.update(duration_eval)
+
+                loss = criterion(output.float(), target)
+
+                # measure accuracy and record loss
+                acc1, acc5 = accuracy(output.float(), target, topk=(1, 5))
+                losses.update(loss.item(), images.size(0))
+                top1.update(acc1[0], images.size(0))
+                top5.update(acc5[0], images.size(0))
+
+                if i % args.print_freq == 0:
+                    progress.display(i + 1)
+
+                # exclude first iteration for calculating througput
+                if i >= warmup_iter and not (args.num_iterations == 0 and i == len(val_loader) - 1):
+                    duration_total += duration_eval
+
+                if (not args.num_iterations == 0) and (args.num_iterations <= warmup_iter):
+                    print("At least ", warmup_iter, " iteartions required for performance measure")
+                    sys.exit(0)
+
+                if i == (args.num_iterations - 1) and args.num_iterations >= warmup_iter:
+                    print('Evalution performance: batch size:%d, throughput:%.2f image/sec, Acc@1:%.2f, Acc@5:%.2f'
+                        % (args.batch_size, (args.batch_size / (duration_total / (args.num_iterations - warmup_iter))), top1.avg, top5.avg))
+                    sys.exit(0)
+                elif args.num_iterations == 0 and i == len(val_loader) - 1:
+                    if args.converge and args.distributed:
+                        top1.all_reduce()
+                        top5.all_reduce()
+                    print('Evalution performance: batch size:%d, throughput:%.2f image/sec, Acc@1:%.2f, Acc@5:%.2f'
+                        % (args.batch_size, (args.batch_size / (duration_total / (len(val_loader) - warmup_iter))), top1.avg, top5.avg))
+                    if args.converge:
+                        global final_top1_acc
+                        global final_top5_acc
+                        global converged
+                        final_top1_acc = top1.avg
+                        final_top5_acc = top5.avg
+                        if final_top1_acc >= args.sota_target:
+                            converged = True
+                    else:
+                        sys.exit(0)
+
+    batch_time = AverageMeter('Time', ':6.3f', Summary.NONE)
+    losses = AverageMeter('Loss', ':.4e', Summary.NONE)
+    top1 = AverageMeter('Acc@1', ':6.2f', Summary.AVERAGE)
+    top5 = AverageMeter('Acc@5', ':6.2f', Summary.AVERAGE)
+    progress = ProgressMeter(
+        len(val_loader) + (args.distributed and (len(val_loader.sampler) * args.world_size < len(val_loader.dataset))),
+        [batch_time, losses, top1, top5],
+        prefix='Test: ')
+
+    # switch to evaluate mode
+    model.eval()
+
+    autocast_dtype = torch.float32
+    if args.fp16:
+        autocast_dtype = torch.float16
+    elif args.bf16:
+        autocast_dtype = torch.bfloat16
+
+    if args.jit_trace and not args.load:
+        trace_input = torch.randn(args.batch_size, 3, 224, 224).to(args.xpu)
+        print('jit trace')
+        with torch.xpu.amp.autocast(enabled=use_autocast, dtype=autocast_dtype, cache_enabled=False):
+            model = torch.jit.trace(model, trace_input)
+    elif args.dynamo:
+        model = compile_model(model, val_loader)
+
+
+    if args.save:
+        if args.jit_trace:
+            torch.jit.save(model, args.save)
+        else:
+            torch.save(model, args.save)
+    run_validate(val_loader, model, autocast_dtype)
+
+    progress.display_summary()
+
+    if args.converge and not args.skip_tensorboard:
+        global tensorboard_data
+        tensorboard_data['eval']['loss'] = losses.avg
+        tensorboard_data['eval']['top1'] = final_top1_acc
+        tensorboard_data['eval']['top5'] = final_top5_acc
+        draw_tensorboard(args)
+
+    if args.distributed:
+        return final_top1_acc
+    else:
+        return top1.avg
+
+def validate_quantization(val_loader, model, criterion, profiling, args):
+    batch_time = AverageMeter('Time', ':6.3f', Summary.NONE)
+    losses = AverageMeter('Loss', ':.4e', Summary.NONE)
+    top1 = AverageMeter('Acc@1', ':6.2f', Summary.AVERAGE)
+    top5 = AverageMeter('Acc@5', ':6.2f', Summary.AVERAGE)
+    progress = ProgressMeter(
+        len(val_loader) + (args.distributed and (len(val_loader.sampler) * args.world_size < len(val_loader.dataset))),
+        [batch_time, losses, top1, top5],
+        prefix='Test: ')
+
+    # switch to evaluate mode
+    model.eval()
+
+    # record time
+    duration_total = 0.0
+
+    # TODO: will fix non blocking H2D perf regression, then remove here option
+    non_blocking = False
+    if args.non_blocking:
+        non_blocking = True
+
+    # config profiler
+    import contextlib
+    def profiler_setup(profiling=False, *args, **kwargs):
+        if profiling:
+            return torch.profiler.profile(*args, **kwargs)
+        else:
+            return contextlib.nullcontext()
+    activities = [torch.profiler.ProfilerActivity.CPU]
+    if args.xpu is not None:
+        activities.append(torch.profiler.ProfilerActivity.XPU)
+    elif args.gpu is not None:
+        activities.append(torch.profiler.ProfilerActivity.CUDA)
+    # fixed wait iters = 1, active iters = 1, warmup_iters = 3, at_least 5 iters
+    # more iters will be skipped, and repeat will be fixed to 1
+    num_iters = args.num_iterations if args.num_iterations else len(val_loader)
+    skip_iters = max(num_iters - 5, 0)
+    schedule = torch.profiler.schedule(skip_first=skip_iters,
+                                       wait=1, warmup=3, active=1)
+    def trace_handle(prof):
+        if args.xpu is not None:
+            torch.save(prof.key_averages().table(sort_by="self_xpu_time_total"), './profiling.int8.inf.pt')
+
+    # start profiler, or none while profiling is false
+    with profiler_setup(profiling, activities=activities, schedule=schedule, on_trace_ready=trace_handle) as prof, torch.inference_mode():
+        for i, (images, target) in enumerate(val_loader):
+            if args.xpu is not None and args.benchmark == 1:
+                images = images.to(args.xpu, non_blocking = non_blocking)
+
+            try:
+                import memory_check
+                memory_check.display_mem("xpu:0")
+            except:
+                pass
+            start = time.time()
+            if args.xpu is not None and args.benchmark == 0:
+                images = images.to(args.xpu, non_blocking = non_blocking)
+            if args.channels_last:
+                images = images.to(memory_format=torch.channels_last)
+
+            # compute output
+            output = model(images)
+
+            # D2H
+            output = output.to("cpu")
+
+            # sync for time measurement
+            torch.xpu.synchronize(args.xpu)
+
+            # measure elapsed time
+            end = time.time()
+            batch_time.update(end - start)
+            duration_eval = end - start
+
+            loss = criterion(output, target)
+
+            if profiling:
+                prof.step()
+
+            # measure accuracy and record loss
+            acc1, acc5 = accuracy(output, target, topk=(1, 5))
+            losses.update(loss.item(), images.size(0))
+            top1.update(acc1[0], images.size(0))
+            top5.update(acc5[0], images.size(0))
+
+            if i % args.print_freq == 0:
+                progress.display(i + 1)
+
+            # exclude first iteration for calculating througput
+            perf_start_iter = 1
+            if args.benchmark == 1 and args.num_iterations >= 500:
+                perf_start_iter = math.floor(args.num_iterations * 0.7)
+            if i >= perf_start_iter:
+                duration_total += duration_eval
+
+            if i == (args.num_iterations - 1) and args.num_iterations >= 2:
+                print('Quantization Evalution performance: batch size:%d, throughput:%.2f image/sec, Acc@1:%.2f, Acc@5:%.2f'
+                    % (args.batch_size, (args.batch_size / (duration_total / (args.num_iterations - perf_start_iter))), top1.avg, top5.avg))
+                sys.exit(0)
+            elif args.num_iterations == 0 and i == len(val_loader) - 1:
+                print('Quantization Evalution performance: batch size:%d, throughput:%.2f image/sec, Acc@1:%.2f, Acc@5:%.2f'
+                    % (args.batch_size, (args.batch_size / (duration_total / (len(val_loader) - 2))), top1.avg, top5.avg))
+                sys.exit(0)
+
+        progress.display_summary()
+
+    return top1.avg
+
+def save_checkpoint(state, is_best, args, filename='checkpoint.pth.tar'):
+    if args.distributed:
+        print('rank ', str(args.rank), ' saving checkpoint')
+        filename = 'checkpoint.rank.' + str(args.rank) + '.pth.tar'
+    else:
+        print('single tile saving checkpoint')
+    torch.save(state, filename)
+
+    if is_best:
+        shutil.copyfile(filename, 'model_best.pth.tar')
+
+def qnormalize(tensor, mean, std, scl):
+    dtype = tensor.dtype
+    mean = torch.as_tensor(mean, dtype=dtype, device=tensor.device)
+    std = torch.as_tensor(std, dtype=dtype, device=tensor.device)
+    if mean.ndim == 1:
+        mean = mean[:, None, None]
+    if std.ndim == 1:
+        std = std[:, None, None]
+
+    tensor.sub_(mean).div_(std)#.mul_(scale)#tensor.sub_(255 * mean).mul(128/255*(1/(1-0.406)))
+    out = torch.quantize_per_tensor(tensor, scale=scl, zero_point=0, dtype=torch.qint8)
+    return out
+
+def compute_scale(val_loader_com):
+    for i, (input, target) in enumerate(val_loader_com):
+        scale = 1.0 / (128 / torch.max(input))
+        return scale
+
+class QNormalize(object):
+    def __init__(self, mean, std, scale):
+        self.mean = mean
+        self.std = std
+        self.scale = scale
+
+    def __call__(self, tensor):
+        return qnormalize(tensor, self.mean, self.std, self.scale)
+
+class Summary(Enum):
+    NONE = 0
+    AVERAGE = 1
+    SUM = 2
+    COUNT = 3
+
+class AverageMeter(object):
+    """Computes and stores the average and current value"""
+    def __init__(self, name, fmt=':f', summary_type=Summary.AVERAGE):
+        self.name = name
+        self.fmt = fmt
+        self.summary_type = summary_type
+        self.reset()
+
+    def reset(self):
+        self.val = 0
+        self.avg = 0
+        self.sum = 0
+        self.count = 0
+
+    def update(self, val, n=1):
+        self.val = val
+        self.sum += val * n
+        self.count += n
+        self.avg = self.sum / self.count
+
+    def all_reduce(self):
+        total = torch.FloatTensor([self.sum, self.count])
+        dist.all_reduce(total, dist.ReduceOp.SUM, async_op=False)
+        self.sum, self.count = total.tolist()
+        self.avg = self.sum / self.count
+
+    def __str__(self):
+        fmtstr = '{name} {val' + self.fmt + '} ({avg' + self.fmt + '})'
+        return fmtstr.format(**self.__dict__)
+
+    def summary(self):
+        fmtstr = ''
+        if self.summary_type is Summary.NONE:
+            fmtstr = ''
+        elif self.summary_type is Summary.AVERAGE:
+            fmtstr = '{name} {avg:.3f}'
+        elif self.summary_type is Summary.SUM:
+            fmtstr = '{name} {sum:.3f}'
+        elif self.summary_type is Summary.COUNT:
+            fmtstr = '{name} {count:.3f}'
+        else:
+            raise ValueError('invalid summary type %r' % self.summary_type)
+
+        return fmtstr.format(**self.__dict__)
+
+class ProgressMeter(object):
+    def __init__(self, num_batches, meters, prefix=""):
+        self.batch_fmtstr = self._get_batch_fmtstr(num_batches)
+        self.meters = meters
+        self.prefix = prefix
+
+    def display(self, batch):
+        entries = [self.prefix + self.batch_fmtstr.format(batch)]
+        entries += [str(meter) for meter in self.meters]
+        print('\t'.join(entries))
+
+    def display_summary(self):
+        entries = [" *"]
+        entries += [meter.summary() for meter in self.meters]
+        print(' '.join(entries))
+
+    def _get_batch_fmtstr(self, num_batches):
+        num_digits = len(str(num_batches // 1))
+        fmt = '{:' + str(num_digits) + 'd}'
+        return '[' + fmt + '/' + fmt.format(num_batches) + ']'
+
+def accuracy(output, target, topk=(1,)):
+    """Computes the accuracy over the k top predictions for the specified values of k"""
+    with torch.no_grad():
+        maxk = max(topk)
+        batch_size = target.size(0)
+
+        _, pred = output.topk(maxk, 1, True, True)
+        pred = pred.t()
+        pred = pred.cpu()
+        correct = pred.eq(target.view(1, -1).expand_as(pred))
+
+        res = []
+        for k in topk:
+            correct_k = correct[:k].reshape(-1).float().sum(0, keepdim=True)
+            res.append(correct_k.mul_(100.0 / batch_size))
+        return res
+
+def MLPerfLRScheduler(optimizer, step, iteration, args):
+    global global_lr
+    warmup_iter = args.warm_up_epoch * iteration
+    decay_steps = args.decay_epochs * iteration
+    power = 2
+    if step <= warmup_iter:
+       lr_rate = args.lr * (step / warmup_iter)
+    else:
+       lr_step = min((step - warmup_iter), decay_steps)
+       lr_rate = ((args.lr - args.end_lr) * (1-(lr_step/decay_steps)) ** power) + args.end_lr
+    global_lr = lr_rate
+    optimizer.param_groups[0]['lr'] = global_lr
+
+def draw_tensorboard(args):
+    global tensorboard_data
+    global global_lr
+    if not args.distributed or (args.distributed and args.rank == 0):
+        epoch = tensorboard_data['epoch']
+
+        train_loss = tensorboard_data['train']['loss']
+        eval_loss = tensorboard_data['eval']['loss']
+
+        train_top1 = tensorboard_data['train']['top1']
+        eval_top1 = tensorboard_data['eval']['top1']
+
+        train_top5 = tensorboard_data['train']['top5']
+        eval_top5 = tensorboard_data['eval']['top5']
+
+        writer.add_scalars('top1 acc', {'train acc': train_top1, 'eval acc': eval_top1}, epoch)
+        writer.add_scalars('top5 acc', {'train acc': train_top5, 'eval acc': eval_top5}, epoch)
+        writer.add_scalar('learning rate', global_lr, epoch)
+        writer.add_scalars('loss value', {'train loss': train_loss, 'eval loss': eval_loss}, epoch)
+
+if __name__ == '__main__':
+    main()
diff --git a/models_v2/pytorch/resnet50v1_5/training/gpu/requirements.txt b/models_v2/pytorch/resnet50v1_5/training/gpu/requirements.txt
new file mode 100644
index 000000000..3868fb16b
--- /dev/null
+++ b/models_v2/pytorch/resnet50v1_5/training/gpu/requirements.txt
@@ -0,0 +1 @@
+pillow
diff --git a/models_v2/pytorch/resnet50v1_5/training/gpu/resnet50.py b/models_v2/pytorch/resnet50v1_5/training/gpu/resnet50.py
new file mode 100644
index 000000000..606bef621
--- /dev/null
+++ b/models_v2/pytorch/resnet50v1_5/training/gpu/resnet50.py
@@ -0,0 +1,35 @@
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Consts for resnet50 inference."""
+
+ACC = {
+    "type": "total",
+    "pattern": r" * Acc@1:(\d+.\d+)",
+    "unit": "acc@1",
+}
+
+PERF = {
+    "type": "total",
+    "pattern": r"throughput:(\d+.\d+) image/sec",
+    "inverse": False,
+    "multiply": False,
+    "use_batch_size": False,
+    "unit": "fps",
+}
+
+FUNCTIONAL = {
+    "pattern": r"Evalution performance: batch size:\d+, throughput:\d+.\d+ image/sec"
+}
diff --git a/models_v2/pytorch/resnet50v1_5/training/gpu/run_model.sh b/models_v2/pytorch/resnet50v1_5/training/gpu/run_model.sh
new file mode 100755
index 000000000..c80880ab7
--- /dev/null
+++ b/models_v2/pytorch/resnet50v1_5/training/gpu/run_model.sh
@@ -0,0 +1,129 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+
+#!/bin/bash
+
+# Create an array of input directories that are expected and then verify that they exist
+declare -A input_envs
+input_envs[DATASET_DIR]=${DATASET_DIR}
+input_envs[MULTI_TILE]=${MULTI_TILE}
+input_envs[PLATFORM]=${PLATFORM}
+input_envs[OUTPUT_DIR]=${OUTPUT_DIR}
+
+for i in "${!input_envs[@]}"; do
+  var_name=$i
+  env_param=${input_envs[$i]}
+
+  if [[ -z $env_param ]]; then
+    echo "The required environment variable $var_name is not set" >&2
+    exit 1
+  fi
+done
+
+if [[ "${PLATFORM}" == "Max" ]]; then
+    BATCH_SIZE=${BATCH_SIZE:-256}
+    PRECISION=${PRECISION:-BF16}
+    NUM_ITERATIONS=${NUM_ITERATIONS:-20}
+elif [[ "${PLATFORM}" == "ARC" ]]; then
+    if [[ "${MULTI_TILE}" == "True" ]]; then
+        echo "ARC not support multitile"
+        exit 1
+    fi
+    BATCH_SIZE=${BATCH_SIZE:-256}
+    PRECISION=${PRECISION:-BF16}
+    NUM_ITERATIONS=${NUM_ITERATIONS:-20}
+
+fi
+
+if [[ "${PRECISION}" == "BF16" ]]; then
+    flag="--bf16 1 "
+elif [[ "${PRECISION}" == "FP32" ]]; then
+    flag=""
+elif [[ "${PRECISION}" == "TF32" ]]; then
+    flag="--tf32 1 "
+else
+    echo -e "Invalid input! Only BF16 FP32 TF32 are supported."
+    exit 1
+fi
+
+echo "resnet50 ${PRECISION} training MultiTile=${MULTI_TILE} BS=${BATCH_SIZE} Iter=${NUM_ITERATIONS}"
+
+
+if [[ ! -d "${DATASET_DIR}" ]] && [[ "${MULTI_TILE}" != "True" ]]; then
+    echo "The DATASET_DIR '${DATASET_DIR}' does not exist"
+    exit 1
+fi
+
+echo 'Running with parameters:'
+echo " PLATFORM: ${PLATFORM}"
+echo " DATASET_PATH: ${DATASET_DIR}"
+echo " OUTPUT_DIR: ${OUTPUT_DIR}"
+echo " PRECISION: ${PRECISION}"
+echo " BATCH_SIZE: ${BATCH_SIZE}"
+echo " NUM_ITERATIONS: ${NUM_ITERATIONS}"
+echo " MULTI_TILE: ${MULTI_TILE}"
+
+# Create the output directory, if it doesn't already exist
+mkdir -p $OUTPUT_DIR
+
+modelname=resnet50
+
+if [[ ${MULTI_TILE} == "False" ]]; then
+    rm ${OUTPUT_DIR}/${modelname}_${PRECISION}_train_t0_raw.log
+    python main.py \
+        -a resnet50 \
+        -b ${BATCH_SIZE} \
+        --xpu 0 \
+        ${DATASET_DIR} \
+        --num-iterations ${NUM_ITERATIONS} \
+        $flag 2>&1 | tee ${OUTPUT_DIR}/${modelname}_${PRECISION}_train_t0_raw.log
+    python common/parse_result.py -m $modelname -l ${OUTPUT_DIR}/${modelname}_${PRECISION}_train_t0_raw.log -b ${BATCH_SIZE}
+    throughput=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_train_t0.log | grep Performance | awk -F ' ' '{print $2}')
+    throughput_unit=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_train_t0.log | grep Performance | awk -F ' ' '{print $3}')
+    latency=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_train_t0.log | grep Latency | awk -F ' ' '{print $2}')
+    acc=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_train_t0.log | grep Accuracy | awk -F ' ' '{print $3}')
+    acc_unit=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_train_t0.log | grep Accuracy | awk -F ' ' '{print $2}')
+else
+    rm ${OUTPUT_DIR}/${modelname}_${PRECISION}_train_raw.log
+    mpiexec -np 2 -ppn 2 --prepend-rank python -u main.py -a resnet50 -b ${BATCH_SIZE} --xpu 0 --dummy --num-iterations ${NUM_ITERATIONS} --bucket-cap 200 --disable-broadcast-buffers ${flag} --large-first-bucket --use-gradient-as-bucket-view --seed 123 2>&1 | tee ${OUTPUT_DIR}/ddp-${modelname}_${PRECISION}_train_raw.log 
+    python common/parse_result.py -m $modelname --ddp -l ${OUTPUT_DIR}/ddp-${modelname}_${PRECISION}_train_raw.log -b ${BATCH_SIZE}
+    throughput=$(cat ${OUTPUT_DIR}/ddp-${modelname}_${PRECISION}_train.log | grep "Sum Performance" | awk -F ' ' '{print $3}')
+    throughput_unit=$(cat ${OUTPUT_DIR}/ddp-${modelname}_${PRECISION}_train.log | grep "Sum Performance" | awk -F ' ' '{print $4}')
+    latency=$(cat ${OUTPUT_DIR}/ddp-${modelname}_${PRECISION}_train.log | grep Latency | awk -F ' ' '{print $2}')
+    acc=$(cat ${OUTPUT_DIR}/ddp-${modelname}_${PRECISION}_train.log | grep Accuracy | awk -F ' ' '{print $3}')
+    acc_unit=$(cat ${OUTPUT_DIR}/ddp-${modelname}_${PRECISION}_train.log | grep Accuracy | awk -F ' ' '{print $2}')
+fi
+
+yaml_content=$(cat <<EOF
+results:
+ - key: throughput
+   value: $throughput
+   unit: $throughput_unit
+ - key: latency
+   value: $latency
+   unit: s
+ - key: accuracy
+   value: $acc
+   unit: $acc_unit
+EOF
+)
+
+# Write the content to a YAML file
+echo "$yaml_content" >  ${OUTPUT_DIR}/results.yaml
+echo "YAML file created."
diff --git a/models_v2/pytorch/resnet50v1_5/training/gpu/setup.sh b/models_v2/pytorch/resnet50v1_5/training/gpu/setup.sh
new file mode 100755
index 000000000..df70afcb7
--- /dev/null
+++ b/models_v2/pytorch/resnet50v1_5/training/gpu/setup.sh
@@ -0,0 +1,28 @@
+#!/usr/bin/env bash
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# setup.sh
+#  - install OS pkgs
+#  - should create virtual env & install pip  requirement.txt
+#  - git clones & applying patches
+
+set -e
+apt-get update && apt-get install -y python3-venv
+
+pip install -r requirements.txt
+
+cp -r ../../../../common .
diff --git a/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/README.md b/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/README.md
new file mode 100644
index 000000000..0639c2577
--- /dev/null
+++ b/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/README.md
@@ -0,0 +1,110 @@
+# SSD-MobilenetV1 Inference
+
+SSD-MobileNetV1 Inference best known configurations with Intel® Extension for PyTorch.
+
+## Model Information
+
+| **Use Case** | **Framework** | **Model Repo** | **Branch/Commit/Tag** | **Optional Patch** |
+|:---:| :---: |:--------------:|:---------------------:|:------------------:|
+|  Inference   |    PyTorch    |       https://github.com/qfgaohao/pytorch-ssd        |           -           |         -          |
+
+# Pre-Requisite
+* Host has Intel® Data Center GPU Arc
+* Host has installed latest Intel® Data Center GPU Arc Series Drivers https://dgpu-docs.intel.com/driver/installation.html
+* The following Intel® oneAPI Base Toolkit components are required:
+  - Intel® oneAPI DPC++ Compiler (Placeholder DPCPPROOT as its installation path)
+  - Intel® oneAPI Math Kernel Library (oneMKL) (Placeholder MKLROOT as its installation path)
+  - Intel® oneAPI MPI Library
+  - Intel® oneAPI TBB Library
+
+  Follow instructions at [Intel® oneAPI Base Toolkit Download page](https://www.intel.com/content/www/us/en/developer/tools/oneapi/base-toolkit-download.html?operatingsystem=linux) to setup the package manager repository.
+
+# Prepare Dataset
+Download, and extract the following dataset, checkpoint and VOC model labels:
+# Dataset: VOC2007
+Download from URL: [VOCtrainval_06-Nov-2007.tar](http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCtest_06-Nov-2007.tar) and extract, you will find the extract folder as below:
+```
+VOCDevkit/
+└── VOC2007
+    ├── Annotations
+    ├── ImageSets
+    ├── JPEGImages
+    ├── SegmentationClass
+    └── SegmentationObject
+```
+please indicator the `DATASET_DIR` to path/to/your/folder/VOCdevkit/VOC2007
+
+# Pretrained model
+Download form [this URL](https://drive.google.com/drive/folders/1pKn-RifvJGWiOx0ZCRLtCXM5GT5lAluu?usp=sharing) to get mobilenet-v1-ssd-mp-0_675.pth and put under `WEIGHT_DIR`
+
+# Voc-model-labels
+Download from [this URL](https://drive.google.com/drive/folders/1pKn-RifvJGWiOx0ZCRLtCXM5GT5lAluu?usp=sharing) to get voc-model-labels.txt and put under `LABEL_DIR`
+
+
+## Inference
+1. `git clone https://github.com/IntelAI/models.git`
+2. `cd models/models_v2/pytorch/ssd-mobilenetv1/inference/gpu`
+3. Create virtual environment `venv` and activate it:
+    ```
+    python3 -m venv venv
+    . ./venv/bin/activate
+    ```
+4. Run setup.sh
+    ```
+    ./setup.sh
+    ```
+5. Install the latest GPU versions of [torch, torchvision and intel_extension_for_pytorch](https://intel.github.io/intel-extension-for-pytorch/index.html#installation):
+  ```
+  python -m pip install torch==<torch_version> torchvision==<torchvvision_version> intel-extension-for-pytorch==<ipex_version> --extra-index-url https://pytorch-extension.intel.com/release-whl-aitools/
+  ```
+6. Set environment variables for Intel® oneAPI Base Toolkit: 
+    Default installation location `{ONEAPI_ROOT}` is `/opt/intel/oneapi` for root account, `${HOME}/intel/oneapi` for other accounts
+    ```bash
+    source {ONEAPI_ROOT}/compiler/latest/env/vars.sh
+    source {ONEAPI_ROOT}/mkl/latest/env/vars.sh
+    source {ONEAPI_ROOT}/tbb/latest/env/vars.sh
+    source {ONEAPI_ROOT}/mpi/latest/env/vars.sh
+    source {ONEAPI_ROOT}/ccl/latest/env/vars.sh
+    ```
+7. Setup required environment paramaters
+
+| **Parameter**                |                                  **export command**                                  |
+|:---------------------------:|:------------------------------------------------------------------------------------:|
+| **MULTI_TILE**               | `export MULTI_TILE=False` (False)                                             |
+| **PLATFORM**                 | `export PLATFORM=ARC` (ARC)                                                   |
+| **DATASET_DIR**              |                               `export DATASET_DIR=` (the dir path to voc2007 in dataset)|
+| **WEIGHT_DIR**              |                               `export WEIGHT_DIR=` (contain .pth files)                             |
+| **LABEL_DIR**              |                               `export LABEL_DIR=` (contain .txt files)                            |
+| **BATCH_SIZE** (optional)    |                               `export BATCH_SIZE=256`                                |
+| **PRECISION**      |                               `export PRECISION=INT8` (INT8, FP16 and FP32 for ARC)  |
+| **OUTPUT_DIR** (optional)    |                               `export OUTPUT_DIR=$PWD`                               |
+|**NUM_ITERATIONS** (optional) |                               `export NUM_ITERATIONS=500`                             |
+| **DATASET_DIR** (optional)   |                               `export DATASET_DIR=--dummy`                           |
+8. Run `run_model.sh`
+
+## Output
+
+Single-tile output will typically looks like:
+
+```
+process image 500
+Load Image: 0.000002 seconds.
+Inference time:  0.02182316780090332
+Post time:  0.003971099853515625
+Prediction: 0.438544 seconds.
+```
+
+```
+Final results of the inference run can be found in `results.yaml` file.
+```
+results:
+ - key: throughput
+   value: 11721.356
+   unit: inst/s
+ - key: latency
+   value: 0.021840476989746095
+   unit: s
+ - key: accuracy
+   value: None
+   unit: accuracy
+```
diff --git a/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/eval_ssd.py b/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/eval_ssd.py
new file mode 100644
index 000000000..8908c1e05
--- /dev/null
+++ b/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/eval_ssd.py
@@ -0,0 +1,477 @@
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+# MIT License
+#
+# Copyright (c) 2019 Hao Gao
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+#
+import os
+import torch
+import torch.nn as nn
+import intel_extension_for_pytorch
+#from vision.ssd.vgg_ssd import create_vgg_ssd, create_vgg_ssd_predictor
+from vision.ssd.mobilenetv1_ssd import create_mobilenetv1_ssd, create_mobilenetv1_ssd_predictor
+#from vision.ssd.mobilenetv1_ssd_lite import create_mobilenetv1_ssd_lite, create_mobilenetv1_ssd_lite_predictor
+#from vision.ssd.squeezenet_ssd_lite import create_squeezenet_ssd_lite, create_squeezenet_ssd_lite_predictor
+from vision.datasets.voc_dataset import VOCDataset
+from vision.datasets.open_images import OpenImagesDataset
+from vision.utils import box_utils, measurements
+from vision.utils.misc import str2bool, Timer
+import argparse
+import pathlib
+import numpy as np
+import logging
+import sys
+#from vision.ssd.mobilenet_v2_ssd_lite import create_mobilenetv2_ssd_lite, create_mobilenetv2_ssd_lite_predictor
+from vision.ssd.data_preprocessing import PredictionTransform
+from vision.ssd.config import mobilenetv1_ssd_config as config
+from torch.quantization.quantize_jit import (
+convert_jit,
+prepare_jit,
+)
+from torch.quantization import default_qconfig
+from torch.jit._recursive import wrap_cpp_module
+torch._C._jit_set_profiling_mode(False)
+torch._C._jit_set_profiling_executor(False)
+
+
+parser = argparse.ArgumentParser(description="SSD Evaluation on VOC Dataset.")
+parser.add_argument('--net', default="vgg16-ssd",
+                    help="The network architecture, it should be of mb1-ssd, \
+                          mb1-ssd-lite, mb2-ssd-lite or vgg16-ssd.")
+parser.add_argument("--trained_model", type=str)
+parser.add_argument("--dataset_type", default="voc", type=str,
+                    help='Specify dataset type. \
+                          Currently support voc and open_images.')
+parser.add_argument("--dataset", type=str,
+    help="The root directory of the VOC dataset or Open Images dataset.")
+parser.add_argument("--label_file", type=str, help="The label file path.")
+parser.add_argument("--use_cuda", type=str2bool, default=False)
+parser.add_argument("--use_xpu", type=str2bool, default=True)
+parser.add_argument('--fp16', action='store_true', help='Datatype used: fp16')
+parser.add_argument('--bf16', action='store_true', help='Datatype used: bf16')
+parser.add_argument("--use_2007_metric", type=str2bool, default=True)
+parser.add_argument("--nms_method", type=str, default="hard")
+parser.add_argument('--iter_num', default=4951, type=int,
+                    help='number of iteration')
+parser.add_argument("--iou_threshold", type=float, default=0.5,
+                    help="The threshold of Intersection over Union.")
+parser.add_argument("--eval_dir", default="eval_results", type=str,
+                    help="The directory to store evaluation results.")
+parser.add_argument('--mb2_width_mult', default=1.0, type=float,
+                    help='Width Multiplifier for MobilenetV2')
+parser.add_argument("--int8", action='store_true', help='use int8', default=False)
+parser.add_argument('--mc', default=False,
+                    help='do calibration with weight per channel quantization')
+parser.add_argument('--channels_last', action='store_true',
+                    help='Dataformat used: channel_last(plain NHWC)')
+parser.add_argument('--calib_num', default=8, type=int,
+                    help='number of calibration iteration')
+parser.add_argument('--dummy', default=0, type=int,
+                    help='dummy for perf')
+parser.add_argument('--batch_size', default=1, type=int,
+                    help='batch_size for dummy')
+parser.add_argument('--image_size', default=300, type=int,
+                    help='image_size for dummy')
+parser.add_argument('--benchmark', default=0, type=int, help='for benchmark performance, move H2D out of end2end calculation')
+parser.add_argument('--profile_iter', default=5, type=int, help='profile iter')
+parser.add_argument('--num-iterations', default=100, type=int, help='iterations for benchmark test')
+parser.add_argument("--save", help='Path to save entile model')
+parser.add_argument("--load", help='Path to load entile model')
+parser.add_argument('--jit', default=-1, type=int,choices=[-1, 0, 1],
+                     help='Select run with jit or impe path, 0 : impe, 1 : jit')
+parser.add_argument("--asymm", action='store_true', help="Asymm quantization infernce")
+parser.add_argument("--uint8", action='store_true', help="Int8 infernce with uint8")
+args = parser.parse_args()
+
+if args.use_xpu:
+    DEVICE = torch.device("xpu")
+else:
+    DEVICE = torch.device(
+        "cuda:0" if torch.cuda.is_available() and args.use_cuda else "cpu")
+
+
+def group_annotation_by_class(dataset):
+    true_case_stat = {}
+    all_gt_boxes = {}
+    all_difficult_cases = {}
+    for i in range(len(dataset)):
+        image_id, annotation = dataset.get_annotation(i)
+        gt_boxes, classes, is_difficult = annotation
+        gt_boxes = torch.from_numpy(gt_boxes)
+        for i, difficult in enumerate(is_difficult):
+            class_index = int(classes[i])
+            gt_box = gt_boxes[i]
+            if not difficult:
+                true_case_stat[class_index] = true_case_stat.get(
+                    class_index, 0) + 1
+
+            if class_index not in all_gt_boxes:
+                all_gt_boxes[class_index] = {}
+            if image_id not in all_gt_boxes[class_index]:
+                all_gt_boxes[class_index][image_id] = []
+            all_gt_boxes[class_index][image_id].append(gt_box)
+            if class_index not in all_difficult_cases:
+                all_difficult_cases[class_index] = {}
+            if image_id not in all_difficult_cases[class_index]:
+                all_difficult_cases[class_index][image_id] = []
+            all_difficult_cases[class_index][image_id].append(difficult)
+
+    for class_index in all_gt_boxes:
+        for image_id in all_gt_boxes[class_index]:
+            all_gt_boxes[class_index][image_id] = torch.stack(
+                all_gt_boxes[class_index][image_id])
+    for class_index in all_difficult_cases:
+        for image_id in all_difficult_cases[class_index]:
+            all_gt_boxes[class_index][image_id] = torch.tensor(
+                all_gt_boxes[class_index][image_id])
+    return true_case_stat, all_gt_boxes, all_difficult_cases
+
+
+def compute_average_precision_per_class(num_true_cases, gt_boxes,
+                                        difficult_cases, prediction_file,
+                                        iou_threshold, use_2007_metric):
+    with open(prediction_file) as f:
+        image_ids = []
+        boxes = []
+        scores = []
+        for line in f:
+            t = line.rstrip().split(" ")
+            image_ids.append(t[0])
+            scores.append(float(t[1]))
+            box = torch.tensor([float(v) for v in t[2:]]).unsqueeze(0)
+            box -= 1.0  # convert to python format where indexes start from 0
+            boxes.append(box)
+        scores = np.array(scores)
+        sorted_indexes = np.argsort(-scores)
+        boxes = [boxes[i] for i in sorted_indexes]
+        image_ids = [image_ids[i] for i in sorted_indexes]
+        true_positive = np.zeros(len(image_ids))
+        false_positive = np.zeros(len(image_ids))
+        matched = set()
+        for i, image_id in enumerate(image_ids):
+            box = boxes[i]
+            if image_id not in gt_boxes:
+                false_positive[i] = 1
+                continue
+
+            gt_box = gt_boxes[image_id]
+            ious = box_utils.iou_of(box, gt_box)
+            max_iou = torch.max(ious).item()
+            max_arg = torch.argmax(ious).item()
+            if max_iou > iou_threshold:
+                if difficult_cases[image_id][max_arg] == 0:
+                    if (image_id, max_arg) not in matched:
+                        true_positive[i] = 1
+                        matched.add((image_id, max_arg))
+                    else:
+                        false_positive[i] = 1
+            else:
+                false_positive[i] = 1
+
+    true_positive = true_positive.cumsum()
+    false_positive = false_positive.cumsum()
+    precision = true_positive / (true_positive + false_positive)
+    recall = true_positive / num_true_cases
+    if use_2007_metric:
+        return measurements.compute_voc2007_average_precision(
+            precision, recall)
+    else:
+        return measurements.compute_average_precision(precision, recall)
+
+def jit(model, dataset):
+    print("run JIT path...")
+
+    model.eval()
+
+    transform = PredictionTransform(config.image_size, mean=config.image_mean, std=config.image_std)
+    image = dataset.get_image(0)
+    height, width, _ = image.shape
+    image = transform(image)
+    images = image.unsqueeze(0)
+    images = images.to("xpu")
+
+    if args.fp16:
+        images = images.half()
+
+    if args.benchmark == 1:
+        modelJit = torch.jit.trace(model, images, check_trace=False)
+    else:
+        modelJit = torch.jit.trace(model, images)
+
+    model = wrap_cpp_module(torch._C._jit_pass_fold_convbn(modelJit._c))
+    return model
+
+def calib(model, dataset):
+    print("Calibration for INT8 ... ")
+    with torch.no_grad():
+        qscheme = torch.per_tensor_affine if args.asymm else torch.per_tensor_symmetric
+        dtype = torch.quint8 if args.uint8 else torch.qint8
+        if args.mc:
+            qconfig = torch.quantization.QConfig(
+                activation=torch.quantization.observer.MinMaxObserver.with_args(
+                    qscheme=qscheme,
+                    reduce_range=False,
+                    dtype=dtype
+                ),
+                weight=torch.quantization.default_per_channel_weight_observer
+            )
+        else:
+            qconfig = torch.quantization.QConfig(
+                activation=torch.quantization.observer.MinMaxObserver.with_args(
+                    qscheme=qscheme,
+                    reduce_range=False,
+                    dtype=dtype
+                ),
+                weight=torch.quantization.default_weight_observer
+            )
+
+        model = prepare_jit(model, {'': qconfig}, True)
+
+        # tune acc through iteration number for calibration
+        for i in range(args.calib_num):
+            image = dataset.get_image(i)
+            height, width, _ = image.shape
+            transform = PredictionTransform(config.image_size, mean=config.image_mean, std=config.image_std)
+            image = transform(image)
+            images = image.unsqueeze(0)
+            images = images.to("xpu")
+            model = model.to("xpu")
+            images = images.to("xpu")
+
+            model(images)
+
+        model = convert_jit(model, True)
+
+    return model
+
+
+
+if __name__ == '__main__':
+    pathlib.Path(args.eval_dir).mkdir(exist_ok=True)
+    eval_path = pathlib.Path(args.eval_dir + "/" + str(os.getpid()))
+    eval_path.mkdir(exist_ok=True)
+    timer = Timer()
+    class_names = [name.strip() for name in open(args.label_file).readlines()]
+
+    if args.dataset_type == "voc":
+        dataset = VOCDataset(args.dataset, args.iter_num, is_test=True)
+    elif args.dataset_type == 'open_images':
+        dataset = OpenImagesDataset(
+            args.dataset, args.iter_num, dataset_type="test")
+
+    true_case_stat, all_gb_boxes, all_difficult_cases = group_annotation_by_class(
+        dataset)
+
+    if args.int8 and args.jit == 0:
+        print("int8 not support impe path")
+        sys.exit(1) 
+
+    if args.load:
+        if os.path.isfile(args.load):
+            load_path = args.load
+            typeFlag = 1 if args.fp16 else 0
+            typeFlag = 2 if args.bf16 else typeFlag
+            channelsFlag = 1 if args.channels_last else 0
+            timer.start("Load Model")
+            if args.jit == -1:
+                if args.int8 or args.fp16:
+                    net = torch.jit.load(load_path)
+                else:
+                    net = torch.load(load_path)
+                    print("Running on", str(net.device))
+            elif args.jit == 0:
+                net = torch.load(load_path)
+                print("Running on", str(net.device))
+            else:
+                net = torch.jit.load(load_path)
+            print(f'It took {timer.end("Load Model")} seconds to load the model.')
+        else:
+            print("=> no saved model found at '{}'".format(args.load))
+    else:
+        if args.net == 'vgg16-ssd':
+            net = create_vgg_ssd(len(class_names), is_test=True)
+        elif args.net == 'mb1-ssd':
+            net = create_mobilenetv1_ssd(
+                len(class_names), is_test=True, device=DEVICE)
+        elif args.net == 'mb1-ssd-lite':
+            net = create_mobilenetv1_ssd_lite(len(class_names), is_test=True)
+        elif args.net == 'sq-ssd-lite':
+            net = create_squeezenet_ssd_lite(len(class_names), is_test=True)
+        elif args.net == 'mb2-ssd-lite':
+            net = create_mobilenetv2_ssd_lite(
+                len(class_names),
+                width_mult=args.mb2_width_mult,
+                is_test=True)
+        else:
+            logging.fatal(
+                "The net type is wrong. \
+                It should be one of vgg16-ssd, mb1-ssd and mb1-ssd-lite.")
+            parser.print_help(sys.stderr)
+            sys.exit(1)
+
+        timer.start("Load Model")
+        net.load(args.trained_model)
+
+        # convert the model to half exclude BatchNorm2d layer
+        # typeFlag 0 Float; typeFlag 1 Half; typeFlag 2 BFloat16
+        typeFlag = 0
+        if args.fp16 or args.bf16:
+            if args.fp16:
+                net.half()
+                typeFlag = 1
+            else:
+                net.bfloat16()
+                typeFlag = 2
+            for layer in net.modules():
+                if isinstance(layer, nn.BatchNorm2d):
+                    layer.float()
+
+        net = net.to(DEVICE)
+
+        if args.jit == -1:
+            if args.fp16 or args.int8:
+                net = jit(net, dataset)
+        elif args.jit == 1:
+            net = jit(net, dataset)
+            
+        if args.int8:
+            net = calib(net, dataset)
+
+        channelsFlag = 0
+        # channelsFlag 0 NCHW; channelsFlag 1 NHWC
+        if args.channels_last:
+            net = net.to(memory_format=torch.channels_last)
+            channelsFlag = 1
+        print("Running on", DEVICE)
+        print(f'It took {timer.end("Load Model")} seconds to load the model.')
+    if args.net == 'vgg16-ssd':
+        predictor = create_vgg_ssd_predictor(
+            net, nms_method=args.nms_method, device=DEVICE)
+    elif args.net == 'mb1-ssd':
+        predictor = create_mobilenetv1_ssd_predictor(
+            net, nms_method=args.nms_method, device=DEVICE)
+    elif args.net == 'mb1-ssd-lite':
+        predictor = create_mobilenetv1_ssd_lite_predictor(
+            net, nms_method=args.nms_method, device=DEVICE)
+    elif args.net == 'sq-ssd-lite':
+        predictor = create_squeezenet_ssd_lite_predictor(
+            net, nms_method=args.nms_method, device=DEVICE)
+    elif args.net == 'mb2-ssd-lite':
+        predictor = create_mobilenetv2_ssd_lite_predictor(
+            net, nms_method=args.nms_method, device=DEVICE)
+    else:
+        logging.fatal(
+            "The net type is wrong. \
+            It should be one of vgg16-ssd, mb1-ssd and mb1-ssd-lite.")
+        parser.print_help(sys.stderr)
+        sys.exit(1)
+
+    results = []
+    for i in range(max(len(dataset), args.num_iterations)):
+        try:
+            import memory_check
+            memory_check.display_mem("xpu:0")
+        except:
+            pass
+        if args.benchmark and i > args.num_iterations:
+            break
+        print("process image", i)
+        timer.start("Load Image")
+        image = dataset.get_image(i) if args.dummy == 0 else None
+        print("Load Image: {:4f} seconds.".format(timer.end("Load Image")))
+        timer.start("Predict")
+
+        profiling = os.environ.get("PROFILE", "OFF").upper() in ["1", "Y", "ON", "YES", "TRUE"]
+        with torch.inference_mode():
+            boxes, labels, probs = predictor.predict(typeFlag, channelsFlag, image, -1, None, profiling, args, i)
+            if profiling and i == args.profile_iter:
+                break
+        # sync for time measurement
+        torch.xpu.synchronize()
+        print("Prediction: {:4f} seconds.".format(timer.end("Predict")))
+        indexes = torch.ones(labels.size(0), 1, dtype=torch.float32) * i
+        results.append(torch.cat([
+            indexes.reshape(-1, 1),
+            labels.reshape(-1, 1).float(),
+            probs.reshape(-1, 1),
+            boxes + 1.0  # matlab's indexes start from 1
+        ], dim=1))
+
+    if args.save:
+        store_path = args.save
+        if args.jit == -1:
+            if args.int8 or args.fp16:
+                torch.jit.save(net, store_path)
+            else:
+                torch.save(net, store_path)
+        elif args.jit == 0:
+            torch.save(net, store_path)
+        else:
+            torch.jit.save(net, store_path)          
+    if args.dummy > 0:
+        sys.exit(0)
+
+    results = torch.cat(results)
+    for class_index, class_name in enumerate(class_names):
+        if class_index == 0:
+            continue  # ignore background
+        prediction_path = eval_path / f"det_test_{class_name}.txt"
+        with open(prediction_path, "w") as f:
+            sub = results[results[:, 1] == class_index, :]
+            for i in range(sub.size(0)):
+                prob_box = sub[i, 2:].numpy()
+                image_id = dataset.ids[int(sub[i, 0])]
+                print(
+                    image_id + " " + " ".join([str(v) for v in prob_box]),
+                    file=f
+                )
+    aps = []
+    print("\n\nAverage Precision Per-class:")
+    for class_index, class_name in enumerate(class_names):
+        if class_index == 0:
+            continue
+        prediction_path = eval_path / f"det_test_{class_name}.txt"
+        ap = compute_average_precision_per_class(
+            true_case_stat[class_index],
+            all_gb_boxes[class_index],
+            all_difficult_cases[class_index],
+            prediction_path,
+            args.iou_threshold,
+            args.use_2007_metric
+        )
+        aps.append(ap)
+        print(f"{class_name}: {ap}")
+
+    print(f"\nAverage Precision Across All Classes:{sum(aps)/len(aps)}")
diff --git a/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/requirements.txt b/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/requirements.txt
new file mode 100644
index 000000000..87fa301d2
--- /dev/null
+++ b/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/requirements.txt
@@ -0,0 +1,3 @@
+pillow
+opencv-python
+pandas
diff --git a/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/run_model.sh b/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/run_model.sh
new file mode 100755
index 000000000..f901ed0b6
--- /dev/null
+++ b/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/run_model.sh
@@ -0,0 +1,143 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+
+#!/bin/bash
+
+# Create an array of input directories that are expected and then verify that they exist
+declare -A input_envs
+input_envs[MULTI_TILE]=${MULTI_TILE}
+input_envs[PLATFORM]=${PLATFORM}
+input_envs[DATASET_DIR]=${DATASET_DIR}
+input_envs[WEIGHT_DIR]=${WEIGHT_DIR}
+input_envs[LABEL_DIR]=${LABEL_DIR}
+
+for i in "${!input_envs[@]}"; do
+  var_name=$i
+  env_param=${input_envs[$i]}
+
+  if [[ -z $env_param ]]; then
+    echo "The required environment variable $var_name is not set" >&2
+    exit 1
+  fi
+done
+
+OUTPUT_DIR=${OUTPUT_DIR:-$PWD}
+
+if [[ "${PLATFORM}" == "ARC" ]]; then
+    if [[ "${MULTI_TILE}" == "True" ]]; then
+	echo "ARC not support multitile"
+	exit 1
+    fi
+    BATCH_SIZE=${BATCH_SIZE:-256}
+    PRECISION=${PRECISION:-INT8}
+    NUM_ITERATIONS=${NUM_ITERATIONS:-500}
+else
+    echo "Only support ARC for platform"
+    exit 1
+fi
+
+if [[ ! -d "${DATASET_DIR}" ]]; then
+    echo "The DATASET_DIR '${DATASET_DIR}' does not exist"
+    exit 1
+fi
+
+if [[ ! -d "${WEIGHT_DIR}" ]]; then
+    echo "The WEIGHT_DIR '${WEIGHT_DIR}' does not exist"
+    exit 1
+fi
+
+if [[ ! -d "${LABEL_DIR}" ]]; then
+    echo "The LABEL_DIR '${LABEL_DIR}' does not exist"
+    exit 1
+fi
+
+echo 'Running with parameters:'
+echo " PLATFORM: ${PLATFORM}"
+echo " DATASET_PATH: ${DATASET_DIR}"
+echo " WEIGHT_PATH: ${WEIGHT_DIR}"
+echo " LABEL_PATH: ${LABEL_DIR}"
+echo " OUTPUT_DIR: ${OUTPUT_DIR}"
+echo " PRECISION: ${PRECISION}"
+echo " BATCH_SIZE: ${BATCH_SIZE}"
+echo " NUM_ITERATIONS: ${NUM_ITERATIONS}"
+echo " MULTI_TILE: ${MULTI_TILE}"
+
+if [[ "${PRECISION}" == "INT8" ]]; then
+    flag="--int8 "
+elif [[ "${PRECISION}" == "FP32" ]]; then
+    flag=" "
+elif [[ "${PRECISION}" == "FP16" ]]; then
+    flag="--fp16 "
+else
+    echo -e "Invalid input! Only INT8 FP32 FP16 are supported."
+    exit 1
+fi
+echo "ssd-mobilenetv1 ${PRECISION} inference plain MultiTile=${MULTI_TILE} BS=${BATCH_SIZE} Iter=${NUM_ITERATIONS}"
+
+# Create the output directory, if it doesn't already exist
+mkdir -p $OUTPUT_DIR
+
+sum_log_analysis() {
+    if [ -f $2 ]; then
+        rm -f $2
+    fi
+    if diff /dev/null ${1}_t0.log |tail -l | grep '^\\ No newline' > /dev/null;then echo >> ${1}_t0.log; fi
+    if diff /dev/null ${1}_t1.log |tail -l | grep '^\\ No newline' > /dev/null;then echo >> ${1}_t1.log; fi
+    bs=$(cat ${1}_t1.log |grep Batch |awk '{print $3}')
+    echo -e "Batch Size: $bs" >$2
+    cat ${1}"_t0.log" ${1}"_t1.log" |grep "Performance" |awk -v tag=$(cat ${1}"_t0.log" ${1}"_t1.log" |grep "Performance" |awk '{sum+=$2} END {printf "%.4f\n",sum}') '{if ( $2=="None" ) {sum="None";nextfile}else  sum=tag} ;END{print "Sum "$1" "sum " "$3}' >> $2
+    cat ${1}"_t0.log" ${1}"_t1.log" |grep "Performance" |awk -v tag=$(cat ${1}"_t0.log" ${1}"_t1.log" |grep "Performance" |awk 'BEGIN {min=1234567890123} {if ($2 <min) {min=$2}}END {printf "%.4f\n",min}') '{if ( $2=="None" ) {min="None";nextfile}else  min=tag} ;END{print "Min "$1" "min " "$3}' >> $2
+    cat ${1}"_t0.log" ${1}"_t1.log" |grep "Latency" |awk '{if ( $2=="N/A" ){avg="N/A";nextfile}else avg=((sum+=$2/2))};END{print "Avg "$1" "avg " "$3}'  >> $2
+    cat ${1}"_t0.log" ${1}"_t1.log" |grep "Accuracy" |awk -v avg=$(cat ${1}"_t0.log" ${1}"_t1.log" |grep "Accuracy" |awk '{sum+=$3}END{printf "%.4f\n",sum/NR}') '{if ( $3=="None" || $2=="N/A" || $3=="nan" || $3=="N/A"){avg="None";nextfile}else avg=avg};END{print "Avg "$1" "$2 " "avg}' >> $2
+    cat ${1}"_t0.log" ${1}"_t1.log" |grep "Functional" | awk -v fail=$(cat ${1}"_t0.log" ${1}"_t1.log" |grep "Functional" |awk '{for(i=1;i<=NF;++i) if($i=="fail") ++sum}END{print sum}') '{if ( fail >= 1 ) tag="fail ";else tag="pass"};END{print $1" "tag}'  >> $2
+    cat ${1}"_t0.log" ${1}"_t1.log" |grep "Error" |awk '{if(a[$1]){a[$1]=a[$1]";"$2}else{a[$1]=$2}}END{for(i in a)print $1" " a[i]}' >> $2
+}
+
+modelname=ssd-mobilenetv1
+if [[ ${MULTI_TILE} == "False" ]]; then
+    rm ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0_raw.log 
+    python -u eval_ssd.py --net mb1-ssd --dataset ${DATASET_DIR} --trained_model ${WEIGHT_DIR}/mobilenet-v1-ssd-mp-0_675.pth --label_file ${LABEL_DIR}/voc-model-labels.txt --dummy 1 --batch_size ${BATCH_SIZE} --benchmark 1 --num-iterations ${NUM_ITERATIONS} $flag 2>&1 | tee ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0_raw.log
+    python common/parse_result.py -m $modelname -l ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0_raw.log -b ${BATCH_SIZE}
+    throughput=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0.log | grep Performance | awk -F ' ' '{print $2}')
+    throughput_unit=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0.log | grep Performance | awk -F ' ' '{print $3}')
+    latency=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0.log | grep Latency | awk -F ' ' '{print $2}')
+    acc=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0.log | grep Accuracy | awk -F ' ' '{print $3}')
+    acc_unit=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0.log | grep Accuracy | awk -F ' ' '{print $2}')
+else
+    echo "not supported MULTI_TILE"
+    exit 1
+fi
+
+yaml_content=$(cat <<EOF
+results:
+ - key: throughput
+   value: $throughput
+   unit: $throughput_unit
+ - key: latency
+   value: $latency
+   unit: s
+ - key: accuracy
+   value: $acc
+   unit: $acc_unit
+EOF
+)
+
+# Write the content to a YAML file
+echo "$yaml_content" >  ${OUTPUT_DIR}/results.yaml
+echo "YAML file created."
diff --git a/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/setup.sh b/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/setup.sh
new file mode 100755
index 000000000..0497e5d23
--- /dev/null
+++ b/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/setup.sh
@@ -0,0 +1,28 @@
+#!/usr/bin/env bash
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# setup.sh
+#  - install OS pkgs
+#  - should create virtual env & install pip  requirement.txt
+#  - git clones & applying patches
+
+set -e
+apt-get update && apt-get install -y python3-venv protobuf-compiler
+
+pip install -r requirements.txt
+
+cp -r ../../../../common .
diff --git a/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/ssd-mobilenetv1.py b/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/ssd-mobilenetv1.py
new file mode 100644
index 000000000..44c447224
--- /dev/null
+++ b/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/ssd-mobilenetv1.py
@@ -0,0 +1,40 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+
+"""Consts for ssd-mobilenetv1 inference."""
+
+ACC = {
+    "type": "total",
+    "pattern": r"Average Precision Across All Classes:(\d+.\d+)",
+    "unit": "accuracy",
+}
+
+PERF = {
+    "type": "mean",
+    "number_of_partials": 10,
+    "pattern": r"Inference time:  (\d+.\d+)",
+    "inverse": True,
+    "multiply": False,
+    "use_batch_size": True,
+    "unit": "inst/s",
+}
+
+FUNCTIONAL = {
+    "pattern": r"Average Precision Across All Classes:\d+.\d+"
+}
diff --git a/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/vision/datasets/open_images.py b/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/vision/datasets/open_images.py
new file mode 100644
index 000000000..17a7d5100
--- /dev/null
+++ b/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/vision/datasets/open_images.py
@@ -0,0 +1,158 @@
+
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+# MIT License
+#
+# Copyright (c) 2019 Hao Gao
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+#
+
+
+import numpy as np
+import pathlib
+import cv2
+import pandas as pd
+import copy
+
+class OpenImagesDataset:
+
+    def __init__(self, root, iter_num,
+                 transform=None, target_transform=None,
+                 dataset_type="train", balance_data=False):
+        self.root = pathlib.Path(root)
+        self.iter_num = iter_num
+        self.transform = transform
+        self.target_transform = target_transform
+        self.dataset_type = dataset_type.lower()
+
+        self.data, self.class_names, self.class_dict = self._read_data()
+        self.balance_data = balance_data
+        self.min_image_num = -1
+        if self.balance_data:
+            self.data = self._balance_data()
+        self.ids = [info['image_id'] for info in self.data]
+
+        self.class_stat = None
+
+    def _getitem(self, index):
+        image_info = self.data[index]
+        image = self._read_image(image_info['image_id'])
+        # duplicate boxes to prevent corruption of dataset
+        boxes = copy.copy(image_info['boxes'])
+        boxes[:, 0] *= image.shape[1]
+        boxes[:, 1] *= image.shape[0]
+        boxes[:, 2] *= image.shape[1]
+        boxes[:, 3] *= image.shape[0]
+        # duplicate labels to prevent corruption of dataset
+        labels = copy.copy(image_info['labels'])
+        if self.transform:
+            image, boxes, labels = self.transform(image, boxes, labels)
+        if self.target_transform:
+            boxes, labels = self.target_transform(boxes, labels)
+        return image_info['image_id'], image, boxes, labels
+
+    def __getitem__(self, index):
+        _, image, boxes, labels = self._getitem(index)
+        return image, boxes, labels
+
+    def get_annotation(self, index):
+        """To conform the eval_ssd implementation that is based on the VOC dataset."""
+        image_id, image, boxes, labels = self._getitem(index)
+        is_difficult = np.zeros(boxes.shape[0], dtype=np.uint8)
+        return image_id, (boxes, labels, is_difficult)
+
+    def get_image(self, index):
+        image_info = self.data[index]
+        image = self._read_image(image_info['image_id'])
+        if self.transform:
+            image, _ = self.transform(image)
+        return image
+
+    def _read_data(self):
+        annotation_file = f"{self.root}/sub-{self.dataset_type}-annotations-bbox.csv"
+        annotations = pd.read_csv(annotation_file)
+        class_names = ['BACKGROUND'] + sorted(list(annotations['ClassName'].unique()))
+        class_dict = {class_name: i for i, class_name in enumerate(class_names)}
+        data = []
+        for image_id, group in annotations.groupby("ImageID"):
+            boxes = group.loc[:, ["XMin", "YMin", "XMax", "YMax"]].values.astype(np.float32)
+            # make labels 64 bits to satisfy the cross_entropy function
+            labels = np.array([class_dict[name] for name in group["ClassName"]], dtype='int64')
+            data.append({
+                'image_id': image_id,
+                'boxes': boxes,
+                'labels': labels
+            })
+        return data, class_names, class_dict
+
+    def __len__(self):
+        return len(self.data) if self.iter_num > len(self.data) else self.iter_num
+
+    def __repr__(self):
+        if self.class_stat is None:
+            self.class_stat = {name: 0 for name in self.class_names[1:]}
+            for example in self.data:
+                for class_index in example['labels']:
+                    class_name = self.class_names[class_index]
+                    self.class_stat[class_name] += 1
+        content = ["Dataset Summary:"
+                   f"Number of Images: {len(self.data)}",
+                   f"Minimum Number of Images for a Class: {self.min_image_num}",
+                   "Label Distribution:"]
+        for class_name, num in self.class_stat.items():
+            content.append(f"\t{class_name}: {num}")
+        return "\n".join(content)
+
+    def _read_image(self, image_id):
+        image_file = self.root / self.dataset_type / f"{image_id}.jpg"
+        image = cv2.imread(str(image_file))
+        if image.shape[2] == 1:
+            image = cv2.cvtColor(image, cv2.COLOR_GRAY2RGB)
+        else:
+            image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
+        return image
+
+    def _balance_data(self):
+        label_image_indexes = [set() for _ in range(len(self.class_names))]
+        for i, image in enumerate(self.data):
+            for label_id in image['labels']:
+                label_image_indexes[label_id].add(i)
+        label_stat = [len(s) for s in label_image_indexes]
+        self.min_image_num = min(label_stat[1:])
+        sample_image_indexes = set()
+        for image_indexes in label_image_indexes[1:]:
+            image_indexes = np.array(list(image_indexes))
+            sub = np.random.permutation(image_indexes)[:self.min_image_num]
+            sample_image_indexes.update(sub)
+        sample_data = [self.data[i] for i in sample_image_indexes]
+        return sample_data
diff --git a/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/vision/datasets/voc_dataset.py b/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/vision/datasets/voc_dataset.py
new file mode 100644
index 000000000..8b8bbc6ca
--- /dev/null
+++ b/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/vision/datasets/voc_dataset.py
@@ -0,0 +1,163 @@
+
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+# MIT License
+#
+# Copyright (c) 2019 Hao Gao
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+#
+
+import numpy as np
+import logging
+import pathlib
+import xml.etree.ElementTree as ET
+import cv2
+import os
+
+
+class VOCDataset:
+
+    def __init__(self, root, iter_num, transform=None, target_transform=None, is_test=False, keep_difficult=False, label_file=None):
+        """Dataset for VOC data.
+        Args:
+            root: the root of the VOC2007 or VOC2012 dataset, the directory contains the following sub-directories:
+                Annotations, ImageSets, JPEGImages, SegmentationClass, SegmentationObject.
+        """
+        self.root = pathlib.Path(root)
+        self.iter_num = iter_num
+        self.transform = transform
+        self.target_transform = target_transform
+        if is_test:
+            image_sets_file = self.root / "ImageSets/Main/test.txt"
+        else:
+            image_sets_file = self.root / "ImageSets/Main/trainval.txt"
+        self.ids = VOCDataset._read_image_ids(image_sets_file)
+        self.keep_difficult = keep_difficult
+
+        # if the labels file exists, read in the class names
+        label_file_name = self.root / "labels.txt"
+
+        if os.path.isfile(label_file_name):
+            class_string = ""
+            with open(label_file_name, 'r') as infile:
+                for line in infile:
+                    class_string += line.rstrip()
+
+            # classes should be a comma separated list
+
+            classes = class_string.split(',')
+            # prepend BACKGROUND as first class
+            classes.insert(0, 'BACKGROUND')
+            classes  = [ elem.replace(" ", "") for elem in classes]
+            self.class_names = tuple(classes)
+            logging.info("VOC Labels read from file: " + str(self.class_names))
+
+        else:
+            logging.info("No labels file, using default VOC classes.")
+            self.class_names = ('BACKGROUND',
+            'aeroplane', 'bicycle', 'bird', 'boat',
+            'bottle', 'bus', 'car', 'cat', 'chair',
+            'cow', 'diningtable', 'dog', 'horse',
+            'motorbike', 'person', 'pottedplant',
+            'sheep', 'sofa', 'train', 'tvmonitor')
+
+
+        self.class_dict = {class_name: i for i, class_name in enumerate(self.class_names)}
+
+    def __getitem__(self, index):
+        image_id = self.ids[index]
+        boxes, labels, is_difficult = self._get_annotation(image_id)
+        if not self.keep_difficult:
+            boxes = boxes[is_difficult == 0]
+            labels = labels[is_difficult == 0]
+        image = self._read_image(image_id)
+        if self.transform:
+            image, boxes, labels = self.transform(image, boxes, labels)
+        if self.target_transform:
+            boxes, labels = self.target_transform(boxes, labels)
+        return image, boxes, labels
+
+    def get_image(self, index):
+        image_id = self.ids[index]
+        image = self._read_image(image_id)
+        if self.transform:
+            image, _ = self.transform(image)
+        return image
+
+    def get_annotation(self, index):
+        image_id = self.ids[index]
+        return image_id, self._get_annotation(image_id)
+
+    def __len__(self):
+        return len(self.ids) if self.iter_num > len(self.ids) else self.iter_num
+
+    @staticmethod
+    def _read_image_ids(image_sets_file):
+        ids = []
+        with open(image_sets_file) as f:
+            for line in f:
+                ids.append(line.rstrip())
+        return ids
+
+    def _get_annotation(self, image_id):
+        annotation_file = self.root / f"Annotations/{image_id}.xml"
+        objects = ET.parse(annotation_file).findall("object")
+        boxes = []
+        labels = []
+        is_difficult = []
+        for object in objects:
+            class_name = object.find('name').text.lower().strip()
+            # we're only concerned with clases in our list
+            if class_name in self.class_dict:
+                bbox = object.find('bndbox')
+
+                # VOC dataset format follows Matlab, in which indexes start from 0
+                x1 = float(bbox.find('xmin').text) - 1
+                y1 = float(bbox.find('ymin').text) - 1
+                x2 = float(bbox.find('xmax').text) - 1
+                y2 = float(bbox.find('ymax').text) - 1
+                boxes.append([x1, y1, x2, y2])
+
+                labels.append(self.class_dict[class_name])
+                is_difficult_str = object.find('difficult').text
+                is_difficult.append(int(is_difficult_str) if is_difficult_str else 0)
+
+        return (np.array(boxes, dtype=np.float32),
+                np.array(labels, dtype=np.int64),
+                np.array(is_difficult, dtype=np.uint8))
+
+    def _read_image(self, image_id):
+        image_file = self.root / f"JPEGImages/{image_id}.jpg"
+        image = cv2.imread(str(image_file))
+        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
+        return image
diff --git a/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/vision/nn/mobilenet.py b/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/vision/nn/mobilenet.py
new file mode 100644
index 000000000..c0043a10f
--- /dev/null
+++ b/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/vision/nn/mobilenet.py
@@ -0,0 +1,75 @@
+# MIT License
+#
+# Copyright (c) 2019 Hao Gao
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+#
+
+# borrowed from "https://github.com/marvis/pytorch-mobilenet"
+
+import torch.nn as nn
+import torch.nn.functional as F
+
+
+class MobileNetV1(nn.Module):
+    def __init__(self, num_classes=1024):
+        super(MobileNetV1, self).__init__()
+
+        def conv_bn(inp, oup, stride):
+            return nn.Sequential(
+                nn.Conv2d(inp, oup, 3, stride, 1, bias=False),
+                nn.BatchNorm2d(oup),
+                nn.ReLU(inplace=True)
+            )
+
+        def conv_dw(inp, oup, stride):
+            return nn.Sequential(
+                nn.Conv2d(inp, inp, 3, stride, 1, groups=inp, bias=False),
+                nn.BatchNorm2d(inp),
+                nn.ReLU(inplace=True),
+
+                nn.Conv2d(inp, oup, 1, 1, 0, bias=False),
+                nn.BatchNorm2d(oup),
+                nn.ReLU(inplace=True),
+            )
+
+        self.model = nn.Sequential(
+            conv_bn(3, 32, 2),
+            conv_dw(32, 64, 1),
+            conv_dw(64, 128, 2),
+            conv_dw(128, 128, 1),
+            conv_dw(128, 256, 2),
+            conv_dw(256, 256, 1),
+            conv_dw(256, 512, 2),
+            conv_dw(512, 512, 1),
+            conv_dw(512, 512, 1),
+            conv_dw(512, 512, 1),
+            conv_dw(512, 512, 1),
+            conv_dw(512, 512, 1),
+            conv_dw(512, 1024, 2),
+            conv_dw(1024, 1024, 1),
+        )
+        self.fc = nn.Linear(1024, num_classes)
+
+    def forward(self, x):
+        x = self.model(x)
+        x = F.avg_pool2d(x, 7)
+        x = x.view(-1, 1024)
+        x = self.fc(x)
+        return x
diff --git a/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/vision/ssd/config/mobilenetv1_ssd_config.py b/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/vision/ssd/config/mobilenetv1_ssd_config.py
new file mode 100644
index 000000000..f81e12400
--- /dev/null
+++ b/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/vision/ssd/config/mobilenetv1_ssd_config.py
@@ -0,0 +1,46 @@
+# MIT License
+#
+# Copyright (c) 2019 Hao Gao
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+#
+
+import numpy as np
+
+from vision.utils.box_utils import SSDSpec, SSDBoxSizes, generate_ssd_priors
+
+
+image_size = 300
+image_mean = np.array([127, 127, 127])  # RGB layout
+image_std = 128.0
+iou_threshold = 0.45
+center_variance = 0.1
+size_variance = 0.2
+
+specs = [
+    SSDSpec(19, 16, SSDBoxSizes(60, 105), [2, 3]),
+    SSDSpec(10, 32, SSDBoxSizes(105, 150), [2, 3]),
+    SSDSpec(5, 64, SSDBoxSizes(150, 195), [2, 3]),
+    SSDSpec(3, 100, SSDBoxSizes(195, 240), [2, 3]),
+    SSDSpec(2, 150, SSDBoxSizes(240, 285), [2, 3]),
+    SSDSpec(1, 300, SSDBoxSizes(285, 330), [2, 3])
+]
+
+
+priors = generate_ssd_priors(specs, image_size)
diff --git a/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/vision/ssd/data_preprocessing.py b/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/vision/ssd/data_preprocessing.py
new file mode 100644
index 000000000..981cdc6a6
--- /dev/null
+++ b/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/vision/ssd/data_preprocessing.py
@@ -0,0 +1,85 @@
+# MIT License
+#
+# Copyright (c) 2019 Hao Gao
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+#
+
+from ..transforms.transforms import *
+
+
+class TrainAugmentation:
+    def __init__(self, size, mean=0, std=1.0):
+        """
+        Args:
+            size: the size the of final image.
+            mean: mean pixel value per channel.
+        """
+        self.mean = mean
+        self.size = size
+        self.augment = Compose([
+            ConvertFromInts(),
+            PhotometricDistort(),
+            Expand(self.mean),
+            RandomSampleCrop(),
+            RandomMirror(),
+            ToPercentCoords(),
+            Resize(self.size),
+            SubtractMeans(self.mean),
+            lambda img, boxes=None, labels=None: (img / std, boxes, labels),
+            ToTensor(),
+        ])
+
+    def __call__(self, img, boxes, labels):
+        """
+
+        Args:
+            img: the output of cv.imread in RGB layout.
+            boxes: boundding boxes in the form of (x1, y1, x2, y2).
+            labels: labels of boxes.
+        """
+        return self.augment(img, boxes, labels)
+
+
+class TestTransform:
+    def __init__(self, size, mean=0.0, std=1.0):
+        self.transform = Compose([
+            ToPercentCoords(),
+            Resize(size),
+            SubtractMeans(mean),
+            lambda img, boxes=None, labels=None: (img / std, boxes, labels),
+            ToTensor(),
+        ])
+
+    def __call__(self, image, boxes, labels):
+        return self.transform(image, boxes, labels)
+
+
+class PredictionTransform:
+    def __init__(self, size, mean=0.0, std=1.0):
+        self.transform = Compose([
+            Resize(size),
+            SubtractMeans(mean),
+            lambda img, boxes=None, labels=None: (img / std, boxes, labels),
+            ToTensor()
+        ])
+
+    def __call__(self, image):
+        image, _, _ = self.transform(image)
+        return image
diff --git a/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/vision/ssd/mobilenetv1_ssd.py b/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/vision/ssd/mobilenetv1_ssd.py
new file mode 100644
index 000000000..8afbd725c
--- /dev/null
+++ b/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/vision/ssd/mobilenetv1_ssd.py
@@ -0,0 +1,113 @@
+
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+# MIT License
+#
+# Copyright (c) 2019 Hao Gao
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+#
+
+import torch
+from torch.nn import Conv2d, Sequential, ModuleList, ReLU
+from ..nn.mobilenet import MobileNetV1
+
+from .ssd import SSD
+from .predictor import Predictor
+from .config import mobilenetv1_ssd_config as config
+
+
+def create_mobilenetv1_ssd(num_classes, is_test=False, device=None):
+    base_net = MobileNetV1(1001).model  # disable dropout layer
+
+    source_layer_indexes = [
+        12,
+        14,
+    ]
+    extras = ModuleList([
+        Sequential(
+            Conv2d(in_channels=1024, out_channels=256, kernel_size=1),
+            ReLU(),
+            Conv2d(in_channels=256, out_channels=512, kernel_size=3, stride=2, padding=1),
+            ReLU()
+        ),
+        Sequential(
+            Conv2d(in_channels=512, out_channels=128, kernel_size=1),
+            ReLU(),
+            Conv2d(in_channels=128, out_channels=256, kernel_size=3, stride=2, padding=1),
+            ReLU()
+        ),
+        Sequential(
+            Conv2d(in_channels=256, out_channels=128, kernel_size=1),
+            ReLU(),
+            Conv2d(in_channels=128, out_channels=256, kernel_size=3, stride=2, padding=1),
+            ReLU()
+        ),
+        Sequential(
+            Conv2d(in_channels=256, out_channels=128, kernel_size=1),
+            ReLU(),
+            Conv2d(in_channels=128, out_channels=256, kernel_size=3, stride=2, padding=1),
+            ReLU()
+        )
+    ])
+
+    regression_headers = ModuleList([
+        Conv2d(in_channels=512, out_channels=6 * 4, kernel_size=3, padding=1),
+        Conv2d(in_channels=1024, out_channels=6 * 4, kernel_size=3, padding=1),
+        Conv2d(in_channels=512, out_channels=6 * 4, kernel_size=3, padding=1),
+        Conv2d(in_channels=256, out_channels=6 * 4, kernel_size=3, padding=1),
+        Conv2d(in_channels=256, out_channels=6 * 4, kernel_size=3, padding=1),
+        Conv2d(in_channels=256, out_channels=6 * 4, kernel_size=3, padding=1), # TODO: change to kernel_size=1, padding=0?
+    ])
+
+    classification_headers = ModuleList([
+        Conv2d(in_channels=512, out_channels=6 * num_classes, kernel_size=3, padding=1),
+        Conv2d(in_channels=1024, out_channels=6 * num_classes, kernel_size=3, padding=1),
+        Conv2d(in_channels=512, out_channels=6 * num_classes, kernel_size=3, padding=1),
+        Conv2d(in_channels=256, out_channels=6 * num_classes, kernel_size=3, padding=1),
+        Conv2d(in_channels=256, out_channels=6 * num_classes, kernel_size=3, padding=1),
+        Conv2d(in_channels=256, out_channels=6 * num_classes, kernel_size=3, padding=1), # TODO: change to kernel_size=1, padding=0?
+    ])
+
+    return SSD(num_classes, base_net, source_layer_indexes,
+               extras, classification_headers, regression_headers, is_test=is_test, config=config, device=device)
+
+
+def create_mobilenetv1_ssd_predictor(net, candidate_size=200, nms_method=None, sigma=0.5, device=None):
+    predictor = Predictor(net, config.image_size, config.image_mean,
+                          config.image_std,
+                          nms_method=nms_method,
+                          iou_threshold=config.iou_threshold,
+                          candidate_size=candidate_size,
+                          sigma=sigma,
+                          device=device)
+    return predictor
diff --git a/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/vision/ssd/predictor.py b/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/vision/ssd/predictor.py
new file mode 100644
index 000000000..7bc8db4dd
--- /dev/null
+++ b/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/vision/ssd/predictor.py
@@ -0,0 +1,169 @@
+
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+# MIT License
+#
+# Copyright (c) 2019 Hao Gao
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+#
+
+import os
+import torch
+import time
+
+from ..utils import box_utils
+from .data_preprocessing import PredictionTransform
+from ..utils.misc import Timer
+
+
+class Predictor:
+    def __init__(self, net, size, mean=0.0, std=1.0, nms_method=None,
+                 iou_threshold=0.45, filter_threshold=0.01, candidate_size=200, sigma=0.5, device=None):
+        self.net = net
+        self.transform = PredictionTransform(size, mean, std)
+        self.iou_threshold = iou_threshold
+        self.filter_threshold = filter_threshold
+        self.candidate_size = candidate_size
+        self.nms_method = nms_method
+
+        self.sigma = sigma
+        if device:
+            self.device = device
+        else:
+            self.device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
+
+        self.net.to(self.device)
+        self.net.eval()
+
+        self.timer = Timer()
+
+    def predict(self, flag, channels_flag, image, top_k=-1, prob_threshold=None, profiling=False, args=None, iter=0):
+        cpu_device = torch.device("cpu")
+        if args is not None and args.dummy > 0:
+            height = args.image_size
+            width = args.image_size
+            images = torch.randn(args.batch_size, 3, height, height)
+        else:
+            height, width, _ = image.shape
+            image = self.transform(image)
+            images = image.unsqueeze(0)
+        if flag == 1:
+            images = images.to(torch.half)
+        elif flag == 2:
+            images = images.to(torch.bfloat16)
+        if channels_flag == 1:
+            images = images.to(memory_format=torch.channels_last)
+        if args is not None and args.benchmark == 1:
+            images = images.to(self.device)
+        with torch.autograd.profiler_legacy.profile(enabled=profiling, use_xpu=True, record_shapes=False) as prof:
+            with torch.inference_mode():
+                self.timer.start()
+                if args is None or args.benchmark == 0:
+                    images = images.to(self.device)
+                scores, boxes = self.net.forward(images)
+
+                if flag > 0:
+                    boxes = boxes[0].to(torch.float)
+                    scores = scores[0].to(torch.float)
+                else:
+                    boxes = boxes[0]
+                    scores = scores[0]
+                if not prob_threshold:
+                    prob_threshold = self.filter_threshold
+
+                # sync for time measurement
+                torch.xpu.synchronize()
+                if args is not None and args.benchmark == 1:
+                    print("Inference time: ", self.timer.end())
+
+        # this version of nms is slower on GPU, so we move data to CPU.
+        boxes = boxes.to(cpu_device)
+        scores = scores.to(cpu_device)
+
+        post_start_time = time.time()
+        picked_box_probs = []
+        picked_labels = []
+        for class_index in range(1, scores.size(1)):
+            probs = scores[:, class_index]
+            mask = probs > prob_threshold
+            probs = probs[mask]
+            if probs.size(0) == 0:
+                continue
+            subset_boxes = boxes[mask, :]
+            box_probs = torch.cat([subset_boxes, probs.reshape(-1, 1)], dim=1)
+            box_probs = box_utils.nms(box_probs, self.nms_method,
+                                      score_threshold=prob_threshold,
+                                      iou_threshold=self.iou_threshold,
+                                      sigma=self.sigma,
+                                      top_k=top_k,
+                                      candidate_size=self.candidate_size)
+            picked_box_probs.append(box_probs)
+            picked_labels.extend([class_index] * box_probs.size(0))
+        if not picked_box_probs:
+            return torch.tensor([]), torch.tensor([]), torch.tensor([])
+        picked_box_probs = torch.cat(picked_box_probs)
+        picked_box_probs[:, 0] *= width
+        picked_box_probs[:, 1] *= height
+        picked_box_probs[:, 2] *= width
+        picked_box_probs[:, 3] *= height
+
+        ret = picked_box_probs[:, :4], torch.tensor(picked_labels), picked_box_probs[:, 4]
+        post_end_time = time.time()
+        print("Post time: ", post_end_time - post_start_time)
+        if args is None or args.benchmark == 0:
+            print("Inference time: ", self.timer.end())
+
+        if profiling and iter==args.profile_iter:
+            title = "/ssd_mobilenetv1_inference_"
+            if args.channels_last:
+                title += "channels_last_"
+            else:
+                title += "block_"
+            if args.bf16:
+                title += "bf16_"
+            if args.fp16:
+                title += "fp16_"
+            if args.int8:
+                title += "int8_"
+            if args.batch_size:
+                title += "bs" + str(args.batch_size) + "_"
+
+            profiling_path = os.getenv('PROFILE_PATH')
+            if not profiling_path:
+                profiling_path = './'
+            torch.save(prof.key_averages().table(sort_by="self_xpu_time_total"), profiling_path + title + 'profiling.pt')
+            torch.save(prof.key_averages(group_by_input_shape=True).table(), profiling_path + title + 'profiling_detailed.pt')
+            prof.export_chrome_trace(profiling_path + title + 'profiling.json')
+            print(prof.key_averages().table(sort_by="self_xpu_time_total"))
+            print(prof.key_averages(group_by_input_shape=True).table())
+        return ret
diff --git a/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/vision/ssd/ssd.py b/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/vision/ssd/ssd.py
new file mode 100644
index 000000000..1ef7a2091
--- /dev/null
+++ b/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/vision/ssd/ssd.py
@@ -0,0 +1,201 @@
+
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+# MIT License
+#
+# Copyright (c) 2019 Hao Gao
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+#
+
+import torch.nn as nn
+import torch
+import numpy as np
+from typing import List, Tuple
+import torch.nn.functional as F
+
+from ..utils import box_utils
+from collections import namedtuple
+GraphPath = namedtuple("GraphPath", ['s0', 'name', 's1'])  #
+
+
+class SSD(nn.Module):
+    def __init__(self, num_classes: int, base_net: nn.ModuleList, source_layer_indexes: List[int],
+                 extras: nn.ModuleList, classification_headers: nn.ModuleList,
+                 regression_headers: nn.ModuleList, is_test=False, config=None, device=None):
+        """Compose a SSD model using the given components.
+        """
+        super(SSD, self).__init__()
+
+        self.num_classes = num_classes
+        self.base_net = base_net
+        self.source_layer_indexes = source_layer_indexes
+        self.extras = extras
+        self.classification_headers = classification_headers
+        self.regression_headers = regression_headers
+        self.is_test = is_test
+        self.config = config
+
+        # register layers in source_layer_indexes by adding them to a module list
+        self.source_layer_add_ons = nn.ModuleList([t[1] for t in source_layer_indexes
+                                                   if isinstance(t, tuple) and not isinstance(t, GraphPath)])
+        if device:
+            self.device = device
+        else:
+            self.device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
+        if is_test:
+            self.config = config
+            self.priors = config.priors.to(self.device)
+
+    def forward(self, x: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]:
+        confidences = []
+        locations = []
+        start_layer_index = 0
+        header_index = 0
+        for end_layer_index in self.source_layer_indexes:
+            if isinstance(end_layer_index, GraphPath):
+                path = end_layer_index
+                end_layer_index = end_layer_index.s0
+                added_layer = None
+            elif isinstance(end_layer_index, tuple):
+                added_layer = end_layer_index[1]
+                end_layer_index = end_layer_index[0]
+                path = None
+            else:
+                added_layer = None
+                path = None
+            for layer in self.base_net[start_layer_index: end_layer_index]:
+                x = layer(x)
+            if added_layer:
+                y = added_layer(x)
+            else:
+                y = x
+            if path:
+                sub = getattr(self.base_net[end_layer_index], path.name)
+                for layer in sub[:path.s1]:
+                    x = layer(x)
+                y = x
+                for layer in sub[path.s1:]:
+                    x = layer(x)
+                end_layer_index += 1
+            start_layer_index = end_layer_index
+            confidence, location = self.compute_header(header_index, y)
+            header_index += 1
+            confidences.append(confidence)
+            locations.append(location)
+
+        for layer in self.base_net[end_layer_index:]:
+            x = layer(x)
+
+        for layer in self.extras:
+            x = layer(x)
+            confidence, location = self.compute_header(header_index, x)
+            header_index += 1
+            confidences.append(confidence)
+            locations.append(location)
+
+        confidences = torch.cat(confidences, 1)
+        # (Todo zhiyuan) Convert to float due to Float16 += BFloat16 not supported
+        # by TensorIterator yet, while Float32 += Half is OK
+        locations = torch.cat(locations, 1).to(torch.float)
+
+        if self.is_test:
+            confidences = F.softmax(confidences, dim=2)
+            boxes = torch.xpu.locations_to_boxes(locations, self.priors, self.config.center_variance, self.config.size_variance);
+            return confidences, boxes
+        else:
+            return confidences, locations
+
+    def compute_header(self, i, x):
+        confidence = self.classification_headers[i](x)
+        confidence = confidence.permute(0, 2, 3, 1).contiguous()
+        confidence = confidence.view(confidence.size(0), -1, self.num_classes)
+
+        location = self.regression_headers[i](x)
+        location = location.permute(0, 2, 3, 1).contiguous()
+        location = location.view(location.size(0), -1, 4)
+
+        return confidence, location
+
+    def init_from_base_net(self, model):
+        self.base_net.load_state_dict(torch.load(model, map_location=lambda storage, loc: storage), strict=True)
+        self.source_layer_add_ons.apply(_xavier_init_)
+        self.extras.apply(_xavier_init_)
+        self.classification_headers.apply(_xavier_init_)
+        self.regression_headers.apply(_xavier_init_)
+
+    def init_from_pretrained_ssd(self, model):
+        state_dict = torch.load(model, map_location=lambda storage, loc: storage)
+        state_dict = {k: v for k, v in state_dict.items() if not (k.startswith("classification_headers") or k.startswith("regression_headers"))}
+        model_dict = self.state_dict()
+        model_dict.update(state_dict)
+        self.load_state_dict(model_dict)
+        self.classification_headers.apply(_xavier_init_)
+        self.regression_headers.apply(_xavier_init_)
+
+    def init(self):
+        self.base_net.apply(_xavier_init_)
+        self.source_layer_add_ons.apply(_xavier_init_)
+        self.extras.apply(_xavier_init_)
+        self.classification_headers.apply(_xavier_init_)
+        self.regression_headers.apply(_xavier_init_)
+
+    def load(self, model):
+        self.load_state_dict(torch.load(model, map_location=lambda storage, loc: storage))
+
+    def save(self, model_path):
+        torch.save(self.state_dict(), model_path)
+
+
+class MatchPrior(object):
+    def __init__(self, center_form_priors, center_variance, size_variance, iou_threshold):
+        self.center_form_priors = center_form_priors
+        self.corner_form_priors = box_utils.center_form_to_corner_form(center_form_priors)
+        self.center_variance = center_variance
+        self.size_variance = size_variance
+        self.iou_threshold = iou_threshold
+
+    def __call__(self, gt_boxes, gt_labels):
+        if type(gt_boxes) is np.ndarray:
+            gt_boxes = torch.from_numpy(gt_boxes)
+        if type(gt_labels) is np.ndarray:
+            gt_labels = torch.from_numpy(gt_labels)
+        boxes, labels = box_utils.assign_priors(gt_boxes, gt_labels,
+                                                self.corner_form_priors, self.iou_threshold)
+        boxes = box_utils.corner_form_to_center_form(boxes)
+        locations = box_utils.convert_boxes_to_locations(boxes, self.center_form_priors, self.center_variance, self.size_variance)
+        return locations, labels
+
+
+def _xavier_init_(m: nn.Module):
+    if isinstance(m, nn.Conv2d):
+        nn.init.xavier_uniform_(m.weight)
diff --git a/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/vision/transforms/transforms.py b/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/vision/transforms/transforms.py
new file mode 100644
index 000000000..33317cd60
--- /dev/null
+++ b/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/vision/transforms/transforms.py
@@ -0,0 +1,430 @@
+# MIT License
+#
+# Copyright (c) 2019 Hao Gao
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+#
+
+# from https://github.com/amdegroot/ssd.pytorch
+
+
+import torch
+from torchvision import transforms
+import cv2
+import numpy as np
+import types
+from numpy import random
+
+
+def intersect(box_a, box_b):
+    max_xy = np.minimum(box_a[:, 2:], box_b[2:])
+    min_xy = np.maximum(box_a[:, :2], box_b[:2])
+    inter = np.clip((max_xy - min_xy), a_min=0, a_max=np.inf)
+    return inter[:, 0] * inter[:, 1]
+
+
+def jaccard_numpy(box_a, box_b):
+    """Compute the jaccard overlap of two sets of boxes.  The jaccard overlap
+    is simply the intersection over union of two boxes.
+    E.g.:
+        A ∩ B / A ∪ B = A ∩ B / (area(A) + area(B) - A ∩ B)
+    Args:
+        box_a: Multiple bounding boxes, Shape: [num_boxes,4]
+        box_b: Single bounding box, Shape: [4]
+    Return:
+        jaccard overlap: Shape: [box_a.shape[0], box_a.shape[1]]
+    """
+    inter = intersect(box_a, box_b)
+    area_a = ((box_a[:, 2]-box_a[:, 0]) *
+              (box_a[:, 3]-box_a[:, 1]))  # [A,B]
+    area_b = ((box_b[2]-box_b[0]) *
+              (box_b[3]-box_b[1]))  # [A,B]
+    union = area_a + area_b - inter
+    return inter / union  # [A,B]
+
+
+class Compose(object):
+    """Composes several augmentations together.
+    Args:
+        transforms (List[Transform]): list of transforms to compose.
+    Example:
+        >>> augmentations.Compose([
+        >>>     transforms.CenterCrop(10),
+        >>>     transforms.ToTensor(),
+        >>> ])
+    """
+
+    def __init__(self, transforms):
+        self.transforms = transforms
+
+    def __call__(self, img, boxes=None, labels=None):
+        for t in self.transforms:
+            img, boxes, labels = t(img, boxes, labels)
+        return img, boxes, labels
+
+
+class Lambda(object):
+    """Applies a lambda as a transform."""
+
+    def __init__(self, lambd):
+        assert isinstance(lambd, types.LambdaType)
+        self.lambd = lambd
+
+    def __call__(self, img, boxes=None, labels=None):
+        return self.lambd(img, boxes, labels)
+
+
+class ConvertFromInts(object):
+    def __call__(self, image, boxes=None, labels=None):
+        return image.astype(np.float32), boxes, labels
+
+
+class SubtractMeans(object):
+    def __init__(self, mean):
+        self.mean = np.array(mean, dtype=np.float32)
+
+    def __call__(self, image, boxes=None, labels=None):
+        image = image.astype(np.float32)
+        image -= self.mean
+        return image.astype(np.float32), boxes, labels
+
+
+class ToAbsoluteCoords(object):
+    def __call__(self, image, boxes=None, labels=None):
+        height, width, channels = image.shape
+        boxes[:, 0] *= width
+        boxes[:, 2] *= width
+        boxes[:, 1] *= height
+        boxes[:, 3] *= height
+
+        return image, boxes, labels
+
+
+class ToPercentCoords(object):
+    def __call__(self, image, boxes=None, labels=None):
+        height, width, channels = image.shape
+        boxes[:, 0] /= width
+        boxes[:, 2] /= width
+        boxes[:, 1] /= height
+        boxes[:, 3] /= height
+
+        return image, boxes, labels
+
+
+class Resize(object):
+    def __init__(self, size=300):
+        self.size = size
+
+    def __call__(self, image, boxes=None, labels=None):
+        image = cv2.resize(image, (self.size,
+                                 self.size))
+        return image, boxes, labels
+
+
+class RandomSaturation(object):
+    def __init__(self, lower=0.5, upper=1.5):
+        self.lower = lower
+        self.upper = upper
+        assert self.upper >= self.lower, "contrast upper must be >= lower."
+        assert self.lower >= 0, "contrast lower must be non-negative."
+
+    def __call__(self, image, boxes=None, labels=None):
+        if random.randint(2):
+            image[:, :, 1] *= random.uniform(self.lower, self.upper)
+
+        return image, boxes, labels
+
+
+class RandomHue(object):
+    def __init__(self, delta=18.0):
+        assert delta >= 0.0 and delta <= 360.0
+        self.delta = delta
+
+    def __call__(self, image, boxes=None, labels=None):
+        if random.randint(2):
+            image[:, :, 0] += random.uniform(-self.delta, self.delta)
+            image[:, :, 0][image[:, :, 0] > 360.0] -= 360.0
+            image[:, :, 0][image[:, :, 0] < 0.0] += 360.0
+        return image, boxes, labels
+
+
+class RandomLightingNoise(object):
+    def __init__(self):
+        self.perms = ((0, 1, 2), (0, 2, 1),
+                      (1, 0, 2), (1, 2, 0),
+                      (2, 0, 1), (2, 1, 0))
+
+    def __call__(self, image, boxes=None, labels=None):
+        if random.randint(2):
+            swap = self.perms[random.randint(len(self.perms))]
+            shuffle = SwapChannels(swap)  # shuffle channels
+            image = shuffle(image)
+        return image, boxes, labels
+
+
+class ConvertColor(object):
+    def __init__(self, current, transform):
+        self.transform = transform
+        self.current = current
+
+    def __call__(self, image, boxes=None, labels=None):
+        if self.current == 'BGR' and self.transform == 'HSV':
+            image = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)
+        elif self.current == 'RGB' and self.transform == 'HSV':
+            image = cv2.cvtColor(image, cv2.COLOR_RGB2HSV)
+        elif self.current == 'BGR' and self.transform == 'RGB':
+            image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
+        elif self.current == 'HSV' and self.transform == 'BGR':
+            image = cv2.cvtColor(image, cv2.COLOR_HSV2BGR)
+        elif self.current == 'HSV' and self.transform == "RGB":
+            image = cv2.cvtColor(image, cv2.COLOR_HSV2RGB)
+        else:
+            raise NotImplementedError
+        return image, boxes, labels
+
+
+class RandomContrast(object):
+    def __init__(self, lower=0.5, upper=1.5):
+        self.lower = lower
+        self.upper = upper
+        assert self.upper >= self.lower, "contrast upper must be >= lower."
+        assert self.lower >= 0, "contrast lower must be non-negative."
+
+    # expects float image
+    def __call__(self, image, boxes=None, labels=None):
+        if random.randint(2):
+            alpha = random.uniform(self.lower, self.upper)
+            image *= alpha
+        return image, boxes, labels
+
+
+class RandomBrightness(object):
+    def __init__(self, delta=32):
+        assert delta >= 0.0
+        assert delta <= 255.0
+        self.delta = delta
+
+    def __call__(self, image, boxes=None, labels=None):
+        if random.randint(2):
+            delta = random.uniform(-self.delta, self.delta)
+            image += delta
+        return image, boxes, labels
+
+
+class ToCV2Image(object):
+    def __call__(self, tensor, boxes=None, labels=None):
+        return tensor.cpu().numpy().astype(np.float32).transpose((1, 2, 0)), boxes, labels
+
+
+class ToTensor(object):
+    def __call__(self, cvimage, boxes=None, labels=None):
+        return torch.from_numpy(cvimage.astype(np.float32)).permute(2, 0, 1), boxes, labels
+
+
+class RandomSampleCrop(object):
+    """Crop
+    Arguments:
+        img (Image): the image being input during training
+        boxes (Tensor): the original bounding boxes in pt form
+        labels (Tensor): the class labels for each bbox
+        mode (float tuple): the min and max jaccard overlaps
+    Return:
+        (img, boxes, classes)
+            img (Image): the cropped image
+            boxes (Tensor): the adjusted bounding boxes in pt form
+            labels (Tensor): the class labels for each bbox
+    """
+    def __init__(self):
+        self.sample_options = (
+            # using entire original input image
+            None,
+            # sample a patch s.t. MIN jaccard w/ obj in .1,.3,.4,.7,.9
+            (0.1, None),
+            (0.3, None),
+            (0.7, None),
+            (0.9, None),
+            # randomly sample a patch
+            (None, None),
+        )
+
+    def __call__(self, image, boxes=None, labels=None):
+        height, width, _ = image.shape
+        while True:
+            # randomly choose a mode
+            mode = random.choice(self.sample_options)
+            if mode is None:
+                return image, boxes, labels
+
+            min_iou, max_iou = mode
+            if min_iou is None:
+                min_iou = float('-inf')
+            if max_iou is None:
+                max_iou = float('inf')
+
+            # max trails (50)
+            for _ in range(50):
+                current_image = image
+
+                w = random.uniform(0.3 * width, width)
+                h = random.uniform(0.3 * height, height)
+
+                # aspect ratio constraint b/t .5 & 2
+                if h / w < 0.5 or h / w > 2:
+                    continue
+
+                left = random.uniform(width - w)
+                top = random.uniform(height - h)
+
+                # convert to integer rect x1,y1,x2,y2
+                rect = np.array([int(left), int(top), int(left+w), int(top+h)])
+
+                # calculate IoU (jaccard overlap) b/t the cropped and gt boxes
+                overlap = jaccard_numpy(boxes, rect)
+
+                # is min and max overlap constraint satisfied? if not try again
+                if overlap.min() < min_iou and max_iou < overlap.max():
+                    continue
+
+                # cut the crop from the image
+                current_image = current_image[rect[1]:rect[3], rect[0]:rect[2],
+                                              :]
+
+                # keep overlap with gt box IF center in sampled patch
+                centers = (boxes[:, :2] + boxes[:, 2:]) / 2.0
+
+                # mask in all gt boxes that above and to the left of centers
+                m1 = (rect[0] < centers[:, 0]) * (rect[1] < centers[:, 1])
+
+                # mask in all gt boxes that under and to the right of centers
+                m2 = (rect[2] > centers[:, 0]) * (rect[3] > centers[:, 1])
+
+                # mask in that both m1 and m2 are true
+                mask = m1 * m2
+
+                # have any valid boxes? try again if not
+                if not mask.any():
+                    continue
+
+                # take only matching gt boxes
+                current_boxes = boxes[mask, :].copy()
+
+                # take only matching gt labels
+                current_labels = labels[mask]
+
+                # should we use the box left and top corner or the crop's
+                current_boxes[:, :2] = np.maximum(current_boxes[:, :2],
+                                                  rect[:2])
+                # adjust to crop (by substracting crop's left,top)
+                current_boxes[:, :2] -= rect[:2]
+
+                current_boxes[:, 2:] = np.minimum(current_boxes[:, 2:],
+                                                  rect[2:])
+                # adjust to crop (by substracting crop's left,top)
+                current_boxes[:, 2:] -= rect[:2]
+
+                return current_image, current_boxes, current_labels
+
+
+class Expand(object):
+    def __init__(self, mean):
+        self.mean = mean
+
+    def __call__(self, image, boxes, labels):
+        if random.randint(2):
+            return image, boxes, labels
+
+        height, width, depth = image.shape
+        ratio = random.uniform(1, 4)
+        left = random.uniform(0, width*ratio - width)
+        top = random.uniform(0, height*ratio - height)
+
+        expand_image = np.zeros(
+            (int(height*ratio), int(width*ratio), depth),
+            dtype=image.dtype)
+        expand_image[:, :, :] = self.mean
+        expand_image[int(top):int(top + height),
+                     int(left):int(left + width)] = image
+        image = expand_image
+
+        boxes = boxes.copy()
+        boxes[:, :2] += (int(left), int(top))
+        boxes[:, 2:] += (int(left), int(top))
+
+        return image, boxes, labels
+
+
+class RandomMirror(object):
+    def __call__(self, image, boxes, classes):
+        _, width, _ = image.shape
+        if random.randint(2):
+            image = image[:, ::-1]
+            boxes = boxes.copy()
+            boxes[:, 0::2] = width - boxes[:, 2::-2]
+        return image, boxes, classes
+
+
+class SwapChannels(object):
+    """Transforms a tensorized image by swapping the channels in the order
+     specified in the swap tuple.
+    Args:
+        swaps (int triple): final order of channels
+            eg: (2, 1, 0)
+    """
+
+    def __init__(self, swaps):
+        self.swaps = swaps
+
+    def __call__(self, image):
+        """
+        Args:
+            image (Tensor): image tensor to be transformed
+        Return:
+            a tensor with channels swapped according to swap
+        """
+        # if torch.is_tensor(image):
+        #     image = image.data.cpu().numpy()
+        # else:
+        #     image = np.array(image)
+        image = image[:, :, self.swaps]
+        return image
+
+
+class PhotometricDistort(object):
+    def __init__(self):
+        self.pd = [
+            RandomContrast(),  # RGB
+            ConvertColor(current="RGB", transform='HSV'),  # HSV
+            RandomSaturation(),  # HSV
+            RandomHue(),  # HSV
+            ConvertColor(current='HSV', transform='RGB'),  # RGB
+            RandomContrast()  # RGB
+        ]
+        self.rand_brightness = RandomBrightness()
+        self.rand_light_noise = RandomLightingNoise()
+
+    def __call__(self, image, boxes, labels):
+        im = image.copy()
+        im, boxes, labels = self.rand_brightness(im, boxes, labels)
+        if random.randint(2):
+            distort = Compose(self.pd[:-1])
+        else:
+            distort = Compose(self.pd[1:])
+        im, boxes, labels = distort(im, boxes, labels)
+        return self.rand_light_noise(im, boxes, labels)
+
diff --git a/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/vision/utils/__init__.py b/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/vision/utils/__init__.py
new file mode 100644
index 000000000..b36757f20
--- /dev/null
+++ b/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/vision/utils/__init__.py
@@ -0,0 +1,24 @@
+# MIT License
+#
+# Copyright (c) 2019 Hao Gao
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+#
+
+from .misc import *
diff --git a/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/vision/utils/box_utils.py b/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/vision/utils/box_utils.py
new file mode 100644
index 000000000..2786e7e41
--- /dev/null
+++ b/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/vision/utils/box_utils.py
@@ -0,0 +1,318 @@
+# MIT License
+#
+# Copyright (c) 2019 Hao Gao
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+#
+
+import collections
+import torch
+import itertools
+from typing import List
+import math
+
+SSDBoxSizes = collections.namedtuple('SSDBoxSizes', ['min', 'max'])
+
+SSDSpec = collections.namedtuple('SSDSpec', ['feature_map_size', 'shrinkage', 'box_sizes', 'aspect_ratios'])
+
+
+def generate_ssd_priors(specs: List[SSDSpec], image_size, clamp=True) -> torch.Tensor:
+    """Generate SSD Prior Boxes.
+
+    It returns the center, height and width of the priors. The values are relative to the image size
+    Args:
+        specs: SSDSpecs about the shapes of sizes of prior boxes. i.e.
+            specs = [
+                SSDSpec(38, 8, SSDBoxSizes(30, 60), [2]),
+                SSDSpec(19, 16, SSDBoxSizes(60, 111), [2, 3]),
+                SSDSpec(10, 32, SSDBoxSizes(111, 162), [2, 3]),
+                SSDSpec(5, 64, SSDBoxSizes(162, 213), [2, 3]),
+                SSDSpec(3, 100, SSDBoxSizes(213, 264), [2]),
+                SSDSpec(1, 300, SSDBoxSizes(264, 315), [2])
+            ]
+        image_size: image size.
+        clamp: if true, clamp the values to make fall between [0.0, 1.0]
+    Returns:
+        priors (num_priors, 4): The prior boxes represented as [[center_x, center_y, w, h]]. All the values
+            are relative to the image size.
+    """
+    priors = []
+    for spec in specs:
+        scale = image_size / spec.shrinkage
+        for j, i in itertools.product(range(spec.feature_map_size), repeat=2):
+            x_center = (i + 0.5) / scale
+            y_center = (j + 0.5) / scale
+
+            # small sized square box
+            size = spec.box_sizes.min
+            h = w = size / image_size
+            priors.append([
+                x_center,
+                y_center,
+                w,
+                h
+            ])
+
+            # big sized square box
+            size = math.sqrt(spec.box_sizes.max * spec.box_sizes.min)
+            h = w = size / image_size
+            priors.append([
+                x_center,
+                y_center,
+                w,
+                h
+            ])
+
+            # change h/w ratio of the small sized box
+            size = spec.box_sizes.min
+            h = w = size / image_size
+            for ratio in spec.aspect_ratios:
+                ratio = math.sqrt(ratio)
+                priors.append([
+                    x_center,
+                    y_center,
+                    w * ratio,
+                    h / ratio
+                ])
+                priors.append([
+                    x_center,
+                    y_center,
+                    w / ratio,
+                    h * ratio
+                ])
+
+    priors = torch.tensor(priors)
+    if clamp:
+        torch.clamp(priors, 0.0, 1.0, out=priors)
+    return priors
+
+
+def convert_locations_to_boxes(locations, priors, center_variance,
+                               size_variance):
+    """Convert regressional location results of SSD into boxes in the form of (center_x, center_y, h, w).
+
+    The conversion:
+        $$predicted\_center * center_variance = \frac {real\_center - prior\_center} {prior\_hw}$$
+        $$exp(predicted\_hw * size_variance) = \frac {real\_hw} {prior\_hw}$$
+    We do it in the inverse direction here.
+    Args:
+        locations (batch_size, num_priors, 4): the regression output of SSD. It will contain the outputs as well.
+        priors (num_priors, 4) or (batch_size/1, num_priors, 4): prior boxes.
+        center_variance: a float used to change the scale of center.
+        size_variance: a float used to change of scale of size.
+    Returns:
+        boxes:  priors: [[center_x, center_y, h, w]]. All the values
+            are relative to the image size.
+    """
+    # priors can have one dimension less.
+    if priors.dim() + 1 == locations.dim():
+        priors = priors.unsqueeze(0)
+    return torch.cat([
+        locations[..., :2] * center_variance * priors[..., 2:] + priors[..., :2],
+        torch.exp(locations[..., 2:] * size_variance) * priors[..., 2:]
+    ], dim=locations.dim() - 1)
+
+
+def convert_boxes_to_locations(center_form_boxes, center_form_priors, center_variance, size_variance):
+    # priors can have one dimension less
+    if center_form_priors.dim() + 1 == center_form_boxes.dim():
+        center_form_priors = center_form_priors.unsqueeze(0)
+    return torch.cat([
+        (center_form_boxes[..., :2] - center_form_priors[..., :2]) / center_form_priors[..., 2:] / center_variance,
+        torch.log(center_form_boxes[..., 2:] / center_form_priors[..., 2:]) / size_variance
+    ], dim=center_form_boxes.dim() - 1)
+
+
+def area_of(left_top, right_bottom) -> torch.Tensor:
+    """Compute the areas of rectangles given two corners.
+
+    Args:
+        left_top (N, 2): left top corner.
+        right_bottom (N, 2): right bottom corner.
+
+    Returns:
+        area (N): return the area.
+    """
+    hw = torch.clamp(right_bottom - left_top, min=0.0)
+    return hw[..., 0] * hw[..., 1]
+
+
+def iou_of(boxes0, boxes1, eps=1e-5):
+    """Return intersection-over-union (Jaccard index) of boxes.
+
+    Args:
+        boxes0 (N, 4): ground truth boxes.
+        boxes1 (N or 1, 4): predicted boxes.
+        eps: a small number to avoid 0 as denominator.
+    Returns:
+        iou (N): IoU values.
+    """
+    overlap_left_top = torch.max(boxes0[..., :2], boxes1[..., :2])
+    overlap_right_bottom = torch.min(boxes0[..., 2:], boxes1[..., 2:])
+
+    overlap_area = area_of(overlap_left_top, overlap_right_bottom)
+    area0 = area_of(boxes0[..., :2], boxes0[..., 2:])
+    area1 = area_of(boxes1[..., :2], boxes1[..., 2:])
+    return overlap_area / (area0 + area1 - overlap_area + eps)
+
+
+def assign_priors(gt_boxes, gt_labels, corner_form_priors,
+                  iou_threshold):
+    """Assign ground truth boxes and targets to priors.
+
+    Args:
+        gt_boxes (num_targets, 4): ground truth boxes.
+        gt_labels (num_targets): labels of targets.
+        priors (num_priors, 4): corner form priors
+    Returns:
+        boxes (num_priors, 4): real values for priors.
+        labels (num_priros): labels for priors.
+    """
+    # size: num_priors x num_targets
+    ious = iou_of(gt_boxes.unsqueeze(0), corner_form_priors.unsqueeze(1))
+    # size: num_priors
+    best_target_per_prior, best_target_per_prior_index = ious.max(1)
+    # size: num_targets
+    best_prior_per_target, best_prior_per_target_index = ious.max(0)
+
+    for target_index, prior_index in enumerate(best_prior_per_target_index):
+        best_target_per_prior_index[prior_index] = target_index
+    # 2.0 is used to make sure every target has a prior assigned
+    best_target_per_prior.index_fill_(0, best_prior_per_target_index, 2)
+    # size: num_priors
+    labels = gt_labels[best_target_per_prior_index]
+    labels[best_target_per_prior < iou_threshold] = 0  # the backgournd id
+    boxes = gt_boxes[best_target_per_prior_index]
+    return boxes, labels
+
+
+def hard_negative_mining(loss, labels, neg_pos_ratio):
+    """
+    It used to suppress the presence of a large number of negative prediction.
+    It works on image level not batch level.
+    For any example/image, it keeps all the positive predictions and
+     cut the number of negative predictions to make sure the ratio
+     between the negative examples and positive examples is no more
+     the given ratio for an image.
+
+    Args:
+        loss (N, num_priors): the loss for each example.
+        labels (N, num_priors): the labels.
+        neg_pos_ratio:  the ratio between the negative examples and positive examples.
+    """
+    pos_mask = labels > 0
+    num_pos = pos_mask.long().sum(dim=1, keepdim=True)
+    num_neg = num_pos * neg_pos_ratio
+
+    loss[pos_mask] = -math.inf
+    _, indexes = loss.sort(dim=1, descending=True)
+    _, orders = indexes.sort(dim=1)
+    neg_mask = orders < num_neg
+    return pos_mask | neg_mask
+
+
+def center_form_to_corner_form(locations):
+    return torch.cat([locations[..., :2] - locations[..., 2:]/2,
+                     locations[..., :2] + locations[..., 2:]/2], locations.dim() - 1) 
+
+
+def corner_form_to_center_form(boxes):
+    return torch.cat([
+        (boxes[..., :2] + boxes[..., 2:]) / 2,
+         boxes[..., 2:] - boxes[..., :2]
+    ], boxes.dim() - 1)
+
+
+def hard_nms(box_scores, iou_threshold, top_k=-1, candidate_size=200):
+    """
+
+    Args:
+        box_scores (N, 5): boxes in corner-form and probabilities.
+        iou_threshold: intersection over union threshold.
+        top_k: keep top_k results. If k <= 0, keep all the results.
+        candidate_size: only consider the candidates with the highest scores.
+    Returns:
+         picked: a list of indexes of the kept boxes
+    """
+    scores = box_scores[:, -1]
+    boxes = box_scores[:, :-1]
+    picked = []
+    _, indexes = scores.sort(descending=True)
+    indexes = indexes[:candidate_size]
+    while len(indexes) > 0:
+        current = indexes[0]
+        picked.append(current.item())
+        if 0 < top_k == len(picked) or len(indexes) == 1:
+            break
+        current_box = boxes[current, :]
+        indexes = indexes[1:]
+        rest_boxes = boxes[indexes, :]
+        iou = iou_of(
+            rest_boxes,
+            current_box.unsqueeze(0),
+        )
+        indexes = indexes[iou <= iou_threshold]
+
+    return box_scores[picked, :]
+
+
+def nms(box_scores, nms_method=None, score_threshold=None, iou_threshold=None,
+        sigma=0.5, top_k=-1, candidate_size=200):
+    if nms_method == "soft":
+        return soft_nms(box_scores, score_threshold, sigma, top_k)
+    else:
+        return hard_nms(box_scores, iou_threshold, top_k, candidate_size=candidate_size)
+
+
+def soft_nms(box_scores, score_threshold, sigma=0.5, top_k=-1):
+    """Soft NMS implementation.
+
+    References:
+        https://arxiv.org/abs/1704.04503
+        https://github.com/facebookresearch/Detectron/blob/master/detectron/utils/cython_nms.pyx
+
+    Args:
+        box_scores (N, 5): boxes in corner-form and probabilities.
+        score_threshold: boxes with scores less than value are not considered.
+        sigma: the parameter in score re-computation.
+            scores[i] = scores[i] * exp(-(iou_i)^2 / simga)
+        top_k: keep top_k results. If k <= 0, keep all the results.
+    Returns:
+         picked_box_scores (K, 5): results of NMS.
+    """
+    picked_box_scores = []
+    while box_scores.size(0) > 0:
+        max_score_index = torch.argmax(box_scores[:, 4])
+        cur_box_prob = torch.tensor(box_scores[max_score_index, :])
+        picked_box_scores.append(cur_box_prob)
+        if len(picked_box_scores) == top_k > 0 or box_scores.size(0) == 1:
+            break
+        cur_box = cur_box_prob[:-1]
+        box_scores[max_score_index, :] = box_scores[-1, :]
+        box_scores = box_scores[:-1, :]
+        ious = iou_of(cur_box.unsqueeze(0), box_scores[:, :-1])
+        box_scores[:, -1] = box_scores[:, -1] * torch.exp(-(ious * ious) / sigma)
+        box_scores = box_scores[box_scores[:, -1] > score_threshold, :]
+    if len(picked_box_scores) > 0:
+        return torch.stack(picked_box_scores)
+    else:
+        return torch.tensor([])
+
+
+
diff --git a/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/vision/utils/measurements.py b/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/vision/utils/measurements.py
new file mode 100644
index 000000000..394ae1f8d
--- /dev/null
+++ b/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/vision/utils/measurements.py
@@ -0,0 +1,55 @@
+# MIT License
+#
+# Copyright (c) 2019 Hao Gao
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+#
+
+import numpy as np
+
+
+def compute_average_precision(precision, recall):
+    """
+    It computes average precision based on the definition of Pascal Competition. It computes the under curve area
+    of precision and recall. Recall follows the normal definition. Precision is a variant.
+    pascal_precision[i] = typical_precision[i:].max()
+    """
+    # identical but faster version of new_precision[i] = old_precision[i:].max()
+    precision = np.concatenate([[0.0], precision, [0.0]])
+    for i in range(len(precision) - 1, 0, -1):
+        precision[i - 1] = np.maximum(precision[i - 1], precision[i])
+
+    # find the index where the value changes
+    recall = np.concatenate([[0.0], recall, [1.0]])
+    changing_points = np.where(recall[1:] != recall[:-1])[0]
+
+    # compute under curve area
+    areas = (recall[changing_points + 1] - recall[changing_points]) * precision[changing_points + 1]
+    return areas.sum()
+
+
+def compute_voc2007_average_precision(precision, recall):
+    ap = 0.
+    for t in np.arange(0., 1.1, 0.1):
+        if np.sum(recall >= t) == 0:
+            p = 0
+        else:
+            p = np.max(precision[recall >= t])
+        ap = ap + p / 11.
+    return ap
diff --git a/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/vision/utils/misc.py b/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/vision/utils/misc.py
new file mode 100644
index 000000000..5388a899d
--- /dev/null
+++ b/models_v2/pytorch/ssd-mobilenetv1/inference/gpu/vision/utils/misc.py
@@ -0,0 +1,68 @@
+# MIT License
+#
+# Copyright (c) 2019 Hao Gao
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy
+# of this software and associated documentation files (the "Software"), to deal
+# in the Software without restriction, including without limitation the rights
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+# copies of the Software, and to permit persons to whom the Software is
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in all
+# copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+# SOFTWARE.
+#
+
+import time
+import torch
+
+
+def str2bool(s):
+    return s.lower() in ('true', '1')
+
+
+class Timer:
+    def __init__(self):
+        self.clock = {}
+
+    def start(self, key="default"):
+        self.clock[key] = time.time()
+
+    def end(self, key="default"):
+        if key not in self.clock:
+            raise Exception(f"{key} is not in the clock.")
+        interval = time.time() - self.clock[key]
+        del self.clock[key]
+        return interval
+        
+
+def save_checkpoint(epoch, net_state_dict, optimizer_state_dict, best_score, checkpoint_path, model_path):
+    torch.save({
+        'epoch': epoch,
+        'model': net_state_dict,
+        'optimizer': optimizer_state_dict,
+        'best_score': best_score
+    }, checkpoint_path)
+    torch.save(net_state_dict, model_path)
+        
+        
+def load_checkpoint(checkpoint_path):
+    return torch.load(checkpoint_path)
+
+
+def freeze_net_layers(net):
+    for param in net.parameters():
+        param.requires_grad = False
+
+
+def store_labels(path, labels):
+    with open(path, "w") as f:
+        f.write("\n".join(labels))
diff --git a/models_v2/pytorch/stable_diffusion/inference/gpu/CONTAINER_FLEX.md b/models_v2/pytorch/stable_diffusion/inference/gpu/CONTAINER_FLEX.md
new file mode 100644
index 000000000..1fb2a4f75
--- /dev/null
+++ b/models_v2/pytorch/stable_diffusion/inference/gpu/CONTAINER_FLEX.md
@@ -0,0 +1,73 @@
+# Running Stable Diffusion inference on Intel® Data Center GPU Flex Series using Intel® Extension for PyTorch*
+
+## Overview
+
+This document has instructions for running Stable Diffusion inference using Intel® Extension for PyTorch on Intel® Flex Series GPU.
+
+## Requirements
+| Item | Detail |
+| ------ | ------- |
+| Host machine  | [Intel® Data Center GPU Flex Series](https://ark.intel.com/content/www/us/en/ark/products/series/230021/intel-data-center-gpu-flex-series.html) 170  |
+| Drivers | GPU-compatible drivers need to be installed: Download [Driver](https://dgpu-docs.intel.com/driver/installation.html) |
+| Software | Docker* |
+
+## Quick Start Scripts
+
+| Script name | Description |
+|-------------|-------------|
+| `run_model.sh` | Inference with batch size 1 on Flex series 170 |
+
+> [!NOTE]
+> At the moment sample supports FP16 precision only (`export PRECISION=fp16`).
+
+## Run Using Docker
+
+### Set up Docker Image
+
+```
+docker pull intel/generative-ai:pytorch-flex-gpu-stable-diffusion-inference
+```
+
+### Run Docker Image
+The stable diffusion inference container includes scripts, model and libraries needed to run FP16 inference. To run the `run_model.sh` quickstart script using this container, you will need to provide an output directory where log files will be written.
+
+```bash
+#Optional
+export PRECISION=fp16
+export BATCH_SIZE=<batch size otherwise (default: 1)>
+
+#Required
+export OUTPUT_DIR=<path to output directory>
+export SCRIPT=run_model.sh
+export MULTI_TILE=False
+export PLATFORM=Flex
+
+IMAGE_NAME=intel/generative-ai:pytorch-flex-gpu-stable-diffusion-inference
+DOCKER_ARGS="--rm -it"
+
+docker run \
+  --privileged \
+  --device=/dev/dri \
+  --ipc=host \
+  --env PRECISION=${PRECISION} \
+  --env OUTPUT_DIR=${OUTPUT_DIR} \
+  --env MULTI_TILE=${MULTI_TILE} \
+  --env PLATFORM=${PLATFORM} \
+  --env http_proxy=${http_proxy} \
+  --env https_proxy=${https_proxy} \
+  --env no_proxy=${no_proxy} \
+  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
+  ${DOCKER_ARGS} \
+  ${IMAGE_NAME} \
+  /bin/bash $SCRIPT
+```
+## Documentation and Sources
+
+[GitHub* Repository](https://github.com/IntelAI/models/tree/master/docker/flex-gpu)
+
+## Support
+Support for Intel® Extension for PyTorch* is found via the [Intel® AI Analytics Toolkit.](https://www.intel.com/content/www/us/en/developer/tools/oneapi/ai-analytics-toolkit.html#gs.qbretz) Additionally, the Intel® Extension for PyTorch* team tracks both bugs and enhancement requests using [GitHub issues](https://github.com/intel/intel-extension-for-pytorch/issues). Before submitting a suggestion or bug report, please search the GitHub issues to see if your issue has already been reported.
+
+## License Agreement
+
+LEGAL NOTICE: By accessing, downloading or using this software and any required dependent software (the “Software Package”), you agree to the terms and conditions of the software license agreements for the Software Package, which may also include notices, disclaimers, or license terms for third party software included with the Software Package. Please refer to the [license file](https://github.com/IntelAI/models/tree/master/third_party) for additional details.
diff --git a/models_v2/pytorch/stable_diffusion/inference/gpu/CONTAINER_MAX.md b/models_v2/pytorch/stable_diffusion/inference/gpu/CONTAINER_MAX.md
new file mode 100644
index 000000000..8511453ed
--- /dev/null
+++ b/models_v2/pytorch/stable_diffusion/inference/gpu/CONTAINER_MAX.md
@@ -0,0 +1,70 @@
+# Running Stable Diffusion inference on Intel® Data Center GPU Max Series using Intel® Extension for PyTorch*
+
+## Overview
+
+This document has instructions for running Stable Diffusion inference using Intel® Extension for PyTorch on Intel® Max Series GPU.
+
+## Requirements
+| Item | Detail |
+| ------ | ------- |
+| Host machine  | [Intel® Data Center GPU Max Series](https://ark.intel.com/content/www/us/en/ark/products/series/232874/intel-data-center-gpu-max-series.html)  |
+| Drivers | GPU-compatible drivers need to be installed: Download [Driver](https://dgpu-docs.intel.com/driver/installation.html) |
+| Software | Docker* |
+
+## Quick Start Scripts
+
+| Script name | Description |
+|-------------|-------------|
+| `run_model.sh` | Inference for specified precision FP16 with batch size 1 on Max Series |
+
+## Run Using Docker
+
+### Set up Docker Image
+
+```
+docker pull intel/generative-ai:pytorch-max-gpu-stable-diffusion-inference
+```
+
+### Run Docker Image
+The stable diffusion inference container includes scripts,model and libraries need to run FP16 inference. To run the `run_model.sh` quickstart script using this container, you will need to provide an output directory where log files will be written. 
+
+```bash
+#Optiional
+export PRECISION=fp16
+export BATCH_SIZE=<provide batch size otherwise, batch size of 1 will be used>
+
+#Required
+export OUTPUT_DIR=<path to output directory>
+export SCRIPT=run_model.sh
+export MULTI_TILE=True
+export PLATFORM=Max
+
+IMAGE_NAME=intel/generative-ai:pytorch-max-gpu-stable-diffusion-inference
+DOCKER_ARGS="--rm -it"
+
+docker run \
+  --privileged \
+  --device=/dev/dri \
+  --ipc=host \
+  --env PRECISION=${PRECISION} \
+  --env OUTPUT_DIR=${OUTPUT_DIR} \
+  --env MULTI_TILE=${MULTI_TILE} \
+  --env PLATFORM=${PLATFORM} \
+  --env http_proxy=${http_proxy} \
+  --env https_proxy=${https_proxy} \
+  --env no_proxy=${no_proxy} \
+  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
+  ${DOCKER_ARGS} \
+  ${IMAGE_NAME} \
+  /bin/bash $SCRIPT
+```
+## Documentation and Sources
+
+[GitHub* Repository](https://github.com/IntelAI/models/tree/master/docker/max-gpu)
+
+## Support
+Support for Intel® Extension for PyTorch* is found via the [Intel® AI Analytics Toolkit.](https://www.intel.com/content/www/us/en/developer/tools/oneapi/ai-analytics-toolkit.html#gs.qbretz) Additionally, the Intel® Extension for PyTorch* team tracks both bugs and enhancement requests using [GitHub issues](https://github.com/intel/intel-extension-for-pytorch/issues). Before submitting a suggestion or bug report, please search the GitHub issues to see if your issue has already been reported.
+
+## License Agreement
+
+LEGAL NOTICE: By accessing, downloading or using this software and any required dependent software (the “Software Package”), you agree to the terms and conditions of the software license agreements for the Software Package, which may also include notices, disclaimers, or license terms for third party software included with the Software Package. Please refer to the [license file](https://github.com/IntelAI/models/tree/master/third_party) for additional details.
diff --git a/models_v2/pytorch/stable_diffusion/inference/gpu/README.md b/models_v2/pytorch/stable_diffusion/inference/gpu/README.md
new file mode 100644
index 000000000..7f3d174b6
--- /dev/null
+++ b/models_v2/pytorch/stable_diffusion/inference/gpu/README.md
@@ -0,0 +1,103 @@
+# Stable Diffusion Inference
+
+Stable Diffusion Inference best known configurations with Intel® Extension for PyTorch.
+
+## Model Information
+
+| **Use Case** | **Framework** | **Model Repo** | **Branch/Commit/Tag** | **Optional Patch** |
+|:---:| :---: |:--------------:|:---------------------:|:------------------:|
+|  Inference   |    Pytorch    |       -        |           -           |         -          |
+
+# Pre-Requisite
+* Host has one of the following GPUs:
+  * **Arc Series** - [Intel® Arc™ A-Series Graphics](https://ark.intel.com/content/www/us/en/ark/products/series/227957/intel-arc-a-series-graphics.html)
+  * **Flex Series** - [Intel® Data Center GPU Flex Series](https://ark.intel.com/content/www/us/en/ark/products/series/230021/intel-data-center-gpu-flex-series.html)
+  * **Max Series** - [Intel® Data Center GPU Max Series](https://ark.intel.com/content/www/us/en/ark/products/series/232874/intel-data-center-gpu-max-series.html)
+* Host has installed latest Intel® Data Center GPU Max & Flex Series Drivers https://dgpu-docs.intel.com/driver/installation.html
+* The following Intel® oneAPI Base Toolkit components are required:
+  - Intel® oneAPI DPC++ Compiler (Placeholder DPCPPROOT as its installation path)
+  - Intel® oneAPI Math Kernel Library (oneMKL) (Placeholder MKLROOT as its installation path)
+  - Intel® oneAPI MPI Library
+  - Intel® oneAPI TBB Library
+
+  Follow instructions at [Intel® oneAPI Base Toolkit Download page](https://www.intel.com/content/www/us/en/developer/tools/oneapi/base-toolkit-download.html?operatingsystem=linux) to setup the package manager repository.
+
+# Dataset
+The scripts will download the dataset automatically. It uses nateraw/parti-prompts (https://huggingface.co/datasets/nateraw/parti-prompts) as the dataset. 
+
+## Inference
+1. `git clone https://github.com/IntelAI/models.git`
+2. `cd models/models_v2/pytorch/stable_diffusion/inference/gpu`
+3. Create virtual environment `venv` and activate it:
+    ```
+    python3 -m venv venv
+    . ./venv/bin/activate
+    ```
+4. Run setup.sh
+    ```
+    ./setup.sh
+    ```
+5. Install the latest GPU versions of [torch, torchvision and intel_extension_for_pytorch](https://intel.github.io/intel-extension-for-pytorch/index.html#installation):
+    ```
+    python -m pip install torch==<torch_version> torchvision==<torchvvision_version> intel-extension-for-pytorch==<ipex_version> --extra-index-url https://pytorch-extension.intel.com/release-whl-aitools/
+    ```
+6. Set environment variables for Intel® oneAPI Base Toolkit: 
+    Default installation location `{ONEAPI_ROOT}` is `/opt/intel/oneapi` for root account, `${HOME}/intel/oneapi` for other accounts
+    ```bash
+    source {ONEAPI_ROOT}/compiler/latest/env/vars.sh
+    source {ONEAPI_ROOT}/mkl/latest/env/vars.sh
+    source {ONEAPI_ROOT}/tbb/latest/env/vars.sh
+    source {ONEAPI_ROOT}/mpi/latest/env/vars.sh
+    source {ONEAPI_ROOT}/ccl/latest/env/vars.sh
+    ```
+7. Setup required environment paramaters
+
+| **Parameter**                |                                  **export command**                                  |
+|:---------------------------:|:------------------------------------------------------------------------------------:|
+| **MULTI_TILE**               | `export MULTI_TILE=True` (True or False)                                             |
+| **PLATFORM**                 | `export PLATFORM=Max` (Max or Flex or ARC)                                                 |
+| **BATCH_SIZE** (optional)    |                               `export BATCH_SIZE=1`                                |
+| **PRECISION** (optional)     |           `export PRECISION=fp16` (fp16 and fp32 are supported for all platform)|
+| **OUTPUT_DIR** (optional)    |                               `export OUTPUT_DIR=$PWD`                               |
+8. Run `run_model.sh`
+
+> [!NOTE]
+> Refer to [CONTAINER_FLEX.md](CONTAINER_FLEX.md) and [CONTAINER_MAX.md](CONTAINER_MAX.md) for Stable Diffusion Inference instructions using docker containers.
+## Output
+
+Single-tile output will typically looks like:
+
+```
+No policy available for current head_size 512
+inference Latency: 3671.8995094299316 ms
+inference Throughput: 0.2723386076966065 samples/s
+CLIP score: 33.59451
+```
+
+Multi-tile output will typically looks like:
+```
+26%|██▌       | 13/50 [00:00<00:01, 20.13it/s]inference Latency: 3714.4706646601358 ms
+inference Throughput: 0.2692173637320938 samples/s
+CLIP score: 33.58945666666667
+100%|██████████| 50/50 [00:02<00:00, 19.64it/s]
+No policy available for current head_size 512
+inference Latency: 3794.5287148157754 ms
+inference Throughput: 0.26353733893104825 samples/s
+CLIP score: 33.58307666666666
+```
+please noted that we have using it/s as the throughput. you can find in the `results.yaml`.
+
+Final results of the inference run can be found in `results.yaml` file.
+```
+results:
+ - key: throughput
+   value: 41.4400
+   unit: it/s
+it/s
+ - key: latency
+   value: 0.0482633
+   unit: s
+ - key: accuracy
+   value: 33.5335
+   unit: accuracy
+```
diff --git a/models_v2/pytorch/stable_diffusion/inference/gpu/main.py b/models_v2/pytorch/stable_diffusion/inference/gpu/main.py
new file mode 100644
index 000000000..633811885
--- /dev/null
+++ b/models_v2/pytorch/stable_diffusion/inference/gpu/main.py
@@ -0,0 +1,287 @@
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+import os
+import time
+import sys
+import torch
+from diffusers import DPMSolverMultistepScheduler, StableDiffusionPipeline
+import argparse
+import numpy as np
+from scipy.linalg import sqrtm
+from PIL import Image
+import pytorch_fid
+
+from datasets import load_dataset
+
+from torchmetrics.functional.multimodal import clip_score
+from functools import partial
+
+parser = argparse.ArgumentParser(description='PyTorch StableDiffusion TexttoImage')
+parser.add_argument('--prompt', default="nateraw/parti-prompts", type=str, help='prompt_dataset')
+parser.add_argument('--batch_size', default=1, type=int, help='batch size')
+parser.add_argument('--idx_start', default=0, type=int, help='select the start index of image')
+parser.add_argument('--precision', choices=["fp32", "fp16", "bf16"],
+                    default="fp16", type=str, help='precision')
+parser.add_argument('--jit', action='store_true', default=False, help='enable JIT')
+parser.add_argument('--iteration', default=30, type=int, help='test iterations')
+parser.add_argument('--warmup_iter', default=2, type=int, help='test warmup')
+parser.add_argument('--device', default='xpu', type=str, help='cpu, cuda or xpu')
+parser.add_argument('--save_image', action='store_true', default=False, help='save image')
+parser.add_argument('--save_tensor', action='store_true', default=False, help='save tensor')
+parser.add_argument('--accuracy', action='store_true', default=False, help='compare the result with cuda')
+parser.add_argument('-m', '--model_id',
+                    choices=["CompVis/stable-diffusion-v1-4", "stabilityai/stable-diffusion-2-1"],
+                    default='stabilityai/stable-diffusion-2-1', type=str, metavar='PATH',
+                    help='path to model structure or weight')
+parser.add_argument('--ref_path', default='', type=str, metavar='PATH',
+                    help='path to reference image (default: none)')
+parser.add_argument('--save_path', default='./xpu_result', type=str, help='output image dir')
+parser.add_argument('--num_inference_steps', default=50, type=int, help='number of unet step')
+parser.add_argument('--evaluate_method', choices=["clip", "fid"],
+                    default="fid", type=str, help='evaluation method, now we suppor clip and fid')
+args = parser.parse_args()
+print(args)
+
+def compare(xpu_res, ref_res):
+    xpu = torch.tensor(xpu_res) 
+    ref = torch.tensor(ref_res) 
+    
+    diff_value = torch.abs((xpu - ref))
+    max_diff = torch.max(diff_value)
+
+    shape = 1
+    for i in range(xpu.dim()):
+        shape = shape * xpu.shape[i]
+
+    value = diff_value > 0.1
+    num = torch.sum(value.contiguous().view(-1))
+    ratio1 = num / shape
+    print("difference larger than 0.1, ratio = {}".format(ratio1))  
+  
+    value = diff_value > 0.01 
+    num = torch.sum(value.contiguous().view(-1))
+    ratio2 = num / shape
+    print("difference larger than 0.01, ratio = {}".format(ratio2))  
+
+    value = diff_value > 0.001 
+    num = torch.sum(value.contiguous().view(-1))
+    ratio3 = num / shape
+    print("difference larger than 0.001, ratio = {}".format(ratio3))  
+
+    if ratio1 < 0.01 and ratio2 < 0.08 and ratio3 < 0.4:
+        print("accuracy pass")
+    else:
+        print("accuracy fail")
+        
+def compare_pil_images(ref_res, cur_res):
+    xpu = torch.tensor(np.array(cur_res)) 
+    ref = torch.tensor(np.array(ref_res)) 
+    
+    diff_value = torch.abs((xpu - ref))
+    max_diff = torch.max(diff_value)
+
+    shape = 1
+    for i in range(xpu.dim()):
+        shape = shape * xpu.shape[i]
+
+    value = diff_value > 0.1
+    num = torch.sum(value.contiguous().view(-1))
+    ratio1 = num / shape
+    print("difference larger than 0.1, ratio = {}".format(ratio1))  
+  
+    value = diff_value > 0.01 
+    num = torch.sum(value.contiguous().view(-1))
+    ratio2 = num / shape
+    print("difference larger than 0.01, ratio = {}".format(ratio2))  
+
+    value = diff_value > 0.001 
+    num = torch.sum(value.contiguous().view(-1))
+    ratio3 = num / shape
+    print("difference larger than 0.001, ratio = {}".format(ratio3))  
+
+    if ratio1 < 0.01 and ratio2 < 0.08 and ratio3 < 0.4:
+        print("accuracy pass")
+    else:
+        print("accuracy fail")
+
+def main():
+    def calculate_clip_score(images, prompts):
+        images_int = (images * 255).astype("uint8")
+        clip_score = clip_score_fn(torch.from_numpy(images_int).permute(0, 3, 1, 2), prompts).detach()
+        return round(float(clip_score), 4)
+
+    profiling = os.environ.get("PROFILE", "OFF").upper() in ["1", "Y", "ON", "YES", "TRUE"]
+
+    # prompt = ["A painting of a squirrel eating a burger"]
+    prompts_dataset = load_dataset(args.prompt, split="train")
+
+    seed = 666
+    prompts_dataset = prompts_dataset.shuffle(seed=seed)
+
+    if args.device == "xpu":
+        import intel_extension_for_pytorch as ipex
+        idx = torch.xpu.current_device()
+        generator = torch.xpu.default_generators[idx]
+        generator.manual_seed(seed)
+    elif args.device == "cuda":
+        generator = torch.Generator(device=args.device).manual_seed(seed)
+    else:
+        generator = torch.Generator(device=args.device)
+
+    amp_enabled = False
+    if args.precision == "fp32":
+        datatype = torch.float
+    elif args.precision == "fp16":
+        datatype = torch.float16
+        # amp_enabled = True
+    elif args.precision == "bf16":
+        datatype = torch.bfloat16
+        amp_enabled = True
+    else:
+        print("unsupported datatype")
+        sys.exit()
+
+    pipe = StableDiffusionPipeline.from_pretrained(args.model_id, torch_dtype=datatype)
+    pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)
+    pipe = pipe.to(args.device)
+    if args.precision == "fp16":
+        pipe = pipe.to(torch.float16)
+
+    if args.device == "xpu":
+        # optimize with ipex
+        pipe.unet = torch.xpu.optimize(pipe.unet.eval(), dtype=datatype, inplace=True)
+        pipe.vae = torch.xpu.optimize(pipe.vae.eval(), dtype=datatype, inplace=True)
+        pipe.text_encoder = torch.xpu.optimize(pipe.text_encoder.eval(), dtype=datatype, inplace=True)
+        print("---- Use ipex optimized model.")
+
+    if args.evaluate_method == "clip":
+        clip_score_fn = partial(clip_score, model_name_or_path="openai/clip-vit-base-patch16")
+        out_type = "np"
+        clip_score_list = []
+    else: 
+        out_type = "pil"
+
+    if args.accuracy or args.save_tensor:
+        out_type = "tensor"
+
+    total_time = 0
+    print("output type is: ", out_type)
+    with torch.no_grad():
+        for step in range(args.warmup_iter):
+            idx1 = args.idx_start + int(step * args.batch_size)
+            idx2 = args.idx_start + int((step + 1) * args.batch_size)
+            input = prompts_dataset[step]["Prompt"]
+            print("input is : ", input)
+            if args.device == "xpu":
+                with torch.xpu.amp.autocast(enabled=amp_enabled, dtype=datatype):
+                    images = pipe(input, generator=generator, num_inference_steps=args.num_inference_steps, output_type=out_type).images
+                torch.xpu.synchronize()
+            elif args.device == "cuda":
+                images = pipe(input, generator=generator, num_inference_steps=args.num_inference_steps, output_type=out_type).images
+                torch.cuda.synchronize()
+            else:
+                images = pipe(input, generator=generator, num_inference_steps=args.num_inference_steps, output_type=out_type).images
+
+        image_before = []
+        iter = 0
+        for iter in range(args.iteration):
+            print("Iteration = {}".format(iter))
+            step = 0
+            idx1 = args.idx_start + int(step * args.batch_size)
+            idx2 = args.idx_start + int((step + 1) * args.batch_size)
+            print("idx1={}".format(idx1))
+            print("idx2={}".format(idx2))
+
+            input = prompts_dataset[iter]["Prompt"]
+            print("input is : ", input)
+
+
+            if args.device == "xpu":
+                with torch.autograd.profiler_legacy.profile(profiling, use_xpu=True, record_shapes=True) as prof:
+                    try:
+                        import memory_check
+                        memory_check.display_mem("xpu:0")
+                    except:
+                        pass
+                    start_time = time.time()
+                    with torch.xpu.amp.autocast(enabled=amp_enabled, dtype=datatype):
+                        images = pipe(input, generator=generator, num_inference_steps=args.num_inference_steps, output_type=out_type).images
+                    torch.xpu.synchronize()
+                    end_time = time.time()
+                if profiling:
+                    torch.save(prof.key_averages().table(sort_by="self_xpu_time_total"), 'stale_diffusion_inf_profile.pt')
+                    torch.save(prof.table(sort_by="id", row_limit=-1), 'stable_diffusion_inf_profile_detailed.pt')
+                    prof.export_chrome_trace('./stable_diffusion_inf_profile_trace.json')
+            elif args.device == "cuda":
+                start_time = time.time()
+                images = pipe(input, generator=generator, num_inference_steps=args.num_inference_steps, output_type=out_type).images
+                torch.cuda.synchronize()
+                end_time = time.time()
+            else:
+                start_time = time.time()
+                images = pipe(input, generator=generator, num_inference_steps=args.num_inference_steps, output_type=out_type).images
+                end_time = time.time()
+
+
+            iter_time = end_time - start_time
+            total_time += iter_time
+            # latency = total_time / (step + 1)
+            # throughput = args.batch_size / latency
+            # print("---latency={} s".format(latency))
+            # print("---throughput={} fps".format(throughput))
+
+            if args.evaluate_method == "clip":
+                sd_clip_score = calculate_clip_score(images, input)
+                clip_score_list.append(sd_clip_score)
+            else:
+                if args.accuracy:
+                    for i in range(args.batch_size):
+                        name = "result_{}_{}.png".format(idx1 + i, iter) if args.save_image else "result_{}_{}.pt".format(idx1 + i, iter)
+                        name = os.path.join(args.ref_path, name)
+                        if args.save_image:
+                            ref_image = Image.open(name)
+                            compare_pil_images(ref_image, images[i])
+                        else:
+                            ref_pt = torch.load(name)
+                            compare(ref_pt, images[i])
+
+                if not os.path.exists(args.save_path):
+                    os.mkdir(args.save_path)
+
+                if args.save_tensor:
+                    for i in range(args.batch_size):
+                        file_name = "./result_{}_{}.pt".format(idx1 + i, iter)
+                        save_path = os.path.join(args.save_path, file_name)
+                        torch.save(images[i], save_path)
+
+                if args.save_image:
+                    for i in range(args.batch_size):
+                        file_name = "./result_{}_{}.png".format(idx1 + i, iter)
+                        save_path = os.path.join(args.save_path, file_name)
+                        images[i].save(save_path)
+            iter += 1
+
+        total_sample = args.iteration * args.batch_size
+        latency = total_time / total_sample * 1000
+        throughput = total_sample / total_time
+        print("inference Latency: {} ms".format(latency))
+        print("inference Throughput: {} samples/s".format(throughput))
+        if args.evaluate_method == "clip":
+            print(f"CLIP score: {np.mean(clip_score_list)}")
+
+
+if __name__ == '__main__':
+    main() 
diff --git a/models_v2/pytorch/stable_diffusion/inference/gpu/requirements.txt b/models_v2/pytorch/stable_diffusion/inference/gpu/requirements.txt
new file mode 100644
index 000000000..9f6bf30fd
--- /dev/null
+++ b/models_v2/pytorch/stable_diffusion/inference/gpu/requirements.txt
@@ -0,0 +1,7 @@
+datasets
+torchmetrics
+diffusers
+transformers
+accelerate
+pytorch-fid
+scipy==1.11.1
diff --git a/models_v2/pytorch/stable_diffusion/inference/gpu/run_model.sh b/models_v2/pytorch/stable_diffusion/inference/gpu/run_model.sh
new file mode 100644
index 000000000..9cf5dc344
--- /dev/null
+++ b/models_v2/pytorch/stable_diffusion/inference/gpu/run_model.sh
@@ -0,0 +1,134 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+
+#!/bin/bash
+
+# Create an array of input directories that are expected and then verify that they exist
+declare -A input_envs
+input_envs[MULTI_TILE]=${MULTI_TILE}
+input_envs[PLATFORM]=${PLATFORM}
+input_envs[OUTPUT_DIR]=${OUTPUT_DIR}
+
+for i in "${!input_envs[@]}"; do
+  var_name=$i
+  env_param=${input_envs[$i]}
+
+  if [[ -z $env_param ]]; then
+    echo "The required environment variable $var_name is not set" >&2
+    exit 1
+  fi
+done
+
+if [[ "${PLATFORM}" == "Max" ]]; then
+    BATCH_SIZE=${BATCH_SIZE:-1}
+    PRECISION=${PRECISION:-fp16}
+elif [[ "${PLATFORM}" == "Flex" ]]; then
+    if [[ "${MULTI_TILE}" == "True" ]]; then
+	echo "Flex GPU does not support multitile"
+	exit 1
+    fi
+    BATCH_SIZE=${BATCH_SIZE:-1}
+    PRECISION=${PRECISION:-fp16}
+elif [[ "${PLATFORM}" == "ARC" ]]; then
+    if [[ "${MULTI_TILE}" == "True" ]]; then
+        echo "ARC not support multitile"
+        exit 1
+    fi
+    BATCH_SIZE=${BATCH_SIZE:-1}
+    PRECISION=${PRECISION:-fp16}
+fi
+
+
+
+# known issue
+if [[ "${MULTI_TILE}" == "True" ]]; then
+    export ZE_FLAT_DEVICE_HIERARCHY=COMPOSITE
+fi
+
+echo 'Running with parameters:'
+echo " PLATFORM: ${PLATFORM}"
+echo " OUTPUT_DIR: ${OUTPUT_DIR}"
+echo " PRECISION: ${PRECISION}"
+echo " BATCH_SIZE: ${BATCH_SIZE}"
+echo " MULTI_TILE: ${MULTI_TILE}"
+
+echo "stable-diffusion ${PRECISION} inference plain MultiTile=${MULTI_TILE} BS=${BATCH_SIZE}"
+
+# Create the output directory, if it doesn't already exist
+mkdir -p $OUTPUT_DIR
+
+sum_log_analysis() {
+    if [ -f $2 ]; then
+        rm -f $2
+    fi
+    if diff /dev/null ${1}_t0.log |tail -l | grep '^\\ No newline' > /dev/null;then echo >> ${1}_t0.log; fi
+    if diff /dev/null ${1}_t1.log |tail -l | grep '^\\ No newline' > /dev/null;then echo >> ${1}_t1.log; fi
+    bs=$(cat ${1}_t1.log |grep Batch |awk '{print $3}')
+    echo -e "Batch Size: $bs" >$2
+    cat ${1}"_t0.log" ${1}"_t1.log" |grep "Performance" |awk -v tag=$(cat ${1}"_t0.log" ${1}"_t1.log" |grep "Performance" |awk '{sum+=$2} END {printf "%.4f\n",sum}') '{if ( $2=="None" ) {sum="None";nextfile}else  sum=tag} ;END{print "Sum "$1" "sum " "$3}' >> $2
+    cat ${1}"_t0.log" ${1}"_t1.log" |grep "Performance" |awk -v tag=$(cat ${1}"_t0.log" ${1}"_t1.log" |grep "Performance" |awk 'BEGIN {min=1234567890123} {if ($2 <min) {min=$2}}END {printf "%.4f\n",min}') '{if ( $2=="None" ) {min="None";nextfile}else  min=tag} ;END{print "Min "$1" "min " "$3}' >> $2
+    cat ${1}"_t0.log" ${1}"_t1.log" |grep "Latency" |awk '{if ( $2=="N/A" ){avg="N/A";nextfile}else avg=((sum+=$2/2))};END{print "Avg "$1" "avg " "$3}'  >> $2
+    cat ${1}"_t0.log" ${1}"_t1.log" |grep "Accuracy" |awk -v avg=$(cat ${1}"_t0.log" ${1}"_t1.log" |grep "Accuracy" |awk '{sum+=$3}END{printf "%.4f\n",sum/NR}') '{if ( $3=="None" || $2=="N/A" || $3=="nan" || $3=="N/A"){avg="None";nextfile}else avg=avg};END{print "Avg "$1" "$2 " "avg}' >> $2
+    cat ${1}"_t0.log" ${1}"_t1.log" |grep "Functional" | awk -v fail=$(cat ${1}"_t0.log" ${1}"_t1.log" |grep "Functional" |awk '{for(i=1;i<=NF;++i) if($i=="fail") ++sum}END{print sum}') '{if ( fail >= 1 ) tag="fail ";else tag="pass"};END{print $1" "tag}'  >> $2
+    cat ${1}"_t0.log" ${1}"_t1.log" |grep "Error" |awk '{if(a[$1]){a[$1]=a[$1]";"$2}else{a[$1]=$2}}END{for(i in a)print $1" " a[i]}' >> $2
+}
+
+modelname=stable-diffusion
+if [[ ${MULTI_TILE} == "False" ]]; then
+    rm ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0_raw.log
+    python -u main.py --save_image --save_path xpu_result0 --precision ${PRECISION}  --evaluate_method clip 2>&1  | tee ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0_raw.log
+    python common/parse_result.py -m $modelname -l ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0_raw.log -b ${BATCH_SIZE}
+    throughput=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0.log | grep Performance | awk -F ' ' '{print $2}')
+    throughput_unit=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0.log | grep Performance | awk -F ' ' '{print $3}')
+    latency=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0.log | grep Latency | awk -F ' ' '{print $2}')
+    acc=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0.log | grep Accuracy | awk -F ' ' '{print $3}')
+    acc_unit=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0.log | grep Accuracy | awk -F ' ' '{print $2}')
+else
+    rm ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0_raw.log
+    rm ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t1_raw.log
+    ZE_AFFINITY_MASK=0.0 python -u main.py --save_image --save_path xpu_result0 --precision ${PRECISION}  --evaluate_method clip 2>&1 | tee ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0_raw.log &
+    ZE_AFFINITY_MASK=0.1 python -u main.py --save_image --save_path xpu_result1 --precision ${PRECISION}  --evaluate_method clip 2>&1 | tee ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t1_raw.log
+    wait
+    python common/parse_result.py -m $modelname -l ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t0_raw.log -b ${BATCH_SIZE}
+    python common/parse_result.py -m $modelname -l ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf_t1_raw.log -b ${BATCH_SIZE}  
+    sum_log_analysis ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf.log
+    throughput=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf.log | grep "Sum Performance" | awk -F ' ' '{print $3}')
+    throughput_unit=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf.log | grep Performance | awk -F ' ' '{print $4}')
+    latency=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf.log | grep Latency | awk -F ' ' '{print $3}')
+    acc=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf.log | grep Accuracy | awk -F ' ' '{print $4}')
+    acc_unit=$(cat ${OUTPUT_DIR}/${modelname}_${PRECISION}_inf.log | grep Accuracy | awk -F ' ' '{print $3}')
+fi
+
+yaml_content=$(cat <<EOF
+results:
+ - key: throughput
+   value: $throughput
+   unit: $throughput_unit
+ - key: latency
+   value: $latency
+   unit: s
+ - key: accuracy
+   value: $acc
+   unit: $acc_unit
+EOF
+)
+
+# Write the content to a YAML file
+echo "$yaml_content" >  ${OUTPUT_DIR}/results.yaml
+echo "YAML file created."
diff --git a/models_v2/pytorch/stable_diffusion/inference/gpu/setup.sh b/models_v2/pytorch/stable_diffusion/inference/gpu/setup.sh
new file mode 100755
index 000000000..a0cf02206
--- /dev/null
+++ b/models_v2/pytorch/stable_diffusion/inference/gpu/setup.sh
@@ -0,0 +1,27 @@
+#!/usr/bin/env bash
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# setup.sh
+#  - install OS pkgs
+#  - should create virtual env & install pip  requirement.txt
+#  - git clones & applying patches
+
+set -e
+apt-get update && apt-get install -y python3-venv
+pip install -r requirements.txt
+
+cp -r ../../../../common .
diff --git a/models_v2/pytorch/stable_diffusion/inference/gpu/stable-diffusion.py b/models_v2/pytorch/stable_diffusion/inference/gpu/stable-diffusion.py
new file mode 100644
index 000000000..fe91ad01b
--- /dev/null
+++ b/models_v2/pytorch/stable_diffusion/inference/gpu/stable-diffusion.py
@@ -0,0 +1,35 @@
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Consts for stable-diffusion inf."""
+
+ACC = {
+    "type": "total",
+    "pattern": r"CLIP score: (\d+.\d+)",
+    "unit": "accuracy",
+}
+
+PERF = {
+    "type": "total",
+    "pattern": r"50/50 .* (\d+.\d+)it/s",
+    "inverse": False,
+    "multiply": False,
+    "use_batch_size": False,
+    "unit": "it/s",
+}
+
+FUNCTIONAL = {
+    "pattern": r"CLIP score: (\d+.\d+)",
+}
diff --git a/models_v2/pytorch/stable_diffusion/inference/gpu/tests.yaml b/models_v2/pytorch/stable_diffusion/inference/gpu/tests.yaml
new file mode 100644
index 000000000..899b8c72e
--- /dev/null
+++ b/models_v2/pytorch/stable_diffusion/inference/gpu/tests.yaml
@@ -0,0 +1,31 @@
+inference-fp16:
+  if: ${{ eq(RUNNER, "flex-gpu") }}
+  cmd: bash $GITHUB_WORKSPACE/tests/cicd/IPEX-XPU/setup.sh ${FRAMEWORK_VERSION} ${FRAMEWORK_EXTENSION_VERSION} ${TORCHVISION_VERSION} ${IS_LKG_DROP} ${AIKIT_VERSION}; bash $GITHUB_WORKSPACE/tests/cicd/IPEX-XPU/stable_diffusion-inference/run_test.sh ${PRECISION} ${IS_LKG_DROP} ${MULTI_TILE} ${PLATFORM}
+  env:
+    PRECISION: "fp16"
+    MULTI_TILE: "False"
+    PLATFORM: "Flex"
+
+inference-fp32:
+  if: ${{ eq(RUNNER, "flex-gpu") }}
+  cmd: bash $GITHUB_WORKSPACE/tests/cicd/IPEX-XPU/setup.sh ${FRAMEWORK_VERSION} ${FRAMEWORK_EXTENSION_VERSION} ${TORCHVISION_VERSION} ${IS_LKG_DROP} ${AIKIT_VERSION}; bash $GITHUB_WORKSPACE/tests/cicd/IPEX-XPU/stable_diffusion-inference/run_test.sh ${PRECISION} ${IS_LKG_DROP} ${MULTI_TILE} ${PLATFORM}
+  env:
+    PRECISION: "fp32"
+    MULTI_TILE: "False"
+    PLATFORM: "Flex"
+
+inference-pvc-fp16:
+  if: ${{ eq(RUNNER, "max-gpu") }}
+  cmd: bash $GITHUB_WORKSPACE/tests/cicd/IPEX-XPU/setup.sh ${FRAMEWORK_VERSION} ${FRAMEWORK_EXTENSION_VERSION} ${TORCHVISION_VERSION} ${IS_LKG_DROP} ${AIKIT_VERSION}; bash $GITHUB_WORKSPACE/tests/cicd/IPEX-XPU/stable_diffusion-inference/run_test.sh ${PRECISION} ${IS_LKG_DROP} ${MULTI_TILE} ${PLATFORM}
+  env:
+    PRECISION: "fp16"
+    MULTI_TILE: "True"
+    PLATFORM: "Max"
+
+inference-pvc-fp32:
+  if: ${{ eq(RUNNER, "max-gpu") }}
+  cmd: bash $GITHUB_WORKSPACE/tests/cicd/IPEX-XPU/setup.sh ${FRAMEWORK_VERSION} ${FRAMEWORK_EXTENSION_VERSION} ${TORCHVISION_VERSION} ${IS_LKG_DROP} ${AIKIT_VERSION}; bash $GITHUB_WORKSPACE/tests/cicd/IPEX-XPU/stable_diffusion-inference/run_test.sh ${PRECISION} ${IS_LKG_DROP} ${MULTI_TILE} ${PLATFORM}
+  env:
+    PRECISION: "fp32"
+    MULTI_TILE: "True"
+    PLATFORM: "Max"
diff --git a/models_v2/pytorch/torchrec_dlrm/inference/gpu/README.md b/models_v2/pytorch/torchrec_dlrm/inference/gpu/README.md
new file mode 100644
index 000000000..e7041d298
--- /dev/null
+++ b/models_v2/pytorch/torchrec_dlrm/inference/gpu/README.md
@@ -0,0 +1,101 @@
+# DLRM v2 Inference
+
+DLRM v2 Inference best known configurations with Intel® Extension for PyTorch.
+
+## Model Information
+
+| **Use Case** | **Framework** | **Model Repo** | **Branch/Commit/Tag** | **Optional Patch** |
+|:---:| :---: |:--------------:|:---------------------:|:------------------:|
+|  Inference   |    PyTorch    |       https://github.com/facebookresearch/dlrm/tree/main/torchrec_dlrm        |           -           |         -          |
+
+# Pre-Requisite
+* Host has 4 Intel® Data Center GPU Max and two tiles for each.
+* Host has installed latest Intel® Data Center GPU Max Series Drivers https://dgpu-docs.intel.com/driver/installation.html
+* Host has installed [Intel® Extension for PyTorch](https://intel.github.io/intel-extension-for-pytorch/xpu/latest/)
+
+# prepare Dataset
+After downloading and uncompressing the [Criteo 1TB Click Logs dataset](consisting of 24 files from day 0 to day 23), process the raw tsv files into the proper format for training by running ./scripts/process_Criteo_1TB_Click_Logs_dataset.sh with necessary command line arguments.
+
+Example usage:
+
+bash ./scripts/process_Criteo_1TB_Click_Logs_dataset.sh \
+./criteo_1tb/raw_input_dataset_dir \
+./criteo_1tb/temp_intermediate_files_dir \
+./criteo_1tb/numpy_contiguous_shuffled_output_dataset_dir
+The script requires 700GB of RAM and takes 1-2 days to run. We currently have features in development to reduce the preproccessing time and memory overhead. MD5 checksums of the expected final preprocessed dataset files are in md5sums_preprocessed_criteo_click_logs_dataset.txt.
+
+the final dataset dir will be like below:
+dataset_dir
+ |_day_0_dense.npy
+ |_day_0_labels.npy
+ |_day_0_sparse_multi_hot.npz
+
+this folder will be used as the parameter DATASET_DIR later
+
+wget https://cloud.mlcommons.org/index.php/s/XzfSeLgW8FYfR3S/download -O weigths.zip
+unzip weights.zip
+and the folder will be used as the parameter WEIGHT_DIR later
+
+
+## Inference
+1. `git clone https://github.com/IntelAI/models.git`
+2. `cd models/models_v2/pytorch/torchrec_dlrm/inference/gpu`
+3. Run `setup.sh` this will install all the required dependencies & create virtual environment `venv`.
+4. Activate virtual env: `. ./venv/bin/activate`
+5. Setup required environment paramaters
+
+| **Parameter**                |                                  **export command**                                  |
+|:---------------------------:|:------------------------------------------------------------------------------------:|
+| **MULTI_TILE**               | `export MULTI_TILE=True` (True or False)                                             |
+| **PLATFORM**                 | `export PLATFORM=PVC` (PVC)                                                 |
+| **WEIGHT_DIR**               | `export WEIGHT_DIR=`                                                                 |
+| **DATASET_DIR**              |                               `export DATASET_DIR=`                                  |
+| **BATCH_SIZE** (optional)    |                               `export BATCH_SIZE=32768`                              |
+| **PRECISION** (optional)     |        `export PRECISION=FP16` (FP16 and FP32 are supported for PVC)                 |
+| **OUTPUT_DIR** (optional)    |                               `export OUTPUT_DIR=$PWD`                               |
+6. Run `run_model.sh`
+
+## Output
+
+Multi-tile output will typically looks like:
+```
+[0] 2024-01-10 21:50:10,779 - __main__ - INFO - avg eval time per iter at ITER: 45, 0.03749502139621311 s
+[6] 2024-01-10 21:50:10,779 - __main__ - INFO - avg eval time per iter at ITER: 45, 0.03693882624308268 s
+[1] 2024-01-10 21:50:10,779 - __main__ - INFO - avg eval time per iter at ITER: 45, 0.03849502139621311 s
+[3] 2024-01-10 21:50:10,779 - __main__ - INFO - avg eval time per iter at ITER: 45, 0.03693882624308268 s
+[7] 2024-01-10 21:50:10,779 - __main__ - INFO - avg eval time per iter at ITER: 45, 0.03858977953592936 s
+[2] 2024-01-10 21:50:10,779 - __main__ - INFO - avg eval time per iter at ITER: 45, 0.03870058589511448 s
+[4] 2024-01-10 21:50:10,780 - __main__ - INFO - avg eval time per iter at ITER: 45, 0.022177388932969836 s
+[5] 2024-01-10 21:50:10,780 - __main__ - INFO - avg eval time per iter at ITER: 45, 0.037547969818115236 s
+[0] AUROC over test set: 0.8147445321083069.
+[0] Number of test samples: 3276800
+[0] 2024:01:10-21:50:11:(34779) |CCL_INFO| finalize atl-mpi
+[0] 2024:01:10-21:50:11:(34779) |CCL_INFO| finalized atl-mpi
+[3] 2024:01:10-21:50:11:(34782) |CCL_INFO| finalizing level-zero
+[7] 2024:01:10-21:50:11:(34786) |CCL_INFO| finalizing level-zero
+[0] 2024:01:10-21:50:11:(34779) |CCL_INFO| finalizing level-zero
+[6] 2024:01:10-21:50:11:(34785) |CCL_INFO| finalizing level-zero
+[4] 2024:01:10-21:50:11:(34783) |CCL_INFO| finalizing level-zero
+[3] 2024:01:10-21:50:11:(34782) |CCL_INFO| finalized level-zero
+[5] 2024:01:10-21:50:11:(34784) |CCL_INFO| finalizing level-zero
+[7] 2024:01:10-21:50:11:(34786) |CCL_INFO| finalized level-zero
+[0] 2024:01:10-21:50:11:(34779) |CCL_INFO| finalized level-zero
+[2] 2024:01:10-21:50:11:(34781) |CCL_INFO| finalizing level-zero
+[6] 2024:01:10-21:50:11:(34785) |CCL_INFO| finalized level-zero
+[4] 2024:01:10-21:50:11:(34783) |CCL_INFO| finalized level-zero
+[5] 2024:01:10-21:50:11:(34784) |CCL_INFO| finalized level-zero
+[2] 2024:01:10-21:50:11:(34781) |CCL_INFO| finalized level-zero
+[1] 2024:01:10-21:50:11:(34780) |CCL_INFO| finalizing level-zero
+[1] 2024:01:10-21:50:11:(34780) |CCL_INFO| finalized level-zero
+```
+
+Final results of the inference run can be found in `results.yaml` file.
+```
+results:
+ - key: throughput
+   value: 1693411.31
+   unit: samples/s
+ - key: accuracy
+   value: 0.815
+   unit: AUROC
+```
diff --git a/models_v2/pytorch/torchrec_dlrm/inference/gpu/cmd_distributed_terabyte_test.sh b/models_v2/pytorch/torchrec_dlrm/inference/gpu/cmd_distributed_terabyte_test.sh
new file mode 100644
index 000000000..8839e5be0
--- /dev/null
+++ b/models_v2/pytorch/torchrec_dlrm/inference/gpu/cmd_distributed_terabyte_test.sh
@@ -0,0 +1,131 @@
+#!/bin/bash
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+set -x
+
+function Parser() {
+    while [ $# -ne 0 ]; do
+        case $1 in
+            -b)
+                shift
+                export GLOBAL_BATCH_SIZE="$1"
+                ;;
+            -fp16)
+                shift
+                FP16="$1"
+                ;;
+            -d)
+                shift
+                DATA="$1"
+                ;;
+            -m)
+                shift
+                WEIGHT="$1"
+                ;;
+            -nd)
+                shift
+                ND="$1"
+                ;;
+            -sp)
+                shift
+                SP="$1"
+                ;;
+            -tf32)
+                shift
+                TF32="$1"
+                ;;
+            -tv)
+                shift
+                TV="$1"
+                ;;
+            -h | --help)
+                echo "Usage: cmd_infer.sh [OPTION...] PAGE..."
+                echo "-b, Optional    Specify the batch size. The default value is 32768"
+                echo "-fp16, Optional    Specify the input dtype is fp16. The default value is true"
+                echo "-d, Optional    Specify the data file"
+                echo "-m, Optional    Specify the weight file"
+                echo "-nd, Optional    Specify the number of node"
+                echo "-sp, Optional    Specify the sharding plan of embedding"
+                echo "-tf32, Optional    Specify the input dtype is tf32. The default value is false"
+                echo "-tv, Optional    Training with val. The default value is false"
+                exit
+                ;;
+            --*|-*)
+                echo ">>> New param: <$1>"
+                ;;
+            *)
+                echo ">>> Parsing mismatch: $1"
+                ;;
+        esac
+        shift
+    done
+}
+
+torch_ccl_path=$(python -c "import torch; import oneccl_bindings_for_pytorch; import os;  print(os.path.abspath(os.path.dirname(oneccl_bindings_for_pytorch.__file__)))")
+source $torch_ccl_path/env/setvars.sh
+export MASTER_ADDR='127.0.0.1'
+#export WORLD_SIZE=2 ;
+export MASTER_PORT='10088'
+export TOTAL_TRAINING_SAMPLES=4195197692;
+export GLOBAL_BATCH_SIZE=65536;
+
+ND=1
+SP="round_robin"
+#export CCL_LOG_LEVEL=DEBUG;
+#export CCL_OP_SYNC=1
+
+DATA=${DATA-'/home/sdp/xw/dlrm-v2/'}
+WEIGHT=${WEIGHT-'/home/sdp/xw/model_weights'}
+
+${FP16:=true}
+${TF32:=false}
+${TV:=false}
+Parser $@
+ARGS+=" --embedding_dim 128"
+ARGS+=" --dense_arch_layer_sizes 512,256,128"
+ARGS+=" --over_arch_layer_sizes 1024,1024,512,256,1"
+ARGS+=" --num_embeddings_per_feature 40000000,39060,17295,7424,20265,3,7122,1543,63,40000000,3067956,405282,10,2209,11938,155,4,976,14,40000000,40000000,40000000,590152,12973,108,36"
+ARGS+=" --validation_freq_within_epoch $((TOTAL_TRAINING_SAMPLES / (GLOBAL_BATCH_SIZE * 20 * 1000)))"
+ARGS+=" --synthetic_multi_hot_criteo_path $DATA"
+ARGS+=" --multi_hot_sizes 3,2,1,2,6,1,1,1,1,7,3,8,1,6,9,5,1,1,1,12,100,27,10,3,1,1"
+#ARGS+=" --multi_hot_distribution_type uniform"
+ARGS+=" --use_xpu"
+ARGS+=" --epochs 1"
+ARGS+=" --pin_memory"
+ARGS+=" --mmap_mode"
+ARGS+=" --batch_size $GLOBAL_BATCH_SIZE"
+ARGS+=" --interaction_type=dcn"
+ARGS+=" --dcn_num_layers=3"
+ARGS+=" --adagrad"
+ARGS+=" --dcn_low_rank_dim=512"
+ARGS+=" --numpy_rand_seed=12345"
+ARGS+=" --log_freq 10"
+ARGS+=" --amp"
+ARGS+=" --inference_only"
+ARGS+=" --snapshot_dir ${WEIGHT}"
+ARGS+=" --limit_test_batches 50"
+ARGS+=" --sharding_plan ${SP}"
+ARGS+=" --num_nodes ${ND}"
+ARGS+=" --learning_rate 0.005"
+
+[ "$TV" = true ]             && ARGS+=" --train_with_val"
+if [ "$TF32" = false ]; then
+        [ "$FP16" = true ]             && ARGS+=" --fp16"
+        echo "${ARGS}"
+	mpirun -np 8 -ppn 8 --prepend-rank python -u dlrm_main.py  ${ARGS}
+else
+        echo "${ARGS}"
+	IPEX_FP32_MATH_MODE=1 mpirun -np 8 -ppn 8 --prepend-rank python -u dlrm_main.py ${ARGS}
+fi
diff --git a/models_v2/pytorch/torchrec_dlrm/inference/gpu/data/dlrm_dataloader.py b/models_v2/pytorch/torchrec_dlrm/inference/gpu/data/dlrm_dataloader.py
new file mode 100644
index 000000000..969fd1f02
--- /dev/null
+++ b/models_v2/pytorch/torchrec_dlrm/inference/gpu/data/dlrm_dataloader.py
@@ -0,0 +1,198 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#!/usr/bin/env python3
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+import argparse
+import os
+from typing import List
+
+from torch import distributed as dist
+from torch.utils.data import DataLoader
+from torchrec.datasets.criteo import (
+    CAT_FEATURE_COUNT,
+    DAYS,
+    DEFAULT_CAT_NAMES,
+    DEFAULT_INT_NAMES,
+    InMemoryBinaryCriteoIterDataPipe,
+)
+# This is for crop dataset
+DAYS_MIN=1
+from torchrec.datasets.random import RandomRecDataset
+
+# OSS import
+try:
+    # pyre-ignore[21]
+    # @manual=//ai_codesign/benchmarks/dlrm/torchrec_dlrm/data:multi_hot_criteo
+    from data.multi_hot_criteo import MultiHotCriteoIterDataPipe
+
+except ImportError:
+    pass
+
+# internal import
+try:
+    from .multi_hot_criteo import MultiHotCriteoIterDataPipe  # noqa F811
+except ImportError:
+    pass
+
+STAGES = ["train", "val", "test"]
+
+
+def _get_random_dataloader(
+    args: argparse.Namespace,
+    stage: str,
+) -> DataLoader:
+    attr = f"limit_{stage}_batches"
+    num_batches = getattr(args, attr)
+    if stage in ["val", "test"] and args.test_batch_size is not None:
+        batch_size = args.test_batch_size
+    else:
+        batch_size = args.batch_size
+    return DataLoader(
+        RandomRecDataset(
+            keys=DEFAULT_CAT_NAMES,
+            batch_size=batch_size,
+            hash_size=args.num_embeddings,
+            hash_sizes=args.num_embeddings_per_feature
+            if hasattr(args, "num_embeddings_per_feature")
+            else None,
+            manual_seed=args.seed if hasattr(args, "seed") else None,
+            ids_per_feature=1,
+            num_dense=len(DEFAULT_INT_NAMES),
+            num_batches=num_batches,
+        ),
+        batch_size=None,
+        batch_sampler=None,
+        pin_memory=args.pin_memory,
+        num_workers=0,
+    )
+
+
+def _get_in_memory_dataloader(
+    args: argparse.Namespace,
+    stage: str,
+) -> DataLoader:
+    if args.in_memory_binary_criteo_path is not None:
+        dir_path = args.in_memory_binary_criteo_path
+        sparse_part = "sparse.npy"
+        datapipe = InMemoryBinaryCriteoIterDataPipe
+    else:
+        dir_path = args.synthetic_multi_hot_criteo_path
+        sparse_part = "sparse_multi_hot.npz"
+        datapipe = MultiHotCriteoIterDataPipe
+
+    if args.dataset_name == "criteo_kaggle":
+        # criteo_kaggle has no validation set, so use 2nd half of training set for now.
+        # Setting stage to "test" will get the 2nd half of the dataset.
+        # Setting root_name to "train" reads from the training set file.
+        (root_name, stage) = ("train", "test") if stage == "val" else stage
+        stage_files: List[List[str]] = [
+            [os.path.join(dir_path, f"{root_name}_dense.npy")],
+            [os.path.join(dir_path, f"{root_name}_{sparse_part}")],
+            [os.path.join(dir_path, f"{root_name}_labels.npy")],
+        ]
+    # criteo_1tb code path uses below two conditionals
+    elif stage == "train":
+        if args.converge:
+            stage_files: List[List[str]] = [
+                [os.path.join(dir_path, f"day_{i}_dense.npy") for i in range(DAYS - 1)],
+                [os.path.join(dir_path, "multihot", f"day_{i}_{sparse_part}") for i in range(DAYS - 1)],
+                [os.path.join(dir_path, f"day_{i}_labels.npy") for i in range(DAYS - 1)],
+            ]
+        else:
+            stage_files: List[List[str]] = [
+                # for crop dataset
+                [os.path.join(dir_path, f"day_{i}_dense.npy") for i in range(DAYS_MIN)],
+                [os.path.join(dir_path, f"day_{i}_{sparse_part}") for i in range(DAYS_MIN)],
+                [os.path.join(dir_path, f"day_{i}_labels.npy") for i in range(DAYS_MIN)],
+            ]
+    elif stage in ["val", "test"]:
+        if args.converge:
+            stage_files: List[List[str]] = [
+                [os.path.join(dir_path, f"day_{DAYS-1}_dense.npy")],
+                [os.path.join(dir_path, "multihot", f"day_{DAYS-1}_{sparse_part}")],
+                [os.path.join(dir_path, f"day_{DAYS-1}_labels.npy")],
+            ]
+        else:
+            stage_files: List[List[str]] = [
+                [os.path.join(dir_path, f"day_{DAYS_MIN-1}_dense.npy")],
+                [os.path.join(dir_path, f"day_{DAYS_MIN-1}_{sparse_part}")],
+                [os.path.join(dir_path, f"day_{DAYS_MIN-1}_labels.npy")],
+            ]
+    if stage in ["val", "test"] and args.test_batch_size is not None:
+        batch_size = args.test_batch_size
+    else:
+        batch_size = args.batch_size
+    dataloader = DataLoader(
+        datapipe(
+            stage,
+            *stage_files,  # pyre-ignore[6]
+            batch_size=batch_size,
+            #rank=dist.get_rank(),
+            #world_size=dist.get_world_size(),
+            # The rand and world_size set for custom dist-dlrm
+            rank=0,
+            world_size=1,
+            drop_last=args.drop_last_training_batch if stage == "train" else False,
+            shuffle_batches=args.shuffle_batches,
+            shuffle_training_set=args.shuffle_training_set,
+            shuffle_training_set_random_seed=args.seed,
+            mmap_mode=args.mmap_mode,
+            hashes=args.num_embeddings_per_feature
+            if args.num_embeddings is None
+            else ([args.num_embeddings] * CAT_FEATURE_COUNT),
+        ),
+        batch_size=None,
+        num_workers=0,
+        pin_memory=args.pin_memory,
+        collate_fn=lambda x: x,
+    )
+    return dataloader
+
+
+def get_dataloader(args: argparse.Namespace, backend: str, stage: str) -> DataLoader:
+    """
+    Gets desired dataloader from dlrm_main command line options. Currently, this
+    function is able to return either a DataLoader wrapped around a RandomRecDataset or
+    a Dataloader wrapped around an InMemoryBinaryCriteoIterDataPipe.
+
+    Args:
+        args (argparse.Namespace): Command line options supplied to dlrm_main.py's main
+            function.
+        backend (str): "nccl" or "gloo".
+        stage (str): "train", "val", or "test".
+
+    Returns:
+        dataloader (DataLoader): PyTorch dataloader for the specified options.
+
+    """
+    stage = stage.lower()
+    if stage not in STAGES:
+        raise ValueError(f"Supplied stage was {stage}. Must be one of {STAGES}.")
+
+    args.pin_memory = (
+        (backend == "nccl") if not hasattr(args, "pin_memory") else args.pin_memory
+    )
+
+    if (
+        args.in_memory_binary_criteo_path is None
+        and args.synthetic_multi_hot_criteo_path is None
+    ):
+        return _get_random_dataloader(args, stage)
+    else:
+        return _get_in_memory_dataloader(args, stage)
diff --git a/models_v2/pytorch/torchrec_dlrm/inference/gpu/data/multi_hot_criteo.py b/models_v2/pytorch/torchrec_dlrm/inference/gpu/data/multi_hot_criteo.py
new file mode 100644
index 000000000..cb520e2ec
--- /dev/null
+++ b/models_v2/pytorch/torchrec_dlrm/inference/gpu/data/multi_hot_criteo.py
@@ -0,0 +1,310 @@
+#!/usr/bin/env python3
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+import zipfile
+from typing import Dict, Iterator, List, Optional
+
+import numpy as np
+import torch
+from iopath.common.file_io import PathManager, PathManagerFactory
+from pyre_extensions import none_throws
+from torch.utils.data import IterableDataset
+from torchrec.datasets.criteo import (
+    CAT_FEATURE_COUNT,
+    DEFAULT_CAT_NAMES,
+)
+from torchrec.datasets.utils import Batch, PATH_MANAGER_KEY
+from torchrec.sparse.jagged_tensor import KeyedJaggedTensor
+
+
+class MultiHotCriteoIterDataPipe(IterableDataset):
+    """
+    Datapipe designed to operate over the MLPerf DLRM v2 synthetic multi-hot dataset.
+    This dataset can be created by following the steps in
+    torchrec_dlrm/scripts/materialize_synthetic_multihot_dataset.py.
+    Each rank reads only the data for the portion of the dataset it is responsible for.
+
+    Args:
+        stage (str): "train", "val", or "test".
+        dense_paths (List[str]): List of path strings to dense npy files.
+        sparse_paths (List[str]): List of path strings to multi-hot sparse npz files.
+        labels_paths (List[str]): List of path strings to labels npy files.
+        batch_size (int): batch size.
+        rank (int): rank.
+        world_size (int): world size.
+        drop_last (Optional[bool]): Whether to drop the last batch if it is incomplete.
+        shuffle_batches (bool): Whether to shuffle batches
+        shuffle_training_set (bool): Whether to shuffle all samples in the dataset.
+        shuffle_training_set_random_seed (int): The random generator seed used when
+            shuffling the training set.
+        hashes (Optional[int]): List of max categorical feature value for each feature.
+            Length of this list should be CAT_FEATURE_COUNT.
+        path_manager_key (str): Path manager key used to load from different
+            filesystems.
+
+    Example::
+
+        datapipe = MultiHotCriteoIterDataPipe(
+            dense_paths=["day_0_dense.npy"],
+            sparse_paths=["day_0_sparse_multi_hot.npz"],
+            labels_paths=["day_0_labels.npy"],
+            batch_size=1024,
+            rank=torch.distributed.get_rank(),
+            world_size=torch.distributed.get_world_size(),
+        )
+        batch = next(iter(datapipe))
+    """
+
+    def __init__(
+        self,
+        stage: str,
+        dense_paths: List[str],
+        sparse_paths: List[str],
+        labels_paths: List[str],
+        batch_size: int,
+        rank: int,
+        world_size: int,
+        drop_last: Optional[bool] = False,
+        shuffle_batches: bool = False,
+        shuffle_training_set: bool = False,
+        shuffle_training_set_random_seed: int = 0,
+        mmap_mode: bool = False,
+        hashes: Optional[List[int]] = None,
+        path_manager_key: str = PATH_MANAGER_KEY,
+    ) -> None:
+        self.stage = stage
+        self.dense_paths = dense_paths
+        self.sparse_paths = sparse_paths
+        self.labels_paths = labels_paths
+        self.batch_size = batch_size
+        self.rank = rank
+        self.world_size = world_size
+        self.drop_last = drop_last
+        self.shuffle_batches = shuffle_batches
+        self.shuffle_training_set = shuffle_training_set
+        np.random.seed(shuffle_training_set_random_seed)
+        self.mmap_mode = mmap_mode
+        # hashes are not used because they were already applied in the
+        # script that generates the multi-hot dataset.
+        self.hashes: np.ndarray = np.array(hashes).reshape((CAT_FEATURE_COUNT, 1))
+        self.path_manager_key = path_manager_key
+        self.path_manager: PathManager = PathManagerFactory().get(path_manager_key)
+
+        if shuffle_training_set and stage == "train":
+            # Currently not implemented for the materialized multi-hot dataset.
+            self._shuffle_and_load_data_for_rank()
+        else:
+            m = "r" if mmap_mode else None
+            self.dense_arrs: List[np.ndarray] = [
+                np.load(f, mmap_mode=m) for f in self.dense_paths
+            ]
+            self.labels_arrs: List[np.ndarray] = [
+                np.load(f, mmap_mode=m) for f in self.labels_paths
+            ]
+            self.sparse_arrs: List = []
+            for sparse_path in self.sparse_paths:
+                multi_hot_ids_l = []
+                for feat_id_num in range(CAT_FEATURE_COUNT):
+                    multi_hot_ft_ids = self._load_from_npz(
+                        sparse_path, f"{feat_id_num}.npy"
+                    )
+                    multi_hot_ids_l.append(multi_hot_ft_ids)
+                self.sparse_arrs.append(multi_hot_ids_l)
+        len_d0 = len(self.dense_arrs[0])
+        second_half_start_index = int(len_d0 // 2 + len_d0 % 2)
+        if stage == "val":
+            self.dense_arrs[0] = self.dense_arrs[0][:second_half_start_index, :]
+            self.labels_arrs[0] = self.labels_arrs[0][:second_half_start_index, :]
+            self.sparse_arrs[0] = [
+                feats[:second_half_start_index, :] for feats in self.sparse_arrs[0]
+            ]
+        elif stage == "test":
+            self.dense_arrs[0] = self.dense_arrs[0][second_half_start_index:, :]
+            self.labels_arrs[0] = self.labels_arrs[0][second_half_start_index:, :]
+            self.sparse_arrs[0] = [
+                feats[second_half_start_index:, :] for feats in self.sparse_arrs[0]
+            ]
+        # When mmap_mode is enabled, sparse features are hashed when
+        # samples are batched in def __iter__. Otherwise, the dataset has been
+        # preloaded with sparse features hashed in the preload stage, here:
+        # if not self.mmap_mode and self.hashes is not None:
+        #     for k, _ in enumerate(self.sparse_arrs):
+        #         self.sparse_arrs[k] = [
+        #             feat % hash
+        #             for (feat, hash) in zip(self.sparse_arrs[k], self.hashes)
+        #         ]
+
+        self.num_rows_per_file: List[int] = list(map(len, self.dense_arrs))
+        total_rows = sum(self.num_rows_per_file)
+        self.num_full_batches: int = (
+            total_rows // batch_size // self.world_size * self.world_size
+        )
+        self.last_batch_sizes: np.ndarray = np.array(
+            [0 for _ in range(self.world_size)]
+        )
+        remainder = total_rows % (self.world_size * batch_size)
+        if not self.drop_last and 0 < remainder:
+            if remainder < self.world_size:
+                self.num_full_batches -= self.world_size
+                self.last_batch_sizes += batch_size
+            else:
+                self.last_batch_sizes += remainder // self.world_size
+            self.last_batch_sizes[: remainder % self.world_size] += 1
+
+        self.multi_hot_sizes: List[int] = [
+            multi_hot_feat.shape[-1] for multi_hot_feat in self.sparse_arrs[0]
+        ]
+
+        # These values are the same for the KeyedJaggedTensors in all batches, so they
+        # are computed once here. This avoids extra work from the KeyedJaggedTensor sync
+        # functions.
+        self.keys: List[str] = DEFAULT_CAT_NAMES
+        self.index_per_key: Dict[str, int] = {
+            key: i for (i, key) in enumerate(self.keys)
+        }
+
+    def _load_from_npz(self, fname, npy_name):
+        # figure out offset of .npy in .npz
+        zf = zipfile.ZipFile(fname)
+        info = zf.NameToInfo[npy_name]
+        assert info.compress_type == 0
+        zf.fp.seek(info.header_offset + len(info.FileHeader()) + 20)
+        # read .npy header
+        zf.open(npy_name, "r")
+        version = np.lib.format.read_magic(zf.fp)
+        shape, fortran_order, dtype = np.lib.format._read_array_header(zf.fp, version)
+        assert (
+            dtype == "int32"
+        ), f"sparse multi-hot dtype is {dtype} but should be int32"
+        offset = zf.fp.tell()
+        # create memmap
+        return np.memmap(
+            zf.filename,
+            dtype=dtype,
+            shape=shape,
+            order="F" if fortran_order else "C",
+            mode="r",
+            offset=offset,
+        )
+
+    def _np_arrays_to_batch(
+        self,
+        dense: np.ndarray,
+        sparse: List[np.ndarray],
+        labels: np.ndarray,
+    ) -> Batch:
+        if self.shuffle_batches:
+            # Shuffle all 3 in unison
+            shuffler = np.random.permutation(len(dense))
+            sparse = [multi_hot_ft[shuffler, :] for multi_hot_ft in sparse]
+            dense = dense[shuffler]
+            labels = labels[shuffler]
+
+        batch_size = len(dense)
+        lengths = torch.ones((CAT_FEATURE_COUNT * batch_size), dtype=torch.int32)
+        for k, multi_hot_size in enumerate(self.multi_hot_sizes):
+            lengths[k * batch_size : (k + 1) * batch_size] = multi_hot_size
+        offsets = torch.cumsum(torch.concat((torch.tensor([0]), lengths)), dim=0)
+        length_per_key = [
+            batch_size * multi_hot_size for multi_hot_size in self.multi_hot_sizes
+        ]
+        offset_per_key = torch.cumsum(
+            torch.concat((torch.tensor([0]), torch.tensor(length_per_key))), dim=0
+        )
+        values = torch.concat([torch.from_numpy(feat).flatten() for feat in sparse])
+        return Batch(
+            dense_features=torch.from_numpy(dense.copy()),
+            sparse_features=KeyedJaggedTensor(
+                keys=self.keys,
+                values=values,
+                lengths=lengths,
+                offsets=offsets,
+                stride=batch_size,
+                length_per_key=length_per_key,
+                offset_per_key=offset_per_key.tolist(),
+                index_per_key=self.index_per_key,
+            ),
+            labels=torch.from_numpy(labels.reshape(-1).copy()),
+        )
+
+    def __iter__(self) -> Iterator[Batch]:
+        # Invariant: buffer never contains more than batch_size rows.
+        buffer: Optional[List[np.ndarray]] = None
+
+        def append_to_buffer(
+            dense: np.ndarray,
+            sparse: List[np.ndarray],
+            labels: np.ndarray,
+        ) -> None:
+            nonlocal buffer
+            if buffer is None:
+                buffer = [dense, sparse, labels]
+            else:
+                buffer[0] = np.concatenate((buffer[0], dense))
+                buffer[1] = [np.concatenate((b, s)) for b, s in zip(buffer[1], sparse)]
+                buffer[2] = np.concatenate((buffer[2], labels))
+
+        # Maintain a buffer that can contain up to batch_size rows. Fill buffer as
+        # much as possible on each iteration. Only return a new batch when batch_size
+        # rows are filled.
+        file_idx = 0
+        row_idx = 0
+        batch_idx = 0
+        buffer_row_count = 0
+        cur_batch_size = (
+            self.batch_size if self.num_full_batches > 0 else self.last_batch_sizes[0]
+        )
+        while (
+            batch_idx
+            < self.num_full_batches + (self.last_batch_sizes[0] > 0) * self.world_size
+        ):
+            if buffer_row_count == cur_batch_size or file_idx == len(self.dense_arrs):
+                if batch_idx % self.world_size == self.rank:
+                    yield self._np_arrays_to_batch(*none_throws(buffer))
+                    buffer = None
+                buffer_row_count = 0
+                batch_idx += 1
+                if (
+                    0 <= batch_idx - self.num_full_batches < self.world_size
+                    and (self.last_batch_sizes[0] > 0)
+                ):
+                    cur_batch_size = self.last_batch_sizes[
+                        batch_idx - self.num_full_batches
+                    ]
+            else:
+                rows_to_get = min(
+                    cur_batch_size - buffer_row_count,
+                    self.num_rows_per_file[file_idx] - row_idx,
+                )
+                buffer_row_count += rows_to_get
+                slice_ = slice(row_idx, row_idx + rows_to_get)
+
+                if batch_idx % self.world_size == self.rank:
+                    dense_inputs = self.dense_arrs[file_idx][slice_, :]
+                    sparse_inputs = [
+                        feats[slice_, :] for feats in self.sparse_arrs[file_idx]
+                    ]
+                    target_labels = self.labels_arrs[file_idx][slice_, :]
+
+                    # if self.mmap_mode and self.hashes is not None:
+                    #     sparse_inputs = [
+                    #         feats % hash
+                    #         for (feats, hash) in zip(sparse_inputs, self.hashes)
+                    #     ]
+
+                    append_to_buffer(
+                        dense_inputs,
+                        sparse_inputs,
+                        target_labels,
+                    )
+                row_idx += rows_to_get
+
+                if row_idx >= self.num_rows_per_file[file_idx]:
+                    file_idx += 1
+                    row_idx = 0
+
+    def __len__(self) -> int:
+        return self.num_full_batches // self.world_size + (self.last_batch_sizes[0] > 0)
diff --git a/models_v2/pytorch/torchrec_dlrm/inference/gpu/dist_models.py b/models_v2/pytorch/torchrec_dlrm/inference/gpu/dist_models.py
new file mode 100644
index 000000000..e3a7d391d
--- /dev/null
+++ b/models_v2/pytorch/torchrec_dlrm/inference/gpu/dist_models.py
@@ -0,0 +1,243 @@
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+import argparse
+import torch
+from torchrec import EmbeddingBagCollection
+import extend_distributed as ext_dist
+from torchrec.models.dlrm import DLRM, InteractionDCNArch, OverArch
+from typing import Dict, List, Optional, Tuple
+from torchrec.sparse.jagged_tensor import KeyedJaggedTensor
+from torchrec.modules.crossnet import LowRankCrossNet
+import time
+
+class DIST_DLRM_DCN(DLRM):
+    """
+    Recsys model with DCN modified from the original model from "Deep Learning Recommendation
+    Model for Personalization and Recommendation Systems"
+    (https://arxiv.org/abs/1906.00091). Similar to DLRM module but has
+    DeepCrossNet https://arxiv.org/pdf/2008.13535.pdf as the interaction layer.
+
+    The module assumes all sparse features have the same embedding dimension
+    (i.e. each EmbeddingBagConfig uses the same embedding_dim).
+
+    The following notation is used throughout the documentation for the models:
+
+    * F: number of sparse features
+    * D: embedding_dimension of sparse features
+    * B: batch size
+    * num_features: number of dense features
+
+    Args:
+        embedding_bag_collection (EmbeddingBagCollection): collection of embedding bags
+            used to define `SparseArch`.
+        dense_in_features (int): the dimensionality of the dense input features.
+        dense_arch_layer_sizes (List[int]): the layer sizes for the `DenseArch`.
+        over_arch_layer_sizes (List[int]): the layer sizes for the `OverArch`.
+            The output dimension of the `InteractionArch` should not be manually
+            specified here.
+        dcn_num_layers (int): the number of DCN layers in the interaction.
+        dcn_low_rank_dim (int): the dimensionality of low rank approximation
+            used in the dcn layers.
+        dense_device (Optional[torch.device]): default compute device.
+
+    Example::
+
+        B = 2
+        D = 8
+
+        eb1_config = EmbeddingBagConfig(
+           name="t1", embedding_dim=D, num_embeddings=100, feature_names=["f1", "f3"]
+        )
+        eb2_config = EmbeddingBagConfig(
+           name="t2",
+           embedding_dim=D,
+           num_embeddings=100,
+           feature_names=["f2"],
+        )
+
+        ebc = EmbeddingBagCollection(tables=[eb1_config, eb2_config])
+        model = DLRM_DCN(
+           embedding_bag_collection=ebc,
+           dense_in_features=100,
+           dense_arch_layer_sizes=[20, D],
+           dcn_num_layers=2,
+           dcn_low_rank_dim=8,
+           over_arch_layer_sizes=[5, 1],
+        )
+
+        features = torch.rand((B, 100))
+
+        #     0       1
+        # 0   [1,2] [4,5]
+        # 1   [4,3] [2,9]
+        # ^
+        # feature
+        sparse_features = KeyedJaggedTensor.from_offsets_sync(
+           keys=["f1", "f3"],
+           values=torch.tensor([1, 2, 4, 5, 4, 3, 2, 9]),
+           offsets=torch.tensor([0, 2, 4, 6, 8]),
+        )
+
+        logits = model(
+           dense_features=features,
+           sparse_features=sparse_features,
+        )
+    """
+
+    def __init__(
+        self,
+        embedding_bag_collection: EmbeddingBagCollection,
+        dense_in_features: int,
+        num_total_sparse_features:int,
+        dense_arch_layer_sizes: List[int],
+        over_arch_layer_sizes: List[int],
+        dcn_num_layers: int,
+        dcn_low_rank_dim: int,
+        n_emb : int,
+        shard_matrix,
+        shard_strategy,
+        args: argparse.Namespace,
+        dense_device: Optional[torch.device] = None,
+    ) -> None:
+        # initialize DLRM
+        # sparse arch and dense arch are initialized via DLRM
+        super().__init__(
+            embedding_bag_collection,
+            dense_in_features,
+            dense_arch_layer_sizes,
+            over_arch_layer_sizes,
+            dense_device,
+        )
+        
+        self.args = args
+        if ext_dist.my_size > 1:
+            if n_emb < ext_dist.my_size:
+                sys.exit(
+                    "only (%d) sparse features for (%d) devices, table partitions will fail"
+                    % (n_emb, ext_dist.my_size)
+                )
+            self.n_global_emb = n_emb
+            #self.n_local_emb, self.n_emb_per_rank = ext_dist.get_split_lengths(
+            #    n_emb
+            #)
+            self.shard_matrix = shard_matrix
+            self.shard_strategy = shard_strategy
+            self.n_emb_per_rank = [len(s_m) for s_m in shard_matrix]
+            #self.local_emb_slice = ext_dist.get_my_slice(n_emb)
+            #self.local_emb_indices = list(range(n_emb))[self.local_emb_slice]
+
+
+        self.embedding_dim: int = embedding_bag_collection.embedding_bag_configs()[
+            0
+        ].embedding_dim
+        #num_sparse_features: int = len(self.sparse_arch.sparse_feature_names)
+
+        # Fix interaction and over arch for DLRM_DCN
+
+        crossnet = LowRankCrossNet(
+            in_features=(num_total_sparse_features + 1) * self.embedding_dim,
+            num_layers=dcn_num_layers,
+            low_rank=dcn_low_rank_dim,
+        )
+
+        self.inter_arch = InteractionDCNArch(
+            num_sparse_features=num_total_sparse_features,
+            crossnet=crossnet,
+        )
+        self.inter_arch.to(dense_device)
+
+        over_in_features: int = (num_total_sparse_features + 1) * self.embedding_dim
+
+        self.over_arch = OverArch(
+            in_features=over_in_features,
+            layer_sizes=over_arch_layer_sizes,
+            device=dense_device,
+        )
+
+    def sequential_forward(
+        self,
+        dense_features: torch.Tensor,
+        sparse_features: KeyedJaggedTensor,
+    ) -> torch.Tensor:
+        """
+        Args:
+            dense_features (torch.Tensor): the dense features.
+            sparse_features (KeyedJaggedTensor): the sparse features.
+
+        Returns:
+            torch.Tensor: logits.
+        """
+        embedded_dense = self.dense_arch(dense_features)
+        embedded_sparse = self.sparse_arch(sparse_features)
+        concatenated_dense = self.inter_arch(
+            dense_features=embedded_dense, sparse_features=embedded_sparse
+        )
+        logits = self.over_arch(concatenated_dense)
+        return logits
+
+    def distributed_forward(
+        self,
+        dense_features: torch.Tensor,
+        sparse_features: KeyedJaggedTensor,
+    ) -> torch.Tensor:
+        """
+        Args:
+            dense_features (torch.Tensor): the dense features.
+            sparse_features (KeyedJaggedTensor): the sparse features.
+
+        Returns:
+            torch.Tensor: logits.
+        """
+        embedded_sparse = self.sparse_arch(sparse_features)
+        a2a_req = ext_dist.alltoall(embedded_sparse, self.n_emb_per_rank)
+        embedded_dense = self.dense_arch(dense_features)
+        embedded_sparse = a2a_req.wait()
+        #['0', '8', '16', '24'], ['1', '9', '17', '25'], ['2', '10', '18'], ['3', '11', '19'], ['4', '12', '20'], ['5', '13', '21'], ['6', '14', '22'], ['7', '15', '23']
+        split_embed = {}
+        for index_rank in range(len(self.shard_matrix)):
+            embed_rank = torch.split(embedded_sparse[index_rank], self.embedding_dim, dim=1)
+            for index in range(len(self.shard_matrix[index_rank])):
+                split_embed[self.shard_matrix[index_rank][index]] = embed_rank[index]
+
+        embedded_sparse = []
+        for i in range(len(self.args.num_embeddings_per_feature)):
+            embedded_sparse.append(split_embed[i])
+        embedded_sparse = torch.cat(list(embedded_sparse), dim=1).reshape(embedded_dense.size(0), -1, embedded_dense.size(-1))
+        if self.args.fp16:
+            embedded_sparse = embedded_sparse.half() if embedded_sparse.dtype != torch.float16 else embedded_sparse
+            embedded_dense = embedded_dense.half() if embedded_dense.dtype != torch.float16 else embedded_dense
+        elif self.args.bf16:
+            embedded_sparse = embedded_sparse.bfloat16() if embedded_sparse.dtype != torch.bfloat16 else embedded_sparse
+            embedded_dense = embedded_dense.bfloat16() if embedded_dense.dtype != torch.bfloat16 else embedded_dense
+
+        concatenated_dense = self.inter_arch(
+            dense_features=embedded_dense, sparse_features=embedded_sparse
+        )
+        logits = self.over_arch(concatenated_dense)
+        return logits
+
+    def forward(
+        self,
+        dense_features: torch.Tensor,
+        sparse_features: KeyedJaggedTensor,
+    ) -> torch.Tensor:
+        if ext_dist.my_size > 1:
+            # multi-node multi-device run
+            return self.distributed_forward(dense_features, sparse_features)
+        else:
+            # single device run
+            return self.sequential_forward(dense_features, sparse_features)
+
diff --git a/models_v2/pytorch/torchrec_dlrm/inference/gpu/dlrm_main.py b/models_v2/pytorch/torchrec_dlrm/inference/gpu/dlrm_main.py
new file mode 100644
index 000000000..5ea76bd85
--- /dev/null
+++ b/models_v2/pytorch/torchrec_dlrm/inference/gpu/dlrm_main.py
@@ -0,0 +1,1100 @@
+#!/usr/bin/env python3
+
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+import argparse
+import itertools
+import os
+import sys
+import time
+from dataclasses import dataclass, field
+from enum import Enum
+from typing import List, Optional
+import numpy as np
+
+import torch
+import torchmetrics as metrics
+from torch.autograd.profiler import record_function
+from pyre_extensions import none_throws
+from torch import distributed as dist
+from torch.utils.data import DataLoader
+from torchrec import EmbeddingBagCollection
+from torchrec.datasets.criteo import DEFAULT_CAT_NAMES, DEFAULT_INT_NAMES
+#from torchrec.distributed import TrainPipelineSparseDist
+from torchrec.distributed.comm import get_local_size
+#from torchrec.distributed.model_parallel import (
+#    DistributedModelParallel,
+#    get_default_sharders,
+#)
+#from torchrec.distributed.planner import EmbeddingShardingPlanner, Topology
+#from torchrec.distributed.planner.storage_reservations import (
+#    HeuristicalStorageReservation,
+#)
+from dist_models import DIST_DLRM_DCN
+from torchrec.models.dlrm import DLRM, DLRM_DCN, DLRM_Projection, DLRMTrain
+from torchrec.modules.embedding_configs import EmbeddingBagConfig
+from torchrec.optim.apply_optimizer_in_backward import apply_optimizer_in_backward
+from torchrec.optim.keyed import CombinedOptimizer, KeyedOptimizerWrapper
+from torchrec.optim.optimizers import in_backward_optimizer_filter
+from tqdm import tqdm
+import extend_distributed as ext_dist
+from extend_distributed import dist, env2int
+from jit_trace_able_utils import unpack, SparseArchTraceAbleWrapper
+
+import logging
+import sharding
+
+logger: logging.Logger = logging.getLogger(__name__)
+logger.setLevel(1)
+
+# OSS import
+try:
+    # pyre-ignore[21]
+    # @manual=//ai_codesign/benchmarks/dlrm/torchrec_dlrm/data:dlrm_dataloader
+    from data.dlrm_dataloader import get_dataloader
+
+    # pyre-ignore[21]
+    # @manual=//ai_codesign/benchmarks/dlrm/torchrec_dlrm:lr_scheduler
+    from lr_scheduler import LRPolicyScheduler
+
+    # pyre-ignore[21]
+    # @manual=//ai_codesign/benchmarks/dlrm/torchrec_dlrm:multi_hot
+    from multi_hot import Multihot, RestartableMap
+except ImportError:
+    pass
+
+# internal import
+try:
+    from .data.dlrm_dataloader import get_dataloader  # noqa F811
+    from .lr_scheduler import LRPolicyScheduler  # noqa F811
+    from .multi_hot import Multihot, RestartableMap  # noqa F811
+except ImportError:
+    pass
+
+TRAIN_PIPELINE_STAGES = 3  # Number of stages in TrainPipelineSparseDist.
+
+n_emb = None
+#local_emb_slice = None
+local_emb_indices = None
+default_embedding_names = None
+idx_name = None
+
+
+def load_checkoutpoint(args, device):
+    checkpoint = None
+    if args.checkpoint_floder is not None:
+        if (not os.path.exists(args.checkpoint_floder)) or (not os.path.isdir(args.checkpoint_floder)):
+            logger.error("The checkpoint dir is not exist.")
+            exit()
+        if  not os.path.exists(args.checkpoint_floder + '/dlrm_training_' + str(dist.get_rank()) + '_save.pt'):
+            logger.warning("The checkpoint " + args.checkpoint_floder + '/dlrm_training_' + str(dist.get_rank()) + '_save.pt' +  " doesn't exist. If converge isn't zero step, your behavior will cause exceptions.")
+        else:
+            print('load...', args.checkpoint_floder + '/dlrm_training_' + str(dist.get_rank()) + '_save.pt', ' to ',
+                  device)
+            checkpoint = torch.load(args.checkpoint_floder + '/dlrm_training_' + str(dist.get_rank()) + '_save.pt', map_location=torch.device('cpu'))
+    return checkpoint
+
+
+
+class InteractionType(Enum):
+    ORIGINAL = "original"
+    DCN = "dcn"
+    PROJECTION = "projection"
+
+    def __str__(self):
+        return self.value
+
+
+def load_snapshot(model, snapshot_dir):
+    from torchsnapshot import Snapshot
+    snapshot = Snapshot(path=snapshot_dir)
+    snapshot.restore(app_state={"model": model})
+
+
+def parse_args(argv: List[str]) -> argparse.Namespace:
+    parser = argparse.ArgumentParser(description="torchrec dlrm example trainer")
+    parser.add_argument(
+        "--epochs",
+        type=int,
+        default=1,
+        help="number of epochs to train",
+    )
+    parser.add_argument(
+        "--batch_size",
+        type=int,
+        default=32,
+        help="batch size to use for training",
+    )
+    parser.add_argument(
+        "--drop_last_training_batch",
+        dest="drop_last_training_batch",
+        action="store_true",
+        help="Drop the last non-full training batch",
+    )
+    parser.add_argument(
+        "--test_batch_size",
+        type=int,
+        default=None,
+        help="batch size to use for validation and testing",
+    )
+    parser.add_argument(
+        "--limit_train_batches",
+        type=int,
+        default=None,
+        help="number of train batches",
+    )
+    parser.add_argument(
+        "--limit_val_batches",
+        type=int,
+        default=None,
+        help="number of validation batches",
+    )
+    parser.add_argument(
+        "--limit_test_batches",
+        type=int,
+        default=None,
+        help="number of test batches",
+    )
+    parser.add_argument(
+        "--dataset_name",
+        type=str,
+        choices=["criteo_1t", "criteo_kaggle"],
+        default="criteo_1t",
+        help="dataset for experiment, current support criteo_1tb, criteo_kaggle",
+    )
+    parser.add_argument(
+        "--num_embeddings",
+        type=int,
+        default=100_000,
+        help="max_ind_size. The number of embeddings in each embedding table. Defaults"
+        " to 100_000 if num_embeddings_per_feature is not supplied.",
+    )
+    parser.add_argument(
+        "--num_embeddings_per_feature",
+        type=str,
+        default=None,
+        help="Comma separated max_ind_size per sparse feature. The number of embeddings"
+        " in each embedding table. 26 values are expected for the Criteo dataset.",
+    )
+    parser.add_argument(
+        "--dense_arch_layer_sizes",
+        type=str,
+        default="512,256,64",
+        help="Comma separated layer sizes for dense arch.",
+    )
+    parser.add_argument(
+        "--over_arch_layer_sizes",
+        type=str,
+        default="512,512,256,1",
+        help="Comma separated layer sizes for over arch.",
+    )
+    parser.add_argument(
+        "--embedding_dim",
+        type=int,
+        default=64,
+        help="Size of each embedding.",
+    )
+    parser.add_argument(
+        "--interaction_branch1_layer_sizes",
+        type=str,
+        default="2048,2048",
+        help="Comma separated layer sizes for interaction branch1 (only on dlrm with projection).",
+    )
+    parser.add_argument(
+        "--interaction_branch2_layer_sizes",
+        type=str,
+        default="2048,2048",
+        help="Comma separated layer sizes for interaction branch2 (only on dlrm with projection).",
+    )
+    parser.add_argument(
+        "--dcn_num_layers",
+        type=int,
+        default=3,
+        help="Number of DCN layers in interaction layer (only on dlrm with DCN).",
+    )
+    parser.add_argument(
+        "--dcn_low_rank_dim",
+        type=int,
+        default=512,
+        help="Low rank dimension for DCN in interaction layer (only on dlrm with DCN).",
+    )
+    parser.add_argument(
+        "--undersampling_rate",
+        type=float,
+        help="Desired proportion of zero-labeled samples to retain (i.e. undersampling zero-labeled rows)."
+        " Ex. 0.3 indicates only 30pct of the rows with label 0 will be kept."
+        " All rows with label 1 will be kept. Value should be between 0 and 1."
+        " When not supplied, no undersampling occurs.",
+    )
+    parser.add_argument(
+        "--seed",
+        type=int,
+        help="Random seed for reproducibility.",
+    )
+    parser.add_argument(
+        "--pin_memory",
+        dest="pin_memory",
+        action="store_true",
+        help="Use pinned memory when loading data.",
+    )
+    parser.add_argument(
+        "--mmap_mode",
+        dest="mmap_mode",
+        action="store_true",
+        help="--mmap_mode mmaps the dataset."
+        " That is, the dataset is kept on disk but is accessed as if it were in memory."
+        " --mmap_mode is intended mostly for faster debugging. Use --mmap_mode to bypass"
+        " preloading the dataset when preloading takes too long or when there is "
+        " insufficient memory available to load the full dataset.",
+    )
+    parser.add_argument(
+        "--in_memory_binary_criteo_path",
+        type=str,
+        default=None,
+        help="Directory path containing the Criteo dataset npy files.",
+    )
+    parser.add_argument(
+        "--synthetic_multi_hot_criteo_path",
+        type=str,
+        default=None,
+        help="Directory path containing the MLPerf v2 synthetic multi-hot dataset npz files.",
+    )
+    parser.add_argument(
+        "--learning_rate",
+        type=float,
+        default=15.0,
+        help="Learning rate.",
+    )
+    parser.add_argument(
+        "--eps",
+        type=float,
+        default=1e-8,
+        help="Epsilon for Adagrad optimizer.",
+    )
+    parser.add_argument(
+        "--shuffle_batches",
+        dest="shuffle_batches",
+        action="store_true",
+        help="Shuffle each batch during training.",
+    )
+    parser.add_argument(
+        "--shuffle_training_set",
+        dest="shuffle_training_set",
+        action="store_true",
+        help="Shuffle the training set in memory. This will override mmap_mode",
+    )
+    parser.add_argument(
+        "--validation_freq_within_epoch",
+        type=int,
+        default=None,
+        help="Frequency at which validation will be run within an epoch.",
+    )
+    parser.set_defaults(
+        pin_memory=None,
+        mmap_mode=None,
+        drop_last=None,
+        shuffle_batches=None,
+        shuffle_training_set=None,
+    )
+    parser.add_argument(
+        "--adagrad",
+        dest="adagrad",
+        action="store_true",
+        help="Flag to determine if adagrad optimizer should be used.",
+    )
+    parser.add_argument(
+        "--interaction_type",
+        type=InteractionType,
+        choices=list(InteractionType),
+        default=InteractionType.ORIGINAL,
+        help="Determine the interaction type to be used (original, dcn, or projection)"
+        " default is original DLRM with pairwise dot product",
+    )
+    parser.add_argument(
+        "--collect_multi_hot_freqs_stats",
+        dest="collect_multi_hot_freqs_stats",
+        action="store_true",
+        help="Flag to determine whether to collect stats on freq of embedding access.",
+    )
+    parser.add_argument(
+        "--multi_hot_sizes",
+        type=str,
+        default=None,
+        help="Comma separated multihot size per sparse feature. 26 values are expected for the Criteo dataset.",
+    )
+    parser.add_argument(
+        "--multi_hot_distribution_type",
+        type=str,
+        choices=["uniform", "pareto"],
+        default=None,
+        help="Multi-hot distribution options.",
+    )
+    parser.add_argument("--lr_warmup_steps", type=int, default=0)
+    parser.add_argument("--lr_decay_start", type=int, default=0)
+    parser.add_argument("--lr_decay_steps", type=int, default=0)
+    parser.add_argument(
+        "--print_lr",
+        action="store_true",
+        help="Print learning rate every iteration.",
+    )
+    parser.add_argument(
+        "--allow_tf32",
+        action="store_true",
+        help="Enable TensorFloat-32 mode for matrix multiplications on A100 (or newer) GPUs.",
+    )
+    parser.add_argument(
+        "--print_sharding_plan",
+        action="store_true",
+        help="Print the sharding plan used for each embedding table.",
+    )
+    parser.add_argument("--use_xpu", action="store_true", default=False)
+    parser.add_argument(
+        "--log_freq",
+        type=int,
+        default=0,
+        help="log_freq to print performance statistics.",
+    )
+    parser.add_argument(
+        "--converge_freq",
+        type=int,
+        default=0,
+        help="converge_freq to save model.",
+    )
+    parser.add_argument("--numpy_rand_seed", type=int, default=123)
+    parser.add_argument("--bf16", action="store_true", default=False)
+    parser.add_argument("--fp16", action="store_true", default=False)
+    parser.add_argument("--amp", action="store_true", default=False)
+    parser.add_argument("--dense", action="store_true", default=False)
+    parser.add_argument("--inference_only", action="store_true", default=False)
+    parser.add_argument("--converge", action="store_true", default=False)
+    parser.add_argument("--train_with_val", action="store_true", default=False)
+    parser.add_argument("--num_nodes", type=int, default=1)
+    parser.add_argument("--trans_data", action="store_true", default=False)
+    parser.add_argument(
+    "--sharding_plan",
+    help="Sharding plan to use",
+    type=str,
+    choices=["round_robin", "uniform", "auto", "hier_auto", "custom", "max_min"],
+    default="round_robin",)
+    parser.add_argument(
+        "--snapshot_dir",
+        help="Weight path for dlrm-v2",
+        type=str,
+    )
+
+    parser.add_argument(
+        "--checkpoint_floder",
+        help="checkpoint path for dlrm-v2",
+        type=str,
+    )
+    parser.add_argument(
+        "--dynamo",
+        action="store_true",
+        help="Use torch compile for model test",
+        default=False
+    )
+
+
+
+    return parser.parse_args(argv)
+
+
+def _evaluate(
+    limit_batches: Optional[int],
+    #eval_pipeline: TrainPipelineSparseDist,
+    eval_model,
+    eval_dataloader: DataLoader,
+    stage: str,
+    log_freq: int,
+    device: torch.device,
+    args: argparse.Namespace,
+) -> float:
+    """
+    Evaluates model. Computes and prints AUROC. Helper function for train_val_test.
+
+    Args:
+        limit_batches (Optional[int]): Limits the dataloader to the first `limit_batches` batches.
+        eval_pipeline (TrainPipelineSparseDist): pipelined model.
+        eval_dataloader (DataLoader): Dataloader for either the validation set or test set.
+        stage (str): "val" or "test".
+
+    Returns:
+        float: auroc result
+    """
+    eval_model.eval()
+    global n_emb, local_emb_indices, default_embedding_names, idx_name
+
+    if args.dynamo:
+        if args.amp:
+            if args.use_device == "xpu":
+                with torch.no_grad():
+                    with torch.xpu.amp.autocast(enabled=args.amp, dtype=args.amp_dtype):
+                        print(" Use dynamo for compiling")
+                        eval_model = torch.compile(eval_model, backend="inductor", options={"freezing": True} )
+
+    def eval_step(model, batch):
+        batch_size = batch.dense_features.size(0)
+        if args.amp:
+            batch.dense_features = batch.dense_features[ext_dist.get_my_slice(batch_size)].to(device)
+        else:
+            batch.dense_features = batch.dense_features[ext_dist.get_my_slice(batch_size)].to(device).to(args.amp_dtype)
+        batch.labels = batch.labels[ext_dist.get_my_slice(batch_size)].to(device)
+        for k, v in batch.sparse_features.items():
+            batch.sparse_features[k]['values'] = batch.sparse_features[k]['values'].to(device)
+            batch.sparse_features[k]['offsets'] = batch.sparse_features[k]['offsets'].to(device)
+
+        #if args.converge and args.dense and (args.use_device == "xpu"):
+        #    torch.xpu.empty_cache()
+        with torch.no_grad():
+            if args.amp:
+                if args.use_device == "xpu":
+                    with torch.xpu.amp.autocast(enabled=args.amp, dtype=args.amp_dtype):
+                        logits = model(batch.dense_features, batch.sparse_features)
+                elif args.use_device == "cuda":
+                    with torch.cuda.amp.autocast(enabled=args.amp, dtype=args.amp_dtype):
+                        logits = model(batch.dense_features, batch.sparse_features)
+                else:
+                    with torch.cpu.amp.autocast(enabled=args.amp, dtype=args.amp_dtype):
+                        logits = model(batch.dense_features, batch.sparse_features)
+            else:
+                logits = model(batch.dense_features, batch.sparse_features)
+        #if args.converge and args.dense and (args.use_device == "xpu"):
+        #    torch.xpu.empty_cache()
+        return logits, batch.labels
+    # Set eval_pipeline._connected to False to cause the pipeline to refill with new batches as if it were newly created and empty.
+    #eval_pipeline._connected = False
+
+    iterator = itertools.islice(iter(eval_dataloader), limit_batches)
+
+    auroc = metrics.AUROC(compute_on_step=False, task='binary').to(device)
+
+    is_rank_zero = dist.get_rank() == 0
+    total_t = 0
+    it = 1
+    with torch.no_grad():
+        while True:
+            try:
+                batch = next(iterator)
+                batch.sparse_features = unpack(batch.sparse_features, default_embedding_names)
+                #ext_dist.barrier()
+                if args.use_device == "xpu":
+                    torch.xpu.synchronize()
+                elif args.use_device == "cuda":
+                    torch.cuda.synchronize()
+                else:
+                    pass
+                t1 = time.time()
+                with (torch.autograd.profiler_legacy.profile(args.enable_profiling, use_xpu=True) if args.use_xpu else torch.profiler.profile(activities=[torch.profiler.ProfilerActivity.CUDA])) as prof:
+                    logits, labels = eval_step(eval_model, batch)
+                    preds = torch.sigmoid(logits)
+                    auroc(preds, labels)
+                    if args.use_device == "xpu":
+                        torch.xpu.synchronize()
+                    elif args.use_device == "cuda":
+                        torch.cuda.synchronize()
+                    else:
+                        pass
+                    if it > 5 and args.inference_only:
+                        total_t += time.time() - t1
+                        logger.info(f"avg eval time per iter at ITER: {it - 5}, {total_t/(it-5)} s")
+            except StopIteration:
+                break
+            if args.enable_profiling and it == 35 and args.inference_only:
+                rank_index = ""
+                if ext_dist.my_size > 1:
+                    rank_index = "rank_" + str(dist.get_rank()) + "_"
+                path ='./report/dlrm_inference_' + rank_index + 'profiling'
+                if args.use_xpu:
+                    torch.save(prof.key_averages().table(sort_by="self_xpu_time_total", row_limit=100000), \
+                        path + '.pt')
+                    prof.export_chrome_trace(path + '.json')
+                    torch.save(prof.table(sort_by="id", row_limit=100000), path + '_detail.pt')
+                elif args.use_device == "cuda":
+                    torch.save(prof.key_averages().table(sort_by="self_cuda_time_total", row_limit=100000), \
+                        path + '.pt')
+                    prof.export_chrome_trace(path + '.json')
+                else:
+                    pass
+            it += 1
+
+    auroc_result = auroc.compute().item()
+    num_samples = torch.tensor(sum(map(len, auroc.target)), device=device)
+    dist.reduce(num_samples, 0, op=dist.ReduceOp.SUM)
+
+    if is_rank_zero:
+        print(f"AUROC over {stage} set: {auroc_result}.")
+        print(f"Number of {stage} samples: {num_samples}")
+    return auroc_result
+
+def _train(
+    train_model,
+    train_optimizer,
+    train_dataloader: DataLoader,
+    val_dataloader: DataLoader,
+    epoch: int,
+    lr_scheduler,
+    print_lr: bool,
+    validation_freq: Optional[int],
+    limit_train_batches: Optional[int],
+    limit_val_batches: Optional[int],
+    log_freq: int,
+    device: torch.device,
+    args: argparse.Namespace,
+) -> bool:
+    """
+    Trains model for 1 epoch. Helper function for train_val_test.
+    Args:
+        train_pipeline (TrainPipelineSparseDist): pipelined model used for training.
+        val_pipeline (TrainPipelineSparseDist): pipelined model used for validation.
+        train_dataloader (DataLoader): Training set's dataloader.
+        val_dataloader (DataLoader): Validation set's dataloader.
+        epoch (int): The number of complete passes through the training set so far.
+        lr_scheduler (LRPolicyScheduler): Learning rate scheduler.
+        print_lr (bool): Whether to print the learning rate every training step.
+        validation_freq (Optional[int]): The number of training steps between validation runs within an epoch.
+        limit_train_batches (Optional[int]): Limits the training set to the first `limit_train_batches` batches.
+        limit_val_batches (Optional[int]): Limits the validation set to the first `limit_val_batches` batches.
+        print_progress (bool): Whether to print tqdm progress bar.
+    Returns:
+        bool: Whether the validation_auroc threshold is reached.
+    """
+    train_model.train()
+    global n_emb, local_emb_indices, default_embedding_names, idx_name
+    def train_step(model, opt, next_batch):
+        batch_size = next_batch.dense_features.size(0)
+        opt.zero_grad(set_to_none=True)
+        if args.amp:
+            next_batch.dense_features = next_batch.dense_features[ext_dist.get_my_slice(batch_size)].to(device)
+        else:
+            next_batch.dense_features = next_batch.dense_features[ext_dist.get_my_slice(batch_size)].to(device).to(args.amp_dtype)
+        next_batch.labels = next_batch.labels[ext_dist.get_my_slice(batch_size)].to(device)
+
+        for k, v in next_batch.sparse_features.items():
+            next_batch.sparse_features[k]['values'] = next_batch.sparse_features[k]['values'].to(device)
+            next_batch.sparse_features[k]['offsets'] = next_batch.sparse_features[k]['offsets'].to(device)
+        #if args.converge and args.dense and (args.use_device == "xpu"):
+        #    torch.xpu.empty_cache()
+        if args.amp:
+            if args.use_device == "xpu":
+                with torch.xpu.amp.autocast(enabled=args.amp, dtype=args.amp_dtype):
+                    losses, _ = model(next_batch)
+                    loss = torch.sum(losses, dim=0)
+            elif args.use_device == "cuda":
+                with torch.cuda.amp.autocast(enabled=args.amp, dtype=args.amp_dtype):
+                    losses, _ = model(next_batch)
+                    loss = torch.sum(losses, dim=0)
+            else:
+                with torch.cpu.amp.autocast(enabled=args.amp, dtype=args.amp_dtype):
+                    losses, _ = model(next_batch)
+                    loss = torch.sum(losses, dim=0)
+        else:
+            losses, _ = model(next_batch)
+            loss = torch.sum(losses, dim=0)
+
+        #if args.converge and args.dense and (args.use_device == "xpu"):
+        #    torch.xpu.empty_cache()
+        loss.backward()
+        opt.step()
+    start_item = 0
+    if args.converge:
+        start_item = args.step
+    iterator = itertools.islice(iter(train_dataloader), start_item, limit_train_batches)
+
+    is_rank_zero = dist.get_rank() == 0
+
+    total_t = 0
+    for it in itertools.count(start_item + 1, 1):
+        try:
+            if is_rank_zero and print_lr:
+                for i, g in enumerate(train_pipeline._optimizer.param_groups):
+                    print(f"lr: {it} {i} {g['lr']:.6f}")
+            next_batch = next(iterator)
+            next_batch.sparse_features = unpack(next_batch.sparse_features, default_embedding_names)
+            #ext_dist.barrier()
+            if args.use_device == "xpu":
+                torch.xpu.synchronize()
+            elif args.use_device == "cuda":
+                torch.cuda.synchronize()
+            else:
+                pass
+            t1 = time.time()
+            with (torch.autograd.profiler_legacy.profile(args.enable_profiling, use_xpu=True, record_shapes=False) if args.use_xpu else torch.profiler.profile(activities=[torch.profiler.ProfilerActivity.CUDA])) as prof:
+                train_step(train_model, train_optimizer, next_batch)
+                if args.use_device == "xpu":
+                    torch.xpu.synchronize()
+                elif args.use_device == "cuda":
+                    torch.cuda.synchronize()
+                else:
+                    pass
+
+                if it > 5:
+                    total_t += time.time() - t1
+                    #if log_freq != 0 and it % log_freq == 0:
+                    logger.info(f"avg training time per iter at ITER: {it - 5}, {total_t/(it - 5)} s")
+            if args.enable_profiling and it == 35:
+                rank_index = ""
+                if ext_dist.my_size > 1:
+                    rank_index = "rank_" + str(dist.get_rank()) + "_"
+                path ='./report/dlrm_training_' + rank_index + 'profiling'
+                if args.use_device == "xpu":
+                    torch.save(prof.key_averages().table(sort_by="self_xpu_time_total", row_limit=100000), \
+                        path + '.pt')
+                    prof.export_chrome_trace(path + '.json')
+                    torch.save(prof.table(sort_by="id", row_limit=100000), path + '_detail.pt')
+                elif args.use_device == "cuda":
+                    torch.save(prof.key_averages().table(sort_by="self_cuda_time_total", row_limit=100000), \
+                        path + '.pt')
+                    prof.export_chrome_trace(path + '.json')
+                else:
+                    torch.save(prof.key_averages().table(sort_by="self_cpu_time_total", row_limit=100000), \
+                        path + '.pt')
+                    prof.export_chrome_trace(path + '.json')
+                    pass
+            lr_scheduler.step()
+
+            if validation_freq and (it % validation_freq == 0) and args.train_with_val:
+                auroc_result = _evaluate(
+                    limit_val_batches,
+                    train_model.model,
+                    val_dataloader,
+                    "val",
+                    log_freq, 
+                    device, 
+                    args
+                )
+                if is_rank_zero:
+                    logger.info(f"The val auc = {auroc_result}")
+                train_model.train()
+
+            if args.converge and it  % args.converge_freq == 0:
+                model_state_dict = {}
+                for k, v in train_model.state_dict().items():
+                    k_split = k.split("module")
+                    k_str = k_split[0][:-1] + k_split[1] if len(k_split) == 2 else k
+                    model_state_dict[k_str] = v
+                torch.save({
+                    "model": model_state_dict, 
+                    "optimizer": train_optimizer.state_dict(),
+                    "lr_scheduler": lr_scheduler.state_dict(),
+                    "epoch": epoch,
+                    "step": it}, 
+                    args.checkpoint_floder + '/dlrm_training_' + str(dist.get_rank()) + '_save.pt')
+        except StopIteration:
+            # Dataset traversal complete
+            break
+
+    logger.info(f"Total number of iterations: {it - 1}")
+
+
+@dataclass
+class TrainValTestResults:
+    val_aurocs: List[float] = field(default_factory=list)
+    test_auroc: Optional[float] = None
+
+
+def train_val_test(
+    args: argparse.Namespace,
+    model: torch.nn.Module,
+    optimizer: torch.optim.Optimizer,
+    device: torch.device,
+    train_dataloader: DataLoader,
+    val_dataloader: DataLoader,
+    test_dataloader: DataLoader,
+    lr_scheduler: LRPolicyScheduler,
+) -> TrainValTestResults:
+    """
+    Train/validation/test loop.
+
+    Args:
+        args (argparse.Namespace): parsed command line args.
+        model (torch.nn.Module): model to train.
+        optimizer (torch.optim.Optimizer): optimizer to use.
+        device (torch.device): device to use.
+        train_dataloader (DataLoader): Training set's dataloader.
+        val_dataloader (DataLoader): Validation set's dataloader.
+        test_dataloader (DataLoader): Test set's dataloader.
+        lr_scheduler (LRPolicyScheduler): Learning rate scheduler.
+
+    Returns:
+        TrainValTestResults.
+    """
+    results = TrainValTestResults()
+    if args.fp16 and (not args.amp):
+        model.to(torch.float16)
+    if args.bf16 and (not args.amp):
+        model.to(torch.bfloat16)
+
+    start_epoch = 0
+    if args.converge:
+        start_epoch = args.epoch
+    if not args.inference_only:
+        for epoch in range(start_epoch, args.epochs):
+            _train(
+                model,
+                optimizer,
+                train_dataloader,
+                val_dataloader,
+                epoch,
+                lr_scheduler,
+                args.print_lr,
+                args.validation_freq_within_epoch,
+                args.limit_train_batches,
+                args.limit_val_batches,
+                args.log_freq,
+                device,
+                args
+            )
+            if args.train_with_val:
+                val_auroc = _evaluate(
+                    args.limit_val_batches, model.model, val_dataloader, "val", args.log_freq, device, args
+                )
+                results.val_aurocs.append(val_auroc)
+        if args.train_with_val:
+            test_auroc = _evaluate(
+                args.limit_test_batches, model.model, test_dataloader, "test", args.log_freq, device, args
+            )
+
+    else:
+        test_auroc = _evaluate(
+            args.limit_test_batches, model.model, test_dataloader, "test", args.log_freq, device, args
+        )
+        results.test_auroc = test_auroc
+
+    return results
+
+
+def main(argv: List[str]) -> None:
+    """
+    Trains, validates, and tests a Deep Learning Recommendation Model (DLRM)
+    (https://arxiv.org/abs/1906.00091). The DLRM model contains both data parallel
+    components (e.g. multi-layer perceptrons & interaction arch) and model parallel
+    components (e.g. embedding tables). The DLRM model is pipelined so that dataloading,
+    data-parallel to model-parallel comms, and forward/backward are overlapped. Can be
+    run with either a random dataloader or an in-memory Criteo 1 TB click logs dataset
+    (https://ailab.criteo.com/download-criteo-1tb-click-logs-dataset/).
+
+    Args:
+        argv (List[str]): command line args.
+
+    Returns:
+        None.
+    """
+    args = parse_args(argv)
+    args.amp_dtype = torch.float32
+    if args.bf16:
+        args.amp_dtype = torch.bfloat16
+    if args.fp16:
+        args.amp_dtype = torch.float16
+    for name, val in vars(args).items():
+        try:
+            vars(args)[name] = list(map(int, val.split(",")))
+        except (ValueError, AttributeError):
+            pass
+    args.enable_profiling = False
+    if os.getenv('PROFILE') == '1': args.enable_profiling=True
+
+    if args.use_xpu:
+        import intel_extension_for_pytorch
+        try:
+            import oneccl_bindings_for_pytorch
+        except ImportError as e:
+            pass
+
+    if not args.use_xpu:
+        torch.backends.cuda.matmul.allow_tf32 = args.allow_tf32
+
+    if args.multi_hot_sizes is not None and args.trans_data:
+        assert (
+            args.num_embeddings_per_feature is not None
+            and len(args.multi_hot_sizes) == len(args.num_embeddings_per_feature)
+            or args.num_embeddings_per_feature is None
+            and len(args.multi_hot_sizes) == len(DEFAULT_CAT_NAMES)
+        ), "--multi_hot_sizes must be a comma delimited list the same size as the number of embedding tables."
+    assert (
+        args.in_memory_binary_criteo_path is None
+        or args.synthetic_multi_hot_criteo_path is None
+    ), "--in_memory_binary_criteo_path and --synthetic_multi_hot_criteo_path are mutually exclusive CLI arguments."
+    assert (
+        args.multi_hot_sizes is None or (not args.trans_data) or args.synthetic_multi_hot_criteo_path is None
+    ), "--multi_hot_sizes is used to convert 1-hot to multi-hot. It's inapplicable with --synthetic_multi_hot_criteo_path."
+    assert (
+        args.multi_hot_distribution_type is None
+        or args.synthetic_multi_hot_criteo_path is None
+    ), "--multi_hot_distribution_type is used to convert 1-hot to multi-hot. It's inapplicable with --synthetic_multi_hot_criteo_path."
+
+    np.random.seed(args.numpy_rand_seed)
+    torch.manual_seed(args.numpy_rand_seed)
+    #rank = int(os.environ["LOCAL_RANK"])
+    rank = env2int(
+        [
+            "MPI_LOCALRANKID",
+            "OMPI_COMM_WORLD_LOCAL_RANK",
+            "MV2_COMM_WORLD_LOCAL_RANK",
+            "LOCAL_RANK",
+        ],
+        0,
+    )
+    args.use_device = "cpu"
+    if args.use_xpu and torch.xpu.is_available():
+        torch.xpu.manual_seed_all(args.numpy_rand_seed)
+        device: torch.device = torch.device("xpu", rank)
+        backend = "ccl"
+        torch.xpu.set_device(device)
+        args.use_device = "xpu"
+    elif torch.cuda.is_available():
+        torch.cuda.manual_seed_all(args.numpy_rand_seed)
+        device: torch.device = torch.device(f"cuda:{rank}")
+        backend = "nccl"
+        torch.cuda.set_device(device)
+        args.use_device = "cuda"
+    else:
+        device: torch.device = torch.device("cpu")
+        backend = "gloo"
+
+    if rank == 0:
+        print(
+            "PARAMS: (lr, batch_size, warmup_steps, decay_start, decay_steps): "
+            f"{(args.learning_rate, args.batch_size, args.lr_warmup_steps, args.lr_decay_start, args.lr_decay_steps)}"
+        )
+    ext_dist.init_distributed(
+        local_rank=-1, use_device=args.use_device, backend=backend
+    )
+
+    if args.num_embeddings_per_feature is not None:
+        args.num_embeddings = None
+
+    # Sets default limits for random dataloader iterations when left unspecified.
+    if (
+        args.in_memory_binary_criteo_path
+        is args.synthetic_multi_hot_criteo_path
+        is None
+    ):
+        for split in ["train", "val", "test"]:
+            attr = f"limit_{split}_batches"
+            if getattr(args, attr) is None:
+                setattr(args, attr, 10)
+
+    train_dataloader = get_dataloader(args, backend, "train")
+    val_dataloader = get_dataloader(args, backend, "val")
+    test_dataloader = get_dataloader(args, backend, "test")
+    # use dist_models, if not you can comment out
+    global n_emb, local_emb_indices, default_embedding_names, idx_name
+    n_emb = len(DEFAULT_CAT_NAMES)
+    #if dist.get_world_size() == 8 and dist.get_rank() == 6:
+    #    local_emb_indices[1] = 25
+    #if dist.get_world_size() == 8 and dist.get_rank() == 7:
+    #    local_emb_indices[2] = 21
+
+    shard_matrix_str, shard_strategy_str, shard_matrix, shard_strategy = sharding.generate_plan(
+    args.num_embeddings_per_feature, args.multi_hot_sizes, args.num_nodes, dist.get_world_size(), args, dist.get_rank() == 0)
+    #local_emb_slice = ext_dist.get_my_slice(n_emb)
+    local_emb_indices = shard_matrix[rank]
+    default_embedding_names = [DEFAULT_CAT_NAMES[i] for i in local_emb_indices]
+    idx_name = zip(local_emb_indices, default_embedding_names)
+
+    eb_configs = [
+        EmbeddingBagConfig(
+            name=f"t_{feature_name}",
+            embedding_dim=args.embedding_dim,
+            num_embeddings=none_throws(args.num_embeddings_per_feature)[feature_idx]
+            if args.num_embeddings is None
+            else args.num_embeddings,
+            feature_names=[feature_name],
+        )
+        #for feature_idx, feature_name in enumerate(DEFAULT_CAT_NAMES)
+        for feature_idx, feature_name in idx_name
+    ]
+    sharded_module_kwargs = {}
+    if args.over_arch_layer_sizes is not None:
+        sharded_module_kwargs["over_arch_layer_sizes"] = args.over_arch_layer_sizes
+
+    if args.interaction_type == InteractionType.ORIGINAL:
+        dlrm_model = DLRM(
+            embedding_bag_collection=EmbeddingBagCollection(
+                tables=eb_configs, device=torch.device("meta")
+            ),
+            dense_in_features=len(DEFAULT_INT_NAMES),
+            dense_arch_layer_sizes=args.dense_arch_layer_sizes,
+            over_arch_layer_sizes=args.over_arch_layer_sizes,
+            dense_device=device,
+        )
+    elif args.interaction_type == InteractionType.DCN:
+        #dlrm_model = DLRM_DCN(
+        dlrm_model = DIST_DLRM_DCN(
+            embedding_bag_collection=EmbeddingBagCollection(
+                #tables=eb_configs, device=torch.device("meta")
+                tables=eb_configs, device="cpu"
+            ),
+            dense_in_features=len(DEFAULT_INT_NAMES),
+            num_total_sparse_features=n_emb,
+            dense_arch_layer_sizes=args.dense_arch_layer_sizes,
+            over_arch_layer_sizes=args.over_arch_layer_sizes,
+            dcn_num_layers=args.dcn_num_layers,
+            dcn_low_rank_dim=args.dcn_low_rank_dim,
+            n_emb=n_emb,
+            shard_matrix=shard_matrix, 
+            shard_strategy = shard_strategy,
+            args=args,
+            dense_device="cpu",
+        )
+    elif args.interaction_type == InteractionType.PROJECTION:
+        dlrm_model = DLRM_Projection(
+            embedding_bag_collection=EmbeddingBagCollection(
+                tables=eb_configs, device=torch.device("meta")
+            ),
+            dense_in_features=len(DEFAULT_INT_NAMES),
+            dense_arch_layer_sizes=args.dense_arch_layer_sizes,
+            over_arch_layer_sizes=args.over_arch_layer_sizes,
+            interaction_branch1_layer_sizes=args.interaction_branch1_layer_sizes,
+            interaction_branch2_layer_sizes=args.interaction_branch2_layer_sizes,
+            dense_device=device,
+        )
+    else:
+        raise ValueError(
+            "Unknown interaction option set. Should be original, dcn, or projection."
+        )
+
+    for embedding_bag in dlrm_model.sparse_arch.embedding_bag_collection.embedding_bags.values():
+        _W = np.random.uniform(
+            low=-np.sqrt(1 / embedding_bag.num_embeddings),
+            high=np.sqrt(1 / embedding_bag.num_embeddings),
+            size=(embedding_bag.num_embeddings, embedding_bag.embedding_dim)).astype(np.float32)
+        embedding_bag.weight.data = torch.tensor(_W, requires_grad=True)
+
+    if not args.amp:
+        dlrm_model.to(args.amp_dtype)
+    train_model = DLRMTrain(dlrm_model)
+    if args.inference_only:
+        assert args.snapshot_dir
+        load_snapshot(train_model, args.snapshot_dir)
+
+
+    # This will apply the Adagrad optimizer in the backward pass for the embeddings (sparse_arch). This means that
+    # the optimizer update will be applied in the backward pass, in this case through a fused op.
+    # TorchRec will use the FBGEMM implementation of EXACT_ADAGRAD. For GPU devices, a fused CUDA kernel is invoked. For CPU, FBGEMM_GPU invokes CPU kernels
+    # https://github.com/pytorch/FBGEMM/blob/2cb8b0dff3e67f9a009c4299defbd6b99cc12b8f/fbgemm_gpu/fbgemm_gpu/split_table_batched_embeddings_ops.py#L676-L678
+
+    # Note that lr_decay, weight_decay and initial_accumulator_value for Adagrad optimizer in FBGEMM v0.3.2
+    # cannot be specified below. This equivalently means that all these parameters are hardcoded to zero.
+    optimizer_kwargs = {"lr": args.learning_rate}
+    if args.adagrad:
+        optimizer_kwargs["eps"] = args.eps
+
+    model = train_model
+    model.model.sparse_arch = SparseArchTraceAbleWrapper(model.model.sparse_arch, args.dense)
+    checkpoint = None
+    args.epoch = 0
+    args.step = 0
+    if args.converge:
+        checkpoint = load_checkoutpoint(args, device)
+        if checkpoint is not None:
+            args.epoch = checkpoint['epoch']
+            args.step = checkpoint['step']
+            print('[info] loadind model state dict...')
+            train_model.load_state_dict(checkpoint['model'])
+        logger.info(f"The start epoch is : {args.epoch}")
+        logger.info(f"The start step is : {args.step}")
+
+
+    optimizer = None
+    if not args.inference_only:
+        if args.adagrad:
+            optimizer = torch.optim.Adagrad(model.parameters(), lr=args.learning_rate, eps=args.eps)
+        else:
+            optimizer = torch.optim.SGD(model.parameters(), lr=args.learning_rate)
+        if args.converge and checkpoint is not None:
+            print('[info] loadind optimizer state dict...')
+            optimizer.load_state_dict(checkpoint['optimizer'])
+
+    if ext_dist.my_size > 1:
+        if args.use_device == "cuda":
+            device_ids = [rank]
+            model.to(device)
+            model.model.dense_arch = ext_dist.DDP(model.model.dense_arch, device_ids=device_ids)
+            model.model.inter_arch = ext_dist.DDP(model.model.inter_arch, device_ids=device_ids)
+            model.model.over_arch = ext_dist.DDP(model.model.over_arch, device_ids=device_ids)
+            if not args.inference_only:
+                for k, v in optimizer.state.items():
+                    for name, value in v.items():
+                        if isinstance(value, torch.Tensor):
+                            optimizer.state[k][name] = value.to(device)
+        elif args.use_device == "xpu":
+            device_ids = [device]
+            model.to(device)
+            model.model.dense_arch = ext_dist.DDP(model.model.dense_arch, device_ids=device_ids)
+            model.model.inter_arch = ext_dist.DDP(model.model.inter_arch, device_ids=device_ids)
+            model.model.over_arch = ext_dist.DDP(model.model.over_arch, device_ids=device_ids)
+            if not args.inference_only:
+                for k, v in optimizer.state.items():
+                    for name, value in v.items():
+                        if isinstance(value, torch.Tensor):
+                            optimizer.state[k][name] = value.to(device)
+        else:
+            model.model.dense_arch = ext_dist.DDP(model.model.dense_arch)
+            model.model.inter_arch = ext_dist.DDP(model.model.inter_arch)
+            model.model.over_arch = ext_dist.DDP(model.model.over_arch)
+
+    lr_scheduler = None
+    if not args.inference_only:
+        lr_scheduler = LRPolicyScheduler(
+            optimizer, args.lr_warmup_steps, args.lr_decay_start, args.lr_decay_steps
+        )
+        if args.converge and checkpoint is not None:
+            print('[info] loadind lr_scheduler state dict...')
+            lr_scheduler.load_state_dict(checkpoint['lr_scheduler'])
+        model.train()
+        if args.use_device == "xpu":
+            model, optimizer = torch.xpu.optimize(model=model, optimizer=optimizer, level="O1",
+                                                  dtype=torch.float32)
+    else:
+        model.eval()
+        if args.use_device == "xpu":
+            model = torch.xpu.optimize(model=model, dtype=args.amp_dtype)
+
+    if args.trans_data and (args.multi_hot_sizes is not None):
+        multihot = Multihot(
+            args.multi_hot_sizes,
+            args.num_embeddings_per_feature,
+            args.batch_size,
+            collect_freqs_stats=args.collect_multi_hot_freqs_stats,
+            dist_type=args.multi_hot_distribution_type,
+        )
+        multihot.pause_stats_collection_during_val_and_test(model)
+        train_dataloader = RestartableMap(
+            multihot.convert_to_multi_hot, train_dataloader
+        )
+        val_dataloader = RestartableMap(multihot.convert_to_multi_hot, val_dataloader)
+        test_dataloader = RestartableMap(multihot.convert_to_multi_hot, test_dataloader)
+    train_val_test(
+        args,
+        model,
+        optimizer,
+        device,
+        train_dataloader,
+        val_dataloader,
+        test_dataloader,
+        lr_scheduler,
+    )
+    if args.collect_multi_hot_freqs_stats:
+        multihot.save_freqs_stats()
+
+
+if __name__ == "__main__":
+    main(sys.argv[1:])
diff --git a/models_v2/pytorch/torchrec_dlrm/inference/gpu/extend_distributed.py b/models_v2/pytorch/torchrec_dlrm/inference/gpu/extend_distributed.py
new file mode 100644
index 000000000..eb417f8b1
--- /dev/null
+++ b/models_v2/pytorch/torchrec_dlrm/inference/gpu/extend_distributed.py
@@ -0,0 +1,679 @@
+#
+# Copyright 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+import builtins
+import os
+import sys
+
+import torch
+import torch.distributed as dist
+from torch.autograd import Function
+from torch.autograd.profiler import record_function
+from torch.nn.parallel import DistributedDataParallel as DDP
+
+try:
+    import intel_extension_for_pytorch
+except ImportError as e:
+    pass
+
+try:
+    if torch.__version__[:6] >= '1.12.0':
+        import oneccl_bindings_for_pytorch
+        torch_ccl = False
+    else:
+        import torch_ccl
+except ImportError as e:
+    # print(e)
+    torch_ccl = False
+
+try:
+    import torch_ucc
+except ImportError as e:
+    torch_ucc = False
+
+
+my_rank = -1
+my_size = -1
+my_local_rank = -1
+my_local_size = -1
+alltoall_supported = False
+a2a_impl = os.environ.get("DLRM_ALLTOALL_IMPL", "")
+
+myreq = None
+
+
+def env2int(env_list, default=-1):
+    for e in env_list:
+        val = int(os.environ.get(e, -1))
+        if val >= 0:
+            return val
+    return default
+
+
+def get_my_slice(n):
+    tmp_size = dist.get_world_size()
+    tmp_rank = dist.get_rank()
+
+    k, m = divmod(n, tmp_size)
+    return slice(
+        tmp_rank * k + min(tmp_rank, m), (tmp_rank + 1) * k + min(tmp_rank + 1, m), 1
+    )
+
+
+def get_split_lengths(n):
+    if my_size >1:
+        tmp_size = dist.get_world_size()
+        tmp_rank = dist.get_rank()
+    else:
+        tmp_size = my_size
+        tmp_rank = my_rank
+
+    k, m = divmod(n, tmp_size)
+    if m == 0:
+        splits = None
+        my_len = k
+    else:
+        splits = [(k + 1) if i < m else k for i in range(tmp_size)]
+        my_len = splits[tmp_rank]
+    return (my_len, splits)
+
+
+def init_distributed(rank=-1, local_rank=-1, size=-1, use_device="cpu", backend=""):
+    global myreq
+    global my_size
+    global my_local_rank
+    global my_local_size
+    global a2a_impl
+    global alltoall_supported
+
+    # guess MPI ranks from env (works for IMPI, OMPI and MVAPICH2)
+    num_mpi_ranks = env2int(
+        ["PMI_SIZE", "OMPI_COMM_WORLD_SIZE", "MV2_COMM_WORLD_SIZE", "WORLD_SIZE"]
+    )
+    if backend == "" and num_mpi_ranks > 1:
+        if torch_ccl or oneccl_bindings_for_pytorch:
+            backend = "ccl"
+        elif use_device=="cuda" and dist.is_nccl_available():
+            backend = "nccl"
+        elif dist.is_mpi_available():
+            backend = "mpi"
+        else:
+            print(
+                "WARNING: MPI multi-process launch detected but PyTorch MPI backend not available."
+            )
+            backend = "gloo"
+
+    if backend != "":
+        # guess Rank and size
+        if rank == -1:
+            rank = env2int(
+                ["PMI_RANK", "OMPI_COMM_WORLD_RANK", "MV2_COMM_WORLD_RANK", "RANK"], 0
+            )
+        if size == -1:
+            size = env2int(
+                [
+                    "PMI_SIZE",
+                    "OMPI_COMM_WORLD_SIZE",
+                    "MV2_COMM_WORLD_SIZE",
+                    "WORLD_SIZE",
+                ],
+                1,
+            )
+        if not os.environ.get("RANK", None) and rank != -1:
+            os.environ["RANK"] = str(rank)
+        if not os.environ.get("WORLD_SIZE", None) and size != -1:
+            os.environ["WORLD_SIZE"] = str(size)
+        if not os.environ.get("MASTER_PORT", None):
+            os.environ["MASTER_PORT"] = "29500"
+        if not os.environ.get("MASTER_ADDR", None):
+            local_size = env2int(
+                [
+                    "MPI_LOCALNRANKS",
+                    "OMPI_COMM_WORLD_LOCAL_SIZE",
+                    "MV2_COMM_WORLD_LOCAL_SIZE",
+                ],
+                1,
+            )
+            if local_size != size and backend != "mpi":
+                print(
+                    "Warning: Looks like distributed multinode run but MASTER_ADDR env not set, using '127.0.0.1' as default"
+                )
+                print(
+                    "If this run hangs, try exporting rank 0's hostname as MASTER_ADDR"
+                )
+            os.environ["MASTER_ADDR"] = "127.0.0.1"
+
+    if size > 1:
+        if local_rank == -1:
+            my_local_rank = env2int(
+                [
+                    "MPI_LOCALRANKID",
+                    "OMPI_COMM_WORLD_LOCAL_RANK",
+                    "MV2_COMM_WORLD_LOCAL_RANK",
+                    "LOCAL_RANK",
+                ],
+                0,
+            )
+        else:
+            my_local_rank = local_rank
+        my_local_size = env2int(
+            [
+                "MPI_LOCALNRANKS",
+                "OMPI_COMM_WORLD_LOCAL_SIZE",
+                "MV2_COMM_WORLD_LOCAL_SIZE",
+            ],
+            1,
+        )
+        if use_device=="cuda":
+            if my_local_size > torch.cuda.device_count():
+                print(
+                    "Not sufficient GPUs available... local_size = %d, ngpus = %d"
+                    % (my_local_size, torch.cuda.device_count())
+                )
+                sys.exit(1)
+            torch.cuda.set_device(my_local_rank)
+        elif use_device=="xpu":
+            if my_local_size > torch.xpu.device_count():
+                print(
+                    "Not sufficient GPUs available... local_size = %d, ngpus = %d"
+                    % (my_local_size, torch.xpu.device_count())
+                )
+                sys.exit(1)
+            torch.xpu.set_device(my_local_rank)
+        dist.init_process_group(backend, rank=rank, world_size=size)
+        this_rank = dist.get_rank()
+        my_size = dist.get_world_size()
+        #my_local_rank = env2int(['MPI_LOCALRANKID', 'OMPI_COMM_WORLD_LOCAL_RANK', 'MV2_COMM_WORLD_LOCAL_RANK'], 0)
+        #my_local_size = env2int(['MPI_LOCALNRANKS', 'OMPI_COMM_WORLD_LOCAL_SIZE', 'MV2_COMM_WORLD_LOCAL_SIZE'], 1)
+        if this_rank == 0:
+            print("Running on %d ranks using %s backend" % (my_size, backend))
+        if hasattr(dist, "all_to_all_single"):
+            try:
+                if use_device == "xpu":
+                    test_device: torch.device = torch.device("xpu", dist.get_rank())
+                elif use_device == "cuda":
+                    test_device: torch.device = torch.device(f"cuda:{dist.get_rank()}")
+                else:
+                    test_device: torch.device = torch.device(f"cpu:{dist.get_rank()}")
+
+                a = torch.rand([1, 3]).to(test_device)
+                dist.broadcast(a, src=0)
+                print("Successfully execute distributed broadcast on the xpu:{}".format(this_rank))
+
+                tensor_list = [torch.zeros(2, dtype=torch.int64).to(test_device) for _ in range(my_size)]
+                tensor = (torch.arange(2, dtype=torch.int64) + 1 + 2 * rank).to(test_device)
+                dist.all_gather(tensor_list, tensor)
+                print("Successfully execute distributed all_gather on the xpu:{}".format(this_rank))
+
+                t = torch.zeros([4*my_size])
+                if use_device == "cuda":
+                    t = t.cuda()
+                elif  use_device == "xpu":
+                    t = t.xpu()
+                dist.all_to_all_single(t, t)
+
+                print("Successfully execute distributed all_to_all_single on the xpu:{}".format(this_rank))
+                alltoall_supported = True
+            except RuntimeError as err:
+                print("fail to enable all_to_all_single primitive: %s" % err)
+                exit()
+        if a2a_impl == "alltoall" and alltoall_supported == False:
+            print(
+                "Requested DLRM_ALLTOALL_IMPL=%s but backend %s does not support it, use scatter/gather based alltoall"
+                % (a2a_impl, backend)
+            )
+            a2a_impl = "scatter"
+        if a2a_impl != "":
+            print("Using DLRM_ALLTOALL_IMPL=%s" % a2a_impl)
+    else:
+        this_rank = 0
+        my_size = 1
+        my_local_rank = 0
+        my_local_size = 1
+    print_all(
+        "world size: %d, current rank: %d, local rank: %d"
+        % (my_size, this_rank, my_local_rank)
+    )
+    myreq = Request()
+
+
+class Request(object):
+    def __init__(self):
+        self.req = None
+        self.tensor = None
+        self.WaitFunction = All2All_Scatter_Wait
+
+    def wait(self):
+        ret = self.WaitFunction.apply(*self.tensor)
+        self.req = None
+        self.tensor = None
+        return ret
+
+
+class All2All_ScatterList_Req(Function):
+    @staticmethod
+    def forward(ctx, a2a_info, *inputs):
+        global myreq
+        tmp_size = dist.get_world_size()
+        tmp_rank = dist.get_rank()
+        batch_split_lengths = (
+            a2a_info.global_batch_partition_slices
+            if a2a_info.global_batch_partition_slices
+            else a2a_info.local_batch_num
+        )
+        table_split_lengths = (
+            a2a_info.global_table_wise_parition_slices
+            if a2a_info.global_table_wise_parition_slices
+            else [a2a_info.local_table_num] * tmp_size
+        )
+        gather_list = []
+        req_list = []
+        for i in range(tmp_size):
+            for j in range(table_split_lengths[i]):
+                out_tensor = inputs[0].new_empty(
+                    [a2a_info.local_batch_num, a2a_info.emb_dim]
+                )
+                scatter_list = (
+                    list(inputs[j].split(batch_split_lengths, dim=0))
+                    if i == tmp_rank
+                    else []
+                )
+                req = dist.scatter(out_tensor, scatter_list, src=i, async_op=True)
+                gather_list.append(out_tensor)
+                req_list.append(req)
+        myreq.req = req_list
+        myreq.tensor = tuple(gather_list)
+        myreq.a2a_info = a2a_info
+        return myreq.tensor
+
+    @staticmethod
+    def backward(ctx, *grad_output):
+        global myreq
+        for r in myreq.req:
+            r.wait()
+        myreq.req = None
+        grad_inputs = myreq.tensor
+        myreq.tensor = None
+        return (None, *grad_inputs)
+
+
+class All2All_ScatterList_Wait(Function):
+    @staticmethod
+    def forward(ctx, *output):
+        global myreq
+        ctx.a2a_info = myreq.a2a_info
+        for r in myreq.req:
+            r.wait()
+        myreq.req = None
+        myreq.tensor = None
+        return output
+
+    @staticmethod
+    def backward(ctx, *grad_output):
+        global myreq
+        tmp_size = dist.get_world_size()
+        tmp_rank = dist.get_rank()
+        a2a_info = ctx.a2a_info
+        grad_output = [t.contiguous() for t in grad_output]
+        batch_split_lengths = (
+            a2a_info.global_batch_partition_slices
+            if a2a_info.global_batch_partition_slices
+            else [a2a_info.local_batch_num] * tmp_size
+        )
+        per_rank_table_splits = (
+            a2a_info.global_table_wise_parition_slices
+            if a2a_info.global_table_wise_parition_slices
+            else [a2a_info.local_table_num] * tmp_size
+        )
+        grad_inputs = [
+            grad_output[0].new_empty([ctx.a2a_info.batch_size, ctx.a2a_info.emb_dim])
+            for _ in range(a2a_info.local_table_num)
+        ]
+        req_list = []
+        ind = 0
+        for i in range(tmp_size):
+            for j in range(per_rank_table_splits[i]):
+                gather_list = (
+                    list(grad_inputs[j].split(batch_split_lengths, dim=0))
+                    if i == tmp_rank
+                    else None
+                )
+                req = dist.gather(grad_output[ind], gather_list, dst=i, async_op=True)
+                req_list.append(req)
+                ind += 1
+        myreq.req = req_list
+        myreq.tensor = grad_inputs
+        return tuple(grad_output)
+
+
+class All2All_Scatter_Req(Function):
+    @staticmethod
+    def forward(ctx, a2a_info, *inputs):
+        tmp_size = dist.get_world_size()
+        tmp_rank = dist.get_rank()
+        global myreq
+        batch_split_lengths = (
+            a2a_info.global_batch_partition_slices
+            if a2a_info.global_batch_partition_slices
+            else a2a_info.local_batch_num
+        )
+        table_split_lengths = (
+            a2a_info.global_table_wise_parition_slices
+            if a2a_info.global_table_wise_parition_slices
+            else [a2a_info.local_table_num] * tmp_size
+        )
+        input = torch.cat(inputs, dim=1)
+        scatter_list = list(input.split(batch_split_lengths, dim=0))
+        gather_list = []
+        req_list = []
+        for i in range(tmp_size):
+            out_tensor = input.new_empty(
+                [a2a_info.local_batch_num, table_split_lengths[i] * a2a_info.emb_dim]
+            )
+            req = dist.scatter(
+                out_tensor, scatter_list if i == tmp_rank else [], src=i, async_op=True
+            )
+            gather_list.append(out_tensor)
+            req_list.append(req)
+        myreq.req = req_list
+        myreq.tensor = tuple(gather_list)
+        myreq.a2a_info = a2a_info
+        ctx.a2a_info = a2a_info
+        return myreq.tensor
+
+    @staticmethod
+    def backward(ctx, *grad_output):
+        global myreq
+        for r in myreq.req:
+            r.wait()
+        myreq.req = None
+        grad_input = myreq.tensor
+        grad_inputs = grad_input.split(ctx.a2a_info.emb_dim, dim=1)
+        myreq.tensor = None
+        return (None, *grad_inputs)
+
+
+class All2All_Scatter_Wait(Function):
+    @staticmethod
+    def forward(ctx, *output):
+        global myreq
+        ctx.a2a_info = myreq.a2a_info
+        for r in myreq.req:
+            r.wait()
+        myreq.req = None
+        myreq.tensor = None
+        return output
+
+    @staticmethod
+    def backward(ctx, *grad_output):
+        tmp_size = dist.get_world_size()
+        tmp_rank = dist.get_rank()
+        global myreq
+        assert len(grad_output) == tmp_size
+        scatter_list = [t.contiguous() for t in grad_output]
+        a2a_info = ctx.a2a_info
+        batch_split_lengths = (
+            a2a_info.global_batch_partition_slices
+            if a2a_info.global_batch_partition_slices
+            else a2a_info.local_batch_num
+        )
+        table_split_lengths = (
+            a2a_info.global_table_wise_parition_slices
+            if a2a_info.global_table_wise_parition_slices
+            else [a2a_info.local_table_num] * tmp_size
+        )
+        grad_input = grad_output[0].new_empty(
+            [a2a_info.batch_size, a2a_info.emb_dim * a2a_info.local_table_num]
+        )
+        gather_list = list(grad_input.split(batch_split_lengths, dim=0))
+        req_list = []
+        for i in range(tmp_size):
+            req = dist.gather(
+                scatter_list[i],
+                gather_list if i == tmp_rank else [],
+                dst=i,
+                async_op=True,
+            )
+            req_list.append(req)
+        myreq.req = req_list
+        myreq.tensor = grad_input
+        return grad_output
+
+
+class All2All_Req(Function):
+    @staticmethod
+    def forward(ctx, a2a_info, *inputs):
+        global myreq
+        with record_function("DLRM alltoall_req_fwd_single"):
+            batch_split_lengths = a2a_info.global_batch_partition_slices
+            if batch_split_lengths:
+                batch_split_lengths = [
+                    m * a2a_info.emb_dim * a2a_info.local_table_num
+                    for m in batch_split_lengths
+                ]
+            table_split_lengths = a2a_info.global_table_wise_parition_slices
+            if table_split_lengths:
+                table_split_lengths = [
+                    a2a_info.local_batch_num * e * a2a_info.emb_dim
+                    for e in table_split_lengths
+                ]
+            input = torch.cat(inputs, dim=1).view([-1])
+            output = input.new_empty(
+                [
+                    a2a_info.global_table_num
+                    * a2a_info.local_batch_num
+                    * a2a_info.emb_dim
+                ]
+            )
+            req = dist.all_to_all_single(
+                output, input, table_split_lengths, batch_split_lengths, async_op=True
+            )
+
+            myreq.req = req
+            myreq.tensor = []
+            myreq.tensor.append(output)
+            myreq.tensor = tuple(myreq.tensor)
+            a2a_info.batch_split_lengths = batch_split_lengths
+            a2a_info.table_split_lengths = table_split_lengths
+            myreq.a2a_info = a2a_info
+            ctx.a2a_info = a2a_info
+            return myreq.tensor
+
+    @staticmethod
+    def backward(ctx, *grad_output):
+        global myreq
+        with record_function("DLRM alltoall_req_bwd_single"):
+            a2a_info = ctx.a2a_info
+            myreq.req.wait()
+            myreq.req = None
+            grad_input = myreq.tensor
+            grad_inputs = grad_input.view([a2a_info.batch_size, -1]).split(
+                a2a_info.emb_dim, dim=1
+            )
+            grad_inputs = [gin.contiguous() for gin in grad_inputs]
+            myreq.tensor = None
+            return (None, *grad_inputs)
+
+
+class All2All_Wait(Function):
+    @staticmethod
+    def forward(ctx, *output):
+        global myreq
+        with record_function("DLRM alltoall_wait_fwd_single"):
+            a2a_info = myreq.a2a_info
+            ctx.a2a_info = a2a_info
+            myreq.req.wait()
+            myreq.req = None
+            myreq.tensor = None
+            table_split_lengths = (
+                a2a_info.table_split_lengths
+                if a2a_info.table_split_lengths
+                else a2a_info.local_table_num
+                * a2a_info.local_batch_num
+                * a2a_info.emb_dim
+            )
+            outputs = output[0].split(table_split_lengths)
+            outputs = tuple(
+                [out.view([a2a_info.local_batch_num, -1]) for out in outputs]
+            )
+            return outputs
+
+    @staticmethod
+    def backward(ctx, *grad_outputs):
+        global myreq
+        with record_function("DLRM alltoall_wait_bwd_single"):
+            a2a_info = ctx.a2a_info
+            grad_outputs = [gout.contiguous().view([-1]) for gout in grad_outputs]
+            grad_output = torch.cat(grad_outputs)
+            grad_input = grad_output.new_empty(
+                [a2a_info.batch_size * a2a_info.local_table_num * a2a_info.emb_dim]
+            )
+            req = dist.all_to_all_single(
+                grad_input,
+                grad_output,
+                a2a_info.batch_split_lengths,
+                a2a_info.table_split_lengths,
+                async_op=True,
+            )
+            myreq.req = req
+            myreq.tensor = grad_input
+            return (grad_output,)
+
+
+class AllGather(Function):
+    @staticmethod
+    def forward(ctx, input, global_lengths, dim=0):
+        tmp_size = dist.get_world_size()
+        tmp_rank = dist.get_rank()
+        if not isinstance(global_lengths, (list, tuple)):
+            global_lengths = [global_lengths] * tmp_size
+
+        assert len(global_lengths) == tmp_size
+        assert global_lengths[tmp_rank] == input.size(dim)
+        local_start = sum(global_lengths[:tmp_rank])
+
+        output_size = list(input.size())
+
+        ctx.dim = dim
+        ctx.local_start = local_start
+        ctx.local_length = global_lengths[tmp_rank]
+
+        input = input.contiguous()
+        if dim == 0:
+            out_len = sum(global_lengths)
+            output_size[dim] = out_len
+            output = input.new_empty(output_size)
+            gather_list = list(output.split(global_lengths, dim=0))
+        else:
+            gather_list = [torch.empty_like(input) for _ in range(tmp_size)]
+            gather_list = []
+            for length in global_lengths:
+                output_size[dim] = length
+                gather_list.append(input.new_empty(output_size))
+
+        dist.all_gather(gather_list, input)
+
+        if dim != 0:
+            output = torch.cat(gather_list, dim=dim)
+
+        return output
+
+    @staticmethod
+    def backward(ctx, grad_output):
+        # print("Inside All2AllBackward")
+        dim = ctx.dim
+        start = ctx.local_start
+        length = ctx.local_length
+
+        grad_input = grad_output.narrow(dim, start, length)
+
+        return (grad_input, None, None)
+
+
+class All2AllInfo(object):
+    pass
+
+
+def alltoall(inputs, per_rank_table_splits):
+    tmp_size = dist.get_world_size()
+    global myreq
+    batch_size, emb_dim = inputs[0].size()
+    a2a_info = All2AllInfo()
+    a2a_info.local_table_num = len(inputs)
+    a2a_info.global_table_wise_parition_slices = per_rank_table_splits
+    (
+        a2a_info.local_batch_num,
+        a2a_info.global_batch_partition_slices,
+    ) = get_split_lengths(batch_size)
+    a2a_info.emb_dim = emb_dim
+    a2a_info.batch_size = batch_size
+    a2a_info.global_table_num = (
+        sum(per_rank_table_splits)
+        if per_rank_table_splits
+        else a2a_info.local_table_num * tmp_size
+    )
+
+    if a2a_impl == "" and alltoall_supported or a2a_impl == "alltoall":
+        # print("Using All2All_Req")
+        output = All2All_Req.apply(a2a_info, *inputs)
+        myreq.WaitFunction = All2All_Wait
+    elif a2a_impl == "" or a2a_impl == "scatter":
+        # print("Using All2All_Scatter_Req")
+        output = All2All_Scatter_Req.apply(a2a_info, *inputs)
+        myreq.WaitFunction = All2All_Scatter_Wait
+    elif a2a_impl == "scatter_list":
+        # print("Using All2All_ScatterList_Req")
+        output = All2All_ScatterList_Req.apply(a2a_info, *inputs)
+        myreq.WaitFunction = All2All_ScatterList_Wait
+    else:
+        print(
+            "Unknown value set for DLRM_ALLTOALL_IMPL (%s), "
+            "please use one of [alltoall, scatter, scatter_list]" % a2a_impl
+        )
+    return myreq
+
+
+def all_gather(input, lengths, dim=0):
+    tmp_size = dist.get_world_size()
+    if not lengths:
+        lengths = [input.size(0)] * tmp_size
+    return AllGather.apply(input, lengths, dim)
+
+
+def barrier():
+    if my_size > 1:
+        dist.barrier()
+
+
+# Override builtin print function to print only from rank 0
+orig_print = builtins.print
+#
+#
+#def rank0_print(*args, **kwargs):
+#    if my_rank <= 0 or kwargs.get("print_all", False):
+#        orig_print(*args, **kwargs)
+#
+#
+#builtins.print = rank0_print
+#
+# Allow printing from all rank with explicit print_all
+def print_all(*args, **kwargs):
+    orig_print(*args, **kwargs)
diff --git a/models_v2/pytorch/torchrec_dlrm/inference/gpu/jit_trace_able_utils.py b/models_v2/pytorch/torchrec_dlrm/inference/gpu/jit_trace_able_utils.py
new file mode 100644
index 000000000..197994c62
--- /dev/null
+++ b/models_v2/pytorch/torchrec_dlrm/inference/gpu/jit_trace_able_utils.py
@@ -0,0 +1,69 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import torch
+from torch import nn
+from torchrec.models.dlrm import SparseArch
+from torchrec.sparse.jagged_tensor import KeyedJaggedTensor
+from typing import List
+
+def unpack(input :KeyedJaggedTensor, default_embedding_names:[str]) -> dict:
+    output = {}
+    for k, v in input.to_dict().items():
+        if k in default_embedding_names:
+            output[k] = {}
+            output[k]['values'] = v._values.to(torch.int64)
+            output[k]['offsets'] = v._offsets.to(torch.int64)
+    return output
+
+
+class SparseArchTraceAbleWrapper(nn.Module):
+
+    def __init__(self, sparse_arch: SparseArch, dense) -> None:
+        super().__init__()
+        self.sparse_arch: SparseArch = sparse_arch
+        if not dense:
+            for embedding_bag in self.sparse_arch.embedding_bag_collection.embedding_bags.values():
+                embedding_bag.sparse = True
+
+
+    def forward(
+        self,
+        features: dict,
+    ) -> torch.Tensor:
+        """
+        Args:
+            features (dict): input tensors of sparse features.
+        Returns:
+            torch.Tensor: tensor of shape B X F X D.
+        """
+        embedding_bag_collection = self.sparse_arch.embedding_bag_collection
+        pooled_embeddings: List[torch.Tensor] = []
+        for i, embedding_bag in enumerate(embedding_bag_collection.embedding_bags.values()):
+            for feature_name in embedding_bag_collection._feature_names[i]:
+                f = features[feature_name]
+                res = embedding_bag(
+                    f['values'],
+                    f['offsets'],
+                    per_sample_weights=None,
+                )
+                pooled_embeddings.append(res)
+        #data = torch.cat(pooled_embeddings, dim=1)
+        #B: int = features['cat_0']['offsets'].numel() - 1
+        #return data.reshape(B, self.sparse_arch.F, self.sparse_arch.D)
+        return pooled_embeddings
diff --git a/models_v2/pytorch/torchrec_dlrm/inference/gpu/lr_scheduler.py b/models_v2/pytorch/torchrec_dlrm/inference/gpu/lr_scheduler.py
new file mode 100644
index 000000000..4aebce416
--- /dev/null
+++ b/models_v2/pytorch/torchrec_dlrm/inference/gpu/lr_scheduler.py
@@ -0,0 +1,48 @@
+#!/usr/bin/env python3
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+# Copied from https://github.com/facebookresearch/dlrm/blob/mlperf/dlrm_s_pytorch.py
+
+import sys
+
+from torch.optim.lr_scheduler import _LRScheduler
+
+
+class LRPolicyScheduler(_LRScheduler):
+    def __init__(self, optimizer, num_warmup_steps, decay_start_step, num_decay_steps):
+        self.num_warmup_steps = num_warmup_steps
+        self.decay_start_step = decay_start_step
+        self.decay_end_step = decay_start_step + num_decay_steps
+        self.num_decay_steps = num_decay_steps
+
+        if self.decay_start_step < self.num_warmup_steps:
+            sys.exit("Learning rate warmup must finish before the decay starts")
+
+        super(LRPolicyScheduler, self).__init__(optimizer)
+
+    def get_lr(self):
+        step_count = self._step_count
+        if step_count < self.num_warmup_steps:
+            # warmup
+            scale = 1.0 - (self.num_warmup_steps - step_count) / self.num_warmup_steps
+            lr = [base_lr * scale for base_lr in self.base_lrs]
+            self.last_lr = lr
+        elif self.decay_start_step <= step_count and step_count < self.decay_end_step:
+            # decay
+            decayed_steps = step_count - self.decay_start_step
+            scale = ((self.num_decay_steps - decayed_steps) / self.num_decay_steps) ** 2
+            min_lr = 0.0000001
+            lr = [max(min_lr, base_lr * scale) for base_lr in self.base_lrs]
+            self.last_lr = lr
+        else:
+            if self.num_decay_steps > 0:
+                # freeze at last, either because we're after decay
+                # or because we're between warmup and decay
+                lr = self.last_lr
+            else:
+                # do not adjust
+                lr = self.base_lrs
+        return lr
diff --git a/models_v2/pytorch/torchrec_dlrm/inference/gpu/md5sums_MLPerf_v2_synthetic_multi_hot_sparse_dataset.txt b/models_v2/pytorch/torchrec_dlrm/inference/gpu/md5sums_MLPerf_v2_synthetic_multi_hot_sparse_dataset.txt
new file mode 100644
index 000000000..9aebba868
--- /dev/null
+++ b/models_v2/pytorch/torchrec_dlrm/inference/gpu/md5sums_MLPerf_v2_synthetic_multi_hot_sparse_dataset.txt
@@ -0,0 +1,24 @@
+9287c283b01087427df915257300bf36  day_0_sparse_multi_hot.npz
+f4fedd921a4b214b03d0dfcd31cc30e6  day_1_sparse_multi_hot.npz
+2a15bdd8d25781c4cdcf5d791dfd19f9  day_2_sparse_multi_hot.npz
+0341aaee2f661f9e939a39d7a6be0aea  day_3_sparse_multi_hot.npz
+e8db54dbf5fe438ecb76fcc2d520f31a  day_4_sparse_multi_hot.npz
+fd35a7a2bc0ba63935b4b1c742eca018  day_5_sparse_multi_hot.npz
+7d5c72b6bbe8be1dfa1f69db5c7e64fd  day_6_sparse_multi_hot.npz
+59bcb9855243d3a5c0ae56daa18e1033  day_7_sparse_multi_hot.npz
+b9f7fbccae6bb9fdabf30259b5e305f0  day_8_sparse_multi_hot.npz
+03da5bf484870a3c77f66befc680ad04  day_9_sparse_multi_hot.npz
+eb048fc4fbd7ffa7932b81a523fe5a39  day_10_sparse_multi_hot.npz
+a2ebee45c9836c8c8598610a6a3e9d60  day_11_sparse_multi_hot.npz
+0dd59855e1a7b65c42f7e7af303c610e  day_12_sparse_multi_hot.npz
+7510698f3fcff9f3d7ef9dd478e637aa  day_13_sparse_multi_hot.npz
+562978b7b93e179f1adb9b9f5e1dc338  day_14_sparse_multi_hot.npz
+042967232f016fbccf0d40d72d0b48bb  day_15_sparse_multi_hot.npz
+7b59170fb0e2d1e78f15cb60cea22723  day_16_sparse_multi_hot.npz
+5054c54515d2574cda0f11646787df44  day_17_sparse_multi_hot.npz
+28d3dbf6c70e68f01df12a4c3298f754  day_18_sparse_multi_hot.npz
+db7554263a1754d3e29341d0f03bc2f0  day_19_sparse_multi_hot.npz
+91ee92ffb4810c26e157c1335ef4de06  day_20_sparse_multi_hot.npz
+2c99fad7b146b0ba581dce34f640f44e  day_21_sparse_multi_hot.npz
+c7ba52c5aaf24d76acca22a0cb13b737  day_22_sparse_multi_hot.npz
+c46b7e31ec6f2f8768fa60bdfc0f6e40  day_23_sparse_multi_hot.npz
diff --git a/models_v2/pytorch/torchrec_dlrm/inference/gpu/md5sums_preprocessed_criteo_click_logs_dataset.txt b/models_v2/pytorch/torchrec_dlrm/inference/gpu/md5sums_preprocessed_criteo_click_logs_dataset.txt
new file mode 100644
index 000000000..f9fa79933
--- /dev/null
+++ b/models_v2/pytorch/torchrec_dlrm/inference/gpu/md5sums_preprocessed_criteo_click_logs_dataset.txt
@@ -0,0 +1,72 @@
+427113b0c4d85a8fceaf793457302067  day_0_dense.npy
+4db255ce4388893e7aa1dcf157077975  day_0_labels.npy
+8b444e74159dbede896e2f3b5ed31ac0  day_0_sparse.npy
+3afc11c56062d8bbea4df300b5a42966  day_1_dense.npy
+fb40746738a7c6f4ee021033bdd518c5  day_1_labels.npy
+61e95a487c955b515155b31611444f32  day_1_sparse.npy
+4e73d5bb330c43826665bec142c6b407  day_2_dense.npy
+f0adfec8191781e3f201d45f923e6ea1  day_2_labels.npy
+0473d30872cd6e582c5da0272a0569f8  day_2_sparse.npy
+df1f3395e0da4a06aa23b2e069ff3ad9  day_3_dense.npy
+69caadf4d219f18b83f3591fe76f17c7  day_3_labels.npy
+d6b0d02ff18da470b7ee17f97d5380e0  day_3_sparse.npy
+27868a93adc66c47d4246acbad8bb689  day_4_dense.npy
+c4a6a16342f0770d67d689c6c173c681  day_4_labels.npy
+ca54008489cb84becc3f37e7b29035c7  day_4_sparse.npy
+e9bc6de06d09b1feebf857d9786ee15c  day_5_dense.npy
+9e3e17f345474cfbde5d62b543e07d6b  day_5_labels.npy
+d1374ee84f80ea147957f8af0e12ebe4  day_5_sparse.npy
+09c8bf0fd4798172e0369134ddc7204a  day_6_dense.npy
+945cef1132ceab8b23f4d0e269522be2  day_6_labels.npy
+e4df1c271e1edd72ee4658a39cca2888  day_6_sparse.npy
+ae718f0d6d29a8b605ae5d12fad3ffcc  day_7_dense.npy
+5ff5e7eef5b88b80ef03d06fc7e81bcf  day_7_labels.npy
+cbcb7501a6b74a45dd5c028c13a4afbc  day_7_sparse.npy
+5a589746fd15819afbc70e2503f94b35  day_8_dense.npy
+43871397750dfdc69cadcbee7e95f2bd  day_8_labels.npy
+c1fb4369c7da27d23f4c7f97c8893250  day_8_sparse.npy
+4bb86eecb92eb4e3368085c2b1bab131  day_9_dense.npy
+f851934555147d436131230ebbdd5609  day_9_labels.npy
+e4ac0fb8a030f0769541f88142c9f931  day_9_sparse.npy
+7fc29f50da6c60185381ca4ad1cb2059  day_10_dense.npy
+e3b3f6f974c4820064db0046bbf954c8  day_10_labels.npy
+1018a9ab88c4a7369325c9d6df73b411  day_10_sparse.npy
+df822ae73cbaa016bf7d371d87313b56  day_11_dense.npy
+26219e9c89c6ce831e7da273da666df1  day_11_labels.npy
+f1596fc0337443a6672a864cd541fb05  day_11_sparse.npy
+015968b4d9940ec9e28cc34788013d6e  day_12_dense.npy
+f0ca7ce0ab6033cdd355df94d11c7ed7  day_12_labels.npy
+03a2ebd22b01cc18b6e338de77b4103f  day_12_sparse.npy
+9d79239a9e976e4dd9b8839c7cbe1eba  day_13_dense.npy
+4b099b9200bbb490afc08b5cd63daa0e  day_13_labels.npy
+2b507e0f97d972ea6ada9b3af64de151  day_13_sparse.npy
+9242e6c974603ec235f163f72fdbc766  day_14_dense.npy
+80cae15e032ffb9eff292738ba4d0dce  day_14_labels.npy
+3dccc979f7c71fae45a10c98ba6c9cb7  day_14_sparse.npy
+64c6c0fcd0940f7e0d7001aa945ec8f8  day_15_dense.npy
+a6a730d1ef55368f3f0b21d32b039662  day_15_labels.npy
+c852516852cc404cb40d4de8626d2ca1  day_15_sparse.npy
+5c75b60e63e9cf98dec13fbb64839c10  day_16_dense.npy
+5a71a29d8df1e8baf6bf28353f1588d4  day_16_labels.npy
+6c838050751697a91bbf3e68ffd4a696  day_16_sparse.npy
+9798bccb5a67c5eac834153ea8bbe110  day_17_dense.npy
+0a814b7eb83f375dd5a555ade6908356  day_17_labels.npy
+40d2bc23fbcccb3ddb1390cc5e694cf0  day_17_sparse.npy
+cda094dfe7f5711877a6486f9863cd4b  day_18_dense.npy
+a4fa26ada0d4c312b7e3354de0f5ee30  day_18_labels.npy
+51711de9194737813a74bfb25c0f5d30  day_18_sparse.npy
+0f0b2c0ed279462cdcc6f79252fd3395  day_19_dense.npy
+b21ad457474b01bd3f95fc46b6b9f04b  day_19_labels.npy
+dd4b72cd704981441d17687f526e42ae  day_19_sparse.npy
+95ffc084f6cafe382afe72cbcae186bc  day_20_dense.npy
+9555e572e8bee22d71db8c2ac121ea8a  day_20_labels.npy
+bc9a8c79c93ea39f32230459b4c4572a  day_20_sparse.npy
+4680683973be5b1a890c9314cfb2e93b  day_21_dense.npy
+672edc866e7ff1928d15338a99e5f336  day_21_labels.npy
+e4a8ae42a6d46893da6edb73e7d8a3f7  day_21_sparse.npy
+3d56f190639398da2bfdc33f87cd34f0  day_22_dense.npy
+733da710c5981cb67d041aa1039e4e6b  day_22_labels.npy
+42ef88d6bb2550a88711fed6fc144846  day_22_sparse.npy
+cdf7af87cbc7e9b468c0be46b1767601  day_23_dense.npy
+dd68f93301812026ed6f58dfb0757fa7  day_23_labels.npy
+0c33f1562529cc3bca7f3708e2be63c9  day_23_sparse.npy
diff --git a/models_v2/pytorch/torchrec_dlrm/inference/gpu/multi_hot.py b/models_v2/pytorch/torchrec_dlrm/inference/gpu/multi_hot.py
new file mode 100644
index 000000000..77b13b9e8
--- /dev/null
+++ b/models_v2/pytorch/torchrec_dlrm/inference/gpu/multi_hot.py
@@ -0,0 +1,175 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+from typing import List, Tuple
+
+import numpy as np
+
+import torch
+from torchrec.datasets.utils import Batch
+from torchrec.sparse.jagged_tensor import KeyedJaggedTensor
+
+
+class RestartableMap:
+    def __init__(self, f, source):
+        self.source = source
+        self.func = f
+
+    def __iter__(self):
+        for x in self.source:
+            yield self.func(x)
+
+    def __len__(self):
+        return len(self.source)
+
+
+class Multihot:
+    def __init__(
+        self,
+        multi_hot_sizes: List[int],
+        num_embeddings_per_feature: List[int],
+        batch_size: int,
+        collect_freqs_stats: bool,
+        dist_type: str = "uniform",
+    ):
+        if dist_type not in {"uniform", "pareto"}:
+            raise ValueError(
+                "Multi-hot distribution type {} is not supported."
+                'Only "uniform" and "pareto" are supported.'.format(dist_type)
+            )
+        self.dist_type = dist_type
+        self.multi_hot_sizes = multi_hot_sizes
+        self.num_embeddings_per_feature = num_embeddings_per_feature
+        self.batch_size = batch_size
+
+        # Generate 1-hot to multi-hot lookup tables, one lookup table per sparse embedding table.
+        self.multi_hot_tables_l = self.__make_multi_hot_indices_tables(
+            dist_type, multi_hot_sizes, num_embeddings_per_feature
+        )
+
+        # Pooling offsets are computed once and reused.
+        self.offsets = self.__make_offsets(
+            multi_hot_sizes, num_embeddings_per_feature, batch_size
+        )
+
+        # For plotting frequency access
+        self.collect_freqs_stats = collect_freqs_stats
+        self.model_to_track = None
+        self.freqs_pre_hash = []
+        self.freqs_post_hash = []
+        for embs_count in num_embeddings_per_feature:
+            self.freqs_pre_hash.append(np.zeros((embs_count)))
+            self.freqs_post_hash.append(np.zeros((embs_count)))
+
+    def save_freqs_stats(self) -> None:
+        if torch.distributed.is_available() and torch.distributed.is_initialized():
+            rank = torch.distributed.get_rank()
+        else:
+            rank = 0
+        pre_dict = {str(k): e for k, e in enumerate(self.freqs_pre_hash)}
+        np.save(f"stats_pre_hash_{rank}_{self.dist_type}.npy", pre_dict)
+        post_dict = {str(k): e for k, e in enumerate(self.freqs_post_hash)}
+        np.save(f"stats_post_hash_{rank}_{self.dist_type}.npy", post_dict)
+
+    def pause_stats_collection_during_val_and_test(
+        self, model: torch.nn.Module
+    ) -> None:
+        self.model_to_track = model
+
+    def __make_multi_hot_indices_tables(
+        self,
+        dist_type: str,
+        multi_hot_sizes: List[int],
+        num_embeddings_per_feature: List[int],
+    ) -> List[np.array]:
+        np.random.seed(
+            0
+        )  # The seed is necessary for all ranks to produce the same lookup values.
+        multi_hot_tables_l = []
+        for embs_count, multi_hot_size in zip(
+            num_embeddings_per_feature, multi_hot_sizes
+        ):
+            embedding_ids = np.arange(embs_count)[:, np.newaxis]
+            if dist_type == "uniform":
+                synthetic_sparse_ids = np.random.randint(
+                    0, embs_count, size=(embs_count, multi_hot_size - 1)
+                )
+            elif dist_type == "pareto":
+                synthetic_sparse_ids = (
+                    np.random.pareto(
+                        a=0.25, size=(embs_count, multi_hot_size - 1)
+                    ).astype(np.int32)
+                    % embs_count
+                )
+            multi_hot_table = np.concatenate(
+                (embedding_ids, synthetic_sparse_ids), axis=-1
+            )
+            multi_hot_tables_l.append(multi_hot_table)
+        multi_hot_tables_l = [
+            torch.from_numpy(multi_hot_table).int()
+            for multi_hot_table in multi_hot_tables_l
+        ]
+        return multi_hot_tables_l
+
+    def __make_offsets(
+        self,
+        multi_hot_sizes: int,
+        num_embeddings_per_feature: List[int],
+        batch_size: int,
+    ) -> List[torch.Tensor]:
+        lS_o = torch.ones(
+            (len(num_embeddings_per_feature) * batch_size), dtype=torch.int32
+        )
+        for k, multi_hot_size in enumerate(multi_hot_sizes):
+            lS_o[k * batch_size : (k + 1) * batch_size] = multi_hot_size
+        lS_o = torch.cumsum(torch.concat((torch.tensor([0]), lS_o)), axis=0)
+        return lS_o
+
+    def __make_new_batch(
+        self,
+        lS_i: torch.Tensor,
+        batch_size: int,
+    ) -> Tuple[torch.Tensor, torch.Tensor]:
+        lS_i = lS_i.reshape(-1, batch_size)
+        multi_hot_ids_l = []
+        for k, (sparse_data_batch_for_table, multi_hot_table) in enumerate(
+            zip(lS_i, self.multi_hot_tables_l)
+        ):
+            multi_hot_ids = torch.nn.functional.embedding(
+                sparse_data_batch_for_table, multi_hot_table
+            )
+            multi_hot_ids = multi_hot_ids.reshape(-1)
+            multi_hot_ids_l.append(multi_hot_ids)
+            if self.collect_freqs_stats and (
+                self.model_to_track is None or self.model_to_track.training
+            ):
+                idx_pre, cnt_pre = np.unique(
+                    sparse_data_batch_for_table, return_counts=True
+                )
+                idx_post, cnt_post = np.unique(multi_hot_ids, return_counts=True)
+                self.freqs_pre_hash[k][idx_pre] += cnt_pre
+                self.freqs_post_hash[k][idx_post] += cnt_post
+        lS_i = torch.cat(multi_hot_ids_l)
+        if batch_size == self.batch_size:
+            return lS_i, self.offsets
+        else:
+            return lS_i, self.__make_offsets(
+                self.multi_hot_sizes, self.num_embeddings_per_feature, batch_size
+            )
+
+    def convert_to_multi_hot(self, batch: Batch) -> Batch:
+        batch_size = len(batch.dense_features)
+        lS_i = batch.sparse_features._values
+        lS_i, lS_o = self.__make_new_batch(lS_i, batch_size)
+        new_sparse_features = KeyedJaggedTensor.from_offsets_sync(
+            keys=batch.sparse_features._keys,
+            values=lS_i,
+            offsets=lS_o,
+        )
+        return Batch(
+            dense_features=batch.dense_features,
+            sparse_features=new_sparse_features,
+            labels=batch.labels,
+        )
diff --git a/models_v2/pytorch/torchrec_dlrm/inference/gpu/requirements.txt b/models_v2/pytorch/torchrec_dlrm/inference/gpu/requirements.txt
new file mode 100644
index 000000000..270005d8d
--- /dev/null
+++ b/models_v2/pytorch/torchrec_dlrm/inference/gpu/requirements.txt
@@ -0,0 +1,4 @@
+fbgemm-gpu==0.3.2
+torchmetrics==0.11.0
+torchrec==0.3.2
+torchsnapshot
diff --git a/models_v2/pytorch/torchrec_dlrm/inference/gpu/run_model.sh b/models_v2/pytorch/torchrec_dlrm/inference/gpu/run_model.sh
new file mode 100755
index 000000000..27c1d8507
--- /dev/null
+++ b/models_v2/pytorch/torchrec_dlrm/inference/gpu/run_model.sh
@@ -0,0 +1,131 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+
+#!/bin/bash
+
+# Create an array of input directories that are expected and then verify that they exist
+declare -A input_envs
+input_envs[DATASET_DIR]=${DATASET_DIR}
+input_envs[WEIGHT_DIR]=${WEIGHT_DIR}
+input_envs[MULTI_TILE]=${MULTI_TILE}
+input_envs[PLATFORM]=${PLATFORM}
+
+for i in "${!input_envs[@]}"; do
+  var_name=$i
+  env_param=${input_envs[$i]}
+
+  if [[ -z $env_param ]]; then
+    echo "The required environment variable $var_name is not set" >&2
+    exit 1
+  fi
+done
+
+OUTPUT_DIR=${OUTPUT_DIR:-$PWD}
+
+if [[ "${PLATFORM}" == "PVC" ]]; then
+    BATCH_SIZE=${BATCH_SIZE:-65536}
+    PRECISION=${PRECISION:-FP16}
+elif [[ "${PLATFORM}" == "ATS-M" ]]; then
+    echo "Only support PVC for platform"
+fi
+
+
+
+if [[ -z "${DATASET_DIR}" ]]; then
+  echo "Using Dummy data since environment variable DATASET_DIR has not been set"
+  DATASET_DIR="--dummy"
+else
+  if [[ ! -d "${DATASET_DIR}" ]]; then
+    echo "The DATASET_DIR '${DATASET_DIR}' does not exist"
+    exit 1
+  fi
+fi
+
+# known issue
+if [[ "${MULTI_TILE}" == "True" ]]; then
+    export ZE_FLAT_DEVICE_HIERARCHY=COMPOSITE
+fi
+
+
+echo 'Running with parameters:'
+echo " PLATFORM: ${PLATFORM}"
+echo " WEIGHT_DIR: ${WEIGHT_DIR}"
+echo " DATASET_DIR: ${DATASET_DIR}"
+echo " OUTPUT_DIR: ${OUTPUT_DIR}"
+echo " PRECISION: ${PRECISION}"
+echo " BATCH_SIZE: ${BATCH_SIZE}"
+echo " MULTI_TILE: ${MULTI_TILE}"
+
+if [[ "${PRECISION}" == "FP16" ]]; then
+    flag="-fp16 true"
+elif [[ "${PRECISION}" == "FP32" ]]; then
+    flag="-fp16 false"
+else
+    echo -e "Invalid input! Only FP32 FP16 are supported."
+    exit 1
+fi
+echo "Dlrmv2 ${PRECISION} inference plain MultiTile=${MULTI_TILE} BS=${BATCH_SIZE}"
+
+# Create the output directory, if it doesn't already exist
+mkdir -p $OUTPUT_DIR
+
+sum_log_analysis() {
+    if [ -f $2 ]; then
+        rm -f $2
+    fi
+    if diff /dev/null ${1}_t0.log |tail -l | grep '^\\ No newline' > /dev/null;then echo >> ${1}_t0.log; fi
+    if diff /dev/null ${1}_t1.log |tail -l | grep '^\\ No newline' > /dev/null;then echo >> ${1}_t1.log; fi
+    bs=$(cat ${1}_t1.log |grep Batch |awk '{print $3}')
+    echo -e "Batch Size: $bs" >$2
+    cat ${1}"_t0.log" ${1}"_t1.log" |grep "Performance" |awk -v tag=$(cat ${1}"_t0.log" ${1}"_t1.log" |grep "Performance" |awk '{sum+=$2} END {printf "%.4f\n",sum}') '{if ( $2=="None" ) {sum="None";nextfile}else  sum=tag} ;END{print "Sum "$1" "sum " "$3}' >> $2
+    cat ${1}"_t0.log" ${1}"_t1.log" |grep "Performance" |awk -v tag=$(cat ${1}"_t0.log" ${1}"_t1.log" |grep "Performance" |awk 'BEGIN {min=1234567890123} {if ($2 <min) {min=$2}}END {printf "%.4f\n",min}') '{if ( $2=="None" ) {min="None";nextfile}else  min=tag} ;END{print "Min "$1" "min " "$3}' >> $2
+    cat ${1}"_t0.log" ${1}"_t1.log" |grep "Latency" |awk '{if ( $2=="N/A" ){avg="N/A";nextfile}else avg=((sum+=$2/2))};END{print "Avg "$1" "avg " "$3}'  >> $2
+    cat ${1}"_t0.log" ${1}"_t1.log" |grep "Accuracy" |awk -v avg=$(cat ${1}"_t0.log" ${1}"_t1.log" |grep "Accuracy" |awk '{sum+=$3}END{printf "%.4f\n",sum/NR}') '{if ( $3=="None" || $2=="N/A" || $3=="nan" || $3=="N/A"){avg="None";nextfile}else avg=avg};END{print "Avg "$1" "$2 " "avg}' >> $2
+    cat ${1}"_t0.log" ${1}"_t1.log" |grep "Functional" | awk -v fail=$(cat ${1}"_t0.log" ${1}"_t1.log" |grep "Functional" |awk '{for(i=1;i<=NF;++i) if($i=="fail") ++sum}END{print sum}') '{if ( fail >= 1 ) tag="fail ";else tag="pass"};END{print $1" "tag}'  >> $2
+    cat ${1}"_t0.log" ${1}"_t1.log" |grep "Error" |awk '{if(a[$1]){a[$1]=a[$1]";"$2}else{a[$1]=$2}}END{for(i in a)print $1" " a[i]}' >> $2
+}
+
+modelname=dlrm-terabyte
+if [[ ${MULTI_TILE} == "False" ]]; then
+	echo -e "do not support MULTI_TILE=False"
+	exit 1
+else
+    rm ${OUTPUT_DIR}/ddp-${modelname}_${PRECISION}_inf_raw.log
+    bash cmd_distributed_terabyte_test.sh -d ${DATASET_DIR} -m ${WEIGHT_DIR} ${flag} 2>&1 | tee ${OUTPUT_DIR}/ddp-${modelname}_${PRECISION}_inf_raw.log
+    python ../../../../../models/common/pytorch/parse_result.py -t ddp -l ${OUTPUT_DIR}/ddp-${modelname}_${PRECISION}_inf_raw.log -b ${BATCH_SIZE}
+    throughput=$(cat ${OUTPUT_DIR}/ddp-${modelname}_${PRECISION}_inf.log | grep "Sum Performance" | awk -F ' ' '{print $3}')
+    throughput_unit=$(cat ${OUTPUT_DIR}/ddp-${modelname}_${PRECISION}_inf.log | grep "Sum Performance" | awk -F ' ' '{print $4}')
+    acc=$(cat ${OUTPUT_DIR}/ddp-${modelname}_${PRECISION}_inf.log | grep Accuracy | awk -F ' ' '{print $3}')
+    acc_unit=$(cat ${OUTPUT_DIR}/ddp-${modelname}_${PRECISION}_inf.log | grep Accuracy | awk -F ' ' '{print $2}')
+fi
+
+yaml_content=$(cat <<EOF
+results:
+ - key: throughput
+   value: $throughput
+   unit: $throughput_unit
+ - key: accuracy
+   value: $acc
+   unit: $acc_unit
+EOF
+)
+
+# Write the content to a YAML file
+echo "$yaml_content" >  ./results.yaml
+echo "YAML file created."
diff --git a/models_v2/pytorch/torchrec_dlrm/inference/gpu/scripts/materialize_synthetic_multihot_dataset.py b/models_v2/pytorch/torchrec_dlrm/inference/gpu/scripts/materialize_synthetic_multihot_dataset.py
new file mode 100644
index 000000000..63d703db0
--- /dev/null
+++ b/models_v2/pytorch/torchrec_dlrm/inference/gpu/scripts/materialize_synthetic_multihot_dataset.py
@@ -0,0 +1,153 @@
+#!/usr/bin/env python3
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+import argparse
+import os
+import pathlib
+import shutil
+import sys
+
+import numpy as np
+import torch
+from torch import distributed as dist, nn
+from torchrec.datasets.criteo import DAYS
+
+p = pathlib.Path(__file__).absolute().parents[1].resolve()
+sys.path.append(os.fspath(p))
+
+# OSS import
+try:
+    # pyre-ignore[21]
+    # @manual=//ai_codesign/benchmarks/dlrm/torchrec_dlrm:multi_hot
+    from multi_hot import Multihot
+except ImportError:
+    pass
+
+# internal import
+try:
+    from .multi_hot import Multihot  # noqa F811
+except ImportError:
+    pass
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser(
+        description="Script to materialize synthetic multi-hot dataset into NumPy npz file format."
+    )
+    parser.add_argument(
+        "--in_memory_binary_criteo_path",
+        type=str,
+        required=True,
+        help="Path to a folder containing the binary (npy) files for the Criteo dataset."
+        " When supplied, InMemoryBinaryCriteoIterDataPipe is used.",
+    )
+    parser.add_argument(
+        "--output_path",
+        type=str,
+        required=True,
+        help="Path to outputted multi-hot sparse dataset.",
+    )
+    parser.add_argument(
+        "--copy_labels_and_dense",
+        dest="copy_labels_and_dense",
+        action="store_true",
+        help="Flag to determine whether to copy labels and dense data to the output directory.",
+    )
+    parser.add_argument(
+        "--num_embeddings_per_feature",
+        type=str,
+        required=True,
+        help="Comma separated max_ind_size per sparse feature. The number of embeddings"
+        " in each embedding table. 26 values are expected for the Criteo dataset.",
+    )
+    parser.add_argument(
+        "--multi_hot_sizes",
+        type=str,
+        required=True,
+        help="Comma separated multihot size per sparse feature. 26 values are expected for the Criteo dataset.",
+    )
+    parser.add_argument(
+        "--multi_hot_distribution_type",
+        type=str,
+        choices=["uniform", "pareto"],
+        default="uniform",
+        help="Multi-hot distribution options.",
+    )
+    return parser.parse_args()
+
+
+def main() -> None:
+    """
+    This script generates and saves the MLPerf v2 multi-hot dataset (4 TB in size).
+    First, run process_Criteo_1TB_Click_Logs_dataset.sh.
+    Then, run this script as follows:
+
+        python materialize_synthetic_multihot_dataset.py \
+            --in_memory_binary_criteo_path $PREPROCESSED_CRITEO_1TB_CLICK_LOGS_DATASET_PATH \
+            --output_path $MATERIALIZED_DATASET_PATH \
+            --num_embeddings_per_feature 40000000,39060,17295,7424,20265,3,7122,1543,63,40000000,3067956,405282,10,2209,11938,155,4,976,14,40000000,40000000,40000000,590152,12973,108,36 \
+            --multi_hot_sizes 3,2,1,2,6,1,1,1,1,7,3,8,1,6,9,5,1,1,1,12,100,27,10,3,1,1 \
+            --multi_hot_distribution_type uniform
+
+    This script takes about 2 hours to run (can be parallelized if needed).
+    """
+    args = parse_args()
+    for name, val in vars(args).items():
+        try:
+            vars(args)[name] = list(map(int, val.split(",")))
+        except (ValueError, AttributeError):
+            pass
+    try:
+        backend = "nccl" if torch.cuda.is_available() else "gloo"
+        if not dist.is_initialized():
+            dist.init_process_group(backend=backend)
+        rank = dist.get_rank()
+        world_size = dist.get_world_size()
+    except (KeyError, ValueError):
+        rank = 0
+        world_size = 1
+
+    print("Generating one-hot to multi-hot lookup table.")
+    multihot = Multihot(
+        multi_hot_sizes=args.multi_hot_sizes,
+        num_embeddings_per_feature=args.num_embeddings_per_feature,
+        batch_size=1,  # Doesn't matter
+        collect_freqs_stats=False,
+        dist_type=args.multi_hot_distribution_type,
+    )
+
+    os.makedirs(args.output_path, exist_ok=True)
+
+    for i in range(rank, DAYS, world_size):
+        input_file_path = os.path.join(
+            args.in_memory_binary_criteo_path, f"day_{i}_sparse.npy"
+        )
+        print(f"Materializing {input_file_path}")
+        sparse_data = np.load(input_file_path, mmap_mode="r")
+        multi_hot_ids_dict = {}
+        for j, (multi_hot_table, hash) in enumerate(
+            zip(multihot.multi_hot_tables_l, args.num_embeddings_per_feature)
+        ):
+            sparse_tensor = torch.from_numpy(sparse_data[:, j] % hash)
+            multi_hot_ids_dict[str(j)] = nn.functional.embedding(
+                sparse_tensor, multi_hot_table
+            ).numpy()
+        output_file_path = os.path.join(
+            args.output_path, f"day_{i}_sparse_multi_hot.npz"
+        )
+        np.savez(output_file_path, **multi_hot_ids_dict)
+        if args.copy_labels_and_dense:
+            for part in ["labels", "dense"]:
+                source_path = os.path.join(
+                    args.in_memory_binary_criteo_path, f"day_{i}_{part}.npy"
+                )
+                output_path = os.path.join(args.output_path, f"day_{i}_{part}.npy")
+                shutil.copyfile(source_path, output_path)
+                print(f"Copying {source_path} to {output_path}")
+
+
+if __name__ == "__main__":
+    main()
diff --git a/models_v2/pytorch/torchrec_dlrm/inference/gpu/scripts/process_Criteo_1TB_Click_Logs_dataset.sh b/models_v2/pytorch/torchrec_dlrm/inference/gpu/scripts/process_Criteo_1TB_Click_Logs_dataset.sh
new file mode 100644
index 000000000..23050647d
--- /dev/null
+++ b/models_v2/pytorch/torchrec_dlrm/inference/gpu/scripts/process_Criteo_1TB_Click_Logs_dataset.sh
@@ -0,0 +1,64 @@
+#!/bin/bash
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+display_help() {
+   echo "Three command line arguments are required."
+   echo "Example usage:"
+   echo "bash process_Criteo_1TB_Click_Logs_dataset.sh \\"
+   echo "./criteo_1tb/raw_input_dataset_dir \\"
+   echo "./criteo_1tb/temp_intermediate_files_dir \\"
+   echo "./criteo_1tb/numpy_contiguous_shuffled_output_dataset_dir"
+   exit 1
+}
+
+[ -z "$1" ] && display_help
+[ -z "$2" ] && display_help
+[ -z "$3" ] && display_help
+
+# Input directory containing the raw Criteo 1TB Click Logs dataset files in tsv format.
+# The 24 dataset filenames in the directory should be day_{0..23} with no .tsv extension.
+raw_tsv_criteo_files_dir=$(readlink -m "$1")
+
+# Directory to store temporary intermediate output files created by preprocessing steps 1 and 2.
+temp_files_dir=$(readlink -m "$2")
+
+# Directory to store temporary intermediate output files created by preprocessing step 1.
+step_1_output_dir="$temp_files_dir/temp_output_of_step_1"
+
+# Directory to store temporary intermediate output files created by preprocessing step 2.
+step_2_output_dir="$temp_files_dir/temp_output_of_step_2"
+
+# Directory to store the final preprocessed Criteo 1TB Click Logs dataset.
+step_3_output_dir=$(readlink -m "$3")
+
+# Step 1. Split the dataset into 3 sets of 24 numpy files:
+# day_{0..23}_dense.npy, day_{0..23}_labels.npy, and day_{0..23}_sparse.npy (~24hrs)
+set -x
+mkdir -p "$step_1_output_dir"
+date
+python -m torchrec.datasets.scripts.npy_preproc_criteo --input_dir "$raw_tsv_criteo_files_dir" --output_dir "$step_1_output_dir" || exit
+
+# Step 2. Convert all sparse indices in day_{0..23}_sparse.npy to contiguous indices and save the output.
+# The output filenames are day_{0..23}_sparse_contig_freq.npy
+mkdir -p "$step_2_output_dir"
+date
+python -m torchrec.datasets.scripts.contiguous_preproc_criteo --input_dir "$step_1_output_dir" --output_dir "$step_2_output_dir" --frequency_threshold 0 || exit
+
+date
+for i in {0..23}
+do
+   name="$step_2_output_dir/day_$i""_sparse_contig_freq.npy"
+   renamed="$step_2_output_dir/day_$i""_sparse.npy"
+   echo "Renaming $name to $renamed"
+   mv "$name" "$renamed"
+done
+
+# Step 3. Shuffle the dataset's samples in days 0 through 22. (~20hrs)
+# Day 23's samples are not shuffled and will be used for the validation set and test set.
+mkdir -p "$step_3_output_dir"
+date
+python -m torchrec.datasets.scripts.shuffle_preproc_criteo --input_dir_labels_and_dense "$step_1_output_dir" --input_dir_sparse "$step_2_output_dir" --output_dir_shuffled "$step_3_output_dir" --random_seed 0 || exit
+date
diff --git a/models_v2/pytorch/torchrec_dlrm/inference/gpu/setup.sh b/models_v2/pytorch/torchrec_dlrm/inference/gpu/setup.sh
new file mode 100755
index 000000000..475cb8af3
--- /dev/null
+++ b/models_v2/pytorch/torchrec_dlrm/inference/gpu/setup.sh
@@ -0,0 +1,27 @@
+#!/usr/bin/env bash
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# setup.sh
+#  - install OS pkgs
+#  - should create virtual env & install pip  requirement.txt
+#  - git clones & applying patches
+
+set -e
+apt-get update && apt-get install -y python3-venv protobuf-compiler
+python3 -m venv $PWD/venv
+. ./venv/bin/activate
+pip install -r requirements.txt
diff --git a/models_v2/pytorch/torchrec_dlrm/inference/gpu/sharding/__init__.py b/models_v2/pytorch/torchrec_dlrm/inference/gpu/sharding/__init__.py
new file mode 100644
index 000000000..17c44d7e2
--- /dev/null
+++ b/models_v2/pytorch/torchrec_dlrm/inference/gpu/sharding/__init__.py
@@ -0,0 +1,24 @@
+#MIT License
+#
+#Copyright (c) Facebook, Inc. and its affiliates.
+#
+#Permission is hereby granted, free of charge, to any person obtaining a copy
+#of this software and associated documentation files (the "Software"), to deal
+#in the Software without restriction, including without limitation the rights
+#to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+#copies of the Software, and to permit persons to whom the Software is
+#furnished to do so, subject to the following conditions:
+#
+#The above copyright notice and this permission notice shall be included in all
+#copies or substantial portions of the Software.
+#
+#THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+#IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+#FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+#AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+#LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+#OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+#SOFTWARE.
+
+from .generate_plan import generate_plan
+from .planner import Cost, CostModel, Planner, ShardingState
diff --git a/models_v2/pytorch/torchrec_dlrm/inference/gpu/sharding/generate_plan.py b/models_v2/pytorch/torchrec_dlrm/inference/gpu/sharding/generate_plan.py
new file mode 100644
index 000000000..ad8b4016e
--- /dev/null
+++ b/models_v2/pytorch/torchrec_dlrm/inference/gpu/sharding/generate_plan.py
@@ -0,0 +1,152 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import logging
+from argparse import Namespace
+from itertools import chain
+from typing import List
+
+from .planner import CostModel, Planner
+
+
+def generate_plan(
+    slot_size_array: List[int],
+    multi_hot_sizes: List[int],
+    num_nodes: int,
+    num_gpus: int,
+    args: Namespace,
+    log_result: bool,
+):
+    def sanity_check(shard_matrix, shard_strategy):
+        # mainly to make sure all the tables are sharded
+        msg = "Not all tables covered in the sharding plan"
+        assert set(chain(*shard_matrix)) == set(range(len(slot_size_array))), msg
+        shard_strategy_list = [x for strategy_pair in shard_strategy for x in strategy_pair[1]]
+        assert set(shard_strategy_list) == set(range(len(slot_size_array))), msg
+
+        for table_list in shard_matrix:
+            if len(table_list) == 0:
+                raise Exception("Currently no empty shard list is allowed")
+
+    def int_to_string(shard_matrix_int, shard_strategy_int):
+        shard_strategy, shard_matrix = [], []
+        for pair in shard_strategy_int:
+            if len(pair[1]) != 0:
+                shard_strategy.append((pair[0], [str(x) for x in pair[1]]))
+        for sub_matrix_ in shard_matrix_int:
+            shard_matrix.append([str(x) for x in sub_matrix_])
+        return shard_matrix, shard_strategy
+
+    if args.sharding_plan in ["round_robin", "uniform"]:
+        # sharding strategies that don't exploit system configs
+        if args.sharding_plan == "round_robin":
+            mp_table = [i for i in range(len(slot_size_array))]
+            shard_matrix_ = [[] for _ in range(num_gpus)]
+            shard_strategy_ = [("mp", [i for i in mp_table])]
+
+            for i, table_id in enumerate(mp_table):
+                target_gpu = i % num_gpus
+                shard_matrix_[target_gpu].append(table_id)
+
+        elif args.sharding_plan == "uniform":
+            shard_matrix_ = [[x for x in range(len(slot_size_array))] for _ in range(num_gpus)]
+            shard_strategy_ = [("mp", [i for i in range(len(slot_size_array))])]
+
+    elif args.sharding_plan in ["auto", "hier_auto"]:
+        # sharding strategies that exploit system configs
+        dram_cap = args.memory_cap_for_embedding
+        if args.optimizer == "adagrad":
+            byte_per_elem = 8
+        elif args.optimizer == "sgd":
+            byte_per_elem = 4
+
+        if args.sharding_plan == "auto":
+            cost_model = CostModel(
+                1,
+                args.mem_comm_bw_ratio / args.mem_comm_work_ratio,
+                args.ev_size * byte_per_elem * 1e-9,
+                dram_cap,
+                slot_size_array,
+            )
+            planner = Planner(multi_hot_sizes, num_gpus, cost_model, log_result=log_result)
+            shard_strategy_, shard_matrix_ = planner.plan()
+
+        elif args.sharding_plan == "hier_auto":
+            if num_nodes <= 1:
+                raise Exception("hier_auto plan is only applicable to configs with more than one node")
+            cost_model = CostModel(
+                1,
+                args.mem_comm_bw_ratio / args.mem_comm_work_ratio,
+                args.ev_size * byte_per_elem * 1e-9,
+                dram_cap * args.num_gpus_per_node,
+                slot_size_array,
+            )
+            planner = Planner(multi_hot_sizes, num_nodes, cost_model, log_result=log_result)
+            shard_strategy_, shard_matrix_node_ = planner.plan()
+            shard_matrix_ = []
+            for node_shard_matrix in shard_matrix_node_:
+                for i in range(args.num_gpus_per_node):
+                    shard_matrix_.append(node_shard_matrix)
+    elif args.sharding_plan in ["max_min"]:
+        mp_table = [i for i in range(len(slot_size_array))]
+        shard_matrix_ = [[] for _ in range(num_gpus)]
+        shard_strategy_ = [("mp", [i for i in mp_table])]
+        dict_feature = {}
+        for k, v in enumerate(slot_size_array):
+            dict_feature[k] = v
+        dict_feature_list = sorted(dict_feature.items(),key=lambda x:x[1], reverse=True)
+        end = len(dict_feature_list) - 1
+        for i in range(len(dict_feature_list)):
+            index = i%num_gpus if int(i/num_gpus) == 0 else (num_gpus -1 - i%num_gpus)
+            shard_matrix_[index].append(dict_feature_list[i][0])
+            if (i >= end):
+                break
+            elif(end - i >= num_gpus - i%num_gpus):
+                shard_matrix_[index].append(dict_feature_list[end][0])
+                end -= 1
+    elif args.sharding_plan in ["custom"]:
+        mp_table = [i for i in range(len(slot_size_array))]
+        shard_matrix_ = [[0, 5], [9, 16], [19, 12], [20, 18], [21, 25], [10, 8, 2, 13, 3], [22, 24, 4, 7, 23], [11, 15, 1, 17, 14, 6]]
+        shard_strategy_ = [("mp", [i for i in mp_table])]
+    else:
+        raise Exception("unknown sharding plan")
+
+    sanity_check(shard_matrix_, shard_strategy_)
+    shard_matrix, shard_strategy = int_to_string(shard_matrix_, shard_strategy_)
+
+    if log_result:
+        logging.info("Provided system info: ")
+        #logging.info("num_gpu_per_nodes: %d", args.num_gpus_per_node)
+        #logging.info("Memory to communication BW ratio: %f", args.mem_comm_bw_ratio)
+        #logging.info("Memory to communication work ratio: %f", args.mem_comm_work_ratio)
+        #logging.info("DRAM capacity: %f GB", args.memory_cap_for_embedding)
+        logging.info("shard_matrix:")
+        logging.info(shard_matrix)
+        logging.info("\n")
+
+    return shard_matrix, shard_strategy, shard_matrix_, shard_strategy_
diff --git a/models_v2/pytorch/torchrec_dlrm/inference/gpu/sharding/planner.py b/models_v2/pytorch/torchrec_dlrm/inference/gpu/sharding/planner.py
new file mode 100644
index 000000000..4fa5c6c37
--- /dev/null
+++ b/models_v2/pytorch/torchrec_dlrm/inference/gpu/sharding/planner.py
@@ -0,0 +1,313 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import logging
+import time
+from typing import List, Tuple
+
+import numpy as np
+
+
+class ShardingState:
+    """
+    Containing the state of a sharding process.
+    The plan iteratively update the sharding state based on a given heuristic and obtain
+    solutions.
+    """
+
+    def __init__(
+        self,
+        array_hotness: np.array,
+        num_bucket: int,
+        dp_table_id: np.array(int) = np.array([]),
+    ) -> None:
+        mp_table_id = np.setdiff1d(np.arange(array_hotness.size), dp_table_id)
+        array_hotness_mp = array_hotness[mp_table_id]
+        sorted_idx = np.argsort(array_hotness_mp)[::-1]
+        self.array_unshard_hotness = array_hotness
+        self.array_hotness = array_hotness_mp[sorted_idx]
+        self.num_bucket = num_bucket
+        self.array_table_id = mp_table_id[sorted_idx]
+        self.array_num_split = np.zeros(self.array_unshard_hotness.size, dtype=int)
+        self.array_num_split[mp_table_id] = 1
+        self.shard_ll = [[] for i in range(self.num_bucket)]
+
+    def split_hot_shard(self):
+        """
+        split the shard with the largest hotness
+        """
+        # shards are sorted based on the hotness. Find the first hot shard that
+        # can be split further
+        for shard_id in range(self.array_table_id.size):
+            table_id = self.array_table_id[shard_id]
+            hotness = self.array_unshard_hotness[table_id]
+            if self.array_num_split[table_id] * 2 <= self.num_bucket:
+                # if this table can be furter splitted and we can put it into
+                # more buckets
+                idx = np.where(self.array_table_id == table_id)[0]
+                self.array_hotness = np.delete(self.array_hotness, idx)
+                self.array_table_id = np.delete(self.array_table_id, idx)
+                self.array_num_split[table_id] *= 2
+                self.array_hotness = np.concatenate(
+                    (
+                        self.array_hotness,
+                        np.ones(self.array_num_split[table_id]) * (hotness / self.array_num_split[table_id]),
+                    )
+                )
+                self.array_table_id = np.concatenate(
+                    (self.array_table_id, np.ones(self.array_num_split[table_id], dtype=int) * table_id)
+                )
+                break
+
+        # sort after splitting to maintain the shard hotness in order
+        sorted_idx = np.argsort(self.array_hotness)[::-1]
+        self.array_hotness = self.array_hotness[sorted_idx]
+        self.array_table_id = self.array_table_id[sorted_idx]
+
+    def split_oom_shard(self, table_id: int) -> bool:
+        hotness = self.array_unshard_hotness[table_id]
+        if self.array_num_split[table_id] * 2 <= self.num_bucket:
+            idx = np.where(self.array_table_id == table_id)[0]
+            self.array_hotness = np.delete(self.array_hotness, idx)
+            self.array_table_id = np.delete(self.array_table_id, idx)
+            self.array_num_split[table_id] *= 2
+            self.array_hotness = np.concatenate(
+                (
+                    self.array_hotness,
+                    np.ones(self.array_num_split[table_id]) * (hotness / self.array_num_split[table_id]),
+                )
+            )
+            self.array_table_id = np.concatenate(
+                (self.array_table_id, np.ones(self.array_num_split[table_id], dtype=int) * table_id)
+            )
+            sorted_idx = np.argsort(self.array_hotness)[::-1]
+            self.array_hotness = self.array_hotness[sorted_idx]
+            self.array_table_id = self.array_table_id[sorted_idx]
+            return True
+        else:
+            return False
+
+    def update_split_num(self):
+        self.array_num_split = np.zeros_like(self.array_unshard_hotness)
+        for shard_list in self.shard_ll:
+            for table_id in shard_list:
+                self.array_num_split[table_id] += 1
+
+    def reset_shard_ll(self):
+        self.shard_ll = [[] for i in range(self.num_bucket)]
+
+    def push_bucket(
+        self,
+        bucket_id: int,
+        table_id: int,
+    ) -> None:
+        self.shard_ll[bucket_id].append(table_id)
+
+    def pop_bucket(
+        self,
+        bucket_id: int,
+    ) -> None:
+        self.shard_ll[bucket_id].pop()
+
+
+class Cost:
+    def __init__(
+        self,
+        cost: np.array(float),
+        hotness_cost: np.array(float),
+        table_cost: np.array(float),
+        mem_cost: np.array(float),
+    ) -> None:
+        self.cost = cost
+        self.hotness_cost = hotness_cost
+        self.table_cost = table_cost
+        self.mem_cost = mem_cost
+
+
+class CostModel:
+    def __init__(
+        self,
+        hotness_cost: float,
+        table_cost: float,
+        mem_cost: float,
+        mem_capacity: float,
+        table_size: List[int],
+    ) -> None:
+        self.unit_hotness_cost = hotness_cost
+        self.unit_table_cost = table_cost
+        self.unit_mem_cost = mem_cost
+        self.mem_capacity = mem_capacity
+        self.array_table_size = np.array(table_size)
+
+    def get_cost(
+        self,
+        ss: ShardingState,
+    ) -> Tuple[Cost, bool]:
+        list_cost = []
+        list_hotness_cost = []
+        list_table_cost = []
+        list_mem_cost = []
+
+        for shard_list in ss.shard_ll:
+            hotness_cost = (
+                self.unit_hotness_cost
+                * (ss.array_unshard_hotness[shard_list] / np.array(ss.array_num_split)[shard_list]).sum()
+            )
+            table_cost = self.unit_table_cost * len(shard_list)
+            mem_cost = (
+                self.unit_mem_cost
+                * (self.array_table_size[shard_list] / np.array(ss.array_num_split)[shard_list]).sum()
+            )
+            list_cost.append(hotness_cost + table_cost)
+            list_hotness_cost.append(hotness_cost)
+            list_table_cost.append(table_cost)
+            list_mem_cost.append(mem_cost)
+
+        return (
+            Cost(np.array(list_cost), np.array(list_hotness_cost), np.array(list_table_cost), np.array(list_mem_cost)),
+            max(list_mem_cost) > self.mem_capacity,
+        )
+
+    def deduct_mem_cap_for_dp(
+        self,
+        dp_table_id: list,
+    ) -> None:
+        self.mem_capacity -= self.array_table_size[dp_table_id].sum() * self.unit_mem_cost
+        if self.mem_capacity < 0:
+            raise Exception("OOM due to DP. Please considering increase the DP threshold")
+
+
+class Planner:
+    """
+    The planner work out a series of plans iteratively.
+    In each iteration, the planner tries to split the hottest shard and place the shards into
+    a bucket based on a give heuristic. When the shard is too large to fit into the best bucket
+    suggested by the heuristic, it finds the next best bucket until it iterates through all the
+    buckets. In that case, it tries to split the shard further. If the shard cannot be split
+    further, the planner aborts and returns the default sharding plan.
+    """
+
+    def __init__(
+        self,
+        list_hotness: list,
+        num_bucket: int,
+        cost_model: CostModel,
+        dp_threshold: int = 0,
+        max_search_iter: int = 20,
+        log_result: bool = False,
+    ) -> None:
+        self.array_hotness = np.array(list_hotness)
+        self.num_bucket = num_bucket
+        self.cost_model = cost_model
+        self.list_candidate = []
+        self.max_search_iter = max_search_iter
+        self.log_result = log_result
+
+        # Create the default sharding plan. Throw if even this default sharding plan cannot fit, as
+        # it should be the most memory-efficient
+        sharding_state_default = ShardingState(self.array_hotness, self.num_bucket)
+        for b in range(self.num_bucket):
+            for t in range(self.array_hotness.size):
+                sharding_state_default.push_bucket(b, t)
+        sharding_state_default.update_split_num()
+        cost, oom = self.cost_model.get_cost(sharding_state_default)
+        if oom:
+            raise Exception("OOM even with the most memory-efficient sharding plan")
+        self.list_candidate.append(
+            (cost.cost.max(), cost.hotness_cost, cost.table_cost, cost.mem_cost, sharding_state_default.shard_ll)
+        )
+
+        # Create DP sharding plan based on the DP threshold
+        self.dp_table_id = np.where(cost_model.array_table_size < dp_threshold / cost_model.unit_mem_cost)[0]
+        self.mp_table_id = np.setdiff1d(np.arange(self.array_hotness.size), self.dp_table_id)
+        self.sharding_state = ShardingState(self.array_hotness, self.num_bucket, self.dp_table_id)
+        self.cost_model.deduct_mem_cap_for_dp(self.dp_table_id)
+
+        logging.basicConfig(level=logging.INFO, format="%(message)s")
+
+    def greedy_plan(self, ss):
+        """
+        This is a heuristic based on greedy policy. The shard is placed to the bucket with the
+        lowest hotness cost
+        """
+        array_cost = np.zeros(ss.num_bucket)
+        ss.reset_shard_ll()
+        for i in range(ss.array_hotness.size):
+            sorted_idx = np.argsort(array_cost)
+            sharded = False
+            for bucket_id in sorted_idx:
+                if ss.array_table_id[i] not in ss.shard_ll[bucket_id]:
+                    # for now, only uniform sharding is supported. Hence cannot put two shards
+                    # from the same table into the same bucket
+                    ss.push_bucket(bucket_id, ss.array_table_id[i])
+                    cost, oom = self.cost_model.get_cost(ss)
+                    if not oom:
+                        sharded = True
+                        array_cost = cost.cost
+                        break
+                    else:
+                        # Currnet bucket cannot fit. Iterate to the next best bucket
+                        ss.pop_bucket(bucket_id)
+            if not sharded:
+                # This means the shard is too large to fit within any bucket
+                return ss.array_table_id[i], ss, cost
+        return None, ss, cost
+
+    def plan(self):
+        t0 = time.time()
+        for i in range(self.max_search_iter):
+            oom_table_id, self.sharding_state, cost = self.greedy_plan(self.sharding_state)
+            if oom_table_id is None:
+                self.list_candidate.append(
+                    (cost.cost.max(), cost.hotness_cost, cost.table_cost, cost.mem_cost, self.sharding_state.shard_ll)
+                )
+                self.sharding_state.split_hot_shard()
+            else:
+                oom_table_can_split = self.sharding_state.split_oom_shard(oom_table_id)
+                if not oom_table_can_split:
+                    break
+
+        self.list_candidate.sort(key=lambda x: x[0])
+
+        shard_strategy = [("mp", self.mp_table_id.tolist())]
+        shard_strategy.append(("dp", self.dp_table_id.tolist()))
+        shard_matrix = self.list_candidate[0][-1]
+        for table_id in self.dp_table_id:
+            for shard_list in shard_matrix:
+                shard_list.append(table_id)
+        if self.log_result:
+            logging.info("Planner took %f sec" % (time.time() - t0))
+            logging.info(shard_strategy)
+            logging.info(shard_matrix)
+            logging.info("hotness cost is:")
+            logging.info(self.list_candidate[0][1])
+            logging.info("table cost is:")
+            logging.info(self.list_candidate[0][2])
+            logging.info("mem cost is:")
+            logging.info(self.list_candidate[0][3])
+        return shard_strategy, shard_matrix
diff --git a/models_v2/pytorch/torchrec_dlrm/training/gpu/README.md b/models_v2/pytorch/torchrec_dlrm/training/gpu/README.md
new file mode 100644
index 000000000..900e8b393
--- /dev/null
+++ b/models_v2/pytorch/torchrec_dlrm/training/gpu/README.md
@@ -0,0 +1,103 @@
+# DLRM v2 Training
+
+DLRM v2 Training best known configurations with Intel® Extension for PyTorch.
+
+## Model Information
+
+| **Use Case** | **Framework** | **Model Repo** | **Branch/Commit/Tag** | **Optional Patch** |
+|:---:| :---: |:--------------:|:---------------------:|:------------------:|
+|  Training   |    PyTorch    |       https://github.com/facebookresearch/dlrm/tree/main/torchrec_dlrm        |           -           |         -          |
+
+# Pre-Requisite
+* Host has 4 Intel® Data Center GPU Max and have two Tiles for each
+* Host has installed latest Intel® Data Center GPU Max Series Drivers https://dgpu-docs.intel.com/driver/installation.html
+* Host has installed [Intel® Extension for PyTorch](https://intel.github.io/intel-extension-for-pytorch/xpu/latest/)
+
+# prepare Dataset
+After downloading and uncompressing the [Criteo 1TB Click Logs dataset](consisting of 24 files from day 0 to day 23), process the raw tsv files into the proper format for training by running ./scripts/process_Criteo_1TB_Click_Logs_dataset.sh with necessary command line arguments.
+
+Example usage:
+
+bash ./scripts/process_Criteo_1TB_Click_Logs_dataset.sh \
+./criteo_1tb/raw_input_dataset_dir \
+./criteo_1tb/temp_intermediate_files_dir \
+./criteo_1tb/numpy_contiguous_shuffled_output_dataset_dir
+The script requires 700GB of RAM and takes 1-2 days to run. We currently have features in development to reduce the preproccessing time and memory overhead. MD5 checksums of the expected final preprocessed dataset files are in md5sums_preprocessed_criteo_click_logs_dataset.txt.
+
+the final dataset dir will be like below:
+dataset_dir
+ |_day_0_dense.npy
+ |_day_0_labels.npy
+ |_day_0_sparse_multi_hot.npz
+
+this folder will be used as the parameter DATASET_DIR later
+
+
+## Training
+1. `git clone https://github.com/IntelAI/models.git`
+2. `cd models/models_v2/pytorch/torchrec_dlrm/training/gpu`
+3. Run `setup.sh` this will install all the required dependencies & create virtual environment `venv`.
+4. Activate virtual env: `. ./venv/bin/activate`
+5. Setup required environment paramaters
+
+| **Parameter**                |                                  **export command**                                  |
+|:---------------------------:|:------------------------------------------------------------------------------------:|
+| **MULTI_TILE**               | `export MULTI_TILE=True` (True or False)                                             |
+| **PLATFORM**                 | `export PLATFORM=PVC` (PVC)                                                 |
+| **DATASET_DIR**              |                               `export DATASET_DIR=`                                  |
+| **BATCH_SIZE** (optional)    |                               `export BATCH_SIZE=65536`                                |
+| **PRECISION** (optional)     |                               `export PRECISION=BF16` (BF16, FP32 and TF32 are supported for PVC)                                |
+| **OUTPUT_DIR** (optional)    |                               `export OUTPUT_DIR=$PWD`                               |
+6. Run `run_model.sh`
+
+## Output
+
+
+Multi-tile output will typically looks like:
+```
+[7] 2024-01-10 22:22:36,284 - __main__ - INFO - avg training time per iter at ITER: 45, 0.07149526278177896 s
+[7] 2024-01-10 22:22:36,284 - __main__ - INFO - Total number of iterations: 50
+[2] 2024-01-10 22:22:36,292 - __main__ - INFO - avg training time per iter at ITER: 45, 0.07994737095303006 s
+[2] 2024-01-10 22:22:36,292 - __main__ - INFO - Total number of iterations: 50
+[0] 2024-01-10 22:22:36,293 - __main__ - INFO - avg training time per iter at ITER: 45, 0.08237933582729763 s
+[0] 2024-01-10 22:22:36,294 - __main__ - INFO - Total number of iterations: 50
+[1] 2024-01-10 22:22:36,296 - __main__ - INFO - avg training time per iter at ITER: 45, 0.08394240803188747 s
+[1] 2024-01-10 22:22:36,296 - __main__ - INFO - Total number of iterations: 50
+[6] 2024-01-10 22:22:36,304 - __main__ - INFO - avg training time per iter at ITER: 45, 0.09519488016764323 s
+[6] 2024-01-10 22:22:36,304 - __main__ - INFO - Total number of iterations: 50
+[5] 2024-01-10 22:22:36,306 - __main__ - INFO - avg training time per iter at ITER: 45, 0.09369233979119194 s
+[5] 2024-01-10 22:22:36,306 - __main__ - INFO - Total number of iterations: 50
+[3] 2024-01-10 22:22:36,309 - __main__ - INFO - avg training time per iter at ITER: 45, 0.09690533743964301 s
+[3] 2024-01-10 22:22:36,309 - __main__ - INFO - Total number of iterations: 50
+[4] 2024-01-10 22:22:36,339 - __main__ - INFO - avg training time per iter at ITER: 45, 0.11025158564249675 s
+[4] 2024-01-10 22:22:36,339 - __main__ - INFO - Total number of iterations: 50
+[0] 2024:01:10-22:22:37:(38583) |CCL_INFO| finalize atl-mpi
+[0] 2024:01:10-22:22:37:(38583) |CCL_INFO| finalized atl-mpi
+[3] 2024:01:10-22:22:37:(38586) |CCL_INFO| finalizing level-zero
+[4] 2024:01:10-22:22:37:(38587) |CCL_INFO| finalizing level-zero
+[3] 2024:01:10-22:22:37:(38586) |CCL_INFO| finalized level-zero
+[2] 2024:01:10-22:22:37:(38585) |CCL_INFO| finalizing level-zero
+[6] 2024:01:10-22:22:37:(38589) |CCL_INFO| finalizing level-zero
+[0] 2024:01:10-22:22:37:(38583) |CCL_INFO| finalizing level-zero
+[7] 2024:01:10-22:22:37:(38590) |CCL_INFO| finalizing level-zero
+[4] 2024:01:10-22:22:37:(38587) |CCL_INFO| finalized level-zero
+[5] 2024:01:10-22:22:37:(38588) |CCL_INFO| finalizing level-zero
+[2] 2024:01:10-22:22:37:(38585) |CCL_INFO| finalized level-zero
+[6] 2024:01:10-22:22:37:(38589) |CCL_INFO| finalized level-zero
+[0] 2024:01:10-22:22:37:(38583) |CCL_INFO| finalized level-zero
+[1] 2024:01:10-22:22:37:(38584) |CCL_INFO| finalizing level-zero
+[7] 2024:01:10-22:22:37:(38590) |CCL_INFO| finalized level-zero
+[5] 2024:01:10-22:22:37:(38588) |CCL_INFO| finalized level-zero
+[1] 2024:01:10-22:22:37:(38584) |CCL_INFO| finalized level-zero
+```
+
+Final results of the inference run can be found in `results.yaml` file.
+```
+results:
+ - key: throughput
+   value: 594422.29
+   unit: samples/s
+ - key: accuracy
+   value: None
+   unit: AUROC
+```
diff --git a/models_v2/pytorch/torchrec_dlrm/training/gpu/cmd_distributed_terabyte_train.sh b/models_v2/pytorch/torchrec_dlrm/training/gpu/cmd_distributed_terabyte_train.sh
new file mode 100644
index 000000000..06573d9a5
--- /dev/null
+++ b/models_v2/pytorch/torchrec_dlrm/training/gpu/cmd_distributed_terabyte_train.sh
@@ -0,0 +1,121 @@
+#!/bin/bash
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+set -x
+
+function Parser() {
+    while [ $# -ne 0 ]; do
+        case $1 in
+            -b)
+                shift
+                export GLOBAL_BATCH_SIZE="$1"
+                ;;
+            -bf16)
+                shift
+                BF16="$1"
+                ;;
+            -d)
+                shift
+                DATA="$1"
+                ;;
+            -nd)
+                shift
+                ND="$1"
+                ;;
+            -sp)
+                shift
+                SP="$1"
+                ;;
+            -tf32)
+                shift
+                TF32="$1"
+                ;;
+            -tv)
+                shift
+                TV="$1"
+                ;;
+            -h | --help)
+                echo "Usage: cmd_infer.sh [OPTION...] PAGE..."
+                echo "-b, Optional    Specify the batch size. The default value is 32768"
+                echo "-bf16, Optional    Specify the input dtype is bf16. The default value is true"
+                echo "-d, Optional    Specify the data file"
+                echo "-nd, Optional    Specify the number of node"
+                echo "-sp, Optional    Specify the sharding plan of embedding"
+                echo "-tf32, Optional    Specify the input dtype is tf32. The default value is false"
+                echo "-tv, Optional    Training with val. The default value is false"
+                exit
+                ;;
+            --*|-*)
+                echo ">>> New param: <$1>"
+                ;;
+            *)
+                echo ">>> Parsing mismatch: $1"
+                ;;
+        esac
+        shift
+    done
+}
+
+torch_ccl_path=$(python -c "import torch; import oneccl_bindings_for_pytorch; import os;  print(os.path.abspath(os.path.dirname(oneccl_bindings_for_pytorch.__file__)))")
+source $torch_ccl_path/env/setvars.sh
+export MASTER_ADDR='127.0.0.1'
+#export WORLD_SIZE=2 ;
+export TOTAL_TRAINING_SAMPLES=4195197692;
+export GLOBAL_BATCH_SIZE=65536;
+ND=1
+SP="round_robin"
+#export CCL_LOG_LEVEL=DEBUG;
+#export CCL_OP_SYNC=1
+
+
+${BF16:=true}
+${TF32:=false}
+${TV:=false}
+Parser $@
+ARGS+=" --embedding_dim 128"
+ARGS+=" --dense_arch_layer_sizes 512,256,128"
+ARGS+=" --over_arch_layer_sizes 1024,1024,512,256,1"
+ARGS+=" --num_embeddings_per_feature 40000000,39060,17295,7424,20265,3,7122,1543,63,40000000,3067956,405282,10,2209,11938,155,4,976,14,40000000,40000000,40000000,590152,12973,108,36"
+ARGS+=" --validation_freq_within_epoch $((TOTAL_TRAINING_SAMPLES / (GLOBAL_BATCH_SIZE * 20 * 1000)))"
+ARGS+=" --synthetic_multi_hot_criteo_path $DATA"
+ARGS+=" --multi_hot_sizes 3,2,1,2,6,1,1,1,1,7,3,8,1,6,9,5,1,1,1,12,100,27,10,3,1,1"
+#ARGS+=" --multi_hot_distribution_type uniform"
+ARGS+=" --use_xpu"
+ARGS+=" --epochs 1"
+ARGS+=" --pin_memory"
+ARGS+=" --mmap_mode"
+ARGS+=" --batch_size $GLOBAL_BATCH_SIZE"
+ARGS+=" --interaction_type=dcn"
+ARGS+=" --dcn_num_layers=3"
+ARGS+=" --adagrad"
+ARGS+=" --dcn_low_rank_dim=512"
+ARGS+=" --numpy_rand_seed=12345"
+ARGS+=" --log_freq 10"
+ARGS+=" --amp"
+ARGS+=" --limit_val_batches 50"
+ARGS+=" --limit_train_batches 50"
+ARGS+=" --limit_test_batches 50"
+ARGS+=" --sharding_plan ${SP}"
+ARGS+=" --num_nodes ${ND}"
+ARGS+=" --learning_rate 0.005"
+[ "$TV" = true ]             && ARGS+=" --train_with_val"
+if [ "$TF32" = false ]; then
+        [ "$BF16" = true ]             && ARGS+=" --bf16"
+        echo "${ARGS}"
+	mpirun -np 8 -ppn 8 --prepend-rank python -u dlrm_main.py  ${ARGS}
+else
+        echo "${ARGS}"
+	IPEX_FP32_MATH_MODE=1 mpirun -np 8 -ppn 8 --prepend-rank python -u dlrm_main.py ${ARGS}
+fi
diff --git a/models_v2/pytorch/torchrec_dlrm/training/gpu/data/dlrm_dataloader.py b/models_v2/pytorch/torchrec_dlrm/training/gpu/data/dlrm_dataloader.py
new file mode 100644
index 000000000..969fd1f02
--- /dev/null
+++ b/models_v2/pytorch/torchrec_dlrm/training/gpu/data/dlrm_dataloader.py
@@ -0,0 +1,198 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#!/usr/bin/env python3
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+import argparse
+import os
+from typing import List
+
+from torch import distributed as dist
+from torch.utils.data import DataLoader
+from torchrec.datasets.criteo import (
+    CAT_FEATURE_COUNT,
+    DAYS,
+    DEFAULT_CAT_NAMES,
+    DEFAULT_INT_NAMES,
+    InMemoryBinaryCriteoIterDataPipe,
+)
+# This is for crop dataset
+DAYS_MIN=1
+from torchrec.datasets.random import RandomRecDataset
+
+# OSS import
+try:
+    # pyre-ignore[21]
+    # @manual=//ai_codesign/benchmarks/dlrm/torchrec_dlrm/data:multi_hot_criteo
+    from data.multi_hot_criteo import MultiHotCriteoIterDataPipe
+
+except ImportError:
+    pass
+
+# internal import
+try:
+    from .multi_hot_criteo import MultiHotCriteoIterDataPipe  # noqa F811
+except ImportError:
+    pass
+
+STAGES = ["train", "val", "test"]
+
+
+def _get_random_dataloader(
+    args: argparse.Namespace,
+    stage: str,
+) -> DataLoader:
+    attr = f"limit_{stage}_batches"
+    num_batches = getattr(args, attr)
+    if stage in ["val", "test"] and args.test_batch_size is not None:
+        batch_size = args.test_batch_size
+    else:
+        batch_size = args.batch_size
+    return DataLoader(
+        RandomRecDataset(
+            keys=DEFAULT_CAT_NAMES,
+            batch_size=batch_size,
+            hash_size=args.num_embeddings,
+            hash_sizes=args.num_embeddings_per_feature
+            if hasattr(args, "num_embeddings_per_feature")
+            else None,
+            manual_seed=args.seed if hasattr(args, "seed") else None,
+            ids_per_feature=1,
+            num_dense=len(DEFAULT_INT_NAMES),
+            num_batches=num_batches,
+        ),
+        batch_size=None,
+        batch_sampler=None,
+        pin_memory=args.pin_memory,
+        num_workers=0,
+    )
+
+
+def _get_in_memory_dataloader(
+    args: argparse.Namespace,
+    stage: str,
+) -> DataLoader:
+    if args.in_memory_binary_criteo_path is not None:
+        dir_path = args.in_memory_binary_criteo_path
+        sparse_part = "sparse.npy"
+        datapipe = InMemoryBinaryCriteoIterDataPipe
+    else:
+        dir_path = args.synthetic_multi_hot_criteo_path
+        sparse_part = "sparse_multi_hot.npz"
+        datapipe = MultiHotCriteoIterDataPipe
+
+    if args.dataset_name == "criteo_kaggle":
+        # criteo_kaggle has no validation set, so use 2nd half of training set for now.
+        # Setting stage to "test" will get the 2nd half of the dataset.
+        # Setting root_name to "train" reads from the training set file.
+        (root_name, stage) = ("train", "test") if stage == "val" else stage
+        stage_files: List[List[str]] = [
+            [os.path.join(dir_path, f"{root_name}_dense.npy")],
+            [os.path.join(dir_path, f"{root_name}_{sparse_part}")],
+            [os.path.join(dir_path, f"{root_name}_labels.npy")],
+        ]
+    # criteo_1tb code path uses below two conditionals
+    elif stage == "train":
+        if args.converge:
+            stage_files: List[List[str]] = [
+                [os.path.join(dir_path, f"day_{i}_dense.npy") for i in range(DAYS - 1)],
+                [os.path.join(dir_path, "multihot", f"day_{i}_{sparse_part}") for i in range(DAYS - 1)],
+                [os.path.join(dir_path, f"day_{i}_labels.npy") for i in range(DAYS - 1)],
+            ]
+        else:
+            stage_files: List[List[str]] = [
+                # for crop dataset
+                [os.path.join(dir_path, f"day_{i}_dense.npy") for i in range(DAYS_MIN)],
+                [os.path.join(dir_path, f"day_{i}_{sparse_part}") for i in range(DAYS_MIN)],
+                [os.path.join(dir_path, f"day_{i}_labels.npy") for i in range(DAYS_MIN)],
+            ]
+    elif stage in ["val", "test"]:
+        if args.converge:
+            stage_files: List[List[str]] = [
+                [os.path.join(dir_path, f"day_{DAYS-1}_dense.npy")],
+                [os.path.join(dir_path, "multihot", f"day_{DAYS-1}_{sparse_part}")],
+                [os.path.join(dir_path, f"day_{DAYS-1}_labels.npy")],
+            ]
+        else:
+            stage_files: List[List[str]] = [
+                [os.path.join(dir_path, f"day_{DAYS_MIN-1}_dense.npy")],
+                [os.path.join(dir_path, f"day_{DAYS_MIN-1}_{sparse_part}")],
+                [os.path.join(dir_path, f"day_{DAYS_MIN-1}_labels.npy")],
+            ]
+    if stage in ["val", "test"] and args.test_batch_size is not None:
+        batch_size = args.test_batch_size
+    else:
+        batch_size = args.batch_size
+    dataloader = DataLoader(
+        datapipe(
+            stage,
+            *stage_files,  # pyre-ignore[6]
+            batch_size=batch_size,
+            #rank=dist.get_rank(),
+            #world_size=dist.get_world_size(),
+            # The rand and world_size set for custom dist-dlrm
+            rank=0,
+            world_size=1,
+            drop_last=args.drop_last_training_batch if stage == "train" else False,
+            shuffle_batches=args.shuffle_batches,
+            shuffle_training_set=args.shuffle_training_set,
+            shuffle_training_set_random_seed=args.seed,
+            mmap_mode=args.mmap_mode,
+            hashes=args.num_embeddings_per_feature
+            if args.num_embeddings is None
+            else ([args.num_embeddings] * CAT_FEATURE_COUNT),
+        ),
+        batch_size=None,
+        num_workers=0,
+        pin_memory=args.pin_memory,
+        collate_fn=lambda x: x,
+    )
+    return dataloader
+
+
+def get_dataloader(args: argparse.Namespace, backend: str, stage: str) -> DataLoader:
+    """
+    Gets desired dataloader from dlrm_main command line options. Currently, this
+    function is able to return either a DataLoader wrapped around a RandomRecDataset or
+    a Dataloader wrapped around an InMemoryBinaryCriteoIterDataPipe.
+
+    Args:
+        args (argparse.Namespace): Command line options supplied to dlrm_main.py's main
+            function.
+        backend (str): "nccl" or "gloo".
+        stage (str): "train", "val", or "test".
+
+    Returns:
+        dataloader (DataLoader): PyTorch dataloader for the specified options.
+
+    """
+    stage = stage.lower()
+    if stage not in STAGES:
+        raise ValueError(f"Supplied stage was {stage}. Must be one of {STAGES}.")
+
+    args.pin_memory = (
+        (backend == "nccl") if not hasattr(args, "pin_memory") else args.pin_memory
+    )
+
+    if (
+        args.in_memory_binary_criteo_path is None
+        and args.synthetic_multi_hot_criteo_path is None
+    ):
+        return _get_random_dataloader(args, stage)
+    else:
+        return _get_in_memory_dataloader(args, stage)
diff --git a/models_v2/pytorch/torchrec_dlrm/training/gpu/data/multi_hot_criteo.py b/models_v2/pytorch/torchrec_dlrm/training/gpu/data/multi_hot_criteo.py
new file mode 100644
index 000000000..cb520e2ec
--- /dev/null
+++ b/models_v2/pytorch/torchrec_dlrm/training/gpu/data/multi_hot_criteo.py
@@ -0,0 +1,310 @@
+#!/usr/bin/env python3
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+import zipfile
+from typing import Dict, Iterator, List, Optional
+
+import numpy as np
+import torch
+from iopath.common.file_io import PathManager, PathManagerFactory
+from pyre_extensions import none_throws
+from torch.utils.data import IterableDataset
+from torchrec.datasets.criteo import (
+    CAT_FEATURE_COUNT,
+    DEFAULT_CAT_NAMES,
+)
+from torchrec.datasets.utils import Batch, PATH_MANAGER_KEY
+from torchrec.sparse.jagged_tensor import KeyedJaggedTensor
+
+
+class MultiHotCriteoIterDataPipe(IterableDataset):
+    """
+    Datapipe designed to operate over the MLPerf DLRM v2 synthetic multi-hot dataset.
+    This dataset can be created by following the steps in
+    torchrec_dlrm/scripts/materialize_synthetic_multihot_dataset.py.
+    Each rank reads only the data for the portion of the dataset it is responsible for.
+
+    Args:
+        stage (str): "train", "val", or "test".
+        dense_paths (List[str]): List of path strings to dense npy files.
+        sparse_paths (List[str]): List of path strings to multi-hot sparse npz files.
+        labels_paths (List[str]): List of path strings to labels npy files.
+        batch_size (int): batch size.
+        rank (int): rank.
+        world_size (int): world size.
+        drop_last (Optional[bool]): Whether to drop the last batch if it is incomplete.
+        shuffle_batches (bool): Whether to shuffle batches
+        shuffle_training_set (bool): Whether to shuffle all samples in the dataset.
+        shuffle_training_set_random_seed (int): The random generator seed used when
+            shuffling the training set.
+        hashes (Optional[int]): List of max categorical feature value for each feature.
+            Length of this list should be CAT_FEATURE_COUNT.
+        path_manager_key (str): Path manager key used to load from different
+            filesystems.
+
+    Example::
+
+        datapipe = MultiHotCriteoIterDataPipe(
+            dense_paths=["day_0_dense.npy"],
+            sparse_paths=["day_0_sparse_multi_hot.npz"],
+            labels_paths=["day_0_labels.npy"],
+            batch_size=1024,
+            rank=torch.distributed.get_rank(),
+            world_size=torch.distributed.get_world_size(),
+        )
+        batch = next(iter(datapipe))
+    """
+
+    def __init__(
+        self,
+        stage: str,
+        dense_paths: List[str],
+        sparse_paths: List[str],
+        labels_paths: List[str],
+        batch_size: int,
+        rank: int,
+        world_size: int,
+        drop_last: Optional[bool] = False,
+        shuffle_batches: bool = False,
+        shuffle_training_set: bool = False,
+        shuffle_training_set_random_seed: int = 0,
+        mmap_mode: bool = False,
+        hashes: Optional[List[int]] = None,
+        path_manager_key: str = PATH_MANAGER_KEY,
+    ) -> None:
+        self.stage = stage
+        self.dense_paths = dense_paths
+        self.sparse_paths = sparse_paths
+        self.labels_paths = labels_paths
+        self.batch_size = batch_size
+        self.rank = rank
+        self.world_size = world_size
+        self.drop_last = drop_last
+        self.shuffle_batches = shuffle_batches
+        self.shuffle_training_set = shuffle_training_set
+        np.random.seed(shuffle_training_set_random_seed)
+        self.mmap_mode = mmap_mode
+        # hashes are not used because they were already applied in the
+        # script that generates the multi-hot dataset.
+        self.hashes: np.ndarray = np.array(hashes).reshape((CAT_FEATURE_COUNT, 1))
+        self.path_manager_key = path_manager_key
+        self.path_manager: PathManager = PathManagerFactory().get(path_manager_key)
+
+        if shuffle_training_set and stage == "train":
+            # Currently not implemented for the materialized multi-hot dataset.
+            self._shuffle_and_load_data_for_rank()
+        else:
+            m = "r" if mmap_mode else None
+            self.dense_arrs: List[np.ndarray] = [
+                np.load(f, mmap_mode=m) for f in self.dense_paths
+            ]
+            self.labels_arrs: List[np.ndarray] = [
+                np.load(f, mmap_mode=m) for f in self.labels_paths
+            ]
+            self.sparse_arrs: List = []
+            for sparse_path in self.sparse_paths:
+                multi_hot_ids_l = []
+                for feat_id_num in range(CAT_FEATURE_COUNT):
+                    multi_hot_ft_ids = self._load_from_npz(
+                        sparse_path, f"{feat_id_num}.npy"
+                    )
+                    multi_hot_ids_l.append(multi_hot_ft_ids)
+                self.sparse_arrs.append(multi_hot_ids_l)
+        len_d0 = len(self.dense_arrs[0])
+        second_half_start_index = int(len_d0 // 2 + len_d0 % 2)
+        if stage == "val":
+            self.dense_arrs[0] = self.dense_arrs[0][:second_half_start_index, :]
+            self.labels_arrs[0] = self.labels_arrs[0][:second_half_start_index, :]
+            self.sparse_arrs[0] = [
+                feats[:second_half_start_index, :] for feats in self.sparse_arrs[0]
+            ]
+        elif stage == "test":
+            self.dense_arrs[0] = self.dense_arrs[0][second_half_start_index:, :]
+            self.labels_arrs[0] = self.labels_arrs[0][second_half_start_index:, :]
+            self.sparse_arrs[0] = [
+                feats[second_half_start_index:, :] for feats in self.sparse_arrs[0]
+            ]
+        # When mmap_mode is enabled, sparse features are hashed when
+        # samples are batched in def __iter__. Otherwise, the dataset has been
+        # preloaded with sparse features hashed in the preload stage, here:
+        # if not self.mmap_mode and self.hashes is not None:
+        #     for k, _ in enumerate(self.sparse_arrs):
+        #         self.sparse_arrs[k] = [
+        #             feat % hash
+        #             for (feat, hash) in zip(self.sparse_arrs[k], self.hashes)
+        #         ]
+
+        self.num_rows_per_file: List[int] = list(map(len, self.dense_arrs))
+        total_rows = sum(self.num_rows_per_file)
+        self.num_full_batches: int = (
+            total_rows // batch_size // self.world_size * self.world_size
+        )
+        self.last_batch_sizes: np.ndarray = np.array(
+            [0 for _ in range(self.world_size)]
+        )
+        remainder = total_rows % (self.world_size * batch_size)
+        if not self.drop_last and 0 < remainder:
+            if remainder < self.world_size:
+                self.num_full_batches -= self.world_size
+                self.last_batch_sizes += batch_size
+            else:
+                self.last_batch_sizes += remainder // self.world_size
+            self.last_batch_sizes[: remainder % self.world_size] += 1
+
+        self.multi_hot_sizes: List[int] = [
+            multi_hot_feat.shape[-1] for multi_hot_feat in self.sparse_arrs[0]
+        ]
+
+        # These values are the same for the KeyedJaggedTensors in all batches, so they
+        # are computed once here. This avoids extra work from the KeyedJaggedTensor sync
+        # functions.
+        self.keys: List[str] = DEFAULT_CAT_NAMES
+        self.index_per_key: Dict[str, int] = {
+            key: i for (i, key) in enumerate(self.keys)
+        }
+
+    def _load_from_npz(self, fname, npy_name):
+        # figure out offset of .npy in .npz
+        zf = zipfile.ZipFile(fname)
+        info = zf.NameToInfo[npy_name]
+        assert info.compress_type == 0
+        zf.fp.seek(info.header_offset + len(info.FileHeader()) + 20)
+        # read .npy header
+        zf.open(npy_name, "r")
+        version = np.lib.format.read_magic(zf.fp)
+        shape, fortran_order, dtype = np.lib.format._read_array_header(zf.fp, version)
+        assert (
+            dtype == "int32"
+        ), f"sparse multi-hot dtype is {dtype} but should be int32"
+        offset = zf.fp.tell()
+        # create memmap
+        return np.memmap(
+            zf.filename,
+            dtype=dtype,
+            shape=shape,
+            order="F" if fortran_order else "C",
+            mode="r",
+            offset=offset,
+        )
+
+    def _np_arrays_to_batch(
+        self,
+        dense: np.ndarray,
+        sparse: List[np.ndarray],
+        labels: np.ndarray,
+    ) -> Batch:
+        if self.shuffle_batches:
+            # Shuffle all 3 in unison
+            shuffler = np.random.permutation(len(dense))
+            sparse = [multi_hot_ft[shuffler, :] for multi_hot_ft in sparse]
+            dense = dense[shuffler]
+            labels = labels[shuffler]
+
+        batch_size = len(dense)
+        lengths = torch.ones((CAT_FEATURE_COUNT * batch_size), dtype=torch.int32)
+        for k, multi_hot_size in enumerate(self.multi_hot_sizes):
+            lengths[k * batch_size : (k + 1) * batch_size] = multi_hot_size
+        offsets = torch.cumsum(torch.concat((torch.tensor([0]), lengths)), dim=0)
+        length_per_key = [
+            batch_size * multi_hot_size for multi_hot_size in self.multi_hot_sizes
+        ]
+        offset_per_key = torch.cumsum(
+            torch.concat((torch.tensor([0]), torch.tensor(length_per_key))), dim=0
+        )
+        values = torch.concat([torch.from_numpy(feat).flatten() for feat in sparse])
+        return Batch(
+            dense_features=torch.from_numpy(dense.copy()),
+            sparse_features=KeyedJaggedTensor(
+                keys=self.keys,
+                values=values,
+                lengths=lengths,
+                offsets=offsets,
+                stride=batch_size,
+                length_per_key=length_per_key,
+                offset_per_key=offset_per_key.tolist(),
+                index_per_key=self.index_per_key,
+            ),
+            labels=torch.from_numpy(labels.reshape(-1).copy()),
+        )
+
+    def __iter__(self) -> Iterator[Batch]:
+        # Invariant: buffer never contains more than batch_size rows.
+        buffer: Optional[List[np.ndarray]] = None
+
+        def append_to_buffer(
+            dense: np.ndarray,
+            sparse: List[np.ndarray],
+            labels: np.ndarray,
+        ) -> None:
+            nonlocal buffer
+            if buffer is None:
+                buffer = [dense, sparse, labels]
+            else:
+                buffer[0] = np.concatenate((buffer[0], dense))
+                buffer[1] = [np.concatenate((b, s)) for b, s in zip(buffer[1], sparse)]
+                buffer[2] = np.concatenate((buffer[2], labels))
+
+        # Maintain a buffer that can contain up to batch_size rows. Fill buffer as
+        # much as possible on each iteration. Only return a new batch when batch_size
+        # rows are filled.
+        file_idx = 0
+        row_idx = 0
+        batch_idx = 0
+        buffer_row_count = 0
+        cur_batch_size = (
+            self.batch_size if self.num_full_batches > 0 else self.last_batch_sizes[0]
+        )
+        while (
+            batch_idx
+            < self.num_full_batches + (self.last_batch_sizes[0] > 0) * self.world_size
+        ):
+            if buffer_row_count == cur_batch_size or file_idx == len(self.dense_arrs):
+                if batch_idx % self.world_size == self.rank:
+                    yield self._np_arrays_to_batch(*none_throws(buffer))
+                    buffer = None
+                buffer_row_count = 0
+                batch_idx += 1
+                if (
+                    0 <= batch_idx - self.num_full_batches < self.world_size
+                    and (self.last_batch_sizes[0] > 0)
+                ):
+                    cur_batch_size = self.last_batch_sizes[
+                        batch_idx - self.num_full_batches
+                    ]
+            else:
+                rows_to_get = min(
+                    cur_batch_size - buffer_row_count,
+                    self.num_rows_per_file[file_idx] - row_idx,
+                )
+                buffer_row_count += rows_to_get
+                slice_ = slice(row_idx, row_idx + rows_to_get)
+
+                if batch_idx % self.world_size == self.rank:
+                    dense_inputs = self.dense_arrs[file_idx][slice_, :]
+                    sparse_inputs = [
+                        feats[slice_, :] for feats in self.sparse_arrs[file_idx]
+                    ]
+                    target_labels = self.labels_arrs[file_idx][slice_, :]
+
+                    # if self.mmap_mode and self.hashes is not None:
+                    #     sparse_inputs = [
+                    #         feats % hash
+                    #         for (feats, hash) in zip(sparse_inputs, self.hashes)
+                    #     ]
+
+                    append_to_buffer(
+                        dense_inputs,
+                        sparse_inputs,
+                        target_labels,
+                    )
+                row_idx += rows_to_get
+
+                if row_idx >= self.num_rows_per_file[file_idx]:
+                    file_idx += 1
+                    row_idx = 0
+
+    def __len__(self) -> int:
+        return self.num_full_batches // self.world_size + (self.last_batch_sizes[0] > 0)
diff --git a/models_v2/pytorch/torchrec_dlrm/training/gpu/dist_models.py b/models_v2/pytorch/torchrec_dlrm/training/gpu/dist_models.py
new file mode 100644
index 000000000..e3a7d391d
--- /dev/null
+++ b/models_v2/pytorch/torchrec_dlrm/training/gpu/dist_models.py
@@ -0,0 +1,243 @@
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+import argparse
+import torch
+from torchrec import EmbeddingBagCollection
+import extend_distributed as ext_dist
+from torchrec.models.dlrm import DLRM, InteractionDCNArch, OverArch
+from typing import Dict, List, Optional, Tuple
+from torchrec.sparse.jagged_tensor import KeyedJaggedTensor
+from torchrec.modules.crossnet import LowRankCrossNet
+import time
+
+class DIST_DLRM_DCN(DLRM):
+    """
+    Recsys model with DCN modified from the original model from "Deep Learning Recommendation
+    Model for Personalization and Recommendation Systems"
+    (https://arxiv.org/abs/1906.00091). Similar to DLRM module but has
+    DeepCrossNet https://arxiv.org/pdf/2008.13535.pdf as the interaction layer.
+
+    The module assumes all sparse features have the same embedding dimension
+    (i.e. each EmbeddingBagConfig uses the same embedding_dim).
+
+    The following notation is used throughout the documentation for the models:
+
+    * F: number of sparse features
+    * D: embedding_dimension of sparse features
+    * B: batch size
+    * num_features: number of dense features
+
+    Args:
+        embedding_bag_collection (EmbeddingBagCollection): collection of embedding bags
+            used to define `SparseArch`.
+        dense_in_features (int): the dimensionality of the dense input features.
+        dense_arch_layer_sizes (List[int]): the layer sizes for the `DenseArch`.
+        over_arch_layer_sizes (List[int]): the layer sizes for the `OverArch`.
+            The output dimension of the `InteractionArch` should not be manually
+            specified here.
+        dcn_num_layers (int): the number of DCN layers in the interaction.
+        dcn_low_rank_dim (int): the dimensionality of low rank approximation
+            used in the dcn layers.
+        dense_device (Optional[torch.device]): default compute device.
+
+    Example::
+
+        B = 2
+        D = 8
+
+        eb1_config = EmbeddingBagConfig(
+           name="t1", embedding_dim=D, num_embeddings=100, feature_names=["f1", "f3"]
+        )
+        eb2_config = EmbeddingBagConfig(
+           name="t2",
+           embedding_dim=D,
+           num_embeddings=100,
+           feature_names=["f2"],
+        )
+
+        ebc = EmbeddingBagCollection(tables=[eb1_config, eb2_config])
+        model = DLRM_DCN(
+           embedding_bag_collection=ebc,
+           dense_in_features=100,
+           dense_arch_layer_sizes=[20, D],
+           dcn_num_layers=2,
+           dcn_low_rank_dim=8,
+           over_arch_layer_sizes=[5, 1],
+        )
+
+        features = torch.rand((B, 100))
+
+        #     0       1
+        # 0   [1,2] [4,5]
+        # 1   [4,3] [2,9]
+        # ^
+        # feature
+        sparse_features = KeyedJaggedTensor.from_offsets_sync(
+           keys=["f1", "f3"],
+           values=torch.tensor([1, 2, 4, 5, 4, 3, 2, 9]),
+           offsets=torch.tensor([0, 2, 4, 6, 8]),
+        )
+
+        logits = model(
+           dense_features=features,
+           sparse_features=sparse_features,
+        )
+    """
+
+    def __init__(
+        self,
+        embedding_bag_collection: EmbeddingBagCollection,
+        dense_in_features: int,
+        num_total_sparse_features:int,
+        dense_arch_layer_sizes: List[int],
+        over_arch_layer_sizes: List[int],
+        dcn_num_layers: int,
+        dcn_low_rank_dim: int,
+        n_emb : int,
+        shard_matrix,
+        shard_strategy,
+        args: argparse.Namespace,
+        dense_device: Optional[torch.device] = None,
+    ) -> None:
+        # initialize DLRM
+        # sparse arch and dense arch are initialized via DLRM
+        super().__init__(
+            embedding_bag_collection,
+            dense_in_features,
+            dense_arch_layer_sizes,
+            over_arch_layer_sizes,
+            dense_device,
+        )
+        
+        self.args = args
+        if ext_dist.my_size > 1:
+            if n_emb < ext_dist.my_size:
+                sys.exit(
+                    "only (%d) sparse features for (%d) devices, table partitions will fail"
+                    % (n_emb, ext_dist.my_size)
+                )
+            self.n_global_emb = n_emb
+            #self.n_local_emb, self.n_emb_per_rank = ext_dist.get_split_lengths(
+            #    n_emb
+            #)
+            self.shard_matrix = shard_matrix
+            self.shard_strategy = shard_strategy
+            self.n_emb_per_rank = [len(s_m) for s_m in shard_matrix]
+            #self.local_emb_slice = ext_dist.get_my_slice(n_emb)
+            #self.local_emb_indices = list(range(n_emb))[self.local_emb_slice]
+
+
+        self.embedding_dim: int = embedding_bag_collection.embedding_bag_configs()[
+            0
+        ].embedding_dim
+        #num_sparse_features: int = len(self.sparse_arch.sparse_feature_names)
+
+        # Fix interaction and over arch for DLRM_DCN
+
+        crossnet = LowRankCrossNet(
+            in_features=(num_total_sparse_features + 1) * self.embedding_dim,
+            num_layers=dcn_num_layers,
+            low_rank=dcn_low_rank_dim,
+        )
+
+        self.inter_arch = InteractionDCNArch(
+            num_sparse_features=num_total_sparse_features,
+            crossnet=crossnet,
+        )
+        self.inter_arch.to(dense_device)
+
+        over_in_features: int = (num_total_sparse_features + 1) * self.embedding_dim
+
+        self.over_arch = OverArch(
+            in_features=over_in_features,
+            layer_sizes=over_arch_layer_sizes,
+            device=dense_device,
+        )
+
+    def sequential_forward(
+        self,
+        dense_features: torch.Tensor,
+        sparse_features: KeyedJaggedTensor,
+    ) -> torch.Tensor:
+        """
+        Args:
+            dense_features (torch.Tensor): the dense features.
+            sparse_features (KeyedJaggedTensor): the sparse features.
+
+        Returns:
+            torch.Tensor: logits.
+        """
+        embedded_dense = self.dense_arch(dense_features)
+        embedded_sparse = self.sparse_arch(sparse_features)
+        concatenated_dense = self.inter_arch(
+            dense_features=embedded_dense, sparse_features=embedded_sparse
+        )
+        logits = self.over_arch(concatenated_dense)
+        return logits
+
+    def distributed_forward(
+        self,
+        dense_features: torch.Tensor,
+        sparse_features: KeyedJaggedTensor,
+    ) -> torch.Tensor:
+        """
+        Args:
+            dense_features (torch.Tensor): the dense features.
+            sparse_features (KeyedJaggedTensor): the sparse features.
+
+        Returns:
+            torch.Tensor: logits.
+        """
+        embedded_sparse = self.sparse_arch(sparse_features)
+        a2a_req = ext_dist.alltoall(embedded_sparse, self.n_emb_per_rank)
+        embedded_dense = self.dense_arch(dense_features)
+        embedded_sparse = a2a_req.wait()
+        #['0', '8', '16', '24'], ['1', '9', '17', '25'], ['2', '10', '18'], ['3', '11', '19'], ['4', '12', '20'], ['5', '13', '21'], ['6', '14', '22'], ['7', '15', '23']
+        split_embed = {}
+        for index_rank in range(len(self.shard_matrix)):
+            embed_rank = torch.split(embedded_sparse[index_rank], self.embedding_dim, dim=1)
+            for index in range(len(self.shard_matrix[index_rank])):
+                split_embed[self.shard_matrix[index_rank][index]] = embed_rank[index]
+
+        embedded_sparse = []
+        for i in range(len(self.args.num_embeddings_per_feature)):
+            embedded_sparse.append(split_embed[i])
+        embedded_sparse = torch.cat(list(embedded_sparse), dim=1).reshape(embedded_dense.size(0), -1, embedded_dense.size(-1))
+        if self.args.fp16:
+            embedded_sparse = embedded_sparse.half() if embedded_sparse.dtype != torch.float16 else embedded_sparse
+            embedded_dense = embedded_dense.half() if embedded_dense.dtype != torch.float16 else embedded_dense
+        elif self.args.bf16:
+            embedded_sparse = embedded_sparse.bfloat16() if embedded_sparse.dtype != torch.bfloat16 else embedded_sparse
+            embedded_dense = embedded_dense.bfloat16() if embedded_dense.dtype != torch.bfloat16 else embedded_dense
+
+        concatenated_dense = self.inter_arch(
+            dense_features=embedded_dense, sparse_features=embedded_sparse
+        )
+        logits = self.over_arch(concatenated_dense)
+        return logits
+
+    def forward(
+        self,
+        dense_features: torch.Tensor,
+        sparse_features: KeyedJaggedTensor,
+    ) -> torch.Tensor:
+        if ext_dist.my_size > 1:
+            # multi-node multi-device run
+            return self.distributed_forward(dense_features, sparse_features)
+        else:
+            # single device run
+            return self.sequential_forward(dense_features, sparse_features)
+
diff --git a/models_v2/pytorch/torchrec_dlrm/training/gpu/dlrm_main.py b/models_v2/pytorch/torchrec_dlrm/training/gpu/dlrm_main.py
new file mode 100644
index 000000000..3435bc33b
--- /dev/null
+++ b/models_v2/pytorch/torchrec_dlrm/training/gpu/dlrm_main.py
@@ -0,0 +1,1099 @@
+#!/usr/bin/env python3
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+import argparse
+import itertools
+import os
+import sys
+import time
+from dataclasses import dataclass, field
+from enum import Enum
+from typing import List, Optional
+import numpy as np
+
+import torch
+import torchmetrics as metrics
+from torch.autograd.profiler import record_function
+from pyre_extensions import none_throws
+from torch import distributed as dist
+from torch.utils.data import DataLoader
+from torchrec import EmbeddingBagCollection
+from torchrec.datasets.criteo import DEFAULT_CAT_NAMES, DEFAULT_INT_NAMES
+#from torchrec.distributed import TrainPipelineSparseDist
+from torchrec.distributed.comm import get_local_size
+#from torchrec.distributed.model_parallel import (
+#    DistributedModelParallel,
+#    get_default_sharders,
+#)
+#from torchrec.distributed.planner import EmbeddingShardingPlanner, Topology
+#from torchrec.distributed.planner.storage_reservations import (
+#    HeuristicalStorageReservation,
+#)
+from dist_models import DIST_DLRM_DCN
+from torchrec.models.dlrm import DLRM, DLRM_DCN, DLRM_Projection, DLRMTrain
+from torchrec.modules.embedding_configs import EmbeddingBagConfig
+from torchrec.optim.apply_optimizer_in_backward import apply_optimizer_in_backward
+from torchrec.optim.keyed import CombinedOptimizer, KeyedOptimizerWrapper
+from torchrec.optim.optimizers import in_backward_optimizer_filter
+from tqdm import tqdm
+import extend_distributed as ext_dist
+from extend_distributed import dist, env2int
+from jit_trace_able_utils import unpack, SparseArchTraceAbleWrapper
+
+import logging
+import sharding
+
+logger: logging.Logger = logging.getLogger(__name__)
+logger.setLevel(1)
+
+# OSS import
+try:
+    # pyre-ignore[21]
+    # @manual=//ai_codesign/benchmarks/dlrm/torchrec_dlrm/data:dlrm_dataloader
+    from data.dlrm_dataloader import get_dataloader
+
+    # pyre-ignore[21]
+    # @manual=//ai_codesign/benchmarks/dlrm/torchrec_dlrm:lr_scheduler
+    from lr_scheduler import LRPolicyScheduler
+
+    # pyre-ignore[21]
+    # @manual=//ai_codesign/benchmarks/dlrm/torchrec_dlrm:multi_hot
+    from multi_hot import Multihot, RestartableMap
+except ImportError:
+    pass
+
+# internal import
+try:
+    from .data.dlrm_dataloader import get_dataloader  # noqa F811
+    from .lr_scheduler import LRPolicyScheduler  # noqa F811
+    from .multi_hot import Multihot, RestartableMap  # noqa F811
+except ImportError:
+    pass
+
+TRAIN_PIPELINE_STAGES = 3  # Number of stages in TrainPipelineSparseDist.
+
+n_emb = None
+#local_emb_slice = None
+local_emb_indices = None
+default_embedding_names = None
+idx_name = None
+
+
+def load_checkoutpoint(args, device):
+    checkpoint = None
+    if args.checkpoint_floder is not None:
+        if (not os.path.exists(args.checkpoint_floder)) or (not os.path.isdir(args.checkpoint_floder)):
+            logger.error("The checkpoint dir is not exist.")
+            exit()
+        if  not os.path.exists(args.checkpoint_floder + '/dlrm_training_' + str(dist.get_rank()) + '_save.pt'):
+            logger.warning("The checkpoint " + args.checkpoint_floder + '/dlrm_training_' + str(dist.get_rank()) + '_save.pt' +  " doesn't exist. If converge isn't zero step, your behavior will cause exceptions.")
+        else:
+            print('load...', args.checkpoint_floder + '/dlrm_training_' + str(dist.get_rank()) + '_save.pt', ' to ',
+                  device)
+            checkpoint = torch.load(args.checkpoint_floder + '/dlrm_training_' + str(dist.get_rank()) + '_save.pt', map_location=torch.device('cpu'))
+    return checkpoint
+
+
+
+class InteractionType(Enum):
+    ORIGINAL = "original"
+    DCN = "dcn"
+    PROJECTION = "projection"
+
+    def __str__(self):
+        return self.value
+
+
+def load_snapshot(model, snapshot_dir):
+    from torchsnapshot import Snapshot
+    snapshot = Snapshot(path=snapshot_dir)
+    snapshot.restore(app_state={"model": model})
+
+
+def parse_args(argv: List[str]) -> argparse.Namespace:
+    parser = argparse.ArgumentParser(description="torchrec dlrm example trainer")
+    parser.add_argument(
+        "--epochs",
+        type=int,
+        default=1,
+        help="number of epochs to train",
+    )
+    parser.add_argument(
+        "--batch_size",
+        type=int,
+        default=32,
+        help="batch size to use for training",
+    )
+    parser.add_argument(
+        "--drop_last_training_batch",
+        dest="drop_last_training_batch",
+        action="store_true",
+        help="Drop the last non-full training batch",
+    )
+    parser.add_argument(
+        "--test_batch_size",
+        type=int,
+        default=None,
+        help="batch size to use for validation and testing",
+    )
+    parser.add_argument(
+        "--limit_train_batches",
+        type=int,
+        default=None,
+        help="number of train batches",
+    )
+    parser.add_argument(
+        "--limit_val_batches",
+        type=int,
+        default=None,
+        help="number of validation batches",
+    )
+    parser.add_argument(
+        "--limit_test_batches",
+        type=int,
+        default=None,
+        help="number of test batches",
+    )
+    parser.add_argument(
+        "--dataset_name",
+        type=str,
+        choices=["criteo_1t", "criteo_kaggle"],
+        default="criteo_1t",
+        help="dataset for experiment, current support criteo_1tb, criteo_kaggle",
+    )
+    parser.add_argument(
+        "--num_embeddings",
+        type=int,
+        default=100_000,
+        help="max_ind_size. The number of embeddings in each embedding table. Defaults"
+        " to 100_000 if num_embeddings_per_feature is not supplied.",
+    )
+    parser.add_argument(
+        "--num_embeddings_per_feature",
+        type=str,
+        default=None,
+        help="Comma separated max_ind_size per sparse feature. The number of embeddings"
+        " in each embedding table. 26 values are expected for the Criteo dataset.",
+    )
+    parser.add_argument(
+        "--dense_arch_layer_sizes",
+        type=str,
+        default="512,256,64",
+        help="Comma separated layer sizes for dense arch.",
+    )
+    parser.add_argument(
+        "--over_arch_layer_sizes",
+        type=str,
+        default="512,512,256,1",
+        help="Comma separated layer sizes for over arch.",
+    )
+    parser.add_argument(
+        "--embedding_dim",
+        type=int,
+        default=64,
+        help="Size of each embedding.",
+    )
+    parser.add_argument(
+        "--interaction_branch1_layer_sizes",
+        type=str,
+        default="2048,2048",
+        help="Comma separated layer sizes for interaction branch1 (only on dlrm with projection).",
+    )
+    parser.add_argument(
+        "--interaction_branch2_layer_sizes",
+        type=str,
+        default="2048,2048",
+        help="Comma separated layer sizes for interaction branch2 (only on dlrm with projection).",
+    )
+    parser.add_argument(
+        "--dcn_num_layers",
+        type=int,
+        default=3,
+        help="Number of DCN layers in interaction layer (only on dlrm with DCN).",
+    )
+    parser.add_argument(
+        "--dcn_low_rank_dim",
+        type=int,
+        default=512,
+        help="Low rank dimension for DCN in interaction layer (only on dlrm with DCN).",
+    )
+    parser.add_argument(
+        "--undersampling_rate",
+        type=float,
+        help="Desired proportion of zero-labeled samples to retain (i.e. undersampling zero-labeled rows)."
+        " Ex. 0.3 indicates only 30pct of the rows with label 0 will be kept."
+        " All rows with label 1 will be kept. Value should be between 0 and 1."
+        " When not supplied, no undersampling occurs.",
+    )
+    parser.add_argument(
+        "--seed",
+        type=int,
+        help="Random seed for reproducibility.",
+    )
+    parser.add_argument(
+        "--pin_memory",
+        dest="pin_memory",
+        action="store_true",
+        help="Use pinned memory when loading data.",
+    )
+    parser.add_argument(
+        "--mmap_mode",
+        dest="mmap_mode",
+        action="store_true",
+        help="--mmap_mode mmaps the dataset."
+        " That is, the dataset is kept on disk but is accessed as if it were in memory."
+        " --mmap_mode is intended mostly for faster debugging. Use --mmap_mode to bypass"
+        " preloading the dataset when preloading takes too long or when there is "
+        " insufficient memory available to load the full dataset.",
+    )
+    parser.add_argument(
+        "--in_memory_binary_criteo_path",
+        type=str,
+        default=None,
+        help="Directory path containing the Criteo dataset npy files.",
+    )
+    parser.add_argument(
+        "--synthetic_multi_hot_criteo_path",
+        type=str,
+        default=None,
+        help="Directory path containing the MLPerf v2 synthetic multi-hot dataset npz files.",
+    )
+    parser.add_argument(
+        "--learning_rate",
+        type=float,
+        default=15.0,
+        help="Learning rate.",
+    )
+    parser.add_argument(
+        "--eps",
+        type=float,
+        default=1e-8,
+        help="Epsilon for Adagrad optimizer.",
+    )
+    parser.add_argument(
+        "--shuffle_batches",
+        dest="shuffle_batches",
+        action="store_true",
+        help="Shuffle each batch during training.",
+    )
+    parser.add_argument(
+        "--shuffle_training_set",
+        dest="shuffle_training_set",
+        action="store_true",
+        help="Shuffle the training set in memory. This will override mmap_mode",
+    )
+    parser.add_argument(
+        "--validation_freq_within_epoch",
+        type=int,
+        default=None,
+        help="Frequency at which validation will be run within an epoch.",
+    )
+    parser.set_defaults(
+        pin_memory=None,
+        mmap_mode=None,
+        drop_last=None,
+        shuffle_batches=None,
+        shuffle_training_set=None,
+    )
+    parser.add_argument(
+        "--adagrad",
+        dest="adagrad",
+        action="store_true",
+        help="Flag to determine if adagrad optimizer should be used.",
+    )
+    parser.add_argument(
+        "--interaction_type",
+        type=InteractionType,
+        choices=list(InteractionType),
+        default=InteractionType.ORIGINAL,
+        help="Determine the interaction type to be used (original, dcn, or projection)"
+        " default is original DLRM with pairwise dot product",
+    )
+    parser.add_argument(
+        "--collect_multi_hot_freqs_stats",
+        dest="collect_multi_hot_freqs_stats",
+        action="store_true",
+        help="Flag to determine whether to collect stats on freq of embedding access.",
+    )
+    parser.add_argument(
+        "--multi_hot_sizes",
+        type=str,
+        default=None,
+        help="Comma separated multihot size per sparse feature. 26 values are expected for the Criteo dataset.",
+    )
+    parser.add_argument(
+        "--multi_hot_distribution_type",
+        type=str,
+        choices=["uniform", "pareto"],
+        default=None,
+        help="Multi-hot distribution options.",
+    )
+    parser.add_argument("--lr_warmup_steps", type=int, default=0)
+    parser.add_argument("--lr_decay_start", type=int, default=0)
+    parser.add_argument("--lr_decay_steps", type=int, default=0)
+    parser.add_argument(
+        "--print_lr",
+        action="store_true",
+        help="Print learning rate every iteration.",
+    )
+    parser.add_argument(
+        "--allow_tf32",
+        action="store_true",
+        help="Enable TensorFloat-32 mode for matrix multiplications on A100 (or newer) GPUs.",
+    )
+    parser.add_argument(
+        "--print_sharding_plan",
+        action="store_true",
+        help="Print the sharding plan used for each embedding table.",
+    )
+    parser.add_argument("--use_xpu", action="store_true", default=False)
+    parser.add_argument(
+        "--log_freq",
+        type=int,
+        default=0,
+        help="log_freq to print performance statistics.",
+    )
+    parser.add_argument(
+        "--converge_freq",
+        type=int,
+        default=0,
+        help="converge_freq to save model.",
+    )
+    parser.add_argument("--numpy_rand_seed", type=int, default=123)
+    parser.add_argument("--bf16", action="store_true", default=False)
+    parser.add_argument("--fp16", action="store_true", default=False)
+    parser.add_argument("--amp", action="store_true", default=False)
+    parser.add_argument("--dense", action="store_true", default=False)
+    parser.add_argument("--inference_only", action="store_true", default=False)
+    parser.add_argument("--converge", action="store_true", default=False)
+    parser.add_argument("--train_with_val", action="store_true", default=False)
+    parser.add_argument("--num_nodes", type=int, default=1)
+    parser.add_argument("--trans_data", action="store_true", default=False)
+    parser.add_argument(
+    "--sharding_plan",
+    help="Sharding plan to use",
+    type=str,
+    choices=["round_robin", "uniform", "auto", "hier_auto", "custom", "max_min"],
+    default="round_robin",)
+    parser.add_argument(
+        "--snapshot_dir",
+        help="Weight path for dlrm-v2",
+        type=str,
+    )
+
+    parser.add_argument(
+        "--checkpoint_floder",
+        help="checkpoint path for dlrm-v2",
+        type=str,
+    )
+    parser.add_argument(
+        "--dynamo",
+        action="store_true",
+        help="Use torch compile for model test",
+        default=False
+    )
+
+
+
+    return parser.parse_args(argv)
+
+
+def _evaluate(
+    limit_batches: Optional[int],
+    #eval_pipeline: TrainPipelineSparseDist,
+    eval_model,
+    eval_dataloader: DataLoader,
+    stage: str,
+    log_freq: int,
+    device: torch.device,
+    args: argparse.Namespace,
+) -> float:
+    """
+    Evaluates model. Computes and prints AUROC. Helper function for train_val_test.
+
+    Args:
+        limit_batches (Optional[int]): Limits the dataloader to the first `limit_batches` batches.
+        eval_pipeline (TrainPipelineSparseDist): pipelined model.
+        eval_dataloader (DataLoader): Dataloader for either the validation set or test set.
+        stage (str): "val" or "test".
+
+    Returns:
+        float: auroc result
+    """
+    eval_model.eval()
+    global n_emb, local_emb_indices, default_embedding_names, idx_name
+
+    if args.dynamo:
+        if args.amp:
+            if args.use_device == "xpu":
+                with torch.no_grad():
+                    with torch.xpu.amp.autocast(enabled=args.amp, dtype=args.amp_dtype):
+                        print(" Use dynamo for compiling")
+                        eval_model = torch.compile(eval_model, backend="inductor", options={"freezing": True} )
+
+    def eval_step(model, batch):
+        batch_size = batch.dense_features.size(0)
+        if args.amp:
+            batch.dense_features = batch.dense_features[ext_dist.get_my_slice(batch_size)].to(device)
+        else:
+            batch.dense_features = batch.dense_features[ext_dist.get_my_slice(batch_size)].to(device).to(args.amp_dtype)
+        batch.labels = batch.labels[ext_dist.get_my_slice(batch_size)].to(device)
+        for k, v in batch.sparse_features.items():
+            batch.sparse_features[k]['values'] = batch.sparse_features[k]['values'].to(device)
+            batch.sparse_features[k]['offsets'] = batch.sparse_features[k]['offsets'].to(device)
+
+        #if args.converge and args.dense and (args.use_device == "xpu"):
+        #    torch.xpu.empty_cache()
+        with torch.no_grad():
+            if args.amp:
+                if args.use_device == "xpu":
+                    with torch.xpu.amp.autocast(enabled=args.amp, dtype=args.amp_dtype):
+                        logits = model(batch.dense_features, batch.sparse_features)
+                elif args.use_device == "cuda":
+                    with torch.cuda.amp.autocast(enabled=args.amp, dtype=args.amp_dtype):
+                        logits = model(batch.dense_features, batch.sparse_features)
+                else:
+                    with torch.cpu.amp.autocast(enabled=args.amp, dtype=args.amp_dtype):
+                        logits = model(batch.dense_features, batch.sparse_features)
+            else:
+                logits = model(batch.dense_features, batch.sparse_features)
+        #if args.converge and args.dense and (args.use_device == "xpu"):
+        #    torch.xpu.empty_cache()
+        return logits, batch.labels
+    # Set eval_pipeline._connected to False to cause the pipeline to refill with new batches as if it were newly created and empty.
+    #eval_pipeline._connected = False
+
+    iterator = itertools.islice(iter(eval_dataloader), limit_batches)
+
+    auroc = metrics.AUROC(compute_on_step=False, task='binary').to(device)
+
+    is_rank_zero = dist.get_rank() == 0
+    total_t = 0
+    it = 1
+    with torch.no_grad():
+        while True:
+            try:
+                batch = next(iterator)
+                batch.sparse_features = unpack(batch.sparse_features, default_embedding_names)
+                #ext_dist.barrier()
+                if args.use_device == "xpu":
+                    torch.xpu.synchronize()
+                elif args.use_device == "cuda":
+                    torch.cuda.synchronize()
+                else:
+                    pass
+                t1 = time.time()
+                with (torch.autograd.profiler_legacy.profile(args.enable_profiling, use_xpu=True) if args.use_xpu else torch.profiler.profile(activities=[torch.profiler.ProfilerActivity.CUDA])) as prof:
+                    logits, labels = eval_step(eval_model, batch)
+                    preds = torch.sigmoid(logits)
+                    auroc(preds, labels)
+                    if args.use_device == "xpu":
+                        torch.xpu.synchronize()
+                    elif args.use_device == "cuda":
+                        torch.cuda.synchronize()
+                    else:
+                        pass
+                    if it > 5 and args.inference_only:
+                        total_t += time.time() - t1
+                        logger.info(f"avg eval time per iter at ITER: {it - 5}, {total_t/(it-5)} s")
+            except StopIteration:
+                break
+            if args.enable_profiling and it == 35 and args.inference_only:
+                rank_index = ""
+                if ext_dist.my_size > 1:
+                    rank_index = "rank_" + str(dist.get_rank()) + "_"
+                path ='./report/dlrm_inference_' + rank_index + 'profiling'
+                if args.use_xpu:
+                    torch.save(prof.key_averages().table(sort_by="self_xpu_time_total", row_limit=100000), \
+                        path + '.pt')
+                    prof.export_chrome_trace(path + '.json')
+                    torch.save(prof.table(sort_by="id", row_limit=100000), path + '_detail.pt')
+                elif args.use_device == "cuda":
+                    torch.save(prof.key_averages().table(sort_by="self_cuda_time_total", row_limit=100000), \
+                        path + '.pt')
+                    prof.export_chrome_trace(path + '.json')
+                else:
+                    pass
+            it += 1
+
+    auroc_result = auroc.compute().item()
+    num_samples = torch.tensor(sum(map(len, auroc.target)), device=device)
+    dist.reduce(num_samples, 0, op=dist.ReduceOp.SUM)
+
+    if is_rank_zero:
+        print(f"AUROC over {stage} set: {auroc_result}.")
+        print(f"Number of {stage} samples: {num_samples}")
+    return auroc_result
+
+def _train(
+    train_model,
+    train_optimizer,
+    train_dataloader: DataLoader,
+    val_dataloader: DataLoader,
+    epoch: int,
+    lr_scheduler,
+    print_lr: bool,
+    validation_freq: Optional[int],
+    limit_train_batches: Optional[int],
+    limit_val_batches: Optional[int],
+    log_freq: int,
+    device: torch.device,
+    args: argparse.Namespace,
+) -> bool:
+    """
+    Trains model for 1 epoch. Helper function for train_val_test.
+    Args:
+        train_pipeline (TrainPipelineSparseDist): pipelined model used for training.
+        val_pipeline (TrainPipelineSparseDist): pipelined model used for validation.
+        train_dataloader (DataLoader): Training set's dataloader.
+        val_dataloader (DataLoader): Validation set's dataloader.
+        epoch (int): The number of complete passes through the training set so far.
+        lr_scheduler (LRPolicyScheduler): Learning rate scheduler.
+        print_lr (bool): Whether to print the learning rate every training step.
+        validation_freq (Optional[int]): The number of training steps between validation runs within an epoch.
+        limit_train_batches (Optional[int]): Limits the training set to the first `limit_train_batches` batches.
+        limit_val_batches (Optional[int]): Limits the validation set to the first `limit_val_batches` batches.
+        print_progress (bool): Whether to print tqdm progress bar.
+    Returns:
+        bool: Whether the validation_auroc threshold is reached.
+    """
+    train_model.train()
+    global n_emb, local_emb_indices, default_embedding_names, idx_name
+    def train_step(model, opt, next_batch):
+        batch_size = next_batch.dense_features.size(0)
+        opt.zero_grad(set_to_none=True)
+        if args.amp:
+            next_batch.dense_features = next_batch.dense_features[ext_dist.get_my_slice(batch_size)].to(device)
+        else:
+            next_batch.dense_features = next_batch.dense_features[ext_dist.get_my_slice(batch_size)].to(device).to(args.amp_dtype)
+        next_batch.labels = next_batch.labels[ext_dist.get_my_slice(batch_size)].to(device)
+
+        for k, v in next_batch.sparse_features.items():
+            next_batch.sparse_features[k]['values'] = next_batch.sparse_features[k]['values'].to(device)
+            next_batch.sparse_features[k]['offsets'] = next_batch.sparse_features[k]['offsets'].to(device)
+        #if args.converge and args.dense and (args.use_device == "xpu"):
+        #    torch.xpu.empty_cache()
+        if args.amp:
+            if args.use_device == "xpu":
+                with torch.xpu.amp.autocast(enabled=args.amp, dtype=args.amp_dtype):
+                    losses, _ = model(next_batch)
+                    loss = torch.sum(losses, dim=0)
+            elif args.use_device == "cuda":
+                with torch.cuda.amp.autocast(enabled=args.amp, dtype=args.amp_dtype):
+                    losses, _ = model(next_batch)
+                    loss = torch.sum(losses, dim=0)
+            else:
+                with torch.cpu.amp.autocast(enabled=args.amp, dtype=args.amp_dtype):
+                    losses, _ = model(next_batch)
+                    loss = torch.sum(losses, dim=0)
+        else:
+            losses, _ = model(next_batch)
+            loss = torch.sum(losses, dim=0)
+
+        #if args.converge and args.dense and (args.use_device == "xpu"):
+        #    torch.xpu.empty_cache()
+        loss.backward()
+        opt.step()
+    start_item = 0
+    if args.converge:
+        start_item = args.step
+    iterator = itertools.islice(iter(train_dataloader), start_item, limit_train_batches)
+
+    is_rank_zero = dist.get_rank() == 0
+
+    total_t = 0
+    for it in itertools.count(start_item + 1, 1):
+        try:
+            if is_rank_zero and print_lr:
+                for i, g in enumerate(train_pipeline._optimizer.param_groups):
+                    print(f"lr: {it} {i} {g['lr']:.6f}")
+            next_batch = next(iterator)
+            next_batch.sparse_features = unpack(next_batch.sparse_features, default_embedding_names)
+            #ext_dist.barrier()
+            if args.use_device == "xpu":
+                torch.xpu.synchronize()
+            elif args.use_device == "cuda":
+                torch.cuda.synchronize()
+            else:
+                pass
+            t1 = time.time()
+            with (torch.autograd.profiler_legacy.profile(args.enable_profiling, use_xpu=True, record_shapes=False) if args.use_xpu else torch.profiler.profile(activities=[torch.profiler.ProfilerActivity.CUDA])) as prof:
+                train_step(train_model, train_optimizer, next_batch)
+                if args.use_device == "xpu":
+                    torch.xpu.synchronize()
+                elif args.use_device == "cuda":
+                    torch.cuda.synchronize()
+                else:
+                    pass
+
+                if it > 5:
+                    total_t += time.time() - t1
+                    #if log_freq != 0 and it % log_freq == 0:
+                    logger.info(f"avg training time per iter at ITER: {it - 5}, {total_t/(it - 5)} s")
+            if args.enable_profiling and it == 35:
+                rank_index = ""
+                if ext_dist.my_size > 1:
+                    rank_index = "rank_" + str(dist.get_rank()) + "_"
+                path ='./report/dlrm_training_' + rank_index + 'profiling'
+                if args.use_device == "xpu":
+                    torch.save(prof.key_averages().table(sort_by="self_xpu_time_total", row_limit=100000), \
+                        path + '.pt')
+                    prof.export_chrome_trace(path + '.json')
+                    torch.save(prof.table(sort_by="id", row_limit=100000), path + '_detail.pt')
+                elif args.use_device == "cuda":
+                    torch.save(prof.key_averages().table(sort_by="self_cuda_time_total", row_limit=100000), \
+                        path + '.pt')
+                    prof.export_chrome_trace(path + '.json')
+                else:
+                    torch.save(prof.key_averages().table(sort_by="self_cpu_time_total", row_limit=100000), \
+                        path + '.pt')
+                    prof.export_chrome_trace(path + '.json')
+                    pass
+            lr_scheduler.step()
+
+            if validation_freq and (it % validation_freq == 0) and args.train_with_val:
+                auroc_result = _evaluate(
+                    limit_val_batches,
+                    train_model.model,
+                    val_dataloader,
+                    "val",
+                    log_freq, 
+                    device, 
+                    args
+                )
+                if is_rank_zero:
+                    logger.info(f"The val auc = {auroc_result}")
+                train_model.train()
+
+            if args.converge and it  % args.converge_freq == 0:
+                model_state_dict = {}
+                for k, v in train_model.state_dict().items():
+                    k_split = k.split("module")
+                    k_str = k_split[0][:-1] + k_split[1] if len(k_split) == 2 else k
+                    model_state_dict[k_str] = v
+                torch.save({
+                    "model": model_state_dict, 
+                    "optimizer": train_optimizer.state_dict(),
+                    "lr_scheduler": lr_scheduler.state_dict(),
+                    "epoch": epoch,
+                    "step": it}, 
+                    args.checkpoint_floder + '/dlrm_training_' + str(dist.get_rank()) + '_save.pt')
+        except StopIteration:
+            # Dataset traversal complete
+            break
+
+    logger.info(f"Total number of iterations: {it - 1}")
+
+
+@dataclass
+class TrainValTestResults:
+    val_aurocs: List[float] = field(default_factory=list)
+    test_auroc: Optional[float] = None
+
+
+def train_val_test(
+    args: argparse.Namespace,
+    model: torch.nn.Module,
+    optimizer: torch.optim.Optimizer,
+    device: torch.device,
+    train_dataloader: DataLoader,
+    val_dataloader: DataLoader,
+    test_dataloader: DataLoader,
+    lr_scheduler: LRPolicyScheduler,
+) -> TrainValTestResults:
+    """
+    Train/validation/test loop.
+
+    Args:
+        args (argparse.Namespace): parsed command line args.
+        model (torch.nn.Module): model to train.
+        optimizer (torch.optim.Optimizer): optimizer to use.
+        device (torch.device): device to use.
+        train_dataloader (DataLoader): Training set's dataloader.
+        val_dataloader (DataLoader): Validation set's dataloader.
+        test_dataloader (DataLoader): Test set's dataloader.
+        lr_scheduler (LRPolicyScheduler): Learning rate scheduler.
+
+    Returns:
+        TrainValTestResults.
+    """
+    results = TrainValTestResults()
+    if args.fp16 and (not args.amp):
+        model.to(torch.float16)
+    if args.bf16 and (not args.amp):
+        model.to(torch.bfloat16)
+
+    start_epoch = 0
+    if args.converge:
+        start_epoch = args.epoch
+    if not args.inference_only:
+        for epoch in range(start_epoch, args.epochs):
+            _train(
+                model,
+                optimizer,
+                train_dataloader,
+                val_dataloader,
+                epoch,
+                lr_scheduler,
+                args.print_lr,
+                args.validation_freq_within_epoch,
+                args.limit_train_batches,
+                args.limit_val_batches,
+                args.log_freq,
+                device,
+                args
+            )
+            if args.train_with_val:
+                val_auroc = _evaluate(
+                    args.limit_val_batches, model.model, val_dataloader, "val", args.log_freq, device, args
+                )
+                results.val_aurocs.append(val_auroc)
+        if args.train_with_val:
+            test_auroc = _evaluate(
+                args.limit_test_batches, model.model, test_dataloader, "test", args.log_freq, device, args
+            )
+
+    else:
+        test_auroc = _evaluate(
+            args.limit_test_batches, model.model, test_dataloader, "test", args.log_freq, device, args
+        )
+        results.test_auroc = test_auroc
+
+    return results
+
+
+def main(argv: List[str]) -> None:
+    """
+    Trains, validates, and tests a Deep Learning Recommendation Model (DLRM)
+    (https://arxiv.org/abs/1906.00091). The DLRM model contains both data parallel
+    components (e.g. multi-layer perceptrons & interaction arch) and model parallel
+    components (e.g. embedding tables). The DLRM model is pipelined so that dataloading,
+    data-parallel to model-parallel comms, and forward/backward are overlapped. Can be
+    run with either a random dataloader or an in-memory Criteo 1 TB click logs dataset
+    (https://ailab.criteo.com/download-criteo-1tb-click-logs-dataset/).
+
+    Args:
+        argv (List[str]): command line args.
+
+    Returns:
+        None.
+    """
+    args = parse_args(argv)
+    args.amp_dtype = torch.float32
+    if args.bf16:
+        args.amp_dtype = torch.bfloat16
+    if args.fp16:
+        args.amp_dtype = torch.float16
+    for name, val in vars(args).items():
+        try:
+            vars(args)[name] = list(map(int, val.split(",")))
+        except (ValueError, AttributeError):
+            pass
+    args.enable_profiling = False
+    if os.getenv('PROFILE') == '1': args.enable_profiling=True
+
+    if args.use_xpu:
+        import intel_extension_for_pytorch
+        try:
+            import oneccl_bindings_for_pytorch
+        except ImportError as e:
+            pass
+
+    if not args.use_xpu:
+        torch.backends.cuda.matmul.allow_tf32 = args.allow_tf32
+
+    if args.multi_hot_sizes is not None and args.trans_data:
+        assert (
+            args.num_embeddings_per_feature is not None
+            and len(args.multi_hot_sizes) == len(args.num_embeddings_per_feature)
+            or args.num_embeddings_per_feature is None
+            and len(args.multi_hot_sizes) == len(DEFAULT_CAT_NAMES)
+        ), "--multi_hot_sizes must be a comma delimited list the same size as the number of embedding tables."
+    assert (
+        args.in_memory_binary_criteo_path is None
+        or args.synthetic_multi_hot_criteo_path is None
+    ), "--in_memory_binary_criteo_path and --synthetic_multi_hot_criteo_path are mutually exclusive CLI arguments."
+    assert (
+        args.multi_hot_sizes is None or (not args.trans_data) or args.synthetic_multi_hot_criteo_path is None
+    ), "--multi_hot_sizes is used to convert 1-hot to multi-hot. It's inapplicable with --synthetic_multi_hot_criteo_path."
+    assert (
+        args.multi_hot_distribution_type is None
+        or args.synthetic_multi_hot_criteo_path is None
+    ), "--multi_hot_distribution_type is used to convert 1-hot to multi-hot. It's inapplicable with --synthetic_multi_hot_criteo_path."
+
+    np.random.seed(args.numpy_rand_seed)
+    torch.manual_seed(args.numpy_rand_seed)
+    #rank = int(os.environ["LOCAL_RANK"])
+    rank = env2int(
+        [
+            "MPI_LOCALRANKID",
+            "OMPI_COMM_WORLD_LOCAL_RANK",
+            "MV2_COMM_WORLD_LOCAL_RANK",
+            "LOCAL_RANK",
+        ],
+        0,
+    )
+    args.use_device = "cpu"
+    if args.use_xpu and torch.xpu.is_available():
+        torch.xpu.manual_seed_all(args.numpy_rand_seed)
+        device: torch.device = torch.device("xpu", rank)
+        backend = "ccl"
+        torch.xpu.set_device(device)
+        args.use_device = "xpu"
+    elif torch.cuda.is_available():
+        torch.cuda.manual_seed_all(args.numpy_rand_seed)
+        device: torch.device = torch.device(f"cuda:{rank}")
+        backend = "nccl"
+        torch.cuda.set_device(device)
+        args.use_device = "cuda"
+    else:
+        device: torch.device = torch.device("cpu")
+        backend = "gloo"
+
+    if rank == 0:
+        print(
+            "PARAMS: (lr, batch_size, warmup_steps, decay_start, decay_steps): "
+            f"{(args.learning_rate, args.batch_size, args.lr_warmup_steps, args.lr_decay_start, args.lr_decay_steps)}"
+        )
+    ext_dist.init_distributed(
+        local_rank=-1, use_device=args.use_device, backend=backend
+    )
+
+    if args.num_embeddings_per_feature is not None:
+        args.num_embeddings = None
+
+    # Sets default limits for random dataloader iterations when left unspecified.
+    if (
+        args.in_memory_binary_criteo_path
+        is args.synthetic_multi_hot_criteo_path
+        is None
+    ):
+        for split in ["train", "val", "test"]:
+            attr = f"limit_{split}_batches"
+            if getattr(args, attr) is None:
+                setattr(args, attr, 10)
+
+    train_dataloader = get_dataloader(args, backend, "train")
+    val_dataloader = get_dataloader(args, backend, "val")
+    test_dataloader = get_dataloader(args, backend, "test")
+    # use dist_models, if not you can comment out
+    global n_emb, local_emb_indices, default_embedding_names, idx_name
+    n_emb = len(DEFAULT_CAT_NAMES)
+    #if dist.get_world_size() == 8 and dist.get_rank() == 6:
+    #    local_emb_indices[1] = 25
+    #if dist.get_world_size() == 8 and dist.get_rank() == 7:
+    #    local_emb_indices[2] = 21
+
+    shard_matrix_str, shard_strategy_str, shard_matrix, shard_strategy = sharding.generate_plan(
+    args.num_embeddings_per_feature, args.multi_hot_sizes, args.num_nodes, dist.get_world_size(), args, dist.get_rank() == 0)
+    #local_emb_slice = ext_dist.get_my_slice(n_emb)
+    local_emb_indices = shard_matrix[rank]
+    default_embedding_names = [DEFAULT_CAT_NAMES[i] for i in local_emb_indices]
+    idx_name = zip(local_emb_indices, default_embedding_names)
+
+    eb_configs = [
+        EmbeddingBagConfig(
+            name=f"t_{feature_name}",
+            embedding_dim=args.embedding_dim,
+            num_embeddings=none_throws(args.num_embeddings_per_feature)[feature_idx]
+            if args.num_embeddings is None
+            else args.num_embeddings,
+            feature_names=[feature_name],
+        )
+        #for feature_idx, feature_name in enumerate(DEFAULT_CAT_NAMES)
+        for feature_idx, feature_name in idx_name
+    ]
+    sharded_module_kwargs = {}
+    if args.over_arch_layer_sizes is not None:
+        sharded_module_kwargs["over_arch_layer_sizes"] = args.over_arch_layer_sizes
+
+    if args.interaction_type == InteractionType.ORIGINAL:
+        dlrm_model = DLRM(
+            embedding_bag_collection=EmbeddingBagCollection(
+                tables=eb_configs, device=torch.device("meta")
+            ),
+            dense_in_features=len(DEFAULT_INT_NAMES),
+            dense_arch_layer_sizes=args.dense_arch_layer_sizes,
+            over_arch_layer_sizes=args.over_arch_layer_sizes,
+            dense_device=device,
+        )
+    elif args.interaction_type == InteractionType.DCN:
+        #dlrm_model = DLRM_DCN(
+        dlrm_model = DIST_DLRM_DCN(
+            embedding_bag_collection=EmbeddingBagCollection(
+                #tables=eb_configs, device=torch.device("meta")
+                tables=eb_configs, device="cpu"
+            ),
+            dense_in_features=len(DEFAULT_INT_NAMES),
+            num_total_sparse_features=n_emb,
+            dense_arch_layer_sizes=args.dense_arch_layer_sizes,
+            over_arch_layer_sizes=args.over_arch_layer_sizes,
+            dcn_num_layers=args.dcn_num_layers,
+            dcn_low_rank_dim=args.dcn_low_rank_dim,
+            n_emb=n_emb,
+            shard_matrix=shard_matrix, 
+            shard_strategy = shard_strategy,
+            args=args,
+            dense_device="cpu",
+        )
+    elif args.interaction_type == InteractionType.PROJECTION:
+        dlrm_model = DLRM_Projection(
+            embedding_bag_collection=EmbeddingBagCollection(
+                tables=eb_configs, device=torch.device("meta")
+            ),
+            dense_in_features=len(DEFAULT_INT_NAMES),
+            dense_arch_layer_sizes=args.dense_arch_layer_sizes,
+            over_arch_layer_sizes=args.over_arch_layer_sizes,
+            interaction_branch1_layer_sizes=args.interaction_branch1_layer_sizes,
+            interaction_branch2_layer_sizes=args.interaction_branch2_layer_sizes,
+            dense_device=device,
+        )
+    else:
+        raise ValueError(
+            "Unknown interaction option set. Should be original, dcn, or projection."
+        )
+
+    for embedding_bag in dlrm_model.sparse_arch.embedding_bag_collection.embedding_bags.values():
+        _W = np.random.uniform(
+            low=-np.sqrt(1 / embedding_bag.num_embeddings),
+            high=np.sqrt(1 / embedding_bag.num_embeddings),
+            size=(embedding_bag.num_embeddings, embedding_bag.embedding_dim)).astype(np.float32)
+        embedding_bag.weight.data = torch.tensor(_W, requires_grad=True)
+
+    if not args.amp:
+        dlrm_model.to(args.amp_dtype)
+    train_model = DLRMTrain(dlrm_model)
+    if args.inference_only:
+        assert args.snapshot_dir
+        load_snapshot(train_model, args.snapshot_dir)
+
+
+    # This will apply the Adagrad optimizer in the backward pass for the embeddings (sparse_arch). This means that
+    # the optimizer update will be applied in the backward pass, in this case through a fused op.
+    # TorchRec will use the FBGEMM implementation of EXACT_ADAGRAD. For GPU devices, a fused CUDA kernel is invoked. For CPU, FBGEMM_GPU invokes CPU kernels
+    # https://github.com/pytorch/FBGEMM/blob/2cb8b0dff3e67f9a009c4299defbd6b99cc12b8f/fbgemm_gpu/fbgemm_gpu/split_table_batched_embeddings_ops.py#L676-L678
+
+    # Note that lr_decay, weight_decay and initial_accumulator_value for Adagrad optimizer in FBGEMM v0.3.2
+    # cannot be specified below. This equivalently means that all these parameters are hardcoded to zero.
+    optimizer_kwargs = {"lr": args.learning_rate}
+    if args.adagrad:
+        optimizer_kwargs["eps"] = args.eps
+
+    model = train_model
+    model.model.sparse_arch = SparseArchTraceAbleWrapper(model.model.sparse_arch, args.dense)
+    checkpoint = None
+    args.epoch = 0
+    args.step = 0
+    if args.converge:
+        checkpoint = load_checkoutpoint(args, device)
+        if checkpoint is not None:
+            args.epoch = checkpoint['epoch']
+            args.step = checkpoint['step']
+            print('[info] loadind model state dict...')
+            train_model.load_state_dict(checkpoint['model'])
+        logger.info(f"The start epoch is : {args.epoch}")
+        logger.info(f"The start step is : {args.step}")
+
+
+    optimizer = None
+    if not args.inference_only:
+        if args.adagrad:
+            optimizer = torch.optim.Adagrad(model.parameters(), lr=args.learning_rate, eps=args.eps)
+        else:
+            optimizer = torch.optim.SGD(model.parameters(), lr=args.learning_rate)
+        if args.converge and checkpoint is not None:
+            print('[info] loadind optimizer state dict...')
+            optimizer.load_state_dict(checkpoint['optimizer'])
+
+    if ext_dist.my_size > 1:
+        if args.use_device == "cuda":
+            device_ids = [rank]
+            model.to(device)
+            model.model.dense_arch = ext_dist.DDP(model.model.dense_arch, device_ids=device_ids)
+            model.model.inter_arch = ext_dist.DDP(model.model.inter_arch, device_ids=device_ids)
+            model.model.over_arch = ext_dist.DDP(model.model.over_arch, device_ids=device_ids)
+            if not args.inference_only:
+                for k, v in optimizer.state.items():
+                    for name, value in v.items():
+                        if isinstance(value, torch.Tensor):
+                            optimizer.state[k][name] = value.to(device)
+        elif args.use_device == "xpu":
+            device_ids = [device]
+            model.to(device)
+            model.model.dense_arch = ext_dist.DDP(model.model.dense_arch, device_ids=device_ids)
+            model.model.inter_arch = ext_dist.DDP(model.model.inter_arch, device_ids=device_ids)
+            model.model.over_arch = ext_dist.DDP(model.model.over_arch, device_ids=device_ids)
+            if not args.inference_only:
+                for k, v in optimizer.state.items():
+                    for name, value in v.items():
+                        if isinstance(value, torch.Tensor):
+                            optimizer.state[k][name] = value.to(device)
+        else:
+            model.model.dense_arch = ext_dist.DDP(model.model.dense_arch)
+            model.model.inter_arch = ext_dist.DDP(model.model.inter_arch)
+            model.model.over_arch = ext_dist.DDP(model.model.over_arch)
+
+    lr_scheduler = None
+    if not args.inference_only:
+        lr_scheduler = LRPolicyScheduler(
+            optimizer, args.lr_warmup_steps, args.lr_decay_start, args.lr_decay_steps
+        )
+        if args.converge and checkpoint is not None:
+            print('[info] loadind lr_scheduler state dict...')
+            lr_scheduler.load_state_dict(checkpoint['lr_scheduler'])
+        model.train()
+        if args.use_device == "xpu":
+            model, optimizer = torch.xpu.optimize(model=model, optimizer=optimizer, level="O1",
+                                                  dtype=torch.float32)
+    else:
+        model.eval()
+        if args.use_device == "xpu":
+            model = torch.xpu.optimize(model=model, dtype=args.amp_dtype)
+
+    if args.trans_data and (args.multi_hot_sizes is not None):
+        multihot = Multihot(
+            args.multi_hot_sizes,
+            args.num_embeddings_per_feature,
+            args.batch_size,
+            collect_freqs_stats=args.collect_multi_hot_freqs_stats,
+            dist_type=args.multi_hot_distribution_type,
+        )
+        multihot.pause_stats_collection_during_val_and_test(model)
+        train_dataloader = RestartableMap(
+            multihot.convert_to_multi_hot, train_dataloader
+        )
+        val_dataloader = RestartableMap(multihot.convert_to_multi_hot, val_dataloader)
+        test_dataloader = RestartableMap(multihot.convert_to_multi_hot, test_dataloader)
+    train_val_test(
+        args,
+        model,
+        optimizer,
+        device,
+        train_dataloader,
+        val_dataloader,
+        test_dataloader,
+        lr_scheduler,
+    )
+    if args.collect_multi_hot_freqs_stats:
+        multihot.save_freqs_stats()
+
+
+if __name__ == "__main__":
+    main(sys.argv[1:])
diff --git a/models_v2/pytorch/torchrec_dlrm/training/gpu/extend_distributed.py b/models_v2/pytorch/torchrec_dlrm/training/gpu/extend_distributed.py
new file mode 100644
index 000000000..e54afc506
--- /dev/null
+++ b/models_v2/pytorch/torchrec_dlrm/training/gpu/extend_distributed.py
@@ -0,0 +1,678 @@
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+import builtins
+import os
+import sys
+
+import torch
+import torch.distributed as dist
+from torch.autograd import Function
+from torch.autograd.profiler import record_function
+from torch.nn.parallel import DistributedDataParallel as DDP
+
+try:
+    import intel_extension_for_pytorch
+except ImportError as e:
+    pass
+
+try:
+    if torch.__version__[:6] >= '1.12.0':
+        import oneccl_bindings_for_pytorch
+        torch_ccl = False
+    else:
+        import torch_ccl
+except ImportError as e:
+    # print(e)
+    torch_ccl = False
+
+try:
+    import torch_ucc
+except ImportError as e:
+    torch_ucc = False
+
+
+my_rank = -1
+my_size = -1
+my_local_rank = -1
+my_local_size = -1
+alltoall_supported = False
+a2a_impl = os.environ.get("DLRM_ALLTOALL_IMPL", "")
+
+myreq = None
+
+
+def env2int(env_list, default=-1):
+    for e in env_list:
+        val = int(os.environ.get(e, -1))
+        if val >= 0:
+            return val
+    return default
+
+
+def get_my_slice(n):
+    tmp_size = dist.get_world_size()
+    tmp_rank = dist.get_rank()
+
+    k, m = divmod(n, tmp_size)
+    return slice(
+        tmp_rank * k + min(tmp_rank, m), (tmp_rank + 1) * k + min(tmp_rank + 1, m), 1
+    )
+
+
+def get_split_lengths(n):
+    if my_size >1:
+        tmp_size = dist.get_world_size()
+        tmp_rank = dist.get_rank()
+    else:
+        tmp_size = my_size
+        tmp_rank = my_rank
+
+    k, m = divmod(n, tmp_size)
+    if m == 0:
+        splits = None
+        my_len = k
+    else:
+        splits = [(k + 1) if i < m else k for i in range(tmp_size)]
+        my_len = splits[tmp_rank]
+    return (my_len, splits)
+
+
+def init_distributed(rank=-1, local_rank=-1, size=-1, use_device="cpu", backend=""):
+    global myreq
+    global my_size
+    global my_local_rank
+    global my_local_size
+    global a2a_impl
+    global alltoall_supported
+
+    # guess MPI ranks from env (works for IMPI, OMPI and MVAPICH2)
+    num_mpi_ranks = env2int(
+        ["PMI_SIZE", "OMPI_COMM_WORLD_SIZE", "MV2_COMM_WORLD_SIZE", "WORLD_SIZE"]
+    )
+    if backend == "" and num_mpi_ranks > 1:
+        if torch_ccl or oneccl_bindings_for_pytorch:
+            backend = "ccl"
+        elif use_device=="cuda" and dist.is_nccl_available():
+            backend = "nccl"
+        elif dist.is_mpi_available():
+            backend = "mpi"
+        else:
+            print(
+                "WARNING: MPI multi-process launch detected but PyTorch MPI backend not available."
+            )
+            backend = "gloo"
+
+    if backend != "":
+        # guess Rank and size
+        if rank == -1:
+            rank = env2int(
+                ["PMI_RANK", "OMPI_COMM_WORLD_RANK", "MV2_COMM_WORLD_RANK", "RANK"], 0
+            )
+        if size == -1:
+            size = env2int(
+                [
+                    "PMI_SIZE",
+                    "OMPI_COMM_WORLD_SIZE",
+                    "MV2_COMM_WORLD_SIZE",
+                    "WORLD_SIZE",
+                ],
+                1,
+            )
+        if not os.environ.get("RANK", None) and rank != -1:
+            os.environ["RANK"] = str(rank)
+        if not os.environ.get("WORLD_SIZE", None) and size != -1:
+            os.environ["WORLD_SIZE"] = str(size)
+        if not os.environ.get("MASTER_PORT", None):
+            os.environ["MASTER_PORT"] = "29500"
+        if not os.environ.get("MASTER_ADDR", None):
+            local_size = env2int(
+                [
+                    "MPI_LOCALNRANKS",
+                    "OMPI_COMM_WORLD_LOCAL_SIZE",
+                    "MV2_COMM_WORLD_LOCAL_SIZE",
+                ],
+                1,
+            )
+            if local_size != size and backend != "mpi":
+                print(
+                    "Warning: Looks like distributed multinode run but MASTER_ADDR env not set, using '127.0.0.1' as default"
+                )
+                print(
+                    "If this run hangs, try exporting rank 0's hostname as MASTER_ADDR"
+                )
+            os.environ["MASTER_ADDR"] = "127.0.0.1"
+
+    if size > 1:
+        if local_rank == -1:
+            my_local_rank = env2int(
+                [
+                    "MPI_LOCALRANKID",
+                    "OMPI_COMM_WORLD_LOCAL_RANK",
+                    "MV2_COMM_WORLD_LOCAL_RANK",
+                    "LOCAL_RANK",
+                ],
+                0,
+            )
+        else:
+            my_local_rank = local_rank
+        my_local_size = env2int(
+            [
+                "MPI_LOCALNRANKS",
+                "OMPI_COMM_WORLD_LOCAL_SIZE",
+                "MV2_COMM_WORLD_LOCAL_SIZE",
+            ],
+            1,
+        )
+        if use_device=="cuda":
+            if my_local_size > torch.cuda.device_count():
+                print(
+                    "Not sufficient GPUs available... local_size = %d, ngpus = %d"
+                    % (my_local_size, torch.cuda.device_count())
+                )
+                sys.exit(1)
+            torch.cuda.set_device(my_local_rank)
+        elif use_device=="xpu":
+            if my_local_size > torch.xpu.device_count():
+                print(
+                    "Not sufficient GPUs available... local_size = %d, ngpus = %d"
+                    % (my_local_size, torch.xpu.device_count())
+                )
+                sys.exit(1)
+            torch.xpu.set_device(my_local_rank)
+        dist.init_process_group(backend, rank=rank, world_size=size)
+        this_rank = dist.get_rank()
+        my_size = dist.get_world_size()
+        #my_local_rank = env2int(['MPI_LOCALRANKID', 'OMPI_COMM_WORLD_LOCAL_RANK', 'MV2_COMM_WORLD_LOCAL_RANK'], 0)
+        #my_local_size = env2int(['MPI_LOCALNRANKS', 'OMPI_COMM_WORLD_LOCAL_SIZE', 'MV2_COMM_WORLD_LOCAL_SIZE'], 1)
+        if this_rank == 0:
+            print("Running on %d ranks using %s backend" % (my_size, backend))
+        if hasattr(dist, "all_to_all_single"):
+            try:
+                if use_device == "xpu":
+                    test_device: torch.device = torch.device("xpu", dist.get_rank())
+                elif use_device == "cuda":
+                    test_device: torch.device = torch.device(f"cuda:{dist.get_rank()}")
+                else:
+                    test_device: torch.device = torch.device(f"cpu:{dist.get_rank()}")
+
+                a = torch.rand([1, 3]).to(test_device)
+                dist.broadcast(a, src=0)
+                print("Successfully execute distributed broadcast on the xpu:{}".format(this_rank))
+
+                tensor_list = [torch.zeros(2, dtype=torch.int64).to(test_device) for _ in range(my_size)]
+                tensor = (torch.arange(2, dtype=torch.int64) + 1 + 2 * rank).to(test_device)
+                dist.all_gather(tensor_list, tensor)
+                print("Successfully execute distributed all_gather on the xpu:{}".format(this_rank))
+
+                t = torch.zeros([4*my_size])
+                if use_device == "cuda":
+                    t = t.cuda()
+                elif  use_device == "xpu":
+                    t = t.xpu()
+                dist.all_to_all_single(t, t)
+
+                print("Successfully execute distributed all_to_all_single on the xpu:{}".format(this_rank))
+                alltoall_supported = True
+            except RuntimeError as err:
+                print("fail to enable all_to_all_single primitive: %s" % err)
+                exit()
+        if a2a_impl == "alltoall" and alltoall_supported == False:
+            print(
+                "Requested DLRM_ALLTOALL_IMPL=%s but backend %s does not support it, use scatter/gather based alltoall"
+                % (a2a_impl, backend)
+            )
+            a2a_impl = "scatter"
+        if a2a_impl != "":
+            print("Using DLRM_ALLTOALL_IMPL=%s" % a2a_impl)
+    else:
+        this_rank = 0
+        my_size = 1
+        my_local_rank = 0
+        my_local_size = 1
+    print_all(
+        "world size: %d, current rank: %d, local rank: %d"
+        % (my_size, this_rank, my_local_rank)
+    )
+    myreq = Request()
+
+
+class Request(object):
+    def __init__(self):
+        self.req = None
+        self.tensor = None
+        self.WaitFunction = All2All_Scatter_Wait
+
+    def wait(self):
+        ret = self.WaitFunction.apply(*self.tensor)
+        self.req = None
+        self.tensor = None
+        return ret
+
+
+class All2All_ScatterList_Req(Function):
+    @staticmethod
+    def forward(ctx, a2a_info, *inputs):
+        global myreq
+        tmp_size = dist.get_world_size()
+        tmp_rank = dist.get_rank()
+        batch_split_lengths = (
+            a2a_info.global_batch_partition_slices
+            if a2a_info.global_batch_partition_slices
+            else a2a_info.local_batch_num
+        )
+        table_split_lengths = (
+            a2a_info.global_table_wise_parition_slices
+            if a2a_info.global_table_wise_parition_slices
+            else [a2a_info.local_table_num] * tmp_size
+        )
+        gather_list = []
+        req_list = []
+        for i in range(tmp_size):
+            for j in range(table_split_lengths[i]):
+                out_tensor = inputs[0].new_empty(
+                    [a2a_info.local_batch_num, a2a_info.emb_dim]
+                )
+                scatter_list = (
+                    list(inputs[j].split(batch_split_lengths, dim=0))
+                    if i == tmp_rank
+                    else []
+                )
+                req = dist.scatter(out_tensor, scatter_list, src=i, async_op=True)
+                gather_list.append(out_tensor)
+                req_list.append(req)
+        myreq.req = req_list
+        myreq.tensor = tuple(gather_list)
+        myreq.a2a_info = a2a_info
+        return myreq.tensor
+
+    @staticmethod
+    def backward(ctx, *grad_output):
+        global myreq
+        for r in myreq.req:
+            r.wait()
+        myreq.req = None
+        grad_inputs = myreq.tensor
+        myreq.tensor = None
+        return (None, *grad_inputs)
+
+
+class All2All_ScatterList_Wait(Function):
+    @staticmethod
+    def forward(ctx, *output):
+        global myreq
+        ctx.a2a_info = myreq.a2a_info
+        for r in myreq.req:
+            r.wait()
+        myreq.req = None
+        myreq.tensor = None
+        return output
+
+    @staticmethod
+    def backward(ctx, *grad_output):
+        global myreq
+        tmp_size = dist.get_world_size()
+        tmp_rank = dist.get_rank()
+        a2a_info = ctx.a2a_info
+        grad_output = [t.contiguous() for t in grad_output]
+        batch_split_lengths = (
+            a2a_info.global_batch_partition_slices
+            if a2a_info.global_batch_partition_slices
+            else [a2a_info.local_batch_num] * tmp_size
+        )
+        per_rank_table_splits = (
+            a2a_info.global_table_wise_parition_slices
+            if a2a_info.global_table_wise_parition_slices
+            else [a2a_info.local_table_num] * tmp_size
+        )
+        grad_inputs = [
+            grad_output[0].new_empty([ctx.a2a_info.batch_size, ctx.a2a_info.emb_dim])
+            for _ in range(a2a_info.local_table_num)
+        ]
+        req_list = []
+        ind = 0
+        for i in range(tmp_size):
+            for j in range(per_rank_table_splits[i]):
+                gather_list = (
+                    list(grad_inputs[j].split(batch_split_lengths, dim=0))
+                    if i == tmp_rank
+                    else None
+                )
+                req = dist.gather(grad_output[ind], gather_list, dst=i, async_op=True)
+                req_list.append(req)
+                ind += 1
+        myreq.req = req_list
+        myreq.tensor = grad_inputs
+        return tuple(grad_output)
+
+
+class All2All_Scatter_Req(Function):
+    @staticmethod
+    def forward(ctx, a2a_info, *inputs):
+        tmp_size = dist.get_world_size()
+        tmp_rank = dist.get_rank()
+        global myreq
+        batch_split_lengths = (
+            a2a_info.global_batch_partition_slices
+            if a2a_info.global_batch_partition_slices
+            else a2a_info.local_batch_num
+        )
+        table_split_lengths = (
+            a2a_info.global_table_wise_parition_slices
+            if a2a_info.global_table_wise_parition_slices
+            else [a2a_info.local_table_num] * tmp_size
+        )
+        input = torch.cat(inputs, dim=1)
+        scatter_list = list(input.split(batch_split_lengths, dim=0))
+        gather_list = []
+        req_list = []
+        for i in range(tmp_size):
+            out_tensor = input.new_empty(
+                [a2a_info.local_batch_num, table_split_lengths[i] * a2a_info.emb_dim]
+            )
+            req = dist.scatter(
+                out_tensor, scatter_list if i == tmp_rank else [], src=i, async_op=True
+            )
+            gather_list.append(out_tensor)
+            req_list.append(req)
+        myreq.req = req_list
+        myreq.tensor = tuple(gather_list)
+        myreq.a2a_info = a2a_info
+        ctx.a2a_info = a2a_info
+        return myreq.tensor
+
+    @staticmethod
+    def backward(ctx, *grad_output):
+        global myreq
+        for r in myreq.req:
+            r.wait()
+        myreq.req = None
+        grad_input = myreq.tensor
+        grad_inputs = grad_input.split(ctx.a2a_info.emb_dim, dim=1)
+        myreq.tensor = None
+        return (None, *grad_inputs)
+
+
+class All2All_Scatter_Wait(Function):
+    @staticmethod
+    def forward(ctx, *output):
+        global myreq
+        ctx.a2a_info = myreq.a2a_info
+        for r in myreq.req:
+            r.wait()
+        myreq.req = None
+        myreq.tensor = None
+        return output
+
+    @staticmethod
+    def backward(ctx, *grad_output):
+        tmp_size = dist.get_world_size()
+        tmp_rank = dist.get_rank()
+        global myreq
+        assert len(grad_output) == tmp_size
+        scatter_list = [t.contiguous() for t in grad_output]
+        a2a_info = ctx.a2a_info
+        batch_split_lengths = (
+            a2a_info.global_batch_partition_slices
+            if a2a_info.global_batch_partition_slices
+            else a2a_info.local_batch_num
+        )
+        table_split_lengths = (
+            a2a_info.global_table_wise_parition_slices
+            if a2a_info.global_table_wise_parition_slices
+            else [a2a_info.local_table_num] * tmp_size
+        )
+        grad_input = grad_output[0].new_empty(
+            [a2a_info.batch_size, a2a_info.emb_dim * a2a_info.local_table_num]
+        )
+        gather_list = list(grad_input.split(batch_split_lengths, dim=0))
+        req_list = []
+        for i in range(tmp_size):
+            req = dist.gather(
+                scatter_list[i],
+                gather_list if i == tmp_rank else [],
+                dst=i,
+                async_op=True,
+            )
+            req_list.append(req)
+        myreq.req = req_list
+        myreq.tensor = grad_input
+        return grad_output
+
+
+class All2All_Req(Function):
+    @staticmethod
+    def forward(ctx, a2a_info, *inputs):
+        global myreq
+        with record_function("DLRM alltoall_req_fwd_single"):
+            batch_split_lengths = a2a_info.global_batch_partition_slices
+            if batch_split_lengths:
+                batch_split_lengths = [
+                    m * a2a_info.emb_dim * a2a_info.local_table_num
+                    for m in batch_split_lengths
+                ]
+            table_split_lengths = a2a_info.global_table_wise_parition_slices
+            if table_split_lengths:
+                table_split_lengths = [
+                    a2a_info.local_batch_num * e * a2a_info.emb_dim
+                    for e in table_split_lengths
+                ]
+            input = torch.cat(inputs, dim=1).view([-1])
+            output = input.new_empty(
+                [
+                    a2a_info.global_table_num
+                    * a2a_info.local_batch_num
+                    * a2a_info.emb_dim
+                ]
+            )
+            req = dist.all_to_all_single(
+                output, input, table_split_lengths, batch_split_lengths, async_op=True
+            )
+
+            myreq.req = req
+            myreq.tensor = []
+            myreq.tensor.append(output)
+            myreq.tensor = tuple(myreq.tensor)
+            a2a_info.batch_split_lengths = batch_split_lengths
+            a2a_info.table_split_lengths = table_split_lengths
+            myreq.a2a_info = a2a_info
+            ctx.a2a_info = a2a_info
+            return myreq.tensor
+
+    @staticmethod
+    def backward(ctx, *grad_output):
+        global myreq
+        with record_function("DLRM alltoall_req_bwd_single"):
+            a2a_info = ctx.a2a_info
+            myreq.req.wait()
+            myreq.req = None
+            grad_input = myreq.tensor
+            grad_inputs = grad_input.view([a2a_info.batch_size, -1]).split(
+                a2a_info.emb_dim, dim=1
+            )
+            grad_inputs = [gin.contiguous() for gin in grad_inputs]
+            myreq.tensor = None
+            return (None, *grad_inputs)
+
+
+class All2All_Wait(Function):
+    @staticmethod
+    def forward(ctx, *output):
+        global myreq
+        with record_function("DLRM alltoall_wait_fwd_single"):
+            a2a_info = myreq.a2a_info
+            ctx.a2a_info = a2a_info
+            myreq.req.wait()
+            myreq.req = None
+            myreq.tensor = None
+            table_split_lengths = (
+                a2a_info.table_split_lengths
+                if a2a_info.table_split_lengths
+                else a2a_info.local_table_num
+                * a2a_info.local_batch_num
+                * a2a_info.emb_dim
+            )
+            outputs = output[0].split(table_split_lengths)
+            outputs = tuple(
+                [out.view([a2a_info.local_batch_num, -1]) for out in outputs]
+            )
+            return outputs
+
+    @staticmethod
+    def backward(ctx, *grad_outputs):
+        global myreq
+        with record_function("DLRM alltoall_wait_bwd_single"):
+            a2a_info = ctx.a2a_info
+            grad_outputs = [gout.contiguous().view([-1]) for gout in grad_outputs]
+            grad_output = torch.cat(grad_outputs)
+            grad_input = grad_output.new_empty(
+                [a2a_info.batch_size * a2a_info.local_table_num * a2a_info.emb_dim]
+            )
+            req = dist.all_to_all_single(
+                grad_input,
+                grad_output,
+                a2a_info.batch_split_lengths,
+                a2a_info.table_split_lengths,
+                async_op=True,
+            )
+            myreq.req = req
+            myreq.tensor = grad_input
+            return (grad_output,)
+
+
+class AllGather(Function):
+    @staticmethod
+    def forward(ctx, input, global_lengths, dim=0):
+        tmp_size = dist.get_world_size()
+        tmp_rank = dist.get_rank()
+        if not isinstance(global_lengths, (list, tuple)):
+            global_lengths = [global_lengths] * tmp_size
+
+        assert len(global_lengths) == tmp_size
+        assert global_lengths[tmp_rank] == input.size(dim)
+        local_start = sum(global_lengths[:tmp_rank])
+
+        output_size = list(input.size())
+
+        ctx.dim = dim
+        ctx.local_start = local_start
+        ctx.local_length = global_lengths[tmp_rank]
+
+        input = input.contiguous()
+        if dim == 0:
+            out_len = sum(global_lengths)
+            output_size[dim] = out_len
+            output = input.new_empty(output_size)
+            gather_list = list(output.split(global_lengths, dim=0))
+        else:
+            gather_list = [torch.empty_like(input) for _ in range(tmp_size)]
+            gather_list = []
+            for length in global_lengths:
+                output_size[dim] = length
+                gather_list.append(input.new_empty(output_size))
+
+        dist.all_gather(gather_list, input)
+
+        if dim != 0:
+            output = torch.cat(gather_list, dim=dim)
+
+        return output
+
+    @staticmethod
+    def backward(ctx, grad_output):
+        # print("Inside All2AllBackward")
+        dim = ctx.dim
+        start = ctx.local_start
+        length = ctx.local_length
+
+        grad_input = grad_output.narrow(dim, start, length)
+
+        return (grad_input, None, None)
+
+
+class All2AllInfo(object):
+    pass
+
+
+def alltoall(inputs, per_rank_table_splits):
+    tmp_size = dist.get_world_size()
+    global myreq
+    batch_size, emb_dim = inputs[0].size()
+    a2a_info = All2AllInfo()
+    a2a_info.local_table_num = len(inputs)
+    a2a_info.global_table_wise_parition_slices = per_rank_table_splits
+    (
+        a2a_info.local_batch_num,
+        a2a_info.global_batch_partition_slices,
+    ) = get_split_lengths(batch_size)
+    a2a_info.emb_dim = emb_dim
+    a2a_info.batch_size = batch_size
+    a2a_info.global_table_num = (
+        sum(per_rank_table_splits)
+        if per_rank_table_splits
+        else a2a_info.local_table_num * tmp_size
+    )
+
+    if a2a_impl == "" and alltoall_supported or a2a_impl == "alltoall":
+        # print("Using All2All_Req")
+        output = All2All_Req.apply(a2a_info, *inputs)
+        myreq.WaitFunction = All2All_Wait
+    elif a2a_impl == "" or a2a_impl == "scatter":
+        # print("Using All2All_Scatter_Req")
+        output = All2All_Scatter_Req.apply(a2a_info, *inputs)
+        myreq.WaitFunction = All2All_Scatter_Wait
+    elif a2a_impl == "scatter_list":
+        # print("Using All2All_ScatterList_Req")
+        output = All2All_ScatterList_Req.apply(a2a_info, *inputs)
+        myreq.WaitFunction = All2All_ScatterList_Wait
+    else:
+        print(
+            "Unknown value set for DLRM_ALLTOALL_IMPL (%s), "
+            "please use one of [alltoall, scatter, scatter_list]" % a2a_impl
+        )
+    return myreq
+
+
+def all_gather(input, lengths, dim=0):
+    tmp_size = dist.get_world_size()
+    if not lengths:
+        lengths = [input.size(0)] * tmp_size
+    return AllGather.apply(input, lengths, dim)
+
+
+def barrier():
+    if my_size > 1:
+        dist.barrier()
+
+
+# Override builtin print function to print only from rank 0
+orig_print = builtins.print
+#
+#
+#def rank0_print(*args, **kwargs):
+#    if my_rank <= 0 or kwargs.get("print_all", False):
+#        orig_print(*args, **kwargs)
+#
+#
+#builtins.print = rank0_print
+#
+# Allow printing from all rank with explicit print_all
+def print_all(*args, **kwargs):
+    orig_print(*args, **kwargs)
diff --git a/models_v2/pytorch/torchrec_dlrm/training/gpu/jit_trace_able_utils.py b/models_v2/pytorch/torchrec_dlrm/training/gpu/jit_trace_able_utils.py
new file mode 100644
index 000000000..197994c62
--- /dev/null
+++ b/models_v2/pytorch/torchrec_dlrm/training/gpu/jit_trace_able_utils.py
@@ -0,0 +1,69 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+import torch
+from torch import nn
+from torchrec.models.dlrm import SparseArch
+from torchrec.sparse.jagged_tensor import KeyedJaggedTensor
+from typing import List
+
+def unpack(input :KeyedJaggedTensor, default_embedding_names:[str]) -> dict:
+    output = {}
+    for k, v in input.to_dict().items():
+        if k in default_embedding_names:
+            output[k] = {}
+            output[k]['values'] = v._values.to(torch.int64)
+            output[k]['offsets'] = v._offsets.to(torch.int64)
+    return output
+
+
+class SparseArchTraceAbleWrapper(nn.Module):
+
+    def __init__(self, sparse_arch: SparseArch, dense) -> None:
+        super().__init__()
+        self.sparse_arch: SparseArch = sparse_arch
+        if not dense:
+            for embedding_bag in self.sparse_arch.embedding_bag_collection.embedding_bags.values():
+                embedding_bag.sparse = True
+
+
+    def forward(
+        self,
+        features: dict,
+    ) -> torch.Tensor:
+        """
+        Args:
+            features (dict): input tensors of sparse features.
+        Returns:
+            torch.Tensor: tensor of shape B X F X D.
+        """
+        embedding_bag_collection = self.sparse_arch.embedding_bag_collection
+        pooled_embeddings: List[torch.Tensor] = []
+        for i, embedding_bag in enumerate(embedding_bag_collection.embedding_bags.values()):
+            for feature_name in embedding_bag_collection._feature_names[i]:
+                f = features[feature_name]
+                res = embedding_bag(
+                    f['values'],
+                    f['offsets'],
+                    per_sample_weights=None,
+                )
+                pooled_embeddings.append(res)
+        #data = torch.cat(pooled_embeddings, dim=1)
+        #B: int = features['cat_0']['offsets'].numel() - 1
+        #return data.reshape(B, self.sparse_arch.F, self.sparse_arch.D)
+        return pooled_embeddings
diff --git a/models_v2/pytorch/torchrec_dlrm/training/gpu/lr_scheduler.py b/models_v2/pytorch/torchrec_dlrm/training/gpu/lr_scheduler.py
new file mode 100644
index 000000000..4aebce416
--- /dev/null
+++ b/models_v2/pytorch/torchrec_dlrm/training/gpu/lr_scheduler.py
@@ -0,0 +1,48 @@
+#!/usr/bin/env python3
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+# Copied from https://github.com/facebookresearch/dlrm/blob/mlperf/dlrm_s_pytorch.py
+
+import sys
+
+from torch.optim.lr_scheduler import _LRScheduler
+
+
+class LRPolicyScheduler(_LRScheduler):
+    def __init__(self, optimizer, num_warmup_steps, decay_start_step, num_decay_steps):
+        self.num_warmup_steps = num_warmup_steps
+        self.decay_start_step = decay_start_step
+        self.decay_end_step = decay_start_step + num_decay_steps
+        self.num_decay_steps = num_decay_steps
+
+        if self.decay_start_step < self.num_warmup_steps:
+            sys.exit("Learning rate warmup must finish before the decay starts")
+
+        super(LRPolicyScheduler, self).__init__(optimizer)
+
+    def get_lr(self):
+        step_count = self._step_count
+        if step_count < self.num_warmup_steps:
+            # warmup
+            scale = 1.0 - (self.num_warmup_steps - step_count) / self.num_warmup_steps
+            lr = [base_lr * scale for base_lr in self.base_lrs]
+            self.last_lr = lr
+        elif self.decay_start_step <= step_count and step_count < self.decay_end_step:
+            # decay
+            decayed_steps = step_count - self.decay_start_step
+            scale = ((self.num_decay_steps - decayed_steps) / self.num_decay_steps) ** 2
+            min_lr = 0.0000001
+            lr = [max(min_lr, base_lr * scale) for base_lr in self.base_lrs]
+            self.last_lr = lr
+        else:
+            if self.num_decay_steps > 0:
+                # freeze at last, either because we're after decay
+                # or because we're between warmup and decay
+                lr = self.last_lr
+            else:
+                # do not adjust
+                lr = self.base_lrs
+        return lr
diff --git a/models_v2/pytorch/torchrec_dlrm/training/gpu/multi_hot.py b/models_v2/pytorch/torchrec_dlrm/training/gpu/multi_hot.py
new file mode 100644
index 000000000..77b13b9e8
--- /dev/null
+++ b/models_v2/pytorch/torchrec_dlrm/training/gpu/multi_hot.py
@@ -0,0 +1,175 @@
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+from typing import List, Tuple
+
+import numpy as np
+
+import torch
+from torchrec.datasets.utils import Batch
+from torchrec.sparse.jagged_tensor import KeyedJaggedTensor
+
+
+class RestartableMap:
+    def __init__(self, f, source):
+        self.source = source
+        self.func = f
+
+    def __iter__(self):
+        for x in self.source:
+            yield self.func(x)
+
+    def __len__(self):
+        return len(self.source)
+
+
+class Multihot:
+    def __init__(
+        self,
+        multi_hot_sizes: List[int],
+        num_embeddings_per_feature: List[int],
+        batch_size: int,
+        collect_freqs_stats: bool,
+        dist_type: str = "uniform",
+    ):
+        if dist_type not in {"uniform", "pareto"}:
+            raise ValueError(
+                "Multi-hot distribution type {} is not supported."
+                'Only "uniform" and "pareto" are supported.'.format(dist_type)
+            )
+        self.dist_type = dist_type
+        self.multi_hot_sizes = multi_hot_sizes
+        self.num_embeddings_per_feature = num_embeddings_per_feature
+        self.batch_size = batch_size
+
+        # Generate 1-hot to multi-hot lookup tables, one lookup table per sparse embedding table.
+        self.multi_hot_tables_l = self.__make_multi_hot_indices_tables(
+            dist_type, multi_hot_sizes, num_embeddings_per_feature
+        )
+
+        # Pooling offsets are computed once and reused.
+        self.offsets = self.__make_offsets(
+            multi_hot_sizes, num_embeddings_per_feature, batch_size
+        )
+
+        # For plotting frequency access
+        self.collect_freqs_stats = collect_freqs_stats
+        self.model_to_track = None
+        self.freqs_pre_hash = []
+        self.freqs_post_hash = []
+        for embs_count in num_embeddings_per_feature:
+            self.freqs_pre_hash.append(np.zeros((embs_count)))
+            self.freqs_post_hash.append(np.zeros((embs_count)))
+
+    def save_freqs_stats(self) -> None:
+        if torch.distributed.is_available() and torch.distributed.is_initialized():
+            rank = torch.distributed.get_rank()
+        else:
+            rank = 0
+        pre_dict = {str(k): e for k, e in enumerate(self.freqs_pre_hash)}
+        np.save(f"stats_pre_hash_{rank}_{self.dist_type}.npy", pre_dict)
+        post_dict = {str(k): e for k, e in enumerate(self.freqs_post_hash)}
+        np.save(f"stats_post_hash_{rank}_{self.dist_type}.npy", post_dict)
+
+    def pause_stats_collection_during_val_and_test(
+        self, model: torch.nn.Module
+    ) -> None:
+        self.model_to_track = model
+
+    def __make_multi_hot_indices_tables(
+        self,
+        dist_type: str,
+        multi_hot_sizes: List[int],
+        num_embeddings_per_feature: List[int],
+    ) -> List[np.array]:
+        np.random.seed(
+            0
+        )  # The seed is necessary for all ranks to produce the same lookup values.
+        multi_hot_tables_l = []
+        for embs_count, multi_hot_size in zip(
+            num_embeddings_per_feature, multi_hot_sizes
+        ):
+            embedding_ids = np.arange(embs_count)[:, np.newaxis]
+            if dist_type == "uniform":
+                synthetic_sparse_ids = np.random.randint(
+                    0, embs_count, size=(embs_count, multi_hot_size - 1)
+                )
+            elif dist_type == "pareto":
+                synthetic_sparse_ids = (
+                    np.random.pareto(
+                        a=0.25, size=(embs_count, multi_hot_size - 1)
+                    ).astype(np.int32)
+                    % embs_count
+                )
+            multi_hot_table = np.concatenate(
+                (embedding_ids, synthetic_sparse_ids), axis=-1
+            )
+            multi_hot_tables_l.append(multi_hot_table)
+        multi_hot_tables_l = [
+            torch.from_numpy(multi_hot_table).int()
+            for multi_hot_table in multi_hot_tables_l
+        ]
+        return multi_hot_tables_l
+
+    def __make_offsets(
+        self,
+        multi_hot_sizes: int,
+        num_embeddings_per_feature: List[int],
+        batch_size: int,
+    ) -> List[torch.Tensor]:
+        lS_o = torch.ones(
+            (len(num_embeddings_per_feature) * batch_size), dtype=torch.int32
+        )
+        for k, multi_hot_size in enumerate(multi_hot_sizes):
+            lS_o[k * batch_size : (k + 1) * batch_size] = multi_hot_size
+        lS_o = torch.cumsum(torch.concat((torch.tensor([0]), lS_o)), axis=0)
+        return lS_o
+
+    def __make_new_batch(
+        self,
+        lS_i: torch.Tensor,
+        batch_size: int,
+    ) -> Tuple[torch.Tensor, torch.Tensor]:
+        lS_i = lS_i.reshape(-1, batch_size)
+        multi_hot_ids_l = []
+        for k, (sparse_data_batch_for_table, multi_hot_table) in enumerate(
+            zip(lS_i, self.multi_hot_tables_l)
+        ):
+            multi_hot_ids = torch.nn.functional.embedding(
+                sparse_data_batch_for_table, multi_hot_table
+            )
+            multi_hot_ids = multi_hot_ids.reshape(-1)
+            multi_hot_ids_l.append(multi_hot_ids)
+            if self.collect_freqs_stats and (
+                self.model_to_track is None or self.model_to_track.training
+            ):
+                idx_pre, cnt_pre = np.unique(
+                    sparse_data_batch_for_table, return_counts=True
+                )
+                idx_post, cnt_post = np.unique(multi_hot_ids, return_counts=True)
+                self.freqs_pre_hash[k][idx_pre] += cnt_pre
+                self.freqs_post_hash[k][idx_post] += cnt_post
+        lS_i = torch.cat(multi_hot_ids_l)
+        if batch_size == self.batch_size:
+            return lS_i, self.offsets
+        else:
+            return lS_i, self.__make_offsets(
+                self.multi_hot_sizes, self.num_embeddings_per_feature, batch_size
+            )
+
+    def convert_to_multi_hot(self, batch: Batch) -> Batch:
+        batch_size = len(batch.dense_features)
+        lS_i = batch.sparse_features._values
+        lS_i, lS_o = self.__make_new_batch(lS_i, batch_size)
+        new_sparse_features = KeyedJaggedTensor.from_offsets_sync(
+            keys=batch.sparse_features._keys,
+            values=lS_i,
+            offsets=lS_o,
+        )
+        return Batch(
+            dense_features=batch.dense_features,
+            sparse_features=new_sparse_features,
+            labels=batch.labels,
+        )
diff --git a/models_v2/pytorch/torchrec_dlrm/training/gpu/requirements.txt b/models_v2/pytorch/torchrec_dlrm/training/gpu/requirements.txt
new file mode 100644
index 000000000..270005d8d
--- /dev/null
+++ b/models_v2/pytorch/torchrec_dlrm/training/gpu/requirements.txt
@@ -0,0 +1,4 @@
+fbgemm-gpu==0.3.2
+torchmetrics==0.11.0
+torchrec==0.3.2
+torchsnapshot
diff --git a/models_v2/pytorch/torchrec_dlrm/training/gpu/run_model.sh b/models_v2/pytorch/torchrec_dlrm/training/gpu/run_model.sh
new file mode 100755
index 000000000..ec2c6f19e
--- /dev/null
+++ b/models_v2/pytorch/torchrec_dlrm/training/gpu/run_model.sh
@@ -0,0 +1,123 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+
+#!/bin/bash
+
+# Create an array of input directories that are expected and then verify that they exist
+declare -A input_envs
+input_envs[DATASET_DIR]=${DATASET_DIR}
+input_envs[MULTI_TILE]=${MULTI_TILE}
+input_envs[PLATFORM]=${PLATFORM}
+
+for i in "${!input_envs[@]}"; do
+  var_name=$i
+  env_param=${input_envs[$i]}
+
+  if [[ -z $env_param ]]; then
+    echo "The required environment variable $var_name is not set" >&2
+    exit 1
+  fi
+done
+
+OUTPUT_DIR=${OUTPUT_DIR:-$PWD}
+
+if [[ "${PLATFORM}" == "PVC" ]]; then
+    BATCH_SIZE=${BATCH_SIZE:-65536}
+    PRECISION=${PRECISION:-BF16}
+elif [[ "${PLATFORM}" == "ATS-M" ]]; then
+    echo "Only support PVC for platform"
+    exit 1
+fi
+
+
+
+if [[ ! -d "${DATASET_DIR}" ]]; then
+    echo "The DATASET_DIR '${DATASET_DIR}' does not exist"
+    exit 1
+fi
+
+
+
+echo 'Running with parameters:'
+echo " PLATFORM: ${PLATFORM}"
+echo " DATASET_PATH: ${DATASET_DIR}"
+echo " OUTPUT_DIR: ${OUTPUT_DIR}"
+echo " PRECISION: ${PRECISION}"
+echo " BATCH_SIZE: ${BATCH_SIZE}"
+echo " MULTI_TILE: ${MULTI_TILE}"
+
+if [[ "${PRECISION}" == "BF16" ]]; then
+    flag="-bf16 true"
+elif [[ "${PRECISION}" == "FP32" ]]; then
+    flag="-bf16 false"
+elif [[ "${PRECISION}" == "TF32" ]]; then
+    flag="-bf16 false -tf32 true"
+else
+    echo -e "Invalid input! Only FP32 BF16 and TF32 are supported."
+    exit 1
+fi
+echo "Dlrmv2 ${PRECISION} Training plain MultiTile=${MULTI_TILE} BS=${BATCH_SIZE}"
+
+# Create the output directory, if it doesn't already exist
+mkdir -p $OUTPUT_DIR
+
+sum_log_analysis() {
+    if [ -f $2 ]; then
+        rm -f $2
+    fi
+    if diff /dev/null ${1}_t0.log |tail -l | grep '^\\ No newline' > /dev/null;then echo >> ${1}_t0.log; fi
+    if diff /dev/null ${1}_t1.log |tail -l | grep '^\\ No newline' > /dev/null;then echo >> ${1}_t1.log; fi
+    bs=$(cat ${1}_t1.log |grep Batch |awk '{print $3}')
+    echo -e "Batch Size: $bs" >$2
+    cat ${1}"_t0.log" ${1}"_t1.log" |grep "Performance" |awk -v tag=$(cat ${1}"_t0.log" ${1}"_t1.log" |grep "Performance" |awk '{sum+=$2} END {printf "%.4f\n",sum}') '{if ( $2=="None" ) {sum="None";nextfile}else  sum=tag} ;END{print "Sum "$1" "sum " "$3}' >> $2
+    cat ${1}"_t0.log" ${1}"_t1.log" |grep "Performance" |awk -v tag=$(cat ${1}"_t0.log" ${1}"_t1.log" |grep "Performance" |awk 'BEGIN {min=1234567890123} {if ($2 <min) {min=$2}}END {printf "%.4f\n",min}') '{if ( $2=="None" ) {min="None";nextfile}else  min=tag} ;END{print "Min "$1" "min " "$3}' >> $2
+    cat ${1}"_t0.log" ${1}"_t1.log" |grep "Latency" |awk '{if ( $2=="N/A" ){avg="N/A";nextfile}else avg=((sum+=$2/2))};END{print "Avg "$1" "avg " "$3}'  >> $2
+    cat ${1}"_t0.log" ${1}"_t1.log" |grep "Accuracy" |awk -v avg=$(cat ${1}"_t0.log" ${1}"_t1.log" |grep "Accuracy" |awk '{sum+=$3}END{printf "%.4f\n",sum/NR}') '{if ( $3=="None" || $2=="N/A" || $3=="nan" || $3=="N/A"){avg="None";nextfile}else avg=avg};END{print "Avg "$1" "$2 " "avg}' >> $2
+    cat ${1}"_t0.log" ${1}"_t1.log" |grep "Functional" | awk -v fail=$(cat ${1}"_t0.log" ${1}"_t1.log" |grep "Functional" |awk '{for(i=1;i<=NF;++i) if($i=="fail") ++sum}END{print sum}') '{if ( fail >= 1 ) tag="fail ";else tag="pass"};END{print $1" "tag}'  >> $2
+    cat ${1}"_t0.log" ${1}"_t1.log" |grep "Error" |awk '{if(a[$1]){a[$1]=a[$1]";"$2}else{a[$1]=$2}}END{for(i in a)print $1" " a[i]}' >> $2
+}
+
+modelname=dlrm-terabyte
+if [[ ${MULTI_TILE} == "False" ]]; then
+	echo -e "do not support MULTI_TILE=False"
+	exit 1
+else
+    rm ${OUTPUT_DIR}/ddp-${modelname}_${PRECISION}_train_raw.log
+    bash cmd_distributed_terabyte_train.sh -d ${DATASET_DIR} ${flag} 2>&1 | tee ${OUTPUT_DIR}/ddp-${modelname}_${PRECISION}_train_raw.log
+    python ../../../../../models/common/pytorch/parse_result.py -t ddp -l ${OUTPUT_DIR}/ddp-${modelname}_${PRECISION}_train_raw.log -b ${BATCH_SIZE}
+    throughput=$(cat ${OUTPUT_DIR}/ddp-${modelname}_${PRECISION}_train.log | grep "Sum Performance" | awk -F ' ' '{print $3}')
+    throughput_unit=$(cat ${OUTPUT_DIR}/ddp-${modelname}_${PRECISION}_train.log | grep "Sum Performance" | awk -F ' ' '{print $4}')
+    acc=$(cat ${OUTPUT_DIR}/ddp-${modelname}_${PRECISION}_train.log | grep Accuracy | awk -F ' ' '{print $3}')
+    acc_unit=$(cat ${OUTPUT_DIR}/ddp-${modelname}_${PRECISION}_train.log | grep Accuracy | awk -F ' ' '{print $2}')
+fi
+
+yaml_content=$(cat <<EOF
+results:
+ - key: throughput
+   value: $throughput
+   unit: $throughput_unit
+ - key: accuracy
+   value: $acc
+   unit: $acc_unit
+EOF
+)
+
+# Write the content to a YAML file
+echo "$yaml_content" >  ./results.yaml
+echo "YAML file created."
diff --git a/models_v2/pytorch/torchrec_dlrm/training/gpu/scripts/materialize_synthetic_multihot_dataset.py b/models_v2/pytorch/torchrec_dlrm/training/gpu/scripts/materialize_synthetic_multihot_dataset.py
new file mode 100644
index 000000000..63d703db0
--- /dev/null
+++ b/models_v2/pytorch/torchrec_dlrm/training/gpu/scripts/materialize_synthetic_multihot_dataset.py
@@ -0,0 +1,153 @@
+#!/usr/bin/env python3
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+import argparse
+import os
+import pathlib
+import shutil
+import sys
+
+import numpy as np
+import torch
+from torch import distributed as dist, nn
+from torchrec.datasets.criteo import DAYS
+
+p = pathlib.Path(__file__).absolute().parents[1].resolve()
+sys.path.append(os.fspath(p))
+
+# OSS import
+try:
+    # pyre-ignore[21]
+    # @manual=//ai_codesign/benchmarks/dlrm/torchrec_dlrm:multi_hot
+    from multi_hot import Multihot
+except ImportError:
+    pass
+
+# internal import
+try:
+    from .multi_hot import Multihot  # noqa F811
+except ImportError:
+    pass
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser(
+        description="Script to materialize synthetic multi-hot dataset into NumPy npz file format."
+    )
+    parser.add_argument(
+        "--in_memory_binary_criteo_path",
+        type=str,
+        required=True,
+        help="Path to a folder containing the binary (npy) files for the Criteo dataset."
+        " When supplied, InMemoryBinaryCriteoIterDataPipe is used.",
+    )
+    parser.add_argument(
+        "--output_path",
+        type=str,
+        required=True,
+        help="Path to outputted multi-hot sparse dataset.",
+    )
+    parser.add_argument(
+        "--copy_labels_and_dense",
+        dest="copy_labels_and_dense",
+        action="store_true",
+        help="Flag to determine whether to copy labels and dense data to the output directory.",
+    )
+    parser.add_argument(
+        "--num_embeddings_per_feature",
+        type=str,
+        required=True,
+        help="Comma separated max_ind_size per sparse feature. The number of embeddings"
+        " in each embedding table. 26 values are expected for the Criteo dataset.",
+    )
+    parser.add_argument(
+        "--multi_hot_sizes",
+        type=str,
+        required=True,
+        help="Comma separated multihot size per sparse feature. 26 values are expected for the Criteo dataset.",
+    )
+    parser.add_argument(
+        "--multi_hot_distribution_type",
+        type=str,
+        choices=["uniform", "pareto"],
+        default="uniform",
+        help="Multi-hot distribution options.",
+    )
+    return parser.parse_args()
+
+
+def main() -> None:
+    """
+    This script generates and saves the MLPerf v2 multi-hot dataset (4 TB in size).
+    First, run process_Criteo_1TB_Click_Logs_dataset.sh.
+    Then, run this script as follows:
+
+        python materialize_synthetic_multihot_dataset.py \
+            --in_memory_binary_criteo_path $PREPROCESSED_CRITEO_1TB_CLICK_LOGS_DATASET_PATH \
+            --output_path $MATERIALIZED_DATASET_PATH \
+            --num_embeddings_per_feature 40000000,39060,17295,7424,20265,3,7122,1543,63,40000000,3067956,405282,10,2209,11938,155,4,976,14,40000000,40000000,40000000,590152,12973,108,36 \
+            --multi_hot_sizes 3,2,1,2,6,1,1,1,1,7,3,8,1,6,9,5,1,1,1,12,100,27,10,3,1,1 \
+            --multi_hot_distribution_type uniform
+
+    This script takes about 2 hours to run (can be parallelized if needed).
+    """
+    args = parse_args()
+    for name, val in vars(args).items():
+        try:
+            vars(args)[name] = list(map(int, val.split(",")))
+        except (ValueError, AttributeError):
+            pass
+    try:
+        backend = "nccl" if torch.cuda.is_available() else "gloo"
+        if not dist.is_initialized():
+            dist.init_process_group(backend=backend)
+        rank = dist.get_rank()
+        world_size = dist.get_world_size()
+    except (KeyError, ValueError):
+        rank = 0
+        world_size = 1
+
+    print("Generating one-hot to multi-hot lookup table.")
+    multihot = Multihot(
+        multi_hot_sizes=args.multi_hot_sizes,
+        num_embeddings_per_feature=args.num_embeddings_per_feature,
+        batch_size=1,  # Doesn't matter
+        collect_freqs_stats=False,
+        dist_type=args.multi_hot_distribution_type,
+    )
+
+    os.makedirs(args.output_path, exist_ok=True)
+
+    for i in range(rank, DAYS, world_size):
+        input_file_path = os.path.join(
+            args.in_memory_binary_criteo_path, f"day_{i}_sparse.npy"
+        )
+        print(f"Materializing {input_file_path}")
+        sparse_data = np.load(input_file_path, mmap_mode="r")
+        multi_hot_ids_dict = {}
+        for j, (multi_hot_table, hash) in enumerate(
+            zip(multihot.multi_hot_tables_l, args.num_embeddings_per_feature)
+        ):
+            sparse_tensor = torch.from_numpy(sparse_data[:, j] % hash)
+            multi_hot_ids_dict[str(j)] = nn.functional.embedding(
+                sparse_tensor, multi_hot_table
+            ).numpy()
+        output_file_path = os.path.join(
+            args.output_path, f"day_{i}_sparse_multi_hot.npz"
+        )
+        np.savez(output_file_path, **multi_hot_ids_dict)
+        if args.copy_labels_and_dense:
+            for part in ["labels", "dense"]:
+                source_path = os.path.join(
+                    args.in_memory_binary_criteo_path, f"day_{i}_{part}.npy"
+                )
+                output_path = os.path.join(args.output_path, f"day_{i}_{part}.npy")
+                shutil.copyfile(source_path, output_path)
+                print(f"Copying {source_path} to {output_path}")
+
+
+if __name__ == "__main__":
+    main()
diff --git a/models_v2/pytorch/torchrec_dlrm/training/gpu/scripts/process_Criteo_1TB_Click_Logs_dataset.sh b/models_v2/pytorch/torchrec_dlrm/training/gpu/scripts/process_Criteo_1TB_Click_Logs_dataset.sh
new file mode 100644
index 000000000..23050647d
--- /dev/null
+++ b/models_v2/pytorch/torchrec_dlrm/training/gpu/scripts/process_Criteo_1TB_Click_Logs_dataset.sh
@@ -0,0 +1,64 @@
+#!/bin/bash
+# Copyright (c) Meta Platforms, Inc. and affiliates.
+#
+# This source code is licensed under the MIT license found in the
+# LICENSE file in the root directory of this source tree.
+
+display_help() {
+   echo "Three command line arguments are required."
+   echo "Example usage:"
+   echo "bash process_Criteo_1TB_Click_Logs_dataset.sh \\"
+   echo "./criteo_1tb/raw_input_dataset_dir \\"
+   echo "./criteo_1tb/temp_intermediate_files_dir \\"
+   echo "./criteo_1tb/numpy_contiguous_shuffled_output_dataset_dir"
+   exit 1
+}
+
+[ -z "$1" ] && display_help
+[ -z "$2" ] && display_help
+[ -z "$3" ] && display_help
+
+# Input directory containing the raw Criteo 1TB Click Logs dataset files in tsv format.
+# The 24 dataset filenames in the directory should be day_{0..23} with no .tsv extension.
+raw_tsv_criteo_files_dir=$(readlink -m "$1")
+
+# Directory to store temporary intermediate output files created by preprocessing steps 1 and 2.
+temp_files_dir=$(readlink -m "$2")
+
+# Directory to store temporary intermediate output files created by preprocessing step 1.
+step_1_output_dir="$temp_files_dir/temp_output_of_step_1"
+
+# Directory to store temporary intermediate output files created by preprocessing step 2.
+step_2_output_dir="$temp_files_dir/temp_output_of_step_2"
+
+# Directory to store the final preprocessed Criteo 1TB Click Logs dataset.
+step_3_output_dir=$(readlink -m "$3")
+
+# Step 1. Split the dataset into 3 sets of 24 numpy files:
+# day_{0..23}_dense.npy, day_{0..23}_labels.npy, and day_{0..23}_sparse.npy (~24hrs)
+set -x
+mkdir -p "$step_1_output_dir"
+date
+python -m torchrec.datasets.scripts.npy_preproc_criteo --input_dir "$raw_tsv_criteo_files_dir" --output_dir "$step_1_output_dir" || exit
+
+# Step 2. Convert all sparse indices in day_{0..23}_sparse.npy to contiguous indices and save the output.
+# The output filenames are day_{0..23}_sparse_contig_freq.npy
+mkdir -p "$step_2_output_dir"
+date
+python -m torchrec.datasets.scripts.contiguous_preproc_criteo --input_dir "$step_1_output_dir" --output_dir "$step_2_output_dir" --frequency_threshold 0 || exit
+
+date
+for i in {0..23}
+do
+   name="$step_2_output_dir/day_$i""_sparse_contig_freq.npy"
+   renamed="$step_2_output_dir/day_$i""_sparse.npy"
+   echo "Renaming $name to $renamed"
+   mv "$name" "$renamed"
+done
+
+# Step 3. Shuffle the dataset's samples in days 0 through 22. (~20hrs)
+# Day 23's samples are not shuffled and will be used for the validation set and test set.
+mkdir -p "$step_3_output_dir"
+date
+python -m torchrec.datasets.scripts.shuffle_preproc_criteo --input_dir_labels_and_dense "$step_1_output_dir" --input_dir_sparse "$step_2_output_dir" --output_dir_shuffled "$step_3_output_dir" --random_seed 0 || exit
+date
diff --git a/models_v2/pytorch/torchrec_dlrm/training/gpu/setup.sh b/models_v2/pytorch/torchrec_dlrm/training/gpu/setup.sh
new file mode 100755
index 000000000..475cb8af3
--- /dev/null
+++ b/models_v2/pytorch/torchrec_dlrm/training/gpu/setup.sh
@@ -0,0 +1,27 @@
+#!/usr/bin/env bash
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# setup.sh
+#  - install OS pkgs
+#  - should create virtual env & install pip  requirement.txt
+#  - git clones & applying patches
+
+set -e
+apt-get update && apt-get install -y python3-venv protobuf-compiler
+python3 -m venv $PWD/venv
+. ./venv/bin/activate
+pip install -r requirements.txt
diff --git a/models_v2/pytorch/torchrec_dlrm/training/gpu/sharding/__init__.py b/models_v2/pytorch/torchrec_dlrm/training/gpu/sharding/__init__.py
new file mode 100644
index 000000000..a875ff9af
--- /dev/null
+++ b/models_v2/pytorch/torchrec_dlrm/training/gpu/sharding/__init__.py
@@ -0,0 +1,23 @@
+#MIT License
+#
+#Copyright (c) Facebook, Inc. and its affiliates.
+#
+#Permission is hereby granted, free of charge, to any person obtaining a copy
+#of this software and associated documentation files (the "Software"), to deal
+#in the Software without restriction, including without limitation the rights
+#to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+#copies of the Software, and to permit persons to whom the Software is
+#furnished to do so, subject to the following conditions:
+#
+#The above copyright notice and this permission notice shall be included in all
+#copies or substantial portions of the Software.
+#
+#THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+#IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+#FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+#AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+#LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+#OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+#SOFTWARE.
+from .generate_plan import generate_plan
+from .planner import Cost, CostModel, Planner, ShardingState
diff --git a/models_v2/pytorch/torchrec_dlrm/training/gpu/sharding/generate_plan.py b/models_v2/pytorch/torchrec_dlrm/training/gpu/sharding/generate_plan.py
new file mode 100644
index 000000000..ad8b4016e
--- /dev/null
+++ b/models_v2/pytorch/torchrec_dlrm/training/gpu/sharding/generate_plan.py
@@ -0,0 +1,152 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import logging
+from argparse import Namespace
+from itertools import chain
+from typing import List
+
+from .planner import CostModel, Planner
+
+
+def generate_plan(
+    slot_size_array: List[int],
+    multi_hot_sizes: List[int],
+    num_nodes: int,
+    num_gpus: int,
+    args: Namespace,
+    log_result: bool,
+):
+    def sanity_check(shard_matrix, shard_strategy):
+        # mainly to make sure all the tables are sharded
+        msg = "Not all tables covered in the sharding plan"
+        assert set(chain(*shard_matrix)) == set(range(len(slot_size_array))), msg
+        shard_strategy_list = [x for strategy_pair in shard_strategy for x in strategy_pair[1]]
+        assert set(shard_strategy_list) == set(range(len(slot_size_array))), msg
+
+        for table_list in shard_matrix:
+            if len(table_list) == 0:
+                raise Exception("Currently no empty shard list is allowed")
+
+    def int_to_string(shard_matrix_int, shard_strategy_int):
+        shard_strategy, shard_matrix = [], []
+        for pair in shard_strategy_int:
+            if len(pair[1]) != 0:
+                shard_strategy.append((pair[0], [str(x) for x in pair[1]]))
+        for sub_matrix_ in shard_matrix_int:
+            shard_matrix.append([str(x) for x in sub_matrix_])
+        return shard_matrix, shard_strategy
+
+    if args.sharding_plan in ["round_robin", "uniform"]:
+        # sharding strategies that don't exploit system configs
+        if args.sharding_plan == "round_robin":
+            mp_table = [i for i in range(len(slot_size_array))]
+            shard_matrix_ = [[] for _ in range(num_gpus)]
+            shard_strategy_ = [("mp", [i for i in mp_table])]
+
+            for i, table_id in enumerate(mp_table):
+                target_gpu = i % num_gpus
+                shard_matrix_[target_gpu].append(table_id)
+
+        elif args.sharding_plan == "uniform":
+            shard_matrix_ = [[x for x in range(len(slot_size_array))] for _ in range(num_gpus)]
+            shard_strategy_ = [("mp", [i for i in range(len(slot_size_array))])]
+
+    elif args.sharding_plan in ["auto", "hier_auto"]:
+        # sharding strategies that exploit system configs
+        dram_cap = args.memory_cap_for_embedding
+        if args.optimizer == "adagrad":
+            byte_per_elem = 8
+        elif args.optimizer == "sgd":
+            byte_per_elem = 4
+
+        if args.sharding_plan == "auto":
+            cost_model = CostModel(
+                1,
+                args.mem_comm_bw_ratio / args.mem_comm_work_ratio,
+                args.ev_size * byte_per_elem * 1e-9,
+                dram_cap,
+                slot_size_array,
+            )
+            planner = Planner(multi_hot_sizes, num_gpus, cost_model, log_result=log_result)
+            shard_strategy_, shard_matrix_ = planner.plan()
+
+        elif args.sharding_plan == "hier_auto":
+            if num_nodes <= 1:
+                raise Exception("hier_auto plan is only applicable to configs with more than one node")
+            cost_model = CostModel(
+                1,
+                args.mem_comm_bw_ratio / args.mem_comm_work_ratio,
+                args.ev_size * byte_per_elem * 1e-9,
+                dram_cap * args.num_gpus_per_node,
+                slot_size_array,
+            )
+            planner = Planner(multi_hot_sizes, num_nodes, cost_model, log_result=log_result)
+            shard_strategy_, shard_matrix_node_ = planner.plan()
+            shard_matrix_ = []
+            for node_shard_matrix in shard_matrix_node_:
+                for i in range(args.num_gpus_per_node):
+                    shard_matrix_.append(node_shard_matrix)
+    elif args.sharding_plan in ["max_min"]:
+        mp_table = [i for i in range(len(slot_size_array))]
+        shard_matrix_ = [[] for _ in range(num_gpus)]
+        shard_strategy_ = [("mp", [i for i in mp_table])]
+        dict_feature = {}
+        for k, v in enumerate(slot_size_array):
+            dict_feature[k] = v
+        dict_feature_list = sorted(dict_feature.items(),key=lambda x:x[1], reverse=True)
+        end = len(dict_feature_list) - 1
+        for i in range(len(dict_feature_list)):
+            index = i%num_gpus if int(i/num_gpus) == 0 else (num_gpus -1 - i%num_gpus)
+            shard_matrix_[index].append(dict_feature_list[i][0])
+            if (i >= end):
+                break
+            elif(end - i >= num_gpus - i%num_gpus):
+                shard_matrix_[index].append(dict_feature_list[end][0])
+                end -= 1
+    elif args.sharding_plan in ["custom"]:
+        mp_table = [i for i in range(len(slot_size_array))]
+        shard_matrix_ = [[0, 5], [9, 16], [19, 12], [20, 18], [21, 25], [10, 8, 2, 13, 3], [22, 24, 4, 7, 23], [11, 15, 1, 17, 14, 6]]
+        shard_strategy_ = [("mp", [i for i in mp_table])]
+    else:
+        raise Exception("unknown sharding plan")
+
+    sanity_check(shard_matrix_, shard_strategy_)
+    shard_matrix, shard_strategy = int_to_string(shard_matrix_, shard_strategy_)
+
+    if log_result:
+        logging.info("Provided system info: ")
+        #logging.info("num_gpu_per_nodes: %d", args.num_gpus_per_node)
+        #logging.info("Memory to communication BW ratio: %f", args.mem_comm_bw_ratio)
+        #logging.info("Memory to communication work ratio: %f", args.mem_comm_work_ratio)
+        #logging.info("DRAM capacity: %f GB", args.memory_cap_for_embedding)
+        logging.info("shard_matrix:")
+        logging.info(shard_matrix)
+        logging.info("\n")
+
+    return shard_matrix, shard_strategy, shard_matrix_, shard_strategy_
diff --git a/models_v2/pytorch/torchrec_dlrm/training/gpu/sharding/planner.py b/models_v2/pytorch/torchrec_dlrm/training/gpu/sharding/planner.py
new file mode 100644
index 000000000..4fa5c6c37
--- /dev/null
+++ b/models_v2/pytorch/torchrec_dlrm/training/gpu/sharding/planner.py
@@ -0,0 +1,313 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import logging
+import time
+from typing import List, Tuple
+
+import numpy as np
+
+
+class ShardingState:
+    """
+    Containing the state of a sharding process.
+    The plan iteratively update the sharding state based on a given heuristic and obtain
+    solutions.
+    """
+
+    def __init__(
+        self,
+        array_hotness: np.array,
+        num_bucket: int,
+        dp_table_id: np.array(int) = np.array([]),
+    ) -> None:
+        mp_table_id = np.setdiff1d(np.arange(array_hotness.size), dp_table_id)
+        array_hotness_mp = array_hotness[mp_table_id]
+        sorted_idx = np.argsort(array_hotness_mp)[::-1]
+        self.array_unshard_hotness = array_hotness
+        self.array_hotness = array_hotness_mp[sorted_idx]
+        self.num_bucket = num_bucket
+        self.array_table_id = mp_table_id[sorted_idx]
+        self.array_num_split = np.zeros(self.array_unshard_hotness.size, dtype=int)
+        self.array_num_split[mp_table_id] = 1
+        self.shard_ll = [[] for i in range(self.num_bucket)]
+
+    def split_hot_shard(self):
+        """
+        split the shard with the largest hotness
+        """
+        # shards are sorted based on the hotness. Find the first hot shard that
+        # can be split further
+        for shard_id in range(self.array_table_id.size):
+            table_id = self.array_table_id[shard_id]
+            hotness = self.array_unshard_hotness[table_id]
+            if self.array_num_split[table_id] * 2 <= self.num_bucket:
+                # if this table can be furter splitted and we can put it into
+                # more buckets
+                idx = np.where(self.array_table_id == table_id)[0]
+                self.array_hotness = np.delete(self.array_hotness, idx)
+                self.array_table_id = np.delete(self.array_table_id, idx)
+                self.array_num_split[table_id] *= 2
+                self.array_hotness = np.concatenate(
+                    (
+                        self.array_hotness,
+                        np.ones(self.array_num_split[table_id]) * (hotness / self.array_num_split[table_id]),
+                    )
+                )
+                self.array_table_id = np.concatenate(
+                    (self.array_table_id, np.ones(self.array_num_split[table_id], dtype=int) * table_id)
+                )
+                break
+
+        # sort after splitting to maintain the shard hotness in order
+        sorted_idx = np.argsort(self.array_hotness)[::-1]
+        self.array_hotness = self.array_hotness[sorted_idx]
+        self.array_table_id = self.array_table_id[sorted_idx]
+
+    def split_oom_shard(self, table_id: int) -> bool:
+        hotness = self.array_unshard_hotness[table_id]
+        if self.array_num_split[table_id] * 2 <= self.num_bucket:
+            idx = np.where(self.array_table_id == table_id)[0]
+            self.array_hotness = np.delete(self.array_hotness, idx)
+            self.array_table_id = np.delete(self.array_table_id, idx)
+            self.array_num_split[table_id] *= 2
+            self.array_hotness = np.concatenate(
+                (
+                    self.array_hotness,
+                    np.ones(self.array_num_split[table_id]) * (hotness / self.array_num_split[table_id]),
+                )
+            )
+            self.array_table_id = np.concatenate(
+                (self.array_table_id, np.ones(self.array_num_split[table_id], dtype=int) * table_id)
+            )
+            sorted_idx = np.argsort(self.array_hotness)[::-1]
+            self.array_hotness = self.array_hotness[sorted_idx]
+            self.array_table_id = self.array_table_id[sorted_idx]
+            return True
+        else:
+            return False
+
+    def update_split_num(self):
+        self.array_num_split = np.zeros_like(self.array_unshard_hotness)
+        for shard_list in self.shard_ll:
+            for table_id in shard_list:
+                self.array_num_split[table_id] += 1
+
+    def reset_shard_ll(self):
+        self.shard_ll = [[] for i in range(self.num_bucket)]
+
+    def push_bucket(
+        self,
+        bucket_id: int,
+        table_id: int,
+    ) -> None:
+        self.shard_ll[bucket_id].append(table_id)
+
+    def pop_bucket(
+        self,
+        bucket_id: int,
+    ) -> None:
+        self.shard_ll[bucket_id].pop()
+
+
+class Cost:
+    def __init__(
+        self,
+        cost: np.array(float),
+        hotness_cost: np.array(float),
+        table_cost: np.array(float),
+        mem_cost: np.array(float),
+    ) -> None:
+        self.cost = cost
+        self.hotness_cost = hotness_cost
+        self.table_cost = table_cost
+        self.mem_cost = mem_cost
+
+
+class CostModel:
+    def __init__(
+        self,
+        hotness_cost: float,
+        table_cost: float,
+        mem_cost: float,
+        mem_capacity: float,
+        table_size: List[int],
+    ) -> None:
+        self.unit_hotness_cost = hotness_cost
+        self.unit_table_cost = table_cost
+        self.unit_mem_cost = mem_cost
+        self.mem_capacity = mem_capacity
+        self.array_table_size = np.array(table_size)
+
+    def get_cost(
+        self,
+        ss: ShardingState,
+    ) -> Tuple[Cost, bool]:
+        list_cost = []
+        list_hotness_cost = []
+        list_table_cost = []
+        list_mem_cost = []
+
+        for shard_list in ss.shard_ll:
+            hotness_cost = (
+                self.unit_hotness_cost
+                * (ss.array_unshard_hotness[shard_list] / np.array(ss.array_num_split)[shard_list]).sum()
+            )
+            table_cost = self.unit_table_cost * len(shard_list)
+            mem_cost = (
+                self.unit_mem_cost
+                * (self.array_table_size[shard_list] / np.array(ss.array_num_split)[shard_list]).sum()
+            )
+            list_cost.append(hotness_cost + table_cost)
+            list_hotness_cost.append(hotness_cost)
+            list_table_cost.append(table_cost)
+            list_mem_cost.append(mem_cost)
+
+        return (
+            Cost(np.array(list_cost), np.array(list_hotness_cost), np.array(list_table_cost), np.array(list_mem_cost)),
+            max(list_mem_cost) > self.mem_capacity,
+        )
+
+    def deduct_mem_cap_for_dp(
+        self,
+        dp_table_id: list,
+    ) -> None:
+        self.mem_capacity -= self.array_table_size[dp_table_id].sum() * self.unit_mem_cost
+        if self.mem_capacity < 0:
+            raise Exception("OOM due to DP. Please considering increase the DP threshold")
+
+
+class Planner:
+    """
+    The planner work out a series of plans iteratively.
+    In each iteration, the planner tries to split the hottest shard and place the shards into
+    a bucket based on a give heuristic. When the shard is too large to fit into the best bucket
+    suggested by the heuristic, it finds the next best bucket until it iterates through all the
+    buckets. In that case, it tries to split the shard further. If the shard cannot be split
+    further, the planner aborts and returns the default sharding plan.
+    """
+
+    def __init__(
+        self,
+        list_hotness: list,
+        num_bucket: int,
+        cost_model: CostModel,
+        dp_threshold: int = 0,
+        max_search_iter: int = 20,
+        log_result: bool = False,
+    ) -> None:
+        self.array_hotness = np.array(list_hotness)
+        self.num_bucket = num_bucket
+        self.cost_model = cost_model
+        self.list_candidate = []
+        self.max_search_iter = max_search_iter
+        self.log_result = log_result
+
+        # Create the default sharding plan. Throw if even this default sharding plan cannot fit, as
+        # it should be the most memory-efficient
+        sharding_state_default = ShardingState(self.array_hotness, self.num_bucket)
+        for b in range(self.num_bucket):
+            for t in range(self.array_hotness.size):
+                sharding_state_default.push_bucket(b, t)
+        sharding_state_default.update_split_num()
+        cost, oom = self.cost_model.get_cost(sharding_state_default)
+        if oom:
+            raise Exception("OOM even with the most memory-efficient sharding plan")
+        self.list_candidate.append(
+            (cost.cost.max(), cost.hotness_cost, cost.table_cost, cost.mem_cost, sharding_state_default.shard_ll)
+        )
+
+        # Create DP sharding plan based on the DP threshold
+        self.dp_table_id = np.where(cost_model.array_table_size < dp_threshold / cost_model.unit_mem_cost)[0]
+        self.mp_table_id = np.setdiff1d(np.arange(self.array_hotness.size), self.dp_table_id)
+        self.sharding_state = ShardingState(self.array_hotness, self.num_bucket, self.dp_table_id)
+        self.cost_model.deduct_mem_cap_for_dp(self.dp_table_id)
+
+        logging.basicConfig(level=logging.INFO, format="%(message)s")
+
+    def greedy_plan(self, ss):
+        """
+        This is a heuristic based on greedy policy. The shard is placed to the bucket with the
+        lowest hotness cost
+        """
+        array_cost = np.zeros(ss.num_bucket)
+        ss.reset_shard_ll()
+        for i in range(ss.array_hotness.size):
+            sorted_idx = np.argsort(array_cost)
+            sharded = False
+            for bucket_id in sorted_idx:
+                if ss.array_table_id[i] not in ss.shard_ll[bucket_id]:
+                    # for now, only uniform sharding is supported. Hence cannot put two shards
+                    # from the same table into the same bucket
+                    ss.push_bucket(bucket_id, ss.array_table_id[i])
+                    cost, oom = self.cost_model.get_cost(ss)
+                    if not oom:
+                        sharded = True
+                        array_cost = cost.cost
+                        break
+                    else:
+                        # Currnet bucket cannot fit. Iterate to the next best bucket
+                        ss.pop_bucket(bucket_id)
+            if not sharded:
+                # This means the shard is too large to fit within any bucket
+                return ss.array_table_id[i], ss, cost
+        return None, ss, cost
+
+    def plan(self):
+        t0 = time.time()
+        for i in range(self.max_search_iter):
+            oom_table_id, self.sharding_state, cost = self.greedy_plan(self.sharding_state)
+            if oom_table_id is None:
+                self.list_candidate.append(
+                    (cost.cost.max(), cost.hotness_cost, cost.table_cost, cost.mem_cost, self.sharding_state.shard_ll)
+                )
+                self.sharding_state.split_hot_shard()
+            else:
+                oom_table_can_split = self.sharding_state.split_oom_shard(oom_table_id)
+                if not oom_table_can_split:
+                    break
+
+        self.list_candidate.sort(key=lambda x: x[0])
+
+        shard_strategy = [("mp", self.mp_table_id.tolist())]
+        shard_strategy.append(("dp", self.dp_table_id.tolist()))
+        shard_matrix = self.list_candidate[0][-1]
+        for table_id in self.dp_table_id:
+            for shard_list in shard_matrix:
+                shard_list.append(table_id)
+        if self.log_result:
+            logging.info("Planner took %f sec" % (time.time() - t0))
+            logging.info(shard_strategy)
+            logging.info(shard_matrix)
+            logging.info("hotness cost is:")
+            logging.info(self.list_candidate[0][1])
+            logging.info("table cost is:")
+            logging.info(self.list_candidate[0][2])
+            logging.info("mem cost is:")
+            logging.info(self.list_candidate[0][3])
+        return shard_strategy, shard_matrix
diff --git a/models_v2/tensorflow/3d_unet/training/gpu/3dunet_itex.patch b/models_v2/tensorflow/3d_unet/training/gpu/3dunet_itex.patch
new file mode 100644
index 000000000..c37494285
--- /dev/null
+++ b/models_v2/tensorflow/3d_unet/training/gpu/3dunet_itex.patch
@@ -0,0 +1,493 @@
+diff --git a/TensorFlow/Segmentation/UNet_3D_Medical/dataset/data_loader.py b/TensorFlow/Segmentation/UNet_3D_Medical/dataset/data_loader.py
+index cd088808..2235ce4c 100644
+--- a/TensorFlow/Segmentation/UNet_3D_Medical/dataset/data_loader.py
++++ b/TensorFlow/Segmentation/UNet_3D_Medical/dataset/data_loader.py
+@@ -14,7 +14,7 @@
+ 
+ import os
+ 
+-import horovod.tensorflow as hvd
++#import horovod.tensorflow as hvd
+ import numpy as np
+ import tensorflow as tf
+ 
+@@ -91,7 +91,7 @@ class Dataset:
+ 
+         ds = tf.data.TFRecordDataset(filenames=self._train)
+ 
+-        ds = ds.shard(hvd.size(), hvd.rank())
++        #ds = ds.shard(hvd.size(), hvd.rank())
+         ds = ds.cache()
+         ds = ds.shuffle(buffer_size=self._batch_size * 8, seed=self._seed)
+         ds = ds.repeat()
+@@ -216,7 +216,7 @@ class Dataset:
+ 
+ def main():
+     from time import time
+-    hvd.init()
++    #hvd.init()
+ 
+     dataset = Dataset(data_dir='/data/BraTS19_tfrecord', batch_size=3)
+ 
+diff --git a/TensorFlow/Segmentation/UNet_3D_Medical/dataset/transforms.py b/TensorFlow/Segmentation/UNet_3D_Medical/dataset/transforms.py
+index 15e6244f..e31de05c 100644
+--- a/TensorFlow/Segmentation/UNet_3D_Medical/dataset/transforms.py
++++ b/TensorFlow/Segmentation/UNet_3D_Medical/dataset/transforms.py
+@@ -12,7 +12,7 @@
+ # See the License for the specific language governing permissions and
+ # limitations under the License.
+ 
+-import tensorflow as tf
++import tensorflow.compat.v1 as tf
+ 
+ 
+ def apply_transforms(x, y, mean, stdev, transforms):
+@@ -48,7 +48,7 @@ class CenterCrop:
+ 
+     def __call__(self, x, y, mean, stdev):
+         shape = x.get_shape()
+-        delta = [(shape[i].value - self.shape[i]) // 2 for i in range(len(self.shape))]
++        delta = [(shape[i] - self.shape[i]) // 2 for i in range(len(self.shape))]
+         x = x[
+             delta[0]:delta[0] + self.shape[0],
+             delta[1]:delta[1] + self.shape[1],
+@@ -72,9 +72,9 @@ class RandomCrop3D:
+     def __call__(self, x, y, mean, stdev):
+         shape = x.get_shape()
+         min = tf.constant(self.margins, dtype=tf.float32)
+-        max = tf.constant([shape[0].value - self.shape[0] - self.margins[0],
+-                           shape[1].value - self.shape[1] - self.margins[1],
+-                           shape[2].value - self.shape[2] - self.margins[2]], dtype=tf.float32)
++        max = tf.constant([shape[0] - self.shape[0] - self.margins[0],
++                           shape[1] - self.shape[1] - self.margins[1],
++                           shape[2] - self.shape[2] - self.margins[2]], dtype=tf.float32)
+         center = tf.random_uniform((len(self.shape),), minval=min, maxval=max)
+         center = tf.cast(center, dtype=tf.int32)
+         x = x[center[0]:center[0] + self.shape[0],
+@@ -165,7 +165,7 @@ class RandomBrightnessCorrection:
+ 
+     def __call__(self, x, y, mean, stdev):
+         mask = tf.math.greater(x, 0)
+-        size = x.get_shape()[-1].value if self._per_channel else 1
++        size = x.get_shape()[-1] if self._per_channel else 1
+         augment = tf.random_uniform([]) > self._threshold
+         correction = tf.random_uniform([size],
+                                        minval=self._alpha_range[0],
+diff --git a/TensorFlow/Segmentation/UNet_3D_Medical/main.py b/TensorFlow/Segmentation/UNet_3D_Medical/main.py
+index cefabd23..be9c793e 100644
+--- a/TensorFlow/Segmentation/UNet_3D_Medical/main.py
++++ b/TensorFlow/Segmentation/UNet_3D_Medical/main.py
+@@ -17,13 +17,67 @@ import logging
+ 
+ import numpy as np
+ import tensorflow as tf
+-import horovod.tensorflow as hvd
++#import horovod.tensorflow as hvd
+ 
+ from dataset.data_loader import Dataset, CLASSES
+ from runtime.hooks import get_hooks, ProfilingHook, TrainingHook
+ from runtime.arguments import PARSER
+-from runtime.setup import prepare_model_dir, build_estimator, set_flags, get_logger
+-
++from runtime.setup import prepare_model_dir, build_estimator, set_flags#, get_logger
++
++import sys
++from tensorflow.core.protobuf import config_pb2
++from tensorflow.python.training.session_run_hook import SessionRunArgs
++from tensorflow.python.training import training_util
++from tensorflow.python.platform import gfile
++from tensorflow.python.client import timeline
++from datetime import datetime
++
++class LoggerHook(tf.estimator.SessionRunHook):
++  """ Logs runtime. """
++  def __init__(self, batch_size, run_profile):
++    self.batch_size = batch_size
++    self.run_profile = run_profile
++
++  def begin(self):
++    self._step = 0
++    self._total_duration = 0
++    self._warmup = 2
++    self._global_step_tensor = training_util._get_or_create_global_step_read()
++
++  def before_run(self, run_context):
++    self._start_time = datetime.now()
++    opts = config_pb2.RunOptions(trace_level=config_pb2.RunOptions.FULL_TRACE)
++    requests = {"global_step": self._global_step_tensor}
++    if self.run_profile:
++      return SessionRunArgs(requests, options=opts)
++    else:
++      return SessionRunArgs(requests)
++
++  def after_run(self, run_context, run_values):
++    self._step += 1
++    duration = datetime.now() - self._start_time
++    ms = duration.total_seconds() * 1000.00
++    if self._step > self._warmup:
++      self._total_duration += ms
++      if self._step % 1 == 0:
++        print("Current step: %d, time in ms: %.2f" %(self._step, ms), flush=True)
++    else:
++      print("Warmup step: %d, time in ms: %.2f" %(self._step, ms), flush=True)
++    sys.stdout.flush()
++    if self._step == 4 and self.run_profile:
++      with gfile.Open('timeline-3dunet-train-bf16.json', "w") as f:
++        trace = timeline.Timeline(run_values.run_metadata.step_stats)
++        f.write(trace.generate_chrome_trace_format())
++
++  def end(self, run_context):
++    print("self._step: %d" %self._step, flush=True)
++    print("Total time spent (after warmup): %.2f ms" %(self._total_duration), flush=True)
++    print("Time spent per iteration (after warmup): %.2f ms" %(self._total_duration/(self._step - self._warmup)), flush=True)
++    time_takes = self._total_duration / (self._step - self._warmup)
++    if self.batch_size == 1:
++      print("Latency is %.3f ms" % (time_takes), flush=True)
++    print("Throughput is %.3f samples/sec" % (self.batch_size * 1000 / time_takes), flush=True)
++    sys.stdout.flush()
+ 
+ def parse_evaluation_results(result):
+     data = {CLASSES[i]: result[CLASSES[i]] for i in range(len(CLASSES))}
+@@ -34,10 +88,10 @@ def parse_evaluation_results(result):
+ 
+ def main():
+     tf.get_logger().setLevel(logging.ERROR)
+-    hvd.init()
++    #hvd.init()
+     params = PARSER.parse_args()
+     model_dir = prepare_model_dir(params)
+-    logger = get_logger(params)
++    #logger = get_logger(params)
+ 
+     dataset = Dataset(data_dir=params.data_dir,
+                       batch_size=params.batch_size,
+@@ -47,10 +101,12 @@ def main():
+ 
+     estimator = build_estimator(params=params, model_dir=model_dir)
+ 
+-    max_steps = params.max_steps // (1 if params.benchmark else hvd.size())
++    max_steps = params.max_steps #// (1 if params.benchmark else hvd.size())
+ 
+     if 'train' in params.exec_mode:
+-        training_hooks = get_hooks(params, logger)
++        training_hooks = []
++        my_hook = LoggerHook(params.batch_size, False)
++        training_hooks.append(my_hook)
+         estimator.train(
+             input_fn=dataset.train_fn,
+             steps=max_steps,
+@@ -59,12 +115,13 @@ def main():
+     if 'evaluate' in params.exec_mode:
+         result = estimator.evaluate(input_fn=dataset.eval_fn, steps=dataset.eval_size)
+         data = parse_evaluation_results(result)
+-        if hvd.rank() == 0:
+-            logger.log(step=(), data=data)
++        if True: #hvd.rank() == 0:
++            #logger.log(step=(), data=data)
++            print(data)
+ 
+     if 'predict' == params.exec_mode:
+-        inference_hooks = get_hooks(params, logger)
+-        if hvd.rank() == 0:
++        inference_hooks = get_hooks(params)
++        if True: #hvd.rank() == 0:
+             count = 1 if not params.benchmark else 2 * params.warmup_steps * params.batch_size // dataset.test_size
+             predictions = estimator.predict(
+                 input_fn=lambda: dataset.test_fn(count=count,
+@@ -76,14 +133,12 @@ def main():
+                     np.save(os.path.join(params.model_dir, "vol_{}.npy".format(idx)), volume)
+ 
+     if 'debug_train' == params.exec_mode:
+-        hooks = [hvd.BroadcastGlobalVariablesHook(0)]
+-        if hvd.rank() == 0:
++        hooks = [] #[hvd.BroadcastGlobalVariablesHook(0)]
++        if True: #hvd.rank() == 0:
+             hooks += [TrainingHook(log_every=params.log_every,
+-                                   logger=logger,
+                                    tensor_names=['total_loss_ref:0']),
+                       ProfilingHook(warmup_steps=params.warmup_steps,
+-                                    global_batch_size=hvd.size() * params.batch_size,
+-                                    logger=logger,
++                                    global_batch_size=params.batch_size,#hvd.size() * params.batch_size,
+                                     mode='train')]
+ 
+         estimator.train(
+@@ -92,10 +147,9 @@ def main():
+             hooks=hooks)
+ 
+     if 'debug_predict' == params.exec_mode:
+-        if hvd.rank() == 0:
++        if True: #hvd.rank() == 0:
+             hooks = [ProfilingHook(warmup_steps=params.warmup_steps,
+                                    global_batch_size=params.batch_size,
+-                                   logger=logger,
+                                    mode='inference')]
+             count = 2 * params.warmup_steps
+             predictions = estimator.predict(input_fn=lambda: dataset.synth_predict_fn(count=count),
+diff --git a/TensorFlow/Segmentation/UNet_3D_Medical/model/layers.py b/TensorFlow/Segmentation/UNet_3D_Medical/model/layers.py
+index bd981651..0bbafd4f 100644
+--- a/TensorFlow/Segmentation/UNet_3D_Medical/model/layers.py
++++ b/TensorFlow/Segmentation/UNet_3D_Medical/model/layers.py
+@@ -12,13 +12,19 @@
+ # See the License for the specific language governing permissions and
+ # limitations under the License.
+ 
+-import tensorflow as tf
++import tensorflow.compat.v1 as tf
++import tensorflow_addons as tfa
+ 
+ 
+ def _normalization(inputs, name, mode):
+     training = mode == tf.estimator.ModeKeys.TRAIN
+ 
+     if name == 'instancenorm':
++        return tfa.layers.InstanceNormalization(
++            center=True,
++            scale=True,
++            epsilon=1e-6,
++            )(inputs)
+         gamma_initializer = tf.constant_initializer(1.0)
+         return tf.contrib.layers.instance_norm(
+             inputs,
+@@ -32,6 +38,7 @@ def _normalization(inputs, name, mode):
+             trainable=True,
+             data_format='NHWC',
+             scope=None)
++        
+ 
+     if name == 'groupnorm':
+         return tf.contrib.layers.group_norm(inputs=inputs,
+diff --git a/TensorFlow/Segmentation/UNet_3D_Medical/model/losses.py b/TensorFlow/Segmentation/UNet_3D_Medical/model/losses.py
+index 26eff0bb..c1e1fd19 100644
+--- a/TensorFlow/Segmentation/UNet_3D_Medical/model/losses.py
++++ b/TensorFlow/Segmentation/UNet_3D_Medical/model/losses.py
+@@ -16,6 +16,8 @@ import tensorflow as tf
+ 
+ 
+ def make_loss(params, y_true, y_pred):
++    y_true = tf.cast(y_true, tf.float32)
++    y_pred = tf.cast(y_pred, tf.float32)
+     if params.loss == 'dice':
+         return _dice(y_true, y_pred)
+     if params.loss == 'ce':
+diff --git a/TensorFlow/Segmentation/UNet_3D_Medical/model/model_fn.py b/TensorFlow/Segmentation/UNet_3D_Medical/model/model_fn.py
+index 9837d588..e53d2aa7 100644
+--- a/TensorFlow/Segmentation/UNet_3D_Medical/model/model_fn.py
++++ b/TensorFlow/Segmentation/UNet_3D_Medical/model/model_fn.py
+@@ -14,8 +14,9 @@
+ 
+ import os
+ 
+-import horovod.tensorflow as hvd
+-import tensorflow as tf
++#import horovod.tensorflow as hvd
++import tensorflow.compat.v1 as tf
++import tensorflow as tf2
+ 
+ from model.unet3d import Builder
+ from model.losses import make_loss, eval_dice, total_dice
+@@ -23,8 +24,12 @@ from dataset.data_loader import CLASSES
+ 
+ 
+ def unet_3d(features, labels, mode, params):
++    #features = tf.cast(features, tf.bfloat16)
++    #tf2.keras.mixed_precision.set_global_policy('mixed_bfloat16')
++    #with tf.compat.v1.tpu.bfloat16_scope():
+ 
+     logits = Builder(n_classes=4, normalization=params.normalization, mode=mode)(features)
++    #logits = tf.cast(logits, tf.float32)
+ 
+     if mode == tf.estimator.ModeKeys.PREDICT:
+         prediction = tf.argmax(input=logits, axis=-1, output_type=tf.dtypes.int32)
+@@ -50,7 +55,7 @@ def unet_3d(features, labels, mode, params):
+     global_step = tf.compat.v1.train.get_or_create_global_step()
+ 
+     optimizer = tf.compat.v1.train.AdamOptimizer(learning_rate=params.learning_rate)
+-    optimizer = hvd.DistributedOptimizer(optimizer)
++    #optimizer = hvd.DistributedOptimizer(optimizer)
+ 
+     # NGC has TF_ENABLE_AUTO_MIXED_PRECISION enabled by default. We cannot use
+     # both graph_rewrite and envar, so if we're not in NGC we do graph_rewrite
+@@ -58,13 +63,13 @@ def unet_3d(features, labels, mode, params):
+         amp_envar = int(os.environ['TF_ENABLE_AUTO_MIXED_PRECISION']) == 1
+     except KeyError:
+         amp_envar = False
+-
++    '''
+     if params.use_amp and not amp_envar:
+         optimizer = tf.train.experimental.enable_mixed_precision_graph_rewrite(
+             optimizer,
+             loss_scale='dynamic'
+         )
+-
++    '''
+     with tf.control_dependencies(tf.get_collection(tf.GraphKeys.UPDATE_OPS)):
+         train_op = optimizer.minimize(loss, global_step=global_step)
+ 
+diff --git a/TensorFlow/Segmentation/UNet_3D_Medical/runtime/hooks.py b/TensorFlow/Segmentation/UNet_3D_Medical/runtime/hooks.py
+index 576b6f5b..03d476d6 100644
+--- a/TensorFlow/Segmentation/UNet_3D_Medical/runtime/hooks.py
++++ b/TensorFlow/Segmentation/UNet_3D_Medical/runtime/hooks.py
+@@ -15,43 +15,40 @@
+ import time
+ 
+ import numpy as np
+-import tensorflow as tf
+-import horovod.tensorflow as hvd
++import tensorflow.compat.v1 as tf
++#import horovod.tensorflow as hvd
++import sys
+ 
+ 
+-def get_hooks(params, logger):
++def get_hooks(params):
+     if 'train' in params.exec_mode:
+-        hooks = [hvd.BroadcastGlobalVariablesHook(0)]
+-        if hvd.rank() == 0:
++        hooks = []#[hvd.BroadcastGlobalVariablesHook(0)]
++        if True:#hvd.rank() == 0:
+             if params.benchmark:
+                 hooks += [ProfilingHook(warmup_steps=params.warmup_steps,
+-                                        global_batch_size=hvd.size() * params.batch_size,
+-                                        logger=logger,
++                                        global_batch_size=params.batch_size, #hvd.size() * params.batch_size,
+                                         mode='train')]
+             else:
+                 hooks += [TrainingHook(log_every=params.log_every,
+-                                       logger=logger,
+                                        tensor_names=['total_loss_ref:0'])]
+         return hooks
+ 
+     elif 'predict' == params.exec_mode:
+         hooks = []
+-        if hvd.rank() == 0:
++        if True: #hvd.rank() == 0:
+             if params.benchmark:
+                 hooks += [ProfilingHook(warmup_steps=params.warmup_steps,
+                                         global_batch_size=params.batch_size,
+-                                        logger=logger,
+                                         mode='test')]
+             return hooks
+ 
+ 
+ class ProfilingHook(tf.estimator.SessionRunHook):
+-    def __init__(self, warmup_steps, global_batch_size, logger, mode):
++    def __init__(self, warmup_steps, global_batch_size, mode):
+         self._warmup_steps = warmup_steps
+         self._global_batch_size = global_batch_size
+         self._step = 0
+         self._timestamps = []
+-        self._logger = logger
+         self._mode = mode
+ 
+     def before_run(self, run_context):
+@@ -65,15 +62,16 @@ class ProfilingHook(tf.estimator.SessionRunHook):
+                                           self._global_batch_size,
+                                           self._mode)
+ 
+-        self._logger.log(step=(), data={metric: float(value) for (metric, value) in stats})
+-        self._logger.flush()
++        #self._logger.log(step=(), data={metric: float(value) for (metric, value) in stats})
++        #self._logger.flush()
++        print({metric: float(value) for (metric, value) in stats})
++        sys.stdout.flush()
+ 
+ 
+ class TrainingHook(tf.estimator.SessionRunHook):
+-    def __init__(self, log_every, logger, tensor_names):
++    def __init__(self, log_every, tensor_names):
+         self._log_every = log_every
+         self._step = 0
+-        self._logger = logger
+         self._tensor_names = tensor_names
+ 
+     def before_run(self, run_context):
+@@ -88,11 +86,15 @@ class TrainingHook(tf.estimator.SessionRunHook):
+                   run_values):
+         if self._step % self._log_every == 0:
+             for i in range(len(self._tensor_names)):
+-                self._logger.log(step=(self._step,), data={self._tensor_names[i]: str(run_values.results[i])})
++                #self._logger.log(step=(self._step,), data={self._tensor_names[i]: str(run_values.results[i])})
++                print('current step = %d'%self._step, end=' ')
++                print({self._tensor_names[i]: str(run_values.results[i])})
++                
+         self._step += 1
+ 
+     def end(self, session):
+-        self._logger.flush()
++        #self._logger.flush()
++        sys.stdout.flush()
+ 
+ 
+ def process_performance_stats(timestamps, batch_size, mode):
+diff --git a/TensorFlow/Segmentation/UNet_3D_Medical/runtime/setup.py b/TensorFlow/Segmentation/UNet_3D_Medical/runtime/setup.py
+index a1bd4cdc..2de6b151 100644
+--- a/TensorFlow/Segmentation/UNet_3D_Medical/runtime/setup.py
++++ b/TensorFlow/Segmentation/UNet_3D_Medical/runtime/setup.py
+@@ -16,10 +16,10 @@ import os
+ import pickle
+ import shutil
+ 
+-import dllogger as logger
++#import dllogger as logger
+ import tensorflow as tf
+-import horovod.tensorflow as hvd
+-from dllogger import StdOutBackend, Verbosity, JSONStreamBackend
++#import horovod.tensorflow as hvd
++#from dllogger import StdOutBackend, Verbosity, JSONStreamBackend
+ 
+ from model.model_fn import unet_3d
+ 
+@@ -38,7 +38,7 @@ def set_flags():
+ 
+ def prepare_model_dir(params):
+     model_dir = os.path.join(params.model_dir, "model_chckpt")
+-    model_dir = model_dir if (hvd.rank() == 0 and not params.benchmark) else None
++    model_dir = model_dir if (not params.benchmark) else None#if (hvd.rank() == 0 and not params.benchmark) else None
+     if model_dir is not None:
+         os.makedirs(model_dir, exist_ok=True)
+         if ('train' in params.exec_mode) and (not params.resume_training):
+@@ -49,17 +49,23 @@ def prepare_model_dir(params):
+ 
+ def build_estimator(params, model_dir):
+     config = tf.compat.v1.ConfigProto(gpu_options=tf.compat.v1.GPUOptions(), allow_soft_placement=True)
++    #config = tf.compat.v1.ConfigProto(allow_soft_placement=True)
++
+ 
+     if params.use_xla:
+         config.graph_options.optimizer_options.global_jit_level = tf.OptimizerOptions.ON_1
+ 
+     config.gpu_options.allow_growth = True
+-    config.gpu_options.visible_device_list = str(hvd.local_rank())
++    #config.gpu_options.visible_device_list = '0'#str(hvd.local_rank())
+ 
++    
+     if params.use_amp:
+-        config.graph_options.rewrite_options.auto_mixed_precision = 1
++        #config.graph_options.rewrite_options.auto_mixed_precision = 1
++        policy = tf.keras.mixed_precision.Policy("mixed_bfloat16")
++        tf.keras.mixed_precision.set_global_policy(policy)
++    
+ 
+-    checkpoint_steps = (params.max_steps // hvd.size()) if hvd.rank() == 0 else None
++    checkpoint_steps = params.max_steps #(params.max_steps // hvd.size()) if hvd.rank() == 0 else None
+     checkpoint_steps = checkpoint_steps if not params.benchmark else None
+     run_config = tf.estimator.RunConfig(
+         save_summary_steps=params.max_steps,
+@@ -73,12 +79,13 @@ def build_estimator(params, model_dir):
+         config=run_config,
+         params=params)
+ 
+-
++'''
+ def get_logger(params):
+     backends = []
+-    if hvd.rank() == 0:
++    if True: #hvd.rank() == 0:
+         backends += [StdOutBackend(Verbosity.VERBOSE)]
+         if params.log_dir:
+             backends += [JSONStreamBackend(Verbosity.VERBOSE, params.log_dir)]
+     logger.init(backends=backends)
+     return logger
++'''
+\ No newline at end of file
diff --git a/models_v2/tensorflow/3d_unet/training/gpu/3dunet_itex_with_horovod.patch b/models_v2/tensorflow/3d_unet/training/gpu/3dunet_itex_with_horovod.patch
new file mode 100644
index 000000000..07ab200dd
--- /dev/null
+++ b/models_v2/tensorflow/3d_unet/training/gpu/3dunet_itex_with_horovod.patch
@@ -0,0 +1,155 @@
+diff --git a/TensorFlow/Segmentation/UNet_3D_Medical/dataset/transforms.py b/TensorFlow/Segmentation/UNet_3D_Medical/dataset/transforms.py
+index 15e6244f..e31de05c 100644
+--- a/TensorFlow/Segmentation/UNet_3D_Medical/dataset/transforms.py
++++ b/TensorFlow/Segmentation/UNet_3D_Medical/dataset/transforms.py
+@@ -12,7 +12,7 @@
+ # See the License for the specific language governing permissions and
+ # limitations under the License.
+ 
+-import tensorflow as tf
++import tensorflow.compat.v1 as tf
+ 
+ 
+ def apply_transforms(x, y, mean, stdev, transforms):
+@@ -48,7 +48,7 @@ class CenterCrop:
+ 
+     def __call__(self, x, y, mean, stdev):
+         shape = x.get_shape()
+-        delta = [(shape[i].value - self.shape[i]) // 2 for i in range(len(self.shape))]
++        delta = [(shape[i] - self.shape[i]) // 2 for i in range(len(self.shape))]
+         x = x[
+             delta[0]:delta[0] + self.shape[0],
+             delta[1]:delta[1] + self.shape[1],
+@@ -72,9 +72,9 @@ class RandomCrop3D:
+     def __call__(self, x, y, mean, stdev):
+         shape = x.get_shape()
+         min = tf.constant(self.margins, dtype=tf.float32)
+-        max = tf.constant([shape[0].value - self.shape[0] - self.margins[0],
+-                           shape[1].value - self.shape[1] - self.margins[1],
+-                           shape[2].value - self.shape[2] - self.margins[2]], dtype=tf.float32)
++        max = tf.constant([shape[0] - self.shape[0] - self.margins[0],
++                           shape[1] - self.shape[1] - self.margins[1],
++                           shape[2] - self.shape[2] - self.margins[2]], dtype=tf.float32)
+         center = tf.random_uniform((len(self.shape),), minval=min, maxval=max)
+         center = tf.cast(center, dtype=tf.int32)
+         x = x[center[0]:center[0] + self.shape[0],
+@@ -165,7 +165,7 @@ class RandomBrightnessCorrection:
+ 
+     def __call__(self, x, y, mean, stdev):
+         mask = tf.math.greater(x, 0)
+-        size = x.get_shape()[-1].value if self._per_channel else 1
++        size = x.get_shape()[-1] if self._per_channel else 1
+         augment = tf.random_uniform([]) > self._threshold
+         correction = tf.random_uniform([size],
+                                        minval=self._alpha_range[0],
+diff --git a/TensorFlow/Segmentation/UNet_3D_Medical/model/layers.py b/TensorFlow/Segmentation/UNet_3D_Medical/model/layers.py
+index bd981651..746f3112 100644
+--- a/TensorFlow/Segmentation/UNet_3D_Medical/model/layers.py
++++ b/TensorFlow/Segmentation/UNet_3D_Medical/model/layers.py
+@@ -12,13 +12,20 @@
+ # See the License for the specific language governing permissions and
+ # limitations under the License.
+ 
+-import tensorflow as tf
++import tensorflow.compat.v1 as tf
++import tensorflow_addons as tfa
+ 
+ 
+ def _normalization(inputs, name, mode):
+     training = mode == tf.estimator.ModeKeys.TRAIN
+ 
+     if name == 'instancenorm':
++        return tfa.layers.InstanceNormalization(
++            center=True,
++            scale=True,
++            epsilon=1e-6,
++            )(inputs)
++
+         gamma_initializer = tf.constant_initializer(1.0)
+         return tf.contrib.layers.instance_norm(
+             inputs,
+diff --git a/TensorFlow/Segmentation/UNet_3D_Medical/model/losses.py b/TensorFlow/Segmentation/UNet_3D_Medical/model/losses.py
+index 26eff0bb..c1e1fd19 100644
+--- a/TensorFlow/Segmentation/UNet_3D_Medical/model/losses.py
++++ b/TensorFlow/Segmentation/UNet_3D_Medical/model/losses.py
+@@ -16,6 +16,8 @@ import tensorflow as tf
+ 
+ 
+ def make_loss(params, y_true, y_pred):
++    y_true = tf.cast(y_true, tf.float32)
++    y_pred = tf.cast(y_pred, tf.float32)
+     if params.loss == 'dice':
+         return _dice(y_true, y_pred)
+     if params.loss == 'ce':
+diff --git a/TensorFlow/Segmentation/UNet_3D_Medical/model/model_fn.py b/TensorFlow/Segmentation/UNet_3D_Medical/model/model_fn.py
+index 9837d588..f3886cef 100644
+--- a/TensorFlow/Segmentation/UNet_3D_Medical/model/model_fn.py
++++ b/TensorFlow/Segmentation/UNet_3D_Medical/model/model_fn.py
+@@ -15,7 +15,7 @@
+ import os
+ 
+ import horovod.tensorflow as hvd
+-import tensorflow as tf
++import tensorflow.compat.v1 as tf
+ 
+ from model.unet3d import Builder
+ from model.losses import make_loss, eval_dice, total_dice
+@@ -59,11 +59,13 @@ def unet_3d(features, labels, mode, params):
+     except KeyError:
+         amp_envar = False
+ 
++    '''
+     if params.use_amp and not amp_envar:
+         optimizer = tf.train.experimental.enable_mixed_precision_graph_rewrite(
+             optimizer,
+             loss_scale='dynamic'
+         )
++    '''
+ 
+     with tf.control_dependencies(tf.get_collection(tf.GraphKeys.UPDATE_OPS)):
+         train_op = optimizer.minimize(loss, global_step=global_step)
+diff --git a/TensorFlow/Segmentation/UNet_3D_Medical/runtime/hooks.py b/TensorFlow/Segmentation/UNet_3D_Medical/runtime/hooks.py
+index 576b6f5b..4b145013 100644
+--- a/TensorFlow/Segmentation/UNet_3D_Medical/runtime/hooks.py
++++ b/TensorFlow/Segmentation/UNet_3D_Medical/runtime/hooks.py
+@@ -15,7 +15,7 @@
+ import time
+ 
+ import numpy as np
+-import tensorflow as tf
++import tensorflow.compat.v1 as tf
+ import horovod.tensorflow as hvd
+ 
+ 
+@@ -102,7 +102,7 @@ def process_performance_stats(timestamps, batch_size, mode):
+     n = np.sqrt(len(timestamps_ms))
+     throughput_imgps = (1000.0 * batch_size / timestamps_ms).mean()
+ 
+-    stats = [("throughput_{}".format(mode), str(throughput_imgps)),
++    stats = [("[Summary of all {} ranks] total_throughput_{}".format(hvd.size(), mode), str(throughput_imgps)),
+              ('latency_{}:'.format(mode), str(latency_ms))]
+     for ci, lvl in zip(["90%:", "95%:", "99%:"],
+                        [1.645, 1.960, 2.576]):
+diff --git a/TensorFlow/Segmentation/UNet_3D_Medical/runtime/setup.py b/TensorFlow/Segmentation/UNet_3D_Medical/runtime/setup.py
+index a1bd4cdc..72516ad7 100644
+--- a/TensorFlow/Segmentation/UNet_3D_Medical/runtime/setup.py
++++ b/TensorFlow/Segmentation/UNet_3D_Medical/runtime/setup.py
+@@ -53,11 +53,17 @@ def build_estimator(params, model_dir):
+     if params.use_xla:
+         config.graph_options.optimizer_options.global_jit_level = tf.OptimizerOptions.ON_1
+ 
++    gpus = tf.config.experimental.list_physical_devices('XPU')
++    for gpu in gpus:
++        tf.config.experimental.set_memory_growth(gpu, True)
++    if gpus:
++        tf.config.experimental.set_visible_devices(gpus[hvd.local_rank()], 'XPU')
+     config.gpu_options.allow_growth = True
+     config.gpu_options.visible_device_list = str(hvd.local_rank())
+ 
+     if params.use_amp:
+-        config.graph_options.rewrite_options.auto_mixed_precision = 1
++        policy = tf.keras.mixed_precision.Policy("mixed_bfloat16")
++        tf.keras.mixed_precision.set_global_policy(policy)
+ 
+     checkpoint_steps = (params.max_steps // hvd.size()) if hvd.rank() == 0 else None
+     checkpoint_steps = checkpoint_steps if not params.benchmark else None
diff --git a/models_v2/tensorflow/3d_unet/training/gpu/CONTAINER.md b/models_v2/tensorflow/3d_unet/training/gpu/CONTAINER.md
new file mode 100644
index 000000000..5cfbdafa0
--- /dev/null
+++ b/models_v2/tensorflow/3d_unet/training/gpu/CONTAINER.md
@@ -0,0 +1,68 @@
+# TensorFlow UNet-3D Training
+
+## Description 
+This document has instructions for running UNet-3D training with BFloat16 precision using Intel® Extension for TensorFlow on Intel® Data Center GPU Max Series.
+
+## Datasets
+
+Download the dataset by registering on [Brain Tumor Segmentation 2019 dataset](https://www.med.upenn.edu/cbica/brats-2019/). After downloading dataset, follow steps in the [Quick Start Guide](https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow/Segmentation/UNet_3D_Medical#quick-start-guide) to pre-process the dataset in tfecords format. 
+
+Set the `DATASET_DIR` to point to the TF records directory when running UNet-3D training. 
+
+## Quick Start Scripts
+
+| Script name | Description |
+|-------------|-------------|
+| `run_model.sh` | Runs BFloat16 Training on single and two tiles |
+
+Requirements:
+
+* Host has [Intel® Data Center GPU Max Series](https://ark.intel.com/content/www/us/en/ark/products/series/232874/intel-data-center-gpu-max-series.html)
+* Host has installed latest Intel® Data Center GPU Max Series [Drivers](https://dgpu-docs.intel.com/driver/installation.html)
+* Docker
+
+### Docker pull command:
+
+```bash
+docker pull intel/image-segmentation:tf-max-gpu-unet-3d-training
+```
+The UNet-3D training container includes scripts, model and libraries needed to run BFloat16 Training. To run the quickstart scripts using this container, you'll need to provide volume mounts of the Brain Tumor Segmentation dataset. You will also need to provide an output directory to store logs.
+
+```bash
+#Required
+export DATASET_DIR=<provide path to pre-processed dataset>
+export PRECISION=bfloat16
+export MULTI_TILE=<provide True for multi-tile training, False for single-tile training>
+export OUTPUT_DIR=<provide path to output logs directory>
+SCRIPT=run_model.sh
+
+#Optional
+export BATCH_SIZE=<provide batch size. Otherwise default is 1>
+IMAGE_NAME=intel/image-segmentation:tf-max-gpu-unet-3d-training
+
+docker run -it \
+--device /dev/dri \
+--env BATCH_SIZE=${BATCH_SIZE} \
+--env MULTI_TILE=${MULTI_TILE} \
+--env DATASET_DIR=${DATASET_DIR} \
+--env PRECISION=${PRECISION} \
+--env OUTPUT_DIR=${OUTPUT_DIR} \
+--env http_proxy=${http_proxy} \
+--env https_proxy=${https_proxy} \
+--env no_proxy=${no_proxy} \
+--volume ${DATASET_DIR}:${DATASET_DIR} \
+--volume /dev/dri:/dev/dri \
+--volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
+${IMAGE_NAME} \
+/bin/bash $SCRIPT
+```
+## Documentation and Sources
+
+[GitHub* Repository](https://github.com/IntelAI/models/tree/master/docker/max-gpu)
+
+## Support
+Support for Intel® Extension for TensorFlow* is available at [Intel® AI Analytics Toolkit](https://www.intel.com/content/www/us/en/developer/tools/oneapi/ai-analytics-toolkit.html#gs.qbretz) Additionally, the Intel® Extension for TensorFlow* team tracks both bugs and enhancement requests using [GitHub issues](https://github.com/intel/intel-extension-for-tensorflow/issues). Before submitting a suggestion or bug report, please search the GitHub issues to see if your issue has already been reported.
+
+## License Agreement
+
+LEGAL NOTICE: By accessing, downloading or using this software and any required dependent software (the “Software Package”), you agree to the terms and conditions of the software license agreements for the Software Package, which may also include notices, disclaimers, or license terms for third party software included with the Software Package. Please refer to the [license file](https://github.com/IntelAI/models/tree/master/third_party) for additional details.
diff --git a/models_v2/tensorflow/3d_unet/training/gpu/README.md b/models_v2/tensorflow/3d_unet/training/gpu/README.md
new file mode 100644
index 000000000..db7e8e3fd
--- /dev/null
+++ b/models_v2/tensorflow/3d_unet/training/gpu/README.md
@@ -0,0 +1,80 @@
+# Unet-3D Model Training for Intel® Extention for TensorFlow
+Best known method of Unet-3D training for Intel® Extention for TensorFlow.
+
+## Model Information
+| **Use Case** |**Framework** | **Model Repo** | **Branch/Commit/Tag** | **Optional Patch**
+| :---: | :---: | :---: | :---: | :---: |
+| Training | TensorFlow | [DeepLearningExamples/UNet_3D](https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow/Segmentation/UNet_3D_Medical) | master | 3dunet_itex.patch <br /> 3dunet_itex_with_horovod.patch |
+
+**Note**: Refer to [CONTAINER.md](CONTAINER.md) for UNet-3D training instructions using docker containers.
+# Pre-Requisite
+* Host has Intel® Data Center GPU Max Series
+* Host has installed latest Intel® Data Center GPU Max Series Drivers https://dgpu-docs.intel.com/driver/installation.html
+* The following Intel® oneAPI Base Toolkit components are required:
+  - Intel® oneAPI DPC++ Compiler (Placeholder DPCPPROOT as its installation path)
+  - Intel® oneAPI Math Kernel Library (oneMKL) (Placeholder MKLROOT as its installation path)
+  - Intel® oneAPI MPI Library
+  - Intel® oneAPI TBB Library
+  - Intel® oneAPI CCL Library
+
+  Follow instructions at [Intel® oneAPI Base Toolkit Download page](https://www.intel.com/content/www/us/en/developer/tools/oneapi/base-toolkit-download.html?operatingsystem=linux) to setup the package manager repository.
+
+# Dataset 
+The 3D-UNet model was trained in the [Brain Tumor Segmentation 2019 dataset](https://www.med.upenn.edu/cbica/brats2019/data.html). Test images provided by the organization were used to produce the resulting masks for submission. Upon registration, the challenge's data is made available through the https//ipp.cbica.upenn.edu service.
+
+
+## Run Model
+1. `git clone https://github.com/IntelAI/models.git`
+2. `cd models/models_v2/tensorflow/3d_unet/training/gpu`
+3. Create virtual environment `venv` and activate it:
+    ```
+    python3 -m venv venv
+    . ./venv/bin/activate
+    ```
+4. Run setup.sh
+    ```
+    ./setup.sh
+    ```
+5. Install [tensorflow and ITEX](https://pypi.org/project/intel-extension-for-tensorflow/)
+6. If you run multi-tile (export MULTI_TILE=True), please install "intel-optimization-for-horovod" with `pip install intel-optimization-for-horovod`
+7. Set environment variables for Intel® oneAPI Base Toolkit: 
+    Default installation location `{ONEAPI_ROOT}` is `/opt/intel/oneapi` for root account, `${HOME}/intel/oneapi` for other accounts
+    ```bash
+    source {ONEAPI_ROOT}/compiler/latest/env/vars.sh
+    source {ONEAPI_ROOT}/mkl/latest/env/vars.sh
+    source {ONEAPI_ROOT}/tbb/latest/env/vars.sh
+    source {ONEAPI_ROOT}/mpi/latest/env/vars.sh
+    source {ONEAPI_ROOT}/ccl/latest/env/vars.sh
+8. Setup required environment paramaters
+
+    |   **Parameter**    | **export command**                                    |
+    | :---: | :--- |
+    |  **DATASET_DIR**   | `export DATASET_DIR=/the/path/to/dataset`             |
+    |   **OUTPUT_DIR**   | `export OUTPUT_DIR=/the/path/to/output_dir`           |
+    |   **MULTI_TILE**   | `export MULTI_TILE=False (False or True)`           |
+    |   **BATCH_SIZE** (optional)   | `export BATCH_SIZE=1`           |
+    |   **PRECISION**   | `export PRECISION=bfloat16` (bfloat16 or fp32)           |
+6. Run `run_model.sh`
+
+## Output
+
+Output will typically looks like:
+```
+Current step: 997, time in ms: 73.95
+Current step: 998, time in ms: 74.81
+Current step: 999, time in ms: 74.31
+Current step: 1000, time in ms: 74.44
+self._step: 1000
+Total time spent (after warmup): 79390.14 ms
+Time spent per iteration (after warmup): 79.55 ms
+Latency is 79.549 ms
+Throughput is xxx samples/sec
+```
+
+Final results of the training run can be found in `results.yaml` file.
+```
+results:
+ - key: throughput
+   value: xxx
+   unit: images/sec
+```
diff --git a/models_v2/tensorflow/3d_unet/training/gpu/requirements.txt b/models_v2/tensorflow/3d_unet/training/gpu/requirements.txt
new file mode 100644
index 000000000..854657d8d
--- /dev/null
+++ b/models_v2/tensorflow/3d_unet/training/gpu/requirements.txt
@@ -0,0 +1,18 @@
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+tfa-nightly
+git+https://github.com/NVIDIA/dllogger.git
\ No newline at end of file
diff --git a/models_v2/tensorflow/3d_unet/training/gpu/run_model.sh b/models_v2/tensorflow/3d_unet/training/gpu/run_model.sh
new file mode 100755
index 000000000..c2c9c9d0a
--- /dev/null
+++ b/models_v2/tensorflow/3d_unet/training/gpu/run_model.sh
@@ -0,0 +1,124 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+
+#!/bin/bash
+
+# Create an array of input directories that are expected and then verify that they exist
+declare -A input_envs
+input_envs[PRECISION]=${PRECISION}
+input_envs[OUTPUT_DIR]=${OUTPUT_DIR}
+input_envs[DATASET_DIR]=${DATASET_DIR}
+input_envs[MULTI_TILE]=${MULTI_TILE}
+
+for i in "${!input_envs[@]}"; do
+  var_name=$i
+  env_param=${input_envs[$i]}
+ 
+  if [[ -z $env_param ]]; then
+    echo "The required environment variable $var_name is not set" >&2
+    exit 1
+  fi
+done
+
+if [ -d ${DATASET_DIR} ];then
+  echo "DATASET_DIR is: "${DATASET_DIR}
+else
+  echo "Error: the path of dataset does not exist!"
+  exit 1
+fi
+
+BATCH_SIZE=${BATCH_SIZE:-1}
+OUTPUT_DIR=${OUTPUT_DIR:-$PWD}
+
+if [ ${PRECISION} == "bfloat16" ];then
+  echo "Datatype is bfloat16"
+  AMP="--amp"
+elif [ ${PRECISION} == "fp32" ];then
+  echo "Datatype is fp32"
+  AMP=""
+else
+  echo "Error: "${PRECISION}" not supported yet!"
+  exit 1
+fi
+
+echo 'Running with parameters:'
+echo " DATASET_PATH: ${DATASET_DIR}"
+echo " OUTPUT_DIR: ${OUTPUT_DIR}"
+echo " PRECISION: ${PRECISION}"
+echo " BATCH_SIZE: $BATCH_SIZE"
+echo " MULTI_TILE: $MULTI_TILE"
+
+current_dir=$(pwd)
+if [ $MULTI_TILE == "True" ];then
+  if [ -d "DeepLearningExamples" ]; then
+    echo "Repository already exists. Skipping clone."
+  else
+    mkdir $current_dir/3d_unet_hvd/ && cd $current_dir/3d_unet_hvd/
+    git clone https://github.com/NVIDIA/DeepLearningExamples.git
+    cd DeepLearningExamples
+    git checkout 88eb3cff2f03dad85035621d041e23a14345999e
+    cd TensorFlow/Segmentation/UNet_3D_Medical/
+    git apply $current_dir/3dunet_itex_with_horovod.patch
+    cd $current_dir
+  fi
+else
+  if [ -d "DeepLearningExamples" ]; then
+    echo "Repository already exists. Skipping clone."
+  else
+    mkdir $current_dir/3d_unet/ && cd $current_dir/3d_unet/
+    git clone https://github.com/NVIDIA/DeepLearningExamples.git
+    cd DeepLearningExamples
+    git checkout 88eb3cff2f03dad85035621d041e23a14345999e
+    cd TensorFlow/Segmentation/UNet_3D_Medical/
+    git apply $current_dir/3dunet_itex.patch
+    cd $current_dir
+  fi
+fi
+
+
+if [ $MULTI_TILE == "True" ];then
+  cd 3d_unet_hvd/DeepLearningExamples/TensorFlow/Segmentation/UNet_3D_Medical/
+  mpirun -np 2 -prepend-rank -ppn 2 \
+  python main.py --data_dir $DATASET_DIR --benchmark --model_dir $OUTPUT_DIR \
+  --exec_mode train --warmup_steps 150 --max_steps 1000 --batch_size $BATCH_SIZE \
+  $AMP |& tee Unet3D_training_${PRECISION}_BS${BATCH_SIZE}_${MULTI_TILE}.log
+  value=$(cat ./Unet3D_training_${PRECISION}_BS${BATCH_SIZE}_${MULTI_TILE}.log | grep -oE 'total_throughput_train : [0-9.]+' | awk '{print $NF}')
+else
+  cd 3d_unet/DeepLearningExamples/TensorFlow/Segmentation/UNet_3D_Medical/
+  python main.py --benchmark --data_dir $DATASET_DIR --model_dir $OUTPUT_DIR \
+  --exec_mode train --batch_size $BATCH_SIZE --warmup_steps 150 --max_steps 1000 --log_every 1 \
+  $AMP |& tee Unet3D_training_${PRECISION}_BS${BATCH_SIZE}_${MULTI_TILE}.log
+  value=$(cat ./Unet3D_training_${PRECISION}_BS${BATCH_SIZE}_${MULTI_TILE}.log | grep "Throughput is" | sed -e "s/.*is//")
+fi
+
+key="throughput"
+unit="images/sec"
+
+yaml_content=$(cat <<EOF
+results:
+ - key: $key
+   value: $value
+   unit: $unit
+EOF
+)
+
+# Write the content to a YAML file
+
+echo "$yaml_content" >  ${OUTPUT_DIR}/results.yaml
+echo "YAML file created, path:${OUTPUT_DIR}/results.yaml"
diff --git a/models_v2/tensorflow/3d_unet/training/gpu/setup.sh b/models_v2/tensorflow/3d_unet/training/gpu/setup.sh
new file mode 100755
index 000000000..17661e1eb
--- /dev/null
+++ b/models_v2/tensorflow/3d_unet/training/gpu/setup.sh
@@ -0,0 +1,25 @@
+#!/usr/bin/env bash
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# setup.sh
+#  - install OS pkgs
+#  - should create virtual env & install pip  requirement.txt
+#  - git clones & applying patches
+
+set -e
+apt-get update && apt-get install -y python3-venv 
+pip install -r requirements.txt
diff --git a/models_v2/tensorflow/3d_unet/training/gpu/tests.yaml b/models_v2/tensorflow/3d_unet/training/gpu/tests.yaml
new file mode 100644
index 000000000..35c4da1f4
--- /dev/null
+++ b/models_v2/tensorflow/3d_unet/training/gpu/tests.yaml
@@ -0,0 +1,25 @@
+training-fp32:
+  cmd: bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/setup.sh ${FRAMEWORK_VERSION} ${FRAMEWORK_EXTENSION_VERSION} ${IS_LKG_DROP} ${AIKIT_VERSION}; bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/3d_unet-training/run_test.sh ${PRECISION} ${IS_LKG_DROP} ${DATASET_DIR} ${MULTI_TILE}
+  env:
+    PRECISION: "float32"
+    DATASET_DIR: "/tf_dataset/dataset/3DUNet/MICCAI_BraTS_2019_Data_Training"  
+    MULTI_TILE: "False"
+training-bfloat16:
+  cmd: bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/setup.sh ${FRAMEWORK_VERSION} ${FRAMEWORK_EXTENSION_VERSION} ${IS_LKG_DROP} ${AIKIT_VERSION}; bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/3d_unet-training/run_test.sh ${PRECISION} ${IS_LKG_DROP} ${DATASET_DIR} ${MULTI_TILE}
+  env:
+    PRECISION: "bfloat16"
+    DATASET_DIR: "/tf_dataset/dataset/3DUNet/MICCAI_BraTS_2019_Data_Training"  
+    MULTI_TILE: "False"
+training-fp32:
+  cmd: bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/setup.sh ${FRAMEWORK_VERSION} ${FRAMEWORK_EXTENSION_VERSION} ${IS_LKG_DROP} ${AIKIT_VERSION}; bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/3d_unet-training/run_test.sh ${PRECISION} ${IS_LKG_DROP} ${DATASET_DIR} ${MULTI_TILE}
+  env:
+    PRECISION: "float32"
+    DATASET_DIR: "/tf_dataset/dataset/3DUNet/MICCAI_BraTS_2019_Data_Training"  
+    MULTI_TILE: "True"
+training-bfloat16:
+  cmd: bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/setup.sh ${FRAMEWORK_VERSION} ${FRAMEWORK_EXTENSION_VERSION} ${IS_LKG_DROP} ${AIKIT_VERSION}; bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/3d_unet-training/run_test.sh ${PRECISION} ${IS_LKG_DROP} ${DATASET_DIR} ${MULTI_TILE}
+  env:
+    PRECISION: "bfloat16"
+    DATASET_DIR: "/tf_dataset/dataset/3DUNet/MICCAI_BraTS_2019_Data_Training"  
+    MULTI_TILE: "True"
+
diff --git a/models_v2/tensorflow/bert_large/training/gpu/CONTAINER.md b/models_v2/tensorflow/bert_large/training/gpu/CONTAINER.md
new file mode 100644
index 000000000..67b3a66b3
--- /dev/null
+++ b/models_v2/tensorflow/bert_large/training/gpu/CONTAINER.md
@@ -0,0 +1,64 @@
+# TensorFlow BERT Large Training 
+
+## Description
+This document has instructions for running BERT Large training with BF16 precision using Intel(R) Extension for TensorFlow on Intel® Data Center GPU Max Series.
+
+## Datasets
+
+Follow instructions [here](README.md#prepare-dataset) to download and prepare the dataset for training. Set the environment variable `DATA_DIR` to point to the dataset directory.
+
+## Quick Start Scripts
+| Script name | Description |
+|-------------|-------------|
+| `run_model.sh` | Runs BERT Large BF16 training on single and two tiles |
+
+Requirements:
+* Host has [Intel® Data Center GPU Max Series](https://ark.intel.com/content/www/us/en/ark/products/series/232874/intel-data-center-gpu-max-series.html)
+* Follow instructions to install GPU-compatible [driver](https://dgpu-docs.intel.com/driver/installation.html)
+* Docker
+
+## Docker pull Command
+```
+docker pull intel/language-modeling:tf-max-gpu-bert-large-training
+```
+The BERT Large training container includes scripts, models and libraries needed to run BF16 training.You wil need to volume mount the dataset directory and the output directory where log files will be generated. 
+
+```bash
+export DATA_DIR=<path to processed dataset>
+export RESULTS_DIR=<path to output log files>
+export MULTI_TILE=<provide True for multi-tile training and False for single tile training>
+export DATATYPE=bf16
+
+IMAGE_NAME=intel/language-modeling:tf-max-gpu-bert-large-training
+DOCKER_ARGS="--rm -it"
+SCRIPT= run_model.sh
+
+docker run \
+  --privileged \
+  --device=/dev/dri \
+  --ipc=host \
+  --env DATA_DIR=${DATA_DIR} \
+  --env RESULTS_DIR=${RESULTS_DIR} \
+  --env MULTI_TILE=${MUTI_TILE} \
+  --env http_proxy=${http_proxy} \
+  --env https_proxy=${https_proxy} \
+  --env no_proxy=${no_proxy} \
+  --env DATATYPE=${DATATYPE} \
+  --volume ${RESULTS_DIR}:${RESULTS_DIR} \
+  --volume ${DATA_DIR}:${DATA_DIR} \
+  --volume /dev/dri:/dev/dri \
+  ${DOCKER_ARGS} \
+  $IMAGE_NAME \
+  /bin/bash $SCRIPT
+  ```
+
+## Documentation and Sources
+
+[GitHub* Repository](https://github.com/IntelAI/models/tree/master/docker/max-gpu)
+
+## Support
+Support for Intel® Extension for TensorFlow* is available at [Intel® AI Analytics Toolkit](https://www.intel.com/content/www/us/en/developer/tools/oneapi/ai-analytics-toolkit.html#gs.qbretz) Additionally, the Intel® Extension for TensorFlow* team tracks both bugs and enhancement requests using [GitHub issues](https://github.com/intel/intel-extension-for-tensorflow/issues). Before submitting a suggestion or bug report, please search the GitHub issues to see if your issue has already been reported.
+
+## License Agreement
+
+LEGAL NOTICE: By accessing, downloading or using this software and any required dependent software (the “Software Package”), you agree to the terms and conditions of the software license agreements for the Software Package, which may also include notices, disclaimers, or license terms for third party software included with the Software Package. Please refer to the [license file](https://github.com/IntelAI/models/tree/master/third_party) for additional details.
diff --git a/models_v2/tensorflow/bert_large/training/gpu/README.md b/models_v2/tensorflow/bert_large/training/gpu/README.md
new file mode 100644
index 000000000..dfe9d1150
--- /dev/null
+++ b/models_v2/tensorflow/bert_large/training/gpu/README.md
@@ -0,0 +1,123 @@
+# Bert-Large Model Training
+
+Bert-Large Model Training using Intel® Extension for TensorFlow.
+
+## Model Information
+
+| **Use Case** | **Framework** |          **Model Repo**           | **Branch/Commit/Tag** |**Optional Patch** |
+| :---: | :---: |:---------------------------------:| :---: | :---: |
+|   training   |  TensorFlow   | [NVIDIA/DeepLearningExamples](https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow2/LanguageModeling/BERT) |        master         | [bert-large-itex-bf16.patch](#bert-large-itex-bf16.patch) |
+
+**Note**: Refer to [CONTAINER.md](CONTAINER.md) for BERT-Large training instructions using docker containers.
+# Pre-Requisite
+
+* Host has Intel® Data Center GPU Max.
+* Host has installed latest Intel® Data Center GPU Max Series Driver https://dgpu-docs.intel.com/driver/installation.html
+* The following Intel® oneAPI Base Toolkit components are required:
+  - Intel® oneAPI DPC++ Compiler (Placeholder DPCPPROOT as its installation path)
+  - Intel® oneAPI Math Kernel Library (oneMKL) (Placeholder MKLROOT as its installation path)
+
+  Recommend to use a package manager like apt, yum or dnf to install the packages above. Follow instructions at [Intel® oneAPI Base Toolkit Download page](https://www.intel.com/content/www/us/en/developer/tools/oneapi/base-toolkit-download.html?operatingsystem=linux) to setup the package manager repository.
+
+# Prepare Dataset
+
+ * Prapare total dataset from scratch. [NV-bert repository](https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow2/LanguageModeling/BERT)  provides scripts to download, verify, and extract the SQuAD dataset and pretrained weights for fine-tuning as well as Wikipedia and BookCorpus dataset for pre-training.
+
+ * You can run below scripts to download datasets for fine-tuning and pretraining. Remember to modify the environment varible ```BERT_PREP_WORKING_DIR ``` in ```data/create_datasets_from_start.sh``` to your real data dir.
+   ```
+   git clone https://github.com/NVIDIA/DeepLearningExamples.git
+   cd DeepLearningExamples/TensorFlow2/LanguageModeling/BERT
+   bash data/create_datasets_from_start.sh pretrained
+   ```
+ * Then use below scripts and patch to create small dataset for bert large training performance.
+    ```
+    git clone https://github.com/IntelAI/models.git
+    cp models/models_v2/tensorflow/bert_large/training/gpu/bert-large-itex-bf16.patch .
+    git am --signoff < bert-large-itex-bf16.patch
+    bash data/create_small_dataset_for_benchmark.sh [dataset_output_dir](default: ${PWD}/dataset)
+    ```
+    The dataset structure will be:
+
+  + [dataset_output_dir]
+     - training/
+     - test/
+     - uncased_L-24_H-1024_A-16/
+
+## Run Model
+
+1. If not already cloned, clone Intel® AI Reference Model repository
+`git clone https://github.com/IntelAI/models.git` 
+2. `cd models/models_v2/tensorflow/bert_large/training/gpu`
+3. Create virtual environment `venv` and activate it:
+    ```
+    python3 -m venv venv
+    . ./venv/bin/activate
+    ```
+4. Run setup.sh
+    ```
+    ./setup.sh
+    ```
+5. Install [tensorflow and ITEX](https://pypi.org/project/intel-extension-for-tensorflow/)
+6. If you want to run with Horovod in multi-tile (export MULTI_TILE=True), please install intel-optimization-for-horovod instead of horovod.
+`pip uninstall horovod && pip install intel-optimization-for-horovod`
+6. Setup required environment paramaters
+
+   | **Parameter**                  |              **export command**              |
+   |:--------------------------------------------:|:----------------------------------------------:|
+   | **DATA_DIR**                |    `export DATA_DIR=/the/path/to/dataset`    |
+   | **RESULTS_DIR**                 | `export RESULTS_DIR=/the/path/to/result_dir` |
+   | **DATATYPE**      | `export DATATYPE=bf16` (bf16, tf32 or fp32)  |
+   | **MULTI_TILE**           |  `export MULTI_TILE=False` (False or True)   |
+
+7. Run `run_model.sh`
+
+## Output
+
+Single-tile output will typicall looks like:
+
+```
+I0907 03:06:10.519348 140707593221952 model_training_utils.py:83] Training Summary: 
+{'total_training_steps': 8, 'train_loss': 10.389582633972168}
+I0907 03:06:10.519572 140707593221952 model_training_utils.py:595] -----------------------------
+I0907 03:06:10.519612 140707593221952 model_training_utils.py:596]   Batch size = 32
+I0907 03:06:10.519635 140707593221952 model_training_utils.py:597]   Num steps = 8
+I0907 03:06:10.519656 140707593221952 model_training_utils.py:598]   LR = 0.0005
+I0907 03:06:10.519677 140707593221952 model_training_utils.py:602] Total Training Time = 339.25 for Sequences = 7680
+I0907 03:06:10.519717 140707593221952 model_training_utils.py:604] Throughput Average (sequences/sec) with overhead = 22.64
+I0907 03:06:10.520012 140707593221952 model_training_utils.py:606] Throughput Average (sequences/sec) = 99.05
+I0907 03:06:10.520030 140707593221952 model_training_utils.py:607] -----------------------------
+decayed_learning_rate_at_crossover_point = 5.000000e-04, adjusted_init_lr = 5.000000e-04
+DLL 2023-09-07 03:06:10.520066 -  throughput_train : 99.048 sequences/s
+DLL 2023-09-07 03:06:10.520158 -  total_loss : 10.3896 
+```
+
+Multi-tile output will typicall looks like:
+
+```
+[0] I1030 10:59:04.732321 139681571383104 model_training_utils.py:83] Training Summary: 
+[0] {'total_training_steps': 8, 'train_loss': 10.069790840148926}
+[0] I1030 10:59:04.732495 139681571383104 model_training_utils.py:595] -----------------------------
+[0] I1030 10:59:04.732545 139681571383104 model_training_utils.py:596]   Batch size = 32
+[0] I1030 10:59:04.732575 139681571383104 model_training_utils.py:597]   Num steps = 8
+[0] I1030 10:59:04.732604 139681571383104 model_training_utils.py:598]   LR = 0.0005
+[0] I1030 10:59:04.732630 139681571383104 model_training_utils.py:600] Multi-GPU training with TF Horovod
+[0] I1030 10:59:04.732667 139681571383104 model_training_utils.py:601] hvd.size() = 2
+[0] I1030 10:59:04.732693 139681571383104 model_training_utils.py:602] Total Training Time = 279.63 for Sequences = 15360
+[0] I1030 10:59:04.732739 139681571383104 model_training_utils.py:604] Throughput Average (sequences/sec) with overhead = 54.93
+[0] I1030 10:59:04.732893 139681571383104 model_training_utils.py:606] Throughput Average (sequences/sec) = 197.41
+[0] I1030 10:59:04.732919 139681571383104 model_training_utils.py:607] -----------------------------
+[0] decayed_learning_rate_at_crossover_point = 1.000000e-03, adjusted_init_lr = 1.000000e-03
+[0] DLL 2023-10-30 10:59:04.732944 -  throughput_train : 197.408 sequences/s
+[0] DLL 2023-10-30 10:59:04.733007 -  total_loss : 10.0698 
+[0] 2023:10:30-10:59:08:(273540) |CCL_INFO| finalizing level-zero
+[0] 2023:10:30-10:59:08:(273540) |CCL_INFO| finalized level-zero
+```
+
+Final results of the training run can be found in `results.yaml` file.
+
+```
+results:
+ - key: throughput
+   value: 99.048
+   unit: sequences/s
+```
diff --git a/models_v2/tensorflow/bert_large/training/gpu/bert-large-itex-bf16.patch b/models_v2/tensorflow/bert_large/training/gpu/bert-large-itex-bf16.patch
new file mode 100644
index 000000000..eec3d88ec
--- /dev/null
+++ b/models_v2/tensorflow/bert_large/training/gpu/bert-large-itex-bf16.patch
@@ -0,0 +1,1044 @@
+From a82f7b321dc4f6b2e11441bdbe49d836432d7c98 Mon Sep 17 00:00:00 2001
+From: "Wang, Yiting" <yiting.wang@intel.com>
+Date: Fri, 3 Nov 2023 00:37:48 -0700
+Subject: [PATCH] 1103 patch for bert large training
+
+---
+ .../LanguageModeling/BERT/common_flags.py     |  5 +
+ .../create_small_dataset_for_benchmark.sh     | 30 ++++++
+ .../official/modeling/model_training_utils.py |  5 +-
+ .../modeling/model_training_utils_test.py     |  4 +-
+ .../BERT/official/nlp/bert_modeling.py        |  9 +-
+ .../BERT/official/nlp/bert_models.py          |  5 +
+ .../official/nlp/modeling/layers/attention.py | 75 ++++++++++-----
+ .../nlp/modeling/layers/transformer.py        | 94 +++++++++++++------
+ .../networks/albert_transformer_encoder.py    |  4 +-
+ .../nlp/modeling/networks/masked_lm.py        | 14 ++-
+ .../modeling/networks/transformer_encoder.py  | 24 +++--
+ .../BERT/official/utils/flags/flags_test.py   |  3 +
+ .../LanguageModeling/BERT/optimization.py     |  2 +-
+ .../LanguageModeling/BERT/run_pretraining.py  | 31 ++++--
+ .../LanguageModeling/BERT/run_squad.py        | 60 +++++++++---
+ .../BERT/scripts/run_pretraining_adam.sh      |  6 +-
+ .../BERT/scripts/run_pretraining_lamb.sh      |  6 +-
+ .../scripts/run_pretraining_lamb_phase1.sh    | 20 ++--
+ .../scripts/run_pretraining_lamb_phase2.sh    | 28 +++---
+ .../BERT/scripts/run_squad.sh                 | 22 ++++-
+ 20 files changed, 327 insertions(+), 120 deletions(-)
+ create mode 100644 TensorFlow2/LanguageModeling/BERT/data/create_small_dataset_for_benchmark.sh
+
+diff --git a/TensorFlow2/LanguageModeling/BERT/common_flags.py b/TensorFlow2/LanguageModeling/BERT/common_flags.py
+index 0c471089..b7c4780e 100644
+--- a/TensorFlow2/LanguageModeling/BERT/common_flags.py
++++ b/TensorFlow2/LanguageModeling/BERT/common_flags.py
+@@ -75,6 +75,9 @@ def define_common_bert_flags():
+   flags.DEFINE_boolean(
+       'use_fp16', False,
+       'Whether to use fp32 or fp16 arithmetic on GPU.')
++  flags.DEFINE_boolean(
++      'use_bf16', False,
++      'Whether to use fp32 or bf16 arithmetic on GPU.')
+   flags.DEFINE_string("optimizer_type", "adam",
+     "Optimizer used for training - LAMB or ADAM")
+   flags.DEFINE_integer(
+@@ -106,6 +109,8 @@ def define_common_bert_flags():
+ def use_float16():
+   return flags_core.get_tf_dtype(flags.FLAGS) == tf.float16
+
++def use_bfloat16():
++  return flags_core.get_tf_dtype(flags.FLAGS) == tf.bfloat16
+
+ def get_loss_scale():
+   return flags_core.get_loss_scale(flags.FLAGS, default_for_fp16='dynamic')
+diff --git a/TensorFlow2/LanguageModeling/BERT/data/create_small_dataset_for_benchmark.sh b/TensorFlow2/LanguageModeling/BERT/data/create_small_dataset_for_benchmark.sh
+new file mode 100644
+index 00000000..61ba6762
+--- /dev/null
++++ b/TensorFlow2/LanguageModeling/BERT/data/create_small_dataset_for_benchmark.sh
+@@ -0,0 +1,30 @@
++#!/bin/bash
++# bash data/create_small_dataset_for_benchmark.sh [output_dir](default: ${PWD}/dataset)
++
++output_dir=${1:-"${PWD}/dataset"}
++echo "output_dir: ${output_dir}"
++if [ ! -d "${output_dir}/training" ]; then
++  mkdir -p ${output_dir}/training
++fi
++if [ ! -d "${output_dir}/test" ]; then
++  mkdir -p ${output_dir}/test
++fi
++
++# prepare pretrained_model
++cp -r data/download/google_pretrained_weights/uncased_L-24_H-1024_A-16/ ${output_dir}
++
++# prepare small training dataset
++for i in {0..23}
++do
++    echo "download training dataset: ${i}"
++    cp ./data/tfrecord/lower_case_1_seq_len_512_max_pred_80_masked_lm_prob_0.15_random_seed_12345_dupe_factor_5_shard_1472_test_split_10/wikicorpus_en/training/wikicorpus_en_training_${i}.tfrecord ${output_dir}/training
++done
++
++# prepare small test dataset
++for i in {0..23}
++do
++    echo "download test dataset: ${i}"
++    cp ./data/tfrecord/lower_case_1_seq_len_512_max_pred_80_masked_lm_prob_0.15_random_seed_12345_dupe_factor_5_shard_1472_test_split_10/wikicorpus_en/test/wikicorpus_en_test_${i}.tfrecord ${output_dir}/test
++done
++
++echo "Create small dataset done."
+\ No newline at end of file
+diff --git a/TensorFlow2/LanguageModeling/BERT/official/modeling/model_training_utils.py b/TensorFlow2/LanguageModeling/BERT/official/modeling/model_training_utils.py
+index f5d080fa..0d568f76 100644
+--- a/TensorFlow2/LanguageModeling/BERT/official/modeling/model_training_utils.py
++++ b/TensorFlow2/LanguageModeling/BERT/official/modeling/model_training_utils.py
+@@ -269,7 +269,8 @@ def run_customized_training_loop(
+           scaled_loss = optimizer.get_scaled_loss(loss)
+
+       if hvd:
+-        tape = hvd.DistributedGradientTape(tape, sparse_as_dense=True, compression=Compression.fp16 if use_float16 else Compression.none)
++        # use grouped_allreduce and set num_groups=1
++        tape = hvd.DistributedGradientTape(tape, sparse_as_dense=True, compression=Compression.fp16 if use_float16 else Compression.none, num_groups=1)
+
+       if use_float16:
+         scaled_grads = tape.gradient(scaled_loss, training_vars)
+@@ -463,7 +464,7 @@ def run_customized_training_loop(
+
+     current_step = optimizer.iterations.numpy()
+     checkpoint_name = 'ctl_step_{step}.ckpt'
+-    manager = tf.train.CheckpointManager(checkpoint, model_dir, max_to_keep=3)
++    manager = tf.train.CheckpointManager(checkpoint, model_dir, max_to_keep=2)
+     FLAGS = params['FLAGS']
+     steps_from_save = 0
+     start_time = time.time()
+diff --git a/TensorFlow2/LanguageModeling/BERT/official/modeling/model_training_utils_test.py b/TensorFlow2/LanguageModeling/BERT/official/modeling/model_training_utils_test.py
+index b473aa9e..86e8267b 100644
+--- a/TensorFlow2/LanguageModeling/BERT/official/modeling/model_training_utils_test.py
++++ b/TensorFlow2/LanguageModeling/BERT/official/modeling/model_training_utils_test.py
+@@ -172,8 +172,8 @@ class ModelTrainingUtilsTest(tf.test.TestCase, parameterized.TestCase):
+   @combinations.generate(eager_gpu_strategy_combinations())
+   def test_train_eager_mixed_precision(self, distribution):
+     model_dir = self.get_temp_dir()
+-    policy = tf.keras.mixed_precision.experimental.Policy('mixed_float16')
+-    tf.keras.mixed_precision.experimental.set_policy(policy)
++    policy = tf.keras.mixed_precision.Policy('mixed_float16')
++    tf.keras.mixed_precision.set_global_policy(policy)
+     self._model_fn = create_model_fn(
+         input_shape=[128], num_classes=3, use_float16=True)
+     self.run_training(
+diff --git a/TensorFlow2/LanguageModeling/BERT/official/nlp/bert_modeling.py b/TensorFlow2/LanguageModeling/BERT/official/nlp/bert_modeling.py
+index 6e4afa5e..3b976666 100644
+--- a/TensorFlow2/LanguageModeling/BERT/official/nlp/bert_modeling.py
++++ b/TensorFlow2/LanguageModeling/BERT/official/nlp/bert_modeling.py
+@@ -27,6 +27,11 @@ import tensorflow as tf
+ from tensorflow.python.util import deprecation
+ from official.modeling import tf_utils
+
++try:
++  import intel_extension_for_tensorflow as itex
++  itex.experimental_ops_override()
++except:
++  pass
+
+ class BertConfig(object):
+   """Configuration for `BertModel`."""
+@@ -258,7 +263,9 @@ class BertModel(tf.keras.layers.Layer):
+     word_embeddings = self.embedding_lookup(input_word_ids)
+     embedding_tensor = self.embedding_postprocessor(
+         word_embeddings=word_embeddings, token_type_ids=input_type_ids)
+-    if self.float_type == tf.float16:
++    if self.float_type == tf.bfloat16:
++      embedding_tensor = tf.cast(embedding_tensor, tf.bfloat16)
++    elif self.float_type == tf.float16:
+       embedding_tensor = tf.cast(embedding_tensor, tf.float16)
+     attention_mask = None
+     if input_mask is not None:
+diff --git a/TensorFlow2/LanguageModeling/BERT/official/nlp/bert_models.py b/TensorFlow2/LanguageModeling/BERT/official/nlp/bert_models.py
+index 4f1f8801..bd9e7f67 100644
+--- a/TensorFlow2/LanguageModeling/BERT/official/nlp/bert_models.py
++++ b/TensorFlow2/LanguageModeling/BERT/official/nlp/bert_models.py
+@@ -29,6 +29,11 @@ from official.nlp.modeling.networks import bert_classifier
+ from official.nlp.modeling.networks import bert_pretrainer
+ from official.nlp.modeling.networks import bert_span_labeler
+
++try:
++  import intel_extension_for_tensorflow as itex
++  itex.experimental_ops_override()
++except:
++  pass
+
+ def gather_indexes(sequence_tensor, positions):
+   """Gathers the vectors at the specific positions.
+diff --git a/TensorFlow2/LanguageModeling/BERT/official/nlp/modeling/layers/attention.py b/TensorFlow2/LanguageModeling/BERT/official/nlp/modeling/layers/attention.py
+index bf7ca5ac..eaf0de64 100644
+--- a/TensorFlow2/LanguageModeling/BERT/official/nlp/modeling/layers/attention.py
++++ b/TensorFlow2/LanguageModeling/BERT/official/nlp/modeling/layers/attention.py
+@@ -24,7 +24,14 @@ import tensorflow as tf
+
+ from official.nlp.modeling.layers import dense_einsum
+ from official.nlp.modeling.layers import masked_softmax
+-
++IS_ITEX=False
++try:
++  import intel_extension_for_tensorflow as itex
++  itex.experimental_ops_override()
++  from intel_extension_for_tensorflow.python.ops.multi_head_attention import scaled_dot_product_attention
++  IS_ITEX=True
++except:
++  IS_ITEX=False
+
+ # @tf.keras.utils.register_keras_serializable(package="Text")
+ class Attention(tf.keras.layers.Layer):
+@@ -59,6 +66,7 @@ class Attention(tf.keras.layers.Layer):
+   """
+
+   def __init__(self,
++               sequence_length,
+                num_heads,
+                head_size,
+                dropout_rate=0.0,
+@@ -71,6 +79,7 @@ class Attention(tf.keras.layers.Layer):
+                bias_constraint=None,
+                **kwargs):
+     super(Attention, self).__init__(**kwargs)
++    self._sequence_length = sequence_length
+     self._num_heads = num_heads
+     self._head_size = head_size
+     self._dropout_rate = dropout_rate
+@@ -81,8 +90,8 @@ class Attention(tf.keras.layers.Layer):
+     self._kernel_constraint = tf.keras.constraints.get(kernel_constraint)
+     self._bias_constraint = tf.keras.constraints.get(bias_constraint)
+
+-    self._query_dense = dense_einsum.DenseEinsum(
+-        output_shape=(self._num_heads, self._head_size),
++    self._query_dense = tf.keras.layers.Dense(
++        self._num_heads * self._head_size,
+         kernel_initializer=self._kernel_initializer,
+         bias_initializer=self._bias_initializer,
+         kernel_regularizer=self._kernel_regularizer,
+@@ -92,8 +101,8 @@ class Attention(tf.keras.layers.Layer):
+         bias_constraint=self._bias_constraint,
+         name="query")
+
+-    self._key_dense = dense_einsum.DenseEinsum(
+-        output_shape=(self._num_heads, self._head_size),
++    self._key_dense = tf.keras.layers.Dense(
++        self._num_heads * self._head_size,
+         kernel_initializer=self._kernel_initializer,
+         bias_initializer=self._bias_initializer,
+         kernel_regularizer=self._kernel_regularizer,
+@@ -103,8 +112,8 @@ class Attention(tf.keras.layers.Layer):
+         bias_constraint=self._bias_constraint,
+         name="key")
+
+-    self._value_dense = dense_einsum.DenseEinsum(
+-        output_shape=(self._num_heads, self._head_size),
++    self._value_dense = tf.keras.layers.Dense(
++        self._num_heads * self._head_size,
+         kernel_initializer=self._kernel_initializer,
+         bias_initializer=self._bias_initializer,
+         kernel_regularizer=self._kernel_regularizer,
+@@ -156,13 +165,17 @@ class Attention(tf.keras.layers.Layer):
+     #   N = `num_attention_heads`
+     #   H = `size_per_head`
+     # `query_tensor` = [B, F, N ,H]
+-    query_tensor = self._query_dense(from_tensor)
++    query_tensor = self._query_dense(tf.reshape(from_tensor, (-1, self._num_heads * self._head_size)))
++    query_tensor = tf.reshape(query_tensor, (-1, self._sequence_length, self._num_heads, self._head_size))
+
+     # `key_tensor` = [B, T, N, H]
+-    key_tensor = self._key_dense(to_tensor)
++    key_tensor = self._key_dense(tf.reshape(to_tensor, (-1, self._num_heads * self._head_size)))
++    key_tensor = tf.reshape(key_tensor, (-1, self._sequence_length, self._num_heads, self._head_size))
+
+     # `value_tensor` = [B, T, N, H]
+-    value_tensor = self._value_dense(to_tensor)
++    value_tensor = self._value_dense(tf.reshape(to_tensor, (-1, self._num_heads * self._head_size)))
++    value_tensor = tf.reshape(value_tensor, (-1, self._sequence_length, self._num_heads, self._head_size))
++
+
+     # Take the dot product between "query" and "key" to get the raw
+     # attention scores.
+@@ -174,22 +187,40 @@ class Attention(tf.keras.layers.Layer):
+     query_tensor = tf.transpose(query_tensor, [0, 2, 1, 3])
+     # `key_tensor` = [B, N, T, H]
+     key_tensor = tf.transpose(key_tensor, [0, 2, 1, 3])
+-    # `attention_scores` = [B, N, F, T]
+-    attention_scores = tf.matmul(query_tensor, key_tensor, transpose_b=True)
++    # `key_tensor` = [B, N, T, H]
++    value_tensor = tf.transpose(value_tensor, [0, 2, 1, 3])
+
+-    attention_scores = tf.multiply(attention_scores,
+-                                   1.0 / math.sqrt(float(self._head_size)))
++    if(IS_ITEX):
++      if attention_mask is not None:
++        attention_mask = tf.expand_dims(attention_mask, axis=[1])
+
+-    # Normalize the attention scores to probabilities.
+-    # `attention_probs` = [B, N, F, T]
+-    attention_probs = self._masked_softmax([attention_scores, attention_mask])
++        # Since attention_mask is 1.0 for positions we want to attend and 0.0 for
++        # masked positions, this operation will create a tensor which is 0.0 for
++        # positions we want to attend and -10000.0 for masked positions.
++        adder = (1.0 - tf.cast(attention_mask, query_tensor.dtype)) * -10000.0
+
+-    # This is actually dropping out entire tokens to attend to, which might
+-    # seem a bit unusual, but is taken from the original Transformer paper.
+-    attention_probs = self._dropout(attention_probs)
++      context_layer = scaled_dot_product_attention(query_tensor, key_tensor, value_tensor, adder, self._dropout_rate, use_fast_attention=True, use_stateless_randomuniform=False)
++    else:  # default attention
++      # `attention_scores` = [B, N, F, T]
++      attention_scores = tf.matmul(query_tensor, key_tensor, transpose_b=True)
+
+-    # `context_layer` = [B, F, N, H]
+-    return tf.einsum("BNFT,BTNH->BFNH", attention_probs, value_tensor)
++      attention_scores = tf.multiply(attention_scores,
++                                    1.0 / math.sqrt(float(self._head_size)))
++
++      # Normalize the attention scores to probabilities.
++      # `attention_probs` = [B, N, F, T]
++      attention_probs = self._masked_softmax([attention_scores, attention_mask])
++
++      # This is actually dropping out entire tokens to attend to, which might
++      # seem a bit unusual, but is taken from the original Transformer paper.
++      attention_probs = self._dropout(attention_probs)
++
++      # `context_layer` = [B, F, N, H]
++      # return tf.einsum("BNFT,BTNH->BFNH", attention_probs, value_tensor)
++      context_layer = tf.matmul(attention_probs, value_tensor)
++      context_layer = tf.transpose(context_layer, [0, 2, 1, 3])
++
++    return tf.reshape(context_layer, (-1, self._num_heads * self._head_size))
+
+
+ # @tf.keras.utils.register_keras_serializable(package="Text")
+diff --git a/TensorFlow2/LanguageModeling/BERT/official/nlp/modeling/layers/transformer.py b/TensorFlow2/LanguageModeling/BERT/official/nlp/modeling/layers/transformer.py
+index 76847be0..6ddb2ad5 100644
+--- a/TensorFlow2/LanguageModeling/BERT/official/nlp/modeling/layers/transformer.py
++++ b/TensorFlow2/LanguageModeling/BERT/official/nlp/modeling/layers/transformer.py
+@@ -20,6 +20,15 @@ from __future__ import division
+ from __future__ import print_function
+
+ import tensorflow as tf
++IS_ITEX=False
++try:
++  import intel_extension_for_tensorflow as itex
++  itex.experimental_ops_override()
++  from intel_extension_for_tensorflow.python.ops.activations import gelu as gelu_itex_impl
++  from intel_extension_for_tensorflow.python.ops.mlp import FusedDenseBiasAddGelu
++  IS_ITEX=True
++except:
++  IS_ITEX=False
+
+ from official.nlp.modeling.layers import attention
+ from official.nlp.modeling.layers import dense_einsum
+@@ -100,6 +109,7 @@ class Transformer(tf.keras.layers.Layer):
+     self._attention_head_size = int(hidden_size // self._num_heads)
+
+     self._attention_layer = attention.Attention(
++        sequence_length=sequence_length,
+         num_heads=self._num_heads,
+         head_size=self._attention_head_size,
+         dropout_rate=self._attention_dropout_rate,
+@@ -111,9 +121,8 @@ class Transformer(tf.keras.layers.Layer):
+         kernel_constraint=self._kernel_constraint,
+         bias_constraint=self._bias_constraint,
+         name="self_attention")
+-    self._attention_output_dense = dense_einsum.DenseEinsum(
+-        output_shape=hidden_size,
+-        num_summed_dimensions=2,
++    self._attention_output_dense = tf.keras.layers.Dense(
++        hidden_size,
+         kernel_initializer=self._kernel_initializer,
+         bias_initializer=self._bias_initializer,
+         kernel_regularizer=self._kernel_regularizer,
+@@ -123,14 +132,33 @@ class Transformer(tf.keras.layers.Layer):
+         bias_constraint=self._bias_constraint,
+         name="self_attention_output")
+     self._attention_dropout = tf.keras.layers.Dropout(rate=self._dropout_rate)
+-    self._attention_layer_norm = (
+-        tf.keras.layers.LayerNormalization(
+-            name="self_attention_layer_norm",
+-            axis=-1,
+-            epsilon=1e-12,
+-            dtype=tf.float32))
+-    self._intermediate_dense = dense_einsum.DenseEinsum(
+-        output_shape=self._intermediate_size,
++    if self.dtype == tf.float16:
++      self._attention_layer_norm = (
++          tf.keras.layers.LayerNormalization(
++              name="self_attention_layer_norm",
++              axis=-1,
++              epsilon=1e-12, dtype=tf.float32))
++    else:
++      self._attention_layer_norm = (
++          tf.keras.layers.LayerNormalization(
++              name="self_attention_layer_norm",
++              axis=-1,
++              epsilon=1e-12))
++    if(IS_ITEX):
++      self._intermediate_dense = FusedDenseBiasAddGelu(
++        self._intermediate_size,
++        kernel_initializer=self._kernel_initializer,
++        bias_initializer=self._bias_initializer,
++        kernel_regularizer=self._kernel_regularizer,
++        bias_regularizer=self._bias_regularizer,
++        activity_regularizer=self._activity_regularizer,
++        kernel_constraint=self._kernel_constraint,
++        bias_constraint=self._bias_constraint,
++        name="intermediate")
++      self._intermediate_activation_layer = gelu_itex_impl
++    else:
++      self._intermediate_dense = tf.keras.layers.Dense(
++        self._intermediate_size,
+         activation=None,
+         kernel_initializer=self._kernel_initializer,
+         bias_initializer=self._bias_initializer,
+@@ -140,10 +168,10 @@ class Transformer(tf.keras.layers.Layer):
+         kernel_constraint=self._kernel_constraint,
+         bias_constraint=self._bias_constraint,
+         name="intermediate")
+-    self._intermediate_activation_layer = tf.keras.layers.Activation(
++      self._intermediate_activation_layer = tf.keras.layers.Activation(
+         self._intermediate_activation)
+-    self._output_dense = dense_einsum.DenseEinsum(
+-        output_shape=hidden_size,
++    self._output_dense = tf.keras.layers.Dense(
++        hidden_size,
+         kernel_initializer=self._kernel_initializer,
+         bias_initializer=self._bias_initializer,
+         kernel_regularizer=self._kernel_regularizer,
+@@ -153,8 +181,12 @@ class Transformer(tf.keras.layers.Layer):
+         bias_constraint=self._bias_constraint,
+         name="output")
+     self._output_dropout = tf.keras.layers.Dropout(rate=self._dropout_rate)
+-    self._output_layer_norm = tf.keras.layers.LayerNormalization(
+-        name="output_layer_norm", axis=-1, epsilon=1e-12, dtype=tf.float32)
++    if self.dtype == tf.float16:
++      self._output_layer_norm = tf.keras.layers.LayerNormalization(
++          name="output_layer_norm", axis=-1, epsilon=1e-12, dtype=tf.float32)
++    else:
++      self._output_layer_norm = tf.keras.layers.LayerNormalization(
++          name="output_layer_norm", axis=-1, epsilon=1e-12)
+
+     super(Transformer, self).build(input_shape)
+
+@@ -207,24 +239,30 @@ class Transformer(tf.keras.layers.Layer):
+     if self.dtype == tf.float16:
+       input_tensor = tf.cast(input_tensor, tf.float32)
+       attention_output = tf.cast(attention_output, tf.float32)
+-    attention_output = self._attention_layer_norm(input_tensor +
+-                                                  attention_output)
++    input_tensor_shape = input_tensor.shape
++    batch_size, sequence_length, hidden_size = input_tensor_shape
++    attention_output = self._attention_layer_norm(tf.reshape(input_tensor, (-1, hidden_size)) +
++                                                  tf.reshape(attention_output, (-1, hidden_size)))
+     intermediate_output = self._intermediate_dense(attention_output)
+-    if self.dtype == tf.float16:
+-      # Casts to float32 so that activation is done in float32.
+-      intermediate_output = tf.cast(intermediate_output, tf.float32)
+-      intermediate_output = self._intermediate_activation_layer(
+-          intermediate_output)
+-      intermediate_output = tf.cast(intermediate_output, tf.float16)
+-    else:
+-      intermediate_output = self._intermediate_activation_layer(
+-          intermediate_output)
++    intermediate_output = tf.reshape(intermediate_output, [-1, self._intermediate_size])
++    if(not IS_ITEX):
++      if self.dtype == tf.float16:
++        # Casts to float32 so that activation is done in float32.
++        intermediate_output = tf.cast(intermediate_output, tf.float32)
++        intermediate_output = self._intermediate_activation_layer(
++            intermediate_output)
++        intermediate_output = tf.cast(intermediate_output, tf.float16)
++      else:
++        intermediate_output = self._intermediate_activation_layer(
++            intermediate_output)
+     layer_output = self._output_dense(intermediate_output)
+     layer_output = self._output_dropout(layer_output)
+     # Use float32 in keras layer norm for numeric stability
+     if self.dtype == tf.float16:
+       layer_output = tf.cast(layer_output, tf.float32)
+-    layer_output = self._output_layer_norm(layer_output + attention_output)
++    layer_output = self._output_layer_norm(tf.reshape(layer_output, (-1, hidden_size)) +
++            tf.reshape(attention_output, (-1, hidden_size)))
++    layer_output = tf.reshape(layer_output, (-1 , sequence_length, hidden_size))
+     if self.dtype == tf.float16:
+       layer_output = tf.cast(layer_output, tf.float16)
+
+diff --git a/TensorFlow2/LanguageModeling/BERT/official/nlp/modeling/networks/albert_transformer_encoder.py b/TensorFlow2/LanguageModeling/BERT/official/nlp/modeling/networks/albert_transformer_encoder.py
+index e18f198a..897f1f9e 100644
+--- a/TensorFlow2/LanguageModeling/BERT/official/nlp/modeling/networks/albert_transformer_encoder.py
++++ b/TensorFlow2/LanguageModeling/BERT/official/nlp/modeling/networks/albert_transformer_encoder.py
+@@ -158,7 +158,9 @@ class AlbertTransformerEncoder(tf.keras.Model):
+           name='embedding_projection')(
+               embeddings)
+
+-    if float_dtype == 'float16':
++    if float_dtype == 'bfloat16':
++      embeddings = tf.cast(embeddings, tf.bfloat16)
++    elif float_dtype == 'float16':
+       embeddings = tf.cast(embeddings, tf.float16)
+
+     data = embeddings
+diff --git a/TensorFlow2/LanguageModeling/BERT/official/nlp/modeling/networks/masked_lm.py b/TensorFlow2/LanguageModeling/BERT/official/nlp/modeling/networks/masked_lm.py
+index 15ec13a3..4af29cb5 100644
+--- a/TensorFlow2/LanguageModeling/BERT/official/nlp/modeling/networks/masked_lm.py
++++ b/TensorFlow2/LanguageModeling/BERT/official/nlp/modeling/networks/masked_lm.py
+@@ -22,7 +22,11 @@ from __future__ import print_function
+ import tensorflow as tf
+
+ from official.modeling import tf_utils
+-
++try:
++  import intel_extension_for_tensorflow as itex
++  itex.experimental_ops_override()
++except:
++  pass
+
+ @tf.keras.utils.register_keras_serializable(package='Text')
+ class MaskedLM(tf.keras.Model):
+@@ -70,7 +74,13 @@ class MaskedLM(tf.keras.Model):
+             activation=activation,
+             kernel_initializer=initializer,
+             name='cls/predictions/transform/dense')(masked_lm_input))
+-    lm_data = tf.keras.layers.LayerNormalization(
++    if float_type == tf.float16:
++      lm_data = tf.keras.layers.LayerNormalization(
++        axis=-1, epsilon=1e-12, name='cls/predictions/transform/LayerNorm')(
++            tf.cast(lm_data, tf.float32))
++      lm_data = tf.cast(lm_data, tf.float16)
++    else:
++      lm_data = tf.keras.layers.LayerNormalization(
+         axis=-1, epsilon=1e-12, name='cls/predictions/transform/LayerNorm')(
+             lm_data)
+     lm_data = tf.keras.layers.Lambda(
+diff --git a/TensorFlow2/LanguageModeling/BERT/official/nlp/modeling/networks/transformer_encoder.py b/TensorFlow2/LanguageModeling/BERT/official/nlp/modeling/networks/transformer_encoder.py
+index 6b8d3f77..b10ee381 100644
+--- a/TensorFlow2/LanguageModeling/BERT/official/nlp/modeling/networks/transformer_encoder.py
++++ b/TensorFlow2/LanguageModeling/BERT/official/nlp/modeling/networks/transformer_encoder.py
+@@ -24,6 +24,11 @@ import tensorflow as tf
+ from official.modeling import activations
+ from official.nlp.modeling import layers
+
++try:
++  import intel_extension_for_tensorflow as itex
++  itex.experimental_ops_override()
++except:
++  pass
+
+ @tf.keras.utils.register_keras_serializable(package='Text')
+ class TransformerEncoder(tf.keras.Model):
+@@ -129,18 +134,25 @@ class TransformerEncoder(tf.keras.Model):
+
+     embeddings = tf.keras.layers.Add()(
+         [word_embeddings, position_embeddings, type_embeddings])
+-    embeddings = (
++    if(float_dtype == 'float16'):
++      embeddings = (
+         tf.keras.layers.LayerNormalization(
+             name='embeddings/layer_norm',
+             axis=-1,
+             epsilon=1e-12,
+             dtype=tf.float32)(embeddings))
+-    embeddings = (
+-        tf.keras.layers.Dropout(rate=dropout_rate,
+-                                dtype=tf.float32)(embeddings))
+-
+-    if float_dtype == 'float16':
++      embeddings = (
++          tf.keras.layers.Dropout(rate=dropout_rate,
++                                  dtype=tf.float32)(embeddings))
+       embeddings = tf.cast(embeddings, tf.float16)
++    else:
++      embeddings = (
++          tf.keras.layers.LayerNormalization(
++              name='embeddings/layer_norm',
++              axis=-1,
++              epsilon=1e-12)(tf.reshape(embeddings,(-1, hidden_size))))
++      embeddings = (
++          tf.keras.layers.Dropout(rate=dropout_rate)(tf.reshape(embeddings, (-1 , sequence_length, hidden_size))))
+
+     data = embeddings
+     attention_mask = layers.SelfAttentionMask()([data, mask])
+diff --git a/TensorFlow2/LanguageModeling/BERT/official/utils/flags/flags_test.py b/TensorFlow2/LanguageModeling/BERT/official/utils/flags/flags_test.py
+index e11a1642..0ecb5921 100644
+--- a/TensorFlow2/LanguageModeling/BERT/official/utils/flags/flags_test.py
++++ b/TensorFlow2/LanguageModeling/BERT/official/utils/flags/flags_test.py
+@@ -90,6 +90,9 @@ class BaseTester(unittest.TestCase):
+     self.assertEqual(flags_core.get_loss_scale(flags.FLAGS,
+                                                default_for_fp16=2), 2)
+
++    flags_core.parse_flags([__file__, "--dtype", "bf16"])
++    self.assertEqual(flags_core.get_tf_dtype(flags.FLAGS), tf.bfloat16)
++
+     flags_core.parse_flags(
+         [__file__, "--dtype", "fp16", "--loss_scale", "5"])
+     self.assertEqual(flags_core.get_loss_scale(flags.FLAGS,
+diff --git a/TensorFlow2/LanguageModeling/BERT/optimization.py b/TensorFlow2/LanguageModeling/BERT/optimization.py
+index e2b75b08..e90ac768 100644
+--- a/TensorFlow2/LanguageModeling/BERT/optimization.py
++++ b/TensorFlow2/LanguageModeling/BERT/optimization.py
+@@ -108,7 +108,7 @@ def create_optimizer(init_lr, num_train_steps, num_warmup_steps, optimizer_type=
+   return optimizer
+
+
+-class AdamWeightDecay(tf.keras.optimizers.Adam):
++class AdamWeightDecay(tf.keras.optimizers.legacy.Adam):
+   """Adam enables L2 weight decay and clip_by_global_norm on gradients.
+   Just adding the square of the weights to the loss function is *not* the
+   correct way of using L2 regularization/weight decay with Adam, since that will
+diff --git a/TensorFlow2/LanguageModeling/BERT/run_pretraining.py b/TensorFlow2/LanguageModeling/BERT/run_pretraining.py
+index a55757b9..15ba78b5 100644
+--- a/TensorFlow2/LanguageModeling/BERT/run_pretraining.py
++++ b/TensorFlow2/LanguageModeling/BERT/run_pretraining.py
+@@ -32,7 +32,7 @@ import model_saving_utils
+ from official.modeling import model_training_utils
+ from official.nlp import bert_modeling as modeling
+ import optimization
+-import gpu_affinity
++# import gpu_affinity
+ import dllogger_class
+ from official.utils.misc import distribution_utils
+ from official.utils.misc import keras_utils
+@@ -109,8 +109,15 @@ def run_customized_training(strategy,
+
+   def _get_pretrain_model():
+     """Gets a pretraining model."""
+-    pretrain_model, core_model = bert_models.pretrain_model(
+-        bert_config, max_seq_length, max_predictions_per_seq, float_type=tf.float16 if FLAGS.use_fp16 else tf.float32)
++    if FLAGS.use_bf16:
++      pretrain_model, core_model = bert_models.pretrain_model(
++          bert_config, max_seq_length, max_predictions_per_seq, float_type=tf.bfloat16)
++    elif FLAGS.use_fp16:
++      pretrain_model, core_model = bert_models.pretrain_model(
++          bert_config, max_seq_length, max_predictions_per_seq, float_type=tf.float16)
++    else:
++      pretrain_model, core_model = bert_models.pretrain_model(
++          bert_config, max_seq_length, max_predictions_per_seq, float_type=tf.float32)
+     pretrain_model.optimizer = optimization.create_optimizer(
+         initial_lr, steps_per_epoch * epochs, warmup_steps, FLAGS.optimizer_type)
+     if FLAGS.use_fp16:
+@@ -176,7 +183,7 @@ def main(_):
+   if not FLAGS.model_dir:
+     FLAGS.model_dir = '/tmp/bert20/'
+
+-  gpus = tf.config.experimental.list_physical_devices('GPU')
++  gpus = tf.config.experimental.list_physical_devices('XPU')
+   for gpu in gpus:
+     tf.config.experimental.set_memory_growth(gpu, True)
+
+@@ -193,12 +200,16 @@ def main(_):
+
+     hvd.init()
+     if gpus:
+-      tf.config.experimental.set_visible_devices(gpus[hvd.local_rank()], 'GPU')
+-      gpu_affinity.set_affinity(hvd.local_rank())
+-
+-  if FLAGS.use_fp16:
+-    policy = tf.keras.mixed_precision.experimental.Policy("mixed_float16")
+-    tf.keras.mixed_precision.experimental.set_policy(policy)
++      tf.config.experimental.set_visible_devices(gpus[hvd.local_rank()], 'XPU')
++      # TODO(Maozhou)
++      # gpu_affinity.set_affinity(hvd.local_rank())
++
++  if FLAGS.use_bf16:
++    policy = tf.keras.mixed_precision.Policy("mixed_bfloat16")
++    tf.keras.mixed_precision.set_global_policy(policy)
++  elif FLAGS.use_fp16:
++    policy = tf.keras.mixed_precision.Policy("mixed_float16")
++    tf.keras.mixed_precision.set_global_policy(policy)
+
+   run_bert_pretrain(strategy)
+
+diff --git a/TensorFlow2/LanguageModeling/BERT/run_squad.py b/TensorFlow2/LanguageModeling/BERT/run_squad.py
+index 87f5f019..a78d130b 100644
+--- a/TensorFlow2/LanguageModeling/BERT/run_squad.py
++++ b/TensorFlow2/LanguageModeling/BERT/run_squad.py
+@@ -46,7 +46,7 @@ import squad_lib as squad_lib_wp
+ # sentence-piece tokenizer based squad_lib
+ import squad_lib_sp
+ import tokenization
+-import gpu_affinity
++# import gpu_affinity
+ import tf_trt
+ from official.utils.misc import distribution_utils
+ from official.utils.misc import keras_utils
+@@ -208,8 +208,15 @@ def predict_squad_customized(strategy, input_meta_data, bert_config,
+
+   else:
+     with distribution_utils.get_strategy_scope(strategy):
+-      squad_model, _ = bert_models.squad_model(
+-          bert_config, input_meta_data['max_seq_length'], float_type=tf.float16 if FLAGS.use_fp16 else tf.float32)
++      if FLAGS.use_bf16:
++        squad_model, _ = bert_models.squad_model(
++          bert_config, input_meta_data['max_seq_length'], float_type=tf.bfloat16)
++      elif FLAGS.use_fp16:
++        squad_model, _ = bert_models.squad_model(
++          bert_config, input_meta_data['max_seq_length'], float_type=tf.float16)
++      else:
++        squad_model, _ = bert_models.squad_model(
++          bert_config, input_meta_data['max_seq_length'], float_type=tf.float32)
+
+     if FLAGS.init_checkpoint:
+       checkpoint = tf.train.Checkpoint(model=squad_model)
+@@ -278,7 +285,10 @@ def predict_squad_customized(strategy, input_meta_data, bert_config,
+   logging.info("Summary Inference Statistics")
+   logging.info("Batch size = %d", FLAGS.predict_batch_size)
+   logging.info("Sequence Length = %d", input_meta_data['max_seq_length'])
+-  logging.info("Precision = %s", "fp16" if FLAGS.use_fp16 else "fp32")
++  if FLAGS.use_bf16:
++    logging.info("Precision = bf16")
++  else:
++    logging.info("Precision = %s", "fp16" if FLAGS.use_fp16 else "fp32")
+   logging.info("Total Inference Time = %0.2f for Sentences = %d", eval_time_elapsed,
+     num_steps * FLAGS.predict_batch_size)
+
+@@ -326,7 +336,11 @@ def train_squad(strategy,
+
+   use_float16 = common_flags.use_float16()
+   if use_float16:
+-    tf.keras.mixed_precision.experimental.set_policy('mixed_float16')
++    tf.keras.mixed_precision.set_global_policy('mixed_float16')
++
++  use_bfloat16 = common_flags.use_bfloat16()
++  if use_bfloat16:
++    tf.keras.mixed_precision.set_global_policy('mixed_bfloat16')
+
+   bert_config = MODEL_CLASSES[FLAGS.model_type][0].from_json_file(
+       FLAGS.bert_config_file)
+@@ -351,10 +365,23 @@ def train_squad(strategy,
+
+   def _get_squad_model():
+     """Get Squad model and optimizer."""
+-    squad_model, core_model = bert_models.squad_model(
++    if FLAGS.use_bf16:
++      squad_model, core_model = bert_models.squad_model(
++        bert_config,
++        max_seq_length,
++        float_type=tf.bfloat16,
++        hub_module_url=FLAGS.hub_module_url)
++    elif FLAGS.use_fp16:
++      squad_model, core_model = bert_models.squad_model(
+         bert_config,
+         max_seq_length,
+-        float_type=tf.float16 if FLAGS.use_fp16 else tf.float32,
++        float_type=tf.float16,
++        hub_module_url=FLAGS.hub_module_url)
++    else:
++      squad_model, core_model = bert_models.squad_model(
++        bert_config,
++        max_seq_length,
++        float_type=tf.float32,
+         hub_module_url=FLAGS.hub_module_url)
+     learning_rate = FLAGS.learning_rate * hvd.size() if FLAGS.use_horovod else FLAGS.learning_rate
+     squad_model.optimizer = optimization.create_optimizer(
+@@ -591,7 +618,7 @@ dynamic_batching {{
+     "max_batch_size": max_batch_size,
+     "seq_length": input_meta_data['max_seq_length'],
+     "dynamic_batching": batching_str,
+-    "gpu_list": ", ".join([x.name.split(":")[-1] for x in tf.config.list_physical_devices('GPU')]),
++    "gpu_list": ", ".join([x.name.split(":")[-1] for x in tf.config.list_physical_devices('XPU')]),
+     "engine_count": FLAGS.triton_engine_count
+   }
+
+@@ -612,7 +639,7 @@ def main(_):
+     export_squad(FLAGS.model_export_path, input_meta_data)
+     return
+
+-  gpus = tf.config.experimental.list_physical_devices('GPU')
++  gpus = tf.config.experimental.list_physical_devices('XPU')
+   for gpu in gpus:
+     tf.config.experimental.set_memory_growth(gpu, True)
+
+@@ -627,12 +654,15 @@ def main(_):
+
+     hvd.init()
+     if gpus:
+-      tf.config.experimental.set_visible_devices(gpus[hvd.local_rank()], 'GPU')
+-      gpu_affinity.set_affinity(hvd.local_rank())
+-
+-  if FLAGS.use_fp16:
+-    policy = tf.keras.mixed_precision.experimental.Policy("mixed_float16")
+-    tf.keras.mixed_precision.experimental.set_policy(policy)
++      tf.config.experimental.set_visible_devices(gpus[hvd.local_rank()], 'XPU')
++      # gpu_affinity.set_affinity(hvd.local_rank())
++
++  if FLAGS.use_bf16:
++    policy = tf.keras.mixed_precision.Policy("mixed_bfloat16")
++    tf.keras.mixed_precision.set_global_policy(policy)
++  elif FLAGS.use_fp16:
++    policy = tf.keras.mixed_precision.Policy("mixed_float16")
++    tf.keras.mixed_precision.set_global_policy(policy)
+
+   os.makedirs(FLAGS.model_dir, exist_ok=True)
+   dllogging = dllogger_class.dllogger_class(FLAGS.dllog_path)
+diff --git a/TensorFlow2/LanguageModeling/BERT/scripts/run_pretraining_adam.sh b/TensorFlow2/LanguageModeling/BERT/scripts/run_pretraining_adam.sh
+index df886b92..fc5d57ab 100755
+--- a/TensorFlow2/LanguageModeling/BERT/scripts/run_pretraining_adam.sh
++++ b/TensorFlow2/LanguageModeling/BERT/scripts/run_pretraining_adam.sh
+@@ -29,7 +29,7 @@ num_accumulation_steps=${9:-1}
+ seq_len=${10:-512}
+ max_pred_per_seq=${11:-80}
+
+-DATA_DIR=data/tfrecord/lower_case_1_seq_len_${seq_len}_max_pred_${max_pred_per_seq}_masked_lm_prob_0.15_random_seed_12345_dupe_factor_5_shard_1472_test_split_10/books_wiki_en_corpus
++DATA_DIR=data/tfrecord/lower_case_1_seq_len_${seq_len}_max_pred_${max_pred_per_seq}_masked_lm_prob_0.15_random_seed_12345_dupe_factor_5_shard_1472_test_split_10/wikicorpus_en
+
+ if [ "$bert_model" = "large" ] ; then
+     export BERT_CONFIG=data/download/google_pretrained_weights/uncased_L-24_H-1024_A-16/bert_config.json
+@@ -40,6 +40,8 @@ fi
+ PREC=""
+ if [ "$precision" = "fp16" ] ; then
+    PREC="--use_fp16"
++elif [ "$precision" = "bf16" ] ; then
++   PREC="--use_bf16"
+ elif [ "$precision" = "fp32" ] || [ "$precision" = "tf32" ] ; then
+    PREC=""
+ else
+@@ -57,7 +59,7 @@ printf -v TAG "tf_bert_pretraining_adam_%s_%s_gbs%d" "$bert_model" "$precision"
+ DATESTAMP=`date +'%y%m%d%H%M%S'`
+
+ #Edit to save logs & checkpoints in a different directory
+-RESULTS_DIR=${RESULTS_DIR:-/results/${TAG}_${DATESTAMP}}
++RESULTS_DIR=${RESULTS_DIR:-./results/${TAG}_${DATESTAMP}}
+ LOGFILE=$RESULTS_DIR/$TAG.$DATESTAMP.log
+ mkdir -m 777 -p $RESULTS_DIR
+ printf "Saving checkpoints to %s\n" "$RESULTS_DIR"
+diff --git a/TensorFlow2/LanguageModeling/BERT/scripts/run_pretraining_lamb.sh b/TensorFlow2/LanguageModeling/BERT/scripts/run_pretraining_lamb.sh
+index 29b0b264..a0072342 100755
+--- a/TensorFlow2/LanguageModeling/BERT/scripts/run_pretraining_lamb.sh
++++ b/TensorFlow2/LanguageModeling/BERT/scripts/run_pretraining_lamb.sh
+@@ -22,7 +22,7 @@ train_batch_size_phase2=${2:-10}
+ eval_batch_size=${3:-8}
+ learning_rate_phase1=${4:-"7.5e-4"}
+ learning_rate_phase2=${5:-"5e-4"}
+-precision=${6:-"fp16"}
++precision=${6:-"fp16"}  # fp16 bf16
+ use_xla=${7:-"true"}
+ num_gpus=${8:-8}
+ warmup_steps_phase1=${9:-"2133"}
+@@ -33,7 +33,7 @@ num_accumulation_steps_phase1=${13:-128}
+ num_accumulation_steps_phase2=${14:-384}
+ bert_model=${15:-"large"}
+
+-DATA_DIR=data
++DATA_DIR=${16:-./data}
+ export DATA_DIR=$DATA_DIR
+
+ GBS1=$(expr $train_batch_size_phase1 \* $num_gpus \* $num_accumulation_steps_phase1)
+@@ -42,7 +42,7 @@ printf -v TAG "tf_bert_pretraining_lamb_%s_%s_gbs1%d_gbs2%d" "$bert_model" "$pre
+ DATESTAMP=`date +'%y%m%d%H%M%S'`
+
+ #Edit to save logs & checkpoints in a different directory
+-RESULTS_DIR=${RESULTS_DIR:-/results/${TAG}_${DATESTAMP}}
++RESULTS_DIR=${17:-./results/${TAG}}
+ LOGFILE=$RESULTS_DIR/$TAG.$DATESTAMP.log
+ mkdir -m 777 -p $RESULTS_DIR
+ printf "Saving checkpoints to %s\n" "$RESULTS_DIR"
+diff --git a/TensorFlow2/LanguageModeling/BERT/scripts/run_pretraining_lamb_phase1.sh b/TensorFlow2/LanguageModeling/BERT/scripts/run_pretraining_lamb_phase1.sh
+index 2fcd54e5..1bb5d5a8 100755
+--- a/TensorFlow2/LanguageModeling/BERT/scripts/run_pretraining_lamb_phase1.sh
++++ b/TensorFlow2/LanguageModeling/BERT/scripts/run_pretraining_lamb_phase1.sh
+@@ -33,19 +33,21 @@ num_accumulation_steps_phase1=${13:-128}
+ num_accumulation_steps_phase2=${14:-384}
+ bert_model=${15:-"large"}
+
+-DATA_DIR=${DATA_DIR:-data}
++DATA_DIR=${DATA_DIR:-./data}
+ #Edit to save logs & checkpoints in a different directory
+-RESULTS_DIR=${RESULTS_DIR:-/results}
++RESULTS_DIR=${RESULTS_DIR:-./results}
+
+ if [ "$bert_model" = "large" ] ; then
+-    export BERT_CONFIG=data/download/google_pretrained_weights/uncased_L-24_H-1024_A-16/bert_config.json
++    export BERT_CONFIG=$DATA_DIR/download/google_pretrained_weights/uncased_L-24_H-1024_A-16/bert_config.json
+ else
+-    export BERT_CONFIG=data/download/google_pretrained_weights/uncased_L-12_H-768_A-12/bert_config.json
++    export BERT_CONFIG=$DATA_DIR/download/google_pretrained_weights/uncased_L-12_H-768_A-12/bert_config.json
+ fi
+
+ PREC=""
+ if [ "$precision" = "fp16" ] ; then
+    PREC="--use_fp16"
++elif [ "$precision" = "bf16" ]; then
++   PREC="--use_bf16"
+ elif [ "$precision" = "fp32" ] || [ "$precision" = "tf32" ] ; then
+    PREC=""
+ else
+@@ -60,21 +62,21 @@ fi
+
+ mpi=""
+ if [ $num_gpus -gt 1 ] ; then
+-   mpi="mpiexec --allow-run-as-root -np $num_gpus"
++   mpi="mpiexec -np $num_gpus -l"
+    horovod="--use_horovod"
+ fi
+
+ #PHASE 1
+
+-train_steps_phase1=$(expr $train_steps \* 9 \/ 10) #Phase 1 is 10% of training
++train_steps_phase1=$(expr $train_steps \* 9 \/ 10) #Phase 1 is 90% of training
+ gbs_phase1=$(expr $train_batch_size_phase1 \* $num_accumulation_steps_phase1)
+ seq_len=128
+ max_pred_per_seq=20
+ RESULTS_DIR_PHASE1=${RESULTS_DIR}/phase_1
+ mkdir -m 777 -p $RESULTS_DIR_PHASE1
+
+-INPUT_FILES="$DATA_DIR/tfrecord/lower_case_1_seq_len_${seq_len}_max_pred_${max_pred_per_seq}_masked_lm_prob_0.15_random_seed_12345_dupe_factor_5_shard_1472_test_split_10/books_wiki_en_corpus/training/*"
+-EVAL_FILES="$DATA_DIR/tfrecord/lower_case_1_seq_len_${seq_len}_max_pred_${max_pred_per_seq}_masked_lm_prob_0.15_random_seed_12345_dupe_factor_5_shard_1472_test_split_10/books_wiki_en_corpus/test"
++INPUT_FILES="$DATA_DIR/tfrecord/lower_case_1_seq_len_${seq_len}_max_pred_${max_pred_per_seq}_masked_lm_prob_0.15_random_seed_12345_dupe_factor_5_shard_1472_test_split_10/wikicorpus_en/training/*"
++EVAL_FILES="$DATA_DIR/tfrecord/lower_case_1_seq_len_${seq_len}_max_pred_${max_pred_per_seq}_masked_lm_prob_0.15_random_seed_12345_dupe_factor_5_shard_1472_test_split_10/wikicorpus_en/test"
+
+ #Check if all necessary files are available before training
+ for DIR_or_file in $DATA_DIR $RESULTS_DIR_PHASE1 $BERT_CONFIG; do
+@@ -84,7 +86,7 @@ for DIR_or_file in $DATA_DIR $RESULTS_DIR_PHASE1 $BERT_CONFIG; do
+   fi
+ done
+
+- $mpi python /workspace/bert_tf2/run_pretraining.py \
++ $mpi python run_pretraining.py \
+      --input_files=$INPUT_FILES \
+      --model_dir=$RESULTS_DIR_PHASE1 \
+      --bert_config_file=$BERT_CONFIG \
+diff --git a/TensorFlow2/LanguageModeling/BERT/scripts/run_pretraining_lamb_phase2.sh b/TensorFlow2/LanguageModeling/BERT/scripts/run_pretraining_lamb_phase2.sh
+index db279129..a16683bf 100755
+--- a/TensorFlow2/LanguageModeling/BERT/scripts/run_pretraining_lamb_phase2.sh
++++ b/TensorFlow2/LanguageModeling/BERT/scripts/run_pretraining_lamb_phase2.sh
+@@ -33,14 +33,15 @@ num_accumulation_steps_phase1=${13:-128}
+ num_accumulation_steps_phase2=${14:-384}
+ bert_model=${15:-"large"}
+
+-DATA_DIR=${DATA_DIR:-data}
++DATA_DIR=${16:-./data}
+ #Edit to save logs & checkpoints in a different directory
+-RESULTS_DIR=${RESULTS_DIR:-/results}
++RESULTS_DIR=${17:-./results}
+
+ if [ "$bert_model" = "large" ] ; then
+-    export BERT_CONFIG=data/download/google_pretrained_weights/uncased_L-24_H-1024_A-16/bert_config.json
++   #  export BERT_CONFIG=$DATA_DIR/download/google_pretrained_weights/uncased_L-24_H-1024_A-16/bert_config.json
++    export BERT_CONFIG=$DATA_DIR/uncased_L-24_H-1024_A-16/bert_config.json
+ else
+-    export BERT_CONFIG=data/download/google_pretrained_weights/uncased_L-12_H-768_A-12/bert_config.json
++    export BERT_CONFIG=$DATA_DIR/download/google_pretrained_weights/uncased_L-12_H-768_A-12/bert_config.json
+ fi
+
+ echo "Container nvidia build = " $NVIDIA_BUILD_ID
+@@ -48,6 +49,8 @@ echo "Container nvidia build = " $NVIDIA_BUILD_ID
+ PREC=""
+ if [ "$precision" = "fp16" ] ; then
+    PREC="--use_fp16"
++elif [ "$precision" = "bf16" ]; then
++   PREC="--use_bf16"
+ elif [ "$precision" = "fp32" ] || [ "$precision" = "tf32" ] ; then
+    PREC=""
+ else
+@@ -62,13 +65,13 @@ fi
+
+ mpi=""
+ if [ $num_gpus -gt 1 ] ; then
+-   mpi="mpiexec --allow-run-as-root -np $num_gpus"
++   mpi="mpiexec -np $num_gpus -l"
+    horovod="--use_horovod"
+ fi
+
+ #PHASE 1 Config
+
+-train_steps_phase1=$(expr $train_steps \* 9 \/ 10) #Phase 1 is 10% of training
++train_steps_phase1=$(expr $train_steps \* 9 \/ 10) #Phase 1 is 90% of training
+ gbs_phase1=$(expr $train_batch_size_phase1 \* $num_accumulation_steps_phase1)
+ PHASE1_CKPT=${RESULTS_DIR}/phase_1/pretrained/bert_model.ckpt-1
+
+@@ -80,15 +83,16 @@ train_steps_phase2=$(expr $train_steps \* 1 \/ 10) #Phase 2 is 10% of training
+ gbs_phase2=$(expr $train_batch_size_phase2 \* $num_accumulation_steps_phase2)
+ train_steps_phase2=$(expr $train_steps_phase2 \* $gbs_phase1 \/ $gbs_phase2) # Adjust for batch size
+
+-RESULTS_DIR_PHASE2=${RESULTS_DIR}/phase_2
++RESULTS_DIR_PHASE2=${RESULTS_DIR}
+ mkdir -m 777 -p $RESULTS_DIR_PHASE2
+
+-INPUT_FILES="$DATA_DIR/tfrecord/lower_case_1_seq_len_${seq_len}_max_pred_${max_pred_per_seq}_masked_lm_prob_0.15_random_seed_12345_dupe_factor_5_shard_1472_test_split_10/books_wiki_en_corpus/training/*"
+-EVAL_FILES="$DATA_DIR/tfrecord/lower_case_1_seq_len_${seq_len}_max_pred_${max_pred_per_seq}_masked_lm_prob_0.15_random_seed_12345_dupe_factor_5_shard_1472_test_split_10/books_wiki_en_corpus/test"
++# INPUT_FILES="$DATA_DIR/tfrecord/lower_case_1_seq_len_${seq_len}_max_pred_${max_pred_per_seq}_masked_lm_prob_0.15_random_seed_12345_dupe_factor_5_shard_1472_test_split_10/wikicorpus_en/training/*"
++# EVAL_FILES="$DATA_DIR/tfrecord/lower_case_1_seq_len_${seq_len}_max_pred_${max_pred_per_seq}_masked_lm_prob_0.15_random_seed_12345_dupe_factor_5_shard_1472_test_split_10/wikicorpus_en/test"
++INPUT_FILES="$DATA_DIR/training/*"
++EVAL_FILES="$DATA_DIR/test"
+
+-$mpi python /workspace/bert_tf2/run_pretraining.py \
++$mpi python run_pretraining.py \
+     --input_files=$INPUT_FILES \
+-    --init_checkpoint=$PHASE1_CKPT \
+     --model_dir=$RESULTS_DIR_PHASE2 \
+     --bert_config_file=$BERT_CONFIG \
+     --train_batch_size=$train_batch_size_phase2 \
+@@ -103,3 +107,5 @@ $mpi python /workspace/bert_tf2/run_pretraining.py \
+     --optimizer_type=LAMB \
+     $horovod $PREC
+
++   #  --init_checkpoint=$PHASE1_CKPT \
++
+diff --git a/TensorFlow2/LanguageModeling/BERT/scripts/run_squad.sh b/TensorFlow2/LanguageModeling/BERT/scripts/run_squad.sh
+index 47cce05a..6e20b81f 100755
+--- a/TensorFlow2/LanguageModeling/BERT/scripts/run_squad.sh
++++ b/TensorFlow2/LanguageModeling/BERT/scripts/run_squad.sh
+@@ -25,6 +25,9 @@ use_xla=${5:-"true"}
+ bert_model=${6:-"large"}
+ squad_version=${7:-"1.1"}
+ epochs=${8:-"2"}
++use_mytrain=${9:-"false"}
++pretrain_path=${10:-"data/results/tf_bert_pretraining_lamb_large_bf16_gbs1_3840_gbs2_1920/phase_2/pretrained/bert_model.ckpt-1"}
++DATA_DIR=${11:-data}
+
+ if [ $num_gpu -gt 1 ] ; then
+     mpi_command="mpirun -np $num_gpu \
+@@ -41,6 +44,9 @@ fi
+ if [ "$precision" = "fp16" ] ; then
+     echo "fp16 activated!"
+     use_fp16="--use_fp16"
++elif [ "$precision" = "bf16" ]; then
++    echo "bf16 activated!"
++    use_fp16="--use_bf16"
+ else
+     use_fp16=""
+ fi
+@@ -53,20 +59,26 @@ else
+ fi
+
+ if [ "$bert_model" = "large" ] ; then
+-    export BERT_BASE_DIR=data/download/google_pretrained_weights/uncased_L-24_H-1024_A-16
++    export BERT_BASE_DIR=$DATA_DIR/download/google_pretrained_weights/uncased_L-24_H-1024_A-16
+ else
+-    export BERT_BASE_DIR=data/download/google_pretrained_weights/uncased_L-12_H-768_A-12
++    export BERT_BASE_DIR=$DATA_DIR/download/google_pretrained_weights/uncased_L-12_H-768_A-12
++fi
++
++if [ "$use_mytrain" = "false" ] ; then
++    export PRETRAIN_FILE=$BERT_BASE_DIR/bert_model.ckpt
++else
++    export PRETRAIN_FILE=$pretrain_path
+ fi
+
+ export SQUAD_VERSION=v$squad_version
+-export SQUAD_DIR=data/download/squad/$SQUAD_VERSION
++export SQUAD_DIR=$DATA_DIR/download/squad/$SQUAD_VERSION
+
+ export GBS=$(expr $batch_size \* $num_gpu)
+ printf -v TAG "tf_bert_finetuning_squad_%s_%s_gbs%d" "$bert_model" "$precision" $GBS
+ DATESTAMP=`date +'%y%m%d%H%M%S'`
+
+ #Edit to save logs & checkpoints in a different directory
+-RESULTS_DIR=/results/${TAG}_${DATESTAMP}
++RESULTS_DIR=${12:-./results/${TAG}}
+ LOGFILE=$RESULTS_DIR/$TAG.$DATESTAMP.log
+ mkdir -m 777 -p $RESULTS_DIR
+ printf "Saving checkpoints to %s\n" "$RESULTS_DIR"
+@@ -80,7 +92,7 @@ $mpi_command python run_squad.py \
+   --predict_file=${SQUAD_DIR}/dev-${SQUAD_VERSION}.json \
+   --vocab_file=${BERT_BASE_DIR}/vocab.txt \
+   --bert_config_file=$BERT_BASE_DIR/bert_config.json \
+-  --init_checkpoint=$BERT_BASE_DIR/bert_model.ckpt \
++  --init_checkpoint=$PRETRAIN_FILE \
+   --train_batch_size=$batch_size \
+   --learning_rate=$learning_rate \
+   --num_train_epochs=$epochs \
+--
+2.34.1
\ No newline at end of file
diff --git a/models_v2/tensorflow/bert_large/training/gpu/requirements.txt b/models_v2/tensorflow/bert_large/training/gpu/requirements.txt
new file mode 100644
index 000000000..723a5e41d
--- /dev/null
+++ b/models_v2/tensorflow/bert_large/training/gpu/requirements.txt
@@ -0,0 +1,26 @@
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+git+https://github.com/NVIDIA/dllogger
+requests
+tqdm
+sentencepiece
+tensorflow_hub
+wget
+progressbar
+nltk
+tensorflow-addons
+horovod
diff --git a/models_v2/tensorflow/bert_large/training/gpu/run_model.sh b/models_v2/tensorflow/bert_large/training/gpu/run_model.sh
new file mode 100755
index 000000000..e85a5c3de
--- /dev/null
+++ b/models_v2/tensorflow/bert_large/training/gpu/run_model.sh
@@ -0,0 +1,124 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+
+#!/bin/bash
+
+# Create an array of input directories that are expected and then verify that they exist
+declare -A input_envs
+input_envs[DATA_DIR]=${DATA_DIR}
+input_envs[RESULTS_DIR]=${RESULTS_DIR}
+input_envs[DATATYPE]=${DATATYPE}
+input_envs[MULTI_TILE]=${MULTI_TILE}
+
+for i in "${!input_envs[@]}"; do
+  var_name=$i
+  env_param=${input_envs[$i]}
+
+  if [[ -z $env_param ]]; then
+    echo "The required environment variable $var_name is not set" >&2
+    exit 1
+  fi
+done
+
+if [ -d ${DATA_DIR} ];then
+  echo "DATA_DIR is: "${DATA_DIR}
+else
+  echo "Error: the path of dataset does not exist!"
+  exit 1
+fi
+
+if [ ${DATATYPE} == "tf32" ];then
+  export ITEX_FP32_MATH_MODE=TF32
+fi
+
+
+echo 'Running with parameters:'
+echo " DATA_DIR: ${DATA_DIR}"
+echo " RESULTS_DIR: ${RESULTS_DIR}"
+echo " DATATYPE: ${DATATYPE}"
+echo " MULTI_TILE: $MULTI_TILE"
+
+rank_number=1
+if [[ $MULTI_TILE == "True" ]];then
+    rank_number=2
+fi
+
+rm -fr $RESULTS_DIR
+
+pwd=$PWD
+cd ./DeepLearningExamples/TensorFlow2/LanguageModeling/BERT
+
+export ITEX_OPS_OVERRIDE=1
+export DATA_DIR=$DATA_DIR
+TRAIN_BATCH_SIZE_PHASE1=60
+TRAIN_BATCH_SIZE_PHASE2=32
+EVAL_BATCH_SIZE=8
+LEARNING_RATE_PHASE1=7.5e-4
+LEARNING_RATE_PHASE2=5e-4
+DATATYPE=$DATATYPE
+USE_XLA=false
+NUM_GPUS=$rank_number
+WARMUP_STEPS_PHASE1=5
+WARMUP_STEPS_PHASE2=1
+TRAIN_STEPS=20
+SAVE_CHECKPOINT_STEPS=2
+NUM_ACCUMULATION_STEPS_PHASE1=64
+NUM_ACCUMULATION_STEPS_PHASE2=30
+BERT_MODEL=large
+
+
+bash scripts/run_pretraining_lamb_phase2.sh \
+    $TRAIN_BATCH_SIZE_PHASE1 \
+    $TRAIN_BATCH_SIZE_PHASE2 \
+    $EVAL_BATCH_SIZE \
+    $LEARNING_RATE_PHASE1 \
+    $LEARNING_RATE_PHASE2 \
+    $DATATYPE \
+    $USE_XLA \
+    $NUM_GPUS \
+    $WARMUP_STEPS_PHASE1 \
+    $WARMUP_STEPS_PHASE2 \
+    $TRAIN_STEPS \
+    $SAVE_CHECKPOINT_STEPS \
+    $NUM_ACCUMULATION_STEPS_PHASE1 \
+    $NUM_ACCUMULATION_STEPS_PHASE2 \
+    $BERT_MODEL \
+    $DATA_DIR \
+    $RESULTS_DIR \
+    |& tee $pwd/bert_large_training_${DATATYPE}.log
+
+cd -
+
+if [[ $MULTI_TILE == "False" ]];then
+    throughput=$(cat $pwd/bert_large_training_${DATATYPE}.log | grep "Throughput Average (sequences/sec)"  | tail -n 1 | awk -F ' ' '{print $9}')
+else
+    throughput=$(cat $pwd/bert_large_training_${DATATYPE}.log | grep "Throughput Average (sequences/sec)"  | tail -n 1 | awk -F ' ' '{print $10}')
+fi
+
+yaml_content=$(cat <<EOF
+results:
+ - key: throughput
+   value: $throughput
+   unit: images/sec
+EOF
+)
+
+# Write the content to a YAML file
+echo "$yaml_content" >  ${RESULTS_DIR}/results.yaml
+echo "YAML file created."
diff --git a/models_v2/tensorflow/bert_large/training/gpu/setup.sh b/models_v2/tensorflow/bert_large/training/gpu/setup.sh
new file mode 100755
index 000000000..1befea450
--- /dev/null
+++ b/models_v2/tensorflow/bert_large/training/gpu/setup.sh
@@ -0,0 +1,43 @@
+#!/usr/bin/env bash
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# setup.sh
+#  - install OS pkgs
+#  - should create virtual env & install pip  requirement.txt
+#  - git clones & applying patches
+
+set -e
+apt-get update && apt-get install -y python3-venv
+
+current_dir=$(pwd)
+if [ -d "pubmed_parser" ]; then
+  echo "Repository already exists. Skipping clone."
+else
+  git clone https://github.com/titipata/pubmed_parser
+  pip install ./pubmed_parser
+fi
+pip install -r requirements.txt
+
+if [ -d "DeepLearningExamples" ]; then
+  echo "Repository already exists. Skipping clone."
+else
+  git clone https://github.com/NVIDIA/DeepLearningExamples.git
+  cd DeepLearningExamples/TensorFlow2/LanguageModeling/BERT
+  cp $current_dir/bert-large-itex-bf16.patch .
+  git am --signoff < bert-large-itex-bf16.patch
+  cd -
+fi
diff --git a/models_v2/tensorflow/bert_large/training/gpu/tests.yaml b/models_v2/tensorflow/bert_large/training/gpu/tests.yaml
new file mode 100644
index 000000000..dabda8db4
--- /dev/null
+++ b/models_v2/tensorflow/bert_large/training/gpu/tests.yaml
@@ -0,0 +1,36 @@
+training-bf16:
+  cmd: bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/setup.sh ${FRAMEWORK_VERSION} ${FRAMEWORK_EXTENSION_VERSION} ${IS_LKG_DROP} ${AIKIT_VERSION}; bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/bert_large-training/run_test.sh ${PRECISION} ${IS_LKG_DROP} ${DATASET} ${MULTI_TILE}
+  env:
+    PRECISION: "bf16"
+    DATASET: "/localdisk/bert_small_data/bert-large-training"
+    MULTI_TILE: "False"
+training-tf32:
+  cmd: bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/setup.sh ${FRAMEWORK_VERSION} ${FRAMEWORK_EXTENSION_VERSION} ${IS_LKG_DROP} ${AIKIT_VERSION}; bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/bert_large-training/run_test.sh ${PRECISION} ${IS_LKG_DROP} ${DATASET} ${MULTI_TILE}
+  env:
+    PRECISION: "tf32"
+    DATASET: "/localdisk/bert_small_data/bert-large-training"
+    MULTI_TILE: "False"
+training-fp32:
+  cmd: bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/setup.sh ${FRAMEWORK_VERSION} ${FRAMEWORK_EXTENSION_VERSION} ${IS_LKG_DROP} ${AIKIT_VERSION}; bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/bert_large-training/run_test.sh ${PRECISION} ${IS_LKG_DROP} ${DATASET} ${MULTI_TILE}
+  env:
+    PRECISION: "fp32"
+    DATASET: "/localdisk/bert_small_data/bert-large-training"
+    MULTI_TILE: "False"
+training-bf16:
+  cmd: bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/setup.sh ${FRAMEWORK_VERSION} ${FRAMEWORK_EXTENSION_VERSION} ${IS_LKG_DROP} ${AIKIT_VERSION}; bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/bert_large-training/run_test.sh ${PRECISION} ${IS_LKG_DROP} ${DATASET} ${MULTI_TILE}
+  env:
+    PRECISION: "bf16"
+    DATASET: "/localdisk/bert_small_data/bert-large-training"
+    MULTI_TILE: "True"
+training-tf32:
+  cmd: bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/setup.sh ${FRAMEWORK_VERSION} ${FRAMEWORK_EXTENSION_VERSION} ${IS_LKG_DROP} ${AIKIT_VERSION}; bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/bert_large-training/run_test.sh ${PRECISION} ${IS_LKG_DROP} ${DATASET} ${MULTI_TILE}
+  env:
+    PRECISION: "tf32"
+    DATASET: "/localdisk/bert_small_data/bert-large-training"
+    MULTI_TILE: "True"
+training-fp32:
+  cmd: bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/setup.sh ${FRAMEWORK_VERSION} ${FRAMEWORK_EXTENSION_VERSION} ${IS_LKG_DROP} ${AIKIT_VERSION}; bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/bert_large-training/run_test.sh ${PRECISION} ${IS_LKG_DROP} ${DATASET} ${MULTI_TILE}
+  env:
+    PRECISION: "fp32"
+    DATASET: "/localdisk/bert_small_data/bert-large-training"
+    MULTI_TILE: "True"
diff --git a/models_v2/tensorflow/efficientnet/inference/gpu/CONTAINER.md b/models_v2/tensorflow/efficientnet/inference/gpu/CONTAINER.md
new file mode 100644
index 000000000..e7e644c97
--- /dev/null
+++ b/models_v2/tensorflow/efficientnet/inference/gpu/CONTAINER.md
@@ -0,0 +1,64 @@
+# Running EfficientNet Inference on Intel® Data Center GPU Flex Series using Intel® Extension for TensorFlow*
+
+## Overview
+
+This document has instructions for running EfficientNet inference using Intel® Extension for TensorFlow* with Intel® Data Center GPU Flex Series.
+
+## Requirements
+| Item | Detail |
+| ------ | ------- |
+| Host machine  | [Intel® Data Center GPU Flex ](https://ark.intel.com/content/www/us/en/ark/products/series/230021/intel-data-center-gpu-flex-series.html) 170 |
+| Drivers | Latest Intel® Data Center GPU Flex Series [Driver](https://dgpu-docs.intel.com/driver/installation.html) |
+| Software | Docker* |
+
+## Quick Start Scripts
+
+| Script name | Description |
+|:-------------:|:-------------:|
+| `run_model` | Runs EfficientNet B0,B3,B4 batch inference for fp16 precision on Flex series 170 |
+
+## Run Using Docker
+
+### Set up Docker Image
+
+```bash
+docker pull intel/image-recognition:tf-flex-gpu-efficientnet-inference
+```
+### Run Docker Image
+
+The EfficientNet inference container contains scripts,models and libraries needed to run fp16 inference. You will need to provide an output directory where log files will be written.
+
+```bash
+#Optional
+export BATCH_SIZE=<provide batch size. Default is 128>
+
+#Required
+export OUTPUT_DIR=<path to output directory>
+MODEL_NAME=<provide either EfficientNetB0,EfficientNetB3 or EfficientNetB4>
+IMAGE_NAME=intel/image-recognition:tf-flex-gpu-efficientnet-inference
+SCRIPT=run_model.sh
+
+docker run \
+  --device=/dev/dri \
+  --ipc=host \
+  --env BATCH_SIZE=${BATCH_SIZE} \
+  --env MODEL_NAME=${MODEL_NAME} \
+  --env OUTPUT_DIR=${OUTPUT_DIR} \
+  --env http_proxy=${http_proxy} \
+  --env https_proxy=${https_proxy} \
+  --env no_proxy=${no_proxy} \
+  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
+  --rm -it \
+  $IMAGE_NAME \
+  /bin/bash $SCRIPT
+  ```
+  ## Documentation and Sources
+
+[GitHub* Repository](https://github.com/IntelAI/models/tree/master/dockers/flex-gpu)
+
+## Support
+Support for Intel® Extension for TensorFlow* is found via the [Intel® AI Analytics Toolkit.](https://www.intel.com/content/www/us/en/developer/tools/oneapi/ai-analytics-toolkit.html#gs.qbretz) Additionally, the Intel® Extension for TensorFlow* team tracks both bugs and enhancement requests using [GitHub issues](https://github.com/intel/intel-extension-for-tensorflow/issues). Before submitting a suggestion or bug report, please search the GitHub issues to see if your issue has already been reported.
+
+## License Agreement
+
+LEGAL NOTICE: By accessing, downloading or using this software and any required dependent software (the “Software Package”), you agree to the terms and conditions of the software license agreements for the Software Package, which may also include notices, disclaimers, or license terms for third party software included with the Software Package. Please refer to the [license file](https://github.com/IntelAI/models/tree/master/third_party) for additional details.
diff --git a/models_v2/tensorflow/efficientnet/inference/gpu/README.md b/models_v2/tensorflow/efficientnet/inference/gpu/README.md
new file mode 100644
index 000000000..1be23bdc8
--- /dev/null
+++ b/models_v2/tensorflow/efficientnet/inference/gpu/README.md
@@ -0,0 +1,60 @@
+# EfficientNet Model Inference
+
+EfficientNet Model Inference using Intel® Extension for TensorFlow.
+
+## Model Information
+
+| **Use Case** | **Framework** | **Model Repo** | **Branch/Commit/Tag** | **Optional Patch** |
+|:---:| :---: |:--------------:|:---------------------:|:------------------:|
+|  Inference   |  TensorFlow   |       -        |           -           |         -          |
+
+**Note**: Refer to [CONTAINER.md](CONTAINER.md) for EfficientNet instructions using docker containers.
+
+# Pre-Requisite
+
+* Host has Intel® Data Center GPU Flex Series
+* Host has installed latest Intel® Data Center GPU Flex Series Driver https://dgpu-docs.intel.com/driver/installation.html
+* Install [Intel® Extension for TensorFlow](https://pypi.org/project/intel-extension-for-tensorflow/)
+
+# Run the model
+1. `git clone https://github.com/IntelAI/models.git`
+2. `cd models/models_v2/tensorflow/efficientnet/inference/gpu`
+3. Create virtual environment `venv` and activate it:
+    ```
+    python3 -m venv venv
+    . ./venv/bin/activate
+    ```
+4. Run setup.sh
+    ```
+    ./setup.sh
+    ```
+5. Install [tensorflow and ITEX](https://pypi.org/project/intel-extension-for-tensorflow/)
+6. Setup required environment paramaters (The precision is fp16 for all the efficientnet models).
+ 
+| **Parameter**             |                                  **export command**                                  |
+|:---------------------------:|:------------------------------------------------------------------------------------:|
+| **MODEL_NAME**          | `export MODEL_NAME=EfficientNetB0` (EfficientNetB0, EfficientNetB3 or EfficientNetB4) |
+| **BATCH_SIZE** (optional) |                               `export BATCH_SIZE=128`                                |
+| **OUTPUT_DIR**          | `export OUTPUT_DIR=<path to output log files>` |
+7. Run `run_model.sh`
+
+## Output
+
+Output will typically look like:
+```
+load data ......
+input shape (128, 224, 224, 3)
+Creating model finished.
+Batchsize is 128
+Avg time: 0.0484589417775472 s.
+Throughput: 2641.4113743463354 img/s.
+```
+
+Final results of the training run can be found in `results.yaml` file.
+
+```
+results:
+ - key: throughput
+   value: 2641.4113743463354
+   unit: img/s
+```
diff --git a/models_v2/tensorflow/efficientnet/inference/gpu/predict.py b/models_v2/tensorflow/efficientnet/inference/gpu/predict.py
new file mode 100644
index 000000000..6286b8227
--- /dev/null
+++ b/models_v2/tensorflow/efficientnet/inference/gpu/predict.py
@@ -0,0 +1,121 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+#
+
+import numpy as np
+import argparse
+import os
+import urllib
+import time
+import tensorflow as tf
+import tensorflow.keras.applications as tka
+from tensorflow.keras.preprocessing import image
+from tensorflow.keras.applications.resnet50 import preprocess_input, decode_predictions
+
+
+def download(url, filename):
+    with urllib.request.urlopen(url) as response, open(filename, 'wb') as out_file:
+        data = response.read()
+        out_file.write(data)
+
+
+@tf.function
+def tf_function_model_predict(model, x_input):
+    return model(x_input, training=False)
+
+
+def main(args):
+    from tensorflow.keras import mixed_precision
+    policy = mixed_precision.Policy('mixed_float16')
+    # policy = mixed_precision.Policy('mixed_bfloat16')
+    mixed_precision.set_global_policy(policy)
+
+    img_size = None
+    if args.model == "EfficientNetB0":
+        img_size = (224, 224)
+    elif args.model == "EfficientNetB3":
+        img_size = (300, 300)
+    elif args.model == "EfficientNetB4":
+        img_size = (380, 380)
+    else:
+        assert (False and "error model name")
+
+    print("download img ......")
+    ImageURL = "https://github.com/intel/caffe/raw/master/examples/images/"
+    image_name = "cat.jpg"
+    url = ImageURL + image_name
+    if not os.path.exists(image_name):
+        download(url, image_name)
+
+    print("load data ......")
+    img = image.load_img(image_name, target_size=img_size)
+    x = image.img_to_array(img)
+    x = np.expand_dims(x, axis=0)
+    x = preprocess_input(x)  # .astype(np.float32)
+
+    rep = np.array([args.batch_size, ], dtype=np.int32)
+    x = tf.repeat(x, rep, axis=0)
+
+    print("input shape", x.shape)
+
+    model = getattr(tka, args.model)(weights='imagenet')
+    model.trainable = False
+    print("Creating model finished.")
+
+    total_iter = 50
+    warmup_iter = 20
+    total_time = 0
+    total_count = 0
+    preds = None
+    for step in range(total_iter):
+        start = time.time()
+        # preds = model.predict(x)
+        # preds = model(x, training=False)
+        preds = tf_function_model_predict(model, x)
+        end = time.time()
+        if step >= warmup_iter:
+            total_time += (end - start)
+            total_count += 1
+    """
+    # resnet50 result
+    # decode the results into a list of tuples (class, description, probability)
+    # (one such list for each sample in the batch)
+    print('Predicted:', decode_predictions(preds, top=3)[0])
+    # Predicted: [(u'n02504013', u'Indian_elephant', 0.82658225), (u'n01871265', u'tusker', 0.1122357), (u'n02504458', u'African_elephant', 0.061040461)
+    """
+    print("Batchsize is", args.batch_size)
+    avg_time = total_time / total_count
+    print("Avg time:", avg_time, "s.")
+    print("Throughput:", args.batch_size / avg_time, "img/s.")
+
+
+def parse_args():
+    parser = argparse.ArgumentParser()
+    parser.add_argument("-m", "--model", dest="model", required=True, help="model name")
+    # parser.add_argument("-d", "--dtype", dest="dtype", required=True, help="data type")
+    # parser.add_argument("-l", "--layout", dest="layout", required=True, help="data layout")
+    parser.add_argument("-b", "--batch_size", dest="batch_size", type=int, required=True, help="batch size")
+    args = parser.parse_args()
+
+    return args
+
+
+if __name__ == "__main__":
+    args = parse_args()
+    main(args)
\ No newline at end of file
diff --git a/models_v2/tensorflow/efficientnet/inference/gpu/run_model.sh b/models_v2/tensorflow/efficientnet/inference/gpu/run_model.sh
new file mode 100755
index 000000000..e12d47805
--- /dev/null
+++ b/models_v2/tensorflow/efficientnet/inference/gpu/run_model.sh
@@ -0,0 +1,55 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+
+#!/bin/bash
+
+# Create an array of input directories that are expected and then verify that they exist
+declare -A input_envs
+input_envs[MODEL_NAME]=${MODEL_NAME}
+input_envs[OUTPUT_DIR]=${OUTPUT_DIR}
+
+for i in "${!input_envs[@]}"; do
+  var_name=$i
+  env_param=${input_envs[$i]}
+ 
+  if [[ -z $env_param ]]; then
+    echo "The required environment variable $var_name is not set" >&2
+    exit 1
+  fi
+done
+
+BATCH_SIZE=${BATCH_SIZE:-128}
+
+echo 'Running with parameters:'
+echo " MODEL_NAME: ${MODEL_NAME}"
+echo " BATCH_SIZE: ${BATCH_SIZE}"
+
+python predict.py -m $MODEL_NAME -b $BATCH_SIZE |& tee ${OUTPUT_DIR}/${MODEL_NAME}_inference_BS${BATCH_SIZE}.log
+throughput=$(cat ${OUTPUT_DIR}/${MODEL_NAME}_inference_BS${BATCH_SIZE}.log | grep Throughput | awk -F ' ' '{print $2}')
+yaml_content=$(cat <<EOF
+results:
+ - key: throughput
+   value: $throughput
+   unit: img/s
+EOF
+)
+
+# Write the content to a YAML file
+echo "$yaml_content" >  ${OUTPUT_DIR}/results.yaml
+echo "YAML file created."
diff --git a/models_v2/tensorflow/efficientnet/inference/gpu/setup.sh b/models_v2/tensorflow/efficientnet/inference/gpu/setup.sh
new file mode 100755
index 000000000..5cea6ea94
--- /dev/null
+++ b/models_v2/tensorflow/efficientnet/inference/gpu/setup.sh
@@ -0,0 +1,20 @@
+#!/usr/bin/env bash
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+set -e
+apt-get update && apt-get install -y python3-venv
+pip install Pillow
diff --git a/models_v2/tensorflow/efficientnet/inference/gpu/tests.yaml b/models_v2/tensorflow/efficientnet/inference/gpu/tests.yaml
new file mode 100644
index 000000000..bbafb7ffa
--- /dev/null
+++ b/models_v2/tensorflow/efficientnet/inference/gpu/tests.yaml
@@ -0,0 +1,12 @@
+inference-efficientnetb0:
+  cmd: bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/setup.sh ${FRAMEWORK_VERSION} ${FRAMEWORK_EXTENSION_VERSION} ${IS_LKG_DROP} ${AIKIT_VERSION}; bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/efficientnet-inference/run_test.sh ${MODEL_NAME} ${IS_LKG_DROP}
+  env:
+    MODEL_NAME: "EfficientNetB0"
+inference-efficientnetb3:
+  cmd: bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/setup.sh ${FRAMEWORK_VERSION} ${FRAMEWORK_EXTENSION_VERSION} ${IS_LKG_DROP} ${AIKIT_VERSION}; bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/efficientnet-inference/run_test.sh ${MODEL_NAME} ${IS_LKG_DROP}
+  env:
+    MODEL_NAME: "EfficientNetB3"
+inference-efficientnetb4:
+  cmd: bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/setup.sh ${FRAMEWORK_VERSION} ${FRAMEWORK_EXTENSION_VERSION} ${IS_LKG_DROP} ${AIKIT_VERSION}; bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/efficientnet-inference/run_test.sh ${MODEL_NAME} ${IS_LKG_DROP}
+  env:
+    MODEL_NAME: "EfficientNetB4"
diff --git a/models_v2/tensorflow/maskrcnn/inference/gpu/CONTAINER.md b/models_v2/tensorflow/maskrcnn/inference/gpu/CONTAINER.md
new file mode 100644
index 000000000..2486d7f4f
--- /dev/null
+++ b/models_v2/tensorflow/maskrcnn/inference/gpu/CONTAINER.md
@@ -0,0 +1,82 @@
+# Running MaskRCNN Inference with FP16 on Intel® Data Center GPU Flex Series using Intel® Extension for TensorFlow*
+
+## Overview
+
+This document has instructions for running MaskRCNN inference using Intel® Extension for TensorFlow* with Intel® Data Center GPU Flex Series.
+
+## Requirements
+| Item | Detail |
+| ------ | ------- |
+| Host machine  | [Intel® Data Center GPU Flex Series](https://ark.intel.com/content/www/us/en/ark/products/series/230021/intel-data-center-gpu-flex-series.html) 170 or 140  |
+| Drivers | GPU-compatible drivers need to be installed: [Download Driver](https://dgpu-docs.intel.com/driver/installation.html)
+| Software | Docker* |
+
+## Get Started
+
+### Download Dataset
+
+This repository provides scripts to download and extract the [COCO 2017 dataset](http://cocodataset.org/#download).
+
+Download and pre-process the datasets using script `download_and_preprocess_coco.sh` provided [here](https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow2/Segmentation/MaskRCNN/dataset). Set the `DATASET_DIR` to point to the TF records directory when running MaskRCNN.
+
+### Quick Start Scripts
+
+| Script name | Description |
+|:-------------:|:-------------:|
+| `run_model.sh` | Runs batch and online inference for FP16 precision on Flex series 170 and 140 |
+
+## Run Using Docker
+
+### Set up Docker Image
+
+```
+docker pull intel/image-segmentation:tf-flex-gpu-maskrcnn-inference
+```
+
+### Run Docker Image
+The MaskRCNN inference container includes scripts,model and libraries needed to run FP16 batch and online inference. To run the inference script using this container, you'll need to provide volume mounts for the COCO processed dataset.You will also need to provide an output directory where log files will be written. 
+
+```bash
+#Optional
+export BATCH_SIZE=<provide batch size. default batch for batch inference is 16>
+
+#Required
+export PRECISION=<provide precision,supports float16>
+export OUTPUT_DIR=<path to output directory>
+export DATASET_DIR=<path to the preprocessed COCO dataset>
+export GPU_TYPE=<provide either flex_170 or flex_140>
+
+IMAGE_NAME=intel/image-segmentation:tf-flex-gpu-maskrcnn-inference
+SCRIPT=run_model.sh
+
+docker run -it \
+  --device=/dev/dri \
+  --ipc=host \
+  --privileged \
+  --env PRECISION=${PRECISION} \
+  --env OUTPUT_DIR=${OUTPUT_DIR} \
+  --env DATASET_DIR=${DATASET_DIR} \
+  --env BATCH_SIZE=${BATCH_SIZE} \
+  --env PRECSION=${PRECISION} \
+  --env GPU_TYPE=${GPU_TYPE} \
+  --env http_proxy=${http_proxy} \
+  --env https_proxy=${https_proxy} \
+  --env no_proxy=${no_proxy} \
+  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
+  --volume ${DATASET_DIR}:${DATASET_DIR} \
+  --rm -it \
+  $IMAGE_NAME \
+  /bin/bash $SCRIPT
+  ```
+**Note:**  Add `--cap-add=SYS_NICE` to the `docker run` command for executing `run_model.sh` on Flex series 140.
+
+## Documentation and Sources
+
+[GitHub* Repository](https://github.com/IntelAI/models/tree/master/docker/flex-gpu)
+
+## Support
+Support for Intel® Extension for TensorFlow* is found via the [Intel® AI Analytics Toolkit.](https://www.intel.com/content/www/us/en/developer/tools/oneapi/ai-analytics-toolkit.html#gs.qbretz) Additionally, the Intel® Extension for TensorFlow* team tracks both bugs and enhancement requests using [GitHub issues](https://github.com/intel/intel-extension-for-tensorflow/issues). Before submitting a suggestion or bug report, please search the GitHub issues to see if your issue has already been reported.
+
+## License Agreement
+
+LEGAL NOTICE: By accessing, downloading or using this software and any required dependent software (the “Software Package”), you agree to the terms and conditions of the software license agreements for the Software Package, which may also include notices, disclaimers, or license terms for third party software included with the Software Package. Please refer to the [license file](https://github.com/IntelAI/models/tree/master/third_party) for additional details.
diff --git a/models_v2/tensorflow/maskrcnn/inference/gpu/EnableInference.patch b/models_v2/tensorflow/maskrcnn/inference/gpu/EnableInference.patch
new file mode 100644
index 000000000..c7f557858
--- /dev/null
+++ b/models_v2/tensorflow/maskrcnn/inference/gpu/EnableInference.patch
@@ -0,0 +1,173 @@
+diff --git a/TensorFlow2/Segmentation/MaskRCNN/main.py b/TensorFlow2/Segmentation/MaskRCNN/main.py
+index 53b59d6e..859396a6 100644
+--- a/TensorFlow2/Segmentation/MaskRCNN/main.py
++++ b/TensorFlow2/Segmentation/MaskRCNN/main.py
+@@ -17,12 +17,12 @@ import os
+ from argparse import Namespace
+ 
+ from mrcnn_tf2.runtime.run import run_training, run_inference, run_evaluation
+-from mrcnn_tf2.utils.dllogger import LoggingBackend
++# from mrcnn_tf2.utils.dllogger import LoggingBackend
+ 
+ os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'
+ os.environ["TF_CPP_VMODULE"] = 'non_max_suppression_op=0,generate_box_proposals_op=0,executor=0'
+ 
+-import dllogger
++# import dllogger
+ 
+ from mrcnn_tf2.arguments import PARSER
+ from mrcnn_tf2.config import CONFIG
+@@ -48,11 +48,14 @@ def main():
+     logging.getLogger('tensorflow').handlers.clear()
+ 
+     # setup dllogger
++    '''
+     dllogger.init(backends=[
+         dllogger.JSONStreamBackend(verbosity=dllogger.Verbosity.VERBOSE, filename=params.log_file),
+         LoggingBackend(verbosity=dllogger.Verbosity.VERBOSE)
+     ])
+     dllogger.log(step='PARAMETER', data=vars(params))
++    '''
++    print(vars(params), flush=True)
+ 
+     # setup dataset
+     dataset = Dataset(params)
+diff --git a/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/ops/roi_ops.py b/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/ops/roi_ops.py
+index 8670753e..250198b7 100644
+--- a/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/ops/roi_ops.py
++++ b/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/ops/roi_ops.py
+@@ -96,8 +96,8 @@ def _propose_rois_gpu(scores,
+         pre_nms_boxes = tf.cast(pre_nms_boxes, dtype=tf.float32)
+         pre_nms_scores = tf.cast(pre_nms_scores, dtype=tf.float32)
+ 
+-        with tf.device('CPU:0'):
+-            boxes, scores, _, _ = tf.image.combined_non_max_suppression(
++        # with tf.device('CPU:0'):
++        boxes, scores, _, _ = tf.image.combined_non_max_suppression(
+                 pre_nms_boxes,
+                 pre_nms_scores,
+                 max_output_size_per_class=topk_limit,
+diff --git a/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/runtime/callbacks.py b/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/runtime/callbacks.py
+index 4e7b56c9..8ffbcdc5 100644
+--- a/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/runtime/callbacks.py
++++ b/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/runtime/callbacks.py
+@@ -23,7 +23,7 @@ class DLLoggerMetricsCallback(KerasCallback):
+     Keras callback that saves metrics using DLLogger.
+     """
+ 
+-    def __init__(self, dllogger, log_every=10, log_learning_rate=False):
++    def __init__(self, log_every=10, log_learning_rate=False):
+         """
+         Args:
+             dllogger (DLLogger): DLLogger instance.
+@@ -32,7 +32,7 @@ class DLLoggerMetricsCallback(KerasCallback):
+                 Cannot be used with AMP enabled as the used hack fails with AMP.
+         """
+         super().__init__()
+-        self._dllogger = dllogger
++        # self._dllogger = dllogger
+         self._log_every = log_every
+         self._log_learning_rate = log_learning_rate
+ 
+@@ -68,7 +68,8 @@ class DLLoggerMetricsCallback(KerasCallback):
+         if not logs:
+             return
+ 
+-        self._dllogger.log(step=step, data=logs)
++        # self._dllogger.log(step=step, data=logs)
++        print(step, {k: v for k, v in logs.items()}, flush=True)
+ 
+ 
+ class DLLoggerPerfCallback(KerasCallback):
+@@ -76,9 +77,9 @@ class DLLoggerPerfCallback(KerasCallback):
+     Keras callback that measures performance and logs it using DLLogger.
+     """
+ 
+-    def __init__(self, dllogger, batch_sizes, warmup_steps=0, log_every=None):
++    def __init__(self, batch_sizes, warmup_steps=0, log_every=None):
+         super().__init__()
+-        self._dllogger = dllogger
++        # self._dllogger = dllogger
+         self._batch_sizes = batch_sizes
+         self._warmup_steps = warmup_steps
+         self._log_every = log_every
+@@ -124,6 +125,17 @@ class DLLoggerPerfCallback(KerasCallback):
+ 
+     def _log_perf(self, deltas, mode, step=tuple()):
+         deltas = np.array(deltas)
++        data={
++                f'{mode}_throughput': self._calculate_throughput(deltas, self._batch_sizes[mode]),
++                f'{mode}_latency': self._calculate_latency(deltas),
++                f'{mode}_latency_90': self._calculate_latency_confidence(deltas, 90.0),
++                f'{mode}_latency_95': self._calculate_latency_confidence(deltas, 95.0),
++                f'{mode}_latency_99': self._calculate_latency_confidence(deltas, 99.0),
++                f'{mode}_time': self._calculate_total_time(self._start_timestamps[mode], time.time())
++            }
++        print(step, data, flush=True)
++
++        '''
+         self._dllogger.log(
+             step=step,
+             data={
+@@ -135,6 +147,7 @@ class DLLoggerPerfCallback(KerasCallback):
+                 f'{mode}_time': self._calculate_total_time(self._start_timestamps[mode], time.time())
+             }
+         )
++        '''
+ 
+     @staticmethod
+     def _calculate_throughput(deltas, batch_size):
+diff --git a/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/runtime/run.py b/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/runtime/run.py
+index d7b001b3..2fecc57c 100644
+--- a/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/runtime/run.py
++++ b/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/runtime/run.py
+@@ -2,7 +2,7 @@ import logging
+ import os
+ 
+ import tensorflow as tf
+-import dllogger
++# import dllogger
+ 
+ from mrcnn_tf2.model.mask_rcnn import MaskRCNN
+ from mrcnn_tf2.runtime.callbacks import DLLoggerMetricsCallback, DLLoggerPerfCallback, PretrainedWeightsLoadingCallback
+@@ -80,10 +80,14 @@ def run_evaluation(dataset, params):
+         include_mask=params.include_mask
+     )
+ 
++    '''
+     dllogger.log(
+         step=tuple(),
+         data={k: float(v) for k, v in eval_results.items()}
+     )
++    '''
++
++    print(tuple(), {k: float(v) for k, v in eval_results.items()}, flush=True)
+ 
+ 
+ def run_inference(dataset, params):
+@@ -112,8 +116,8 @@ def setup(params):
+         logging.info('XLA is activated')
+ 
+     if params.amp:
+-        policy = tf.keras.mixed_precision.experimental.Policy("mixed_float16", loss_scale="dynamic")
+-        tf.keras.mixed_precision.experimental.set_policy(policy)
++        policy = tf.keras.mixed_precision.Policy("mixed_float16")
++        tf.keras.mixed_precision.set_global_policy(policy)
+         logging.info('AMP is activated')
+ 
+ 
+@@ -143,12 +147,12 @@ def create_model(params):
+ 
+ def create_callbacks(params):
+     yield DLLoggerMetricsCallback(
+-        dllogger=dllogger,
++        # dllogger=dllogger,
+         log_every=params.log_every
+     )
+ 
+     yield DLLoggerPerfCallback(
+-        dllogger=dllogger,
++        # dllogger=dllogger,
+         batch_sizes={
+             'train': params.train_batch_size * getattr(params, 'replicas', 1),
+             'test': params.eval_batch_size * getattr(params, 'replicas', 1),
diff --git a/models_v2/tensorflow/maskrcnn/inference/gpu/README.md b/models_v2/tensorflow/maskrcnn/inference/gpu/README.md
new file mode 100644
index 000000000..0d9f554f9
--- /dev/null
+++ b/models_v2/tensorflow/maskrcnn/inference/gpu/README.md
@@ -0,0 +1,90 @@
+# Mask RCNN Inference
+
+Mask RCNN Inference using Intel® Extension for TensorFlow.
+
+## Model Information
+
+| **Use Case** | **Framework** | **Model Repo** | **Branch/Commit/Tag** | **Weight** | **Optional Patch** |
+| :---: | :---: | :---: | :---: | :---: | :---: |
+|   Inference   |  Tensorflow   | [DeepLearningExamples/MaskRCNN](https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow2/Segmentation/MaskRCNN) |        master         | See Section [Prerequisites](#weight) | [EnableInference.patch](#inference patch) |
+
+**Note**: Refer to [CONTAINER.md](CONTAINER.md) for MaskRCNN Inference instructions using docker containers.
+
+# Pre-Requisite
+* Host has Intel® Data Center GPU Flex Series
+* Host has installed latest Intel® Data Center GPU Flex Series Drivers https://dgpu-docs.intel.com/driver/installation.html
+* Install [Intel® Extension for TensorFlow](https://pypi.org/project/intel-extension-for-tensorflow/)
+* The following Intel® oneAPI Base Toolkit components are required:
+  - Intel® oneAPI DPC++ Compiler (Placeholder DPCPPROOT as its installation path)
+  - Intel® oneAPI Math Kernel Library (oneMKL) (Placeholder MKLROOT as its installation path)
+  - Intel® oneAPI MPI Library
+  - Intel® oneAPI TBB Library
+  - Intel® oneAPI CCL Library
+
+  Follow instructions at [Intel® oneAPI Base Toolkit Download page](https://www.intel.com/content/www/us/en/developer/tools/oneapi/base-toolkit-download.html?operatingsystem=linux) to setup the package manager repository.
+
+# Dataset 
+Download & preprocess COCO 2017 dataset. 
+```
+export DATASET_DIR=/path/to/dataset/dir
+git clone https://github.com/NVIDIA/DeepLearningExamples.git
+cd ./DeepLearningExamples/TensorFlow2/Segmentation/MaskRCNN/dataset
+bash download_and_preprocess_coco.sh $DATASET_DIR
+```
+
+## Inference
+1. `git clone https://github.com/IntelAI/models.git`
+2. `cd models/models_v2/tensorflow/maskrcnn/inference/gpu`
+3. Create virtual environment `venv` and activate it:
+    ```
+    python3 -m venv venv
+    . ./venv/bin/activate
+    ```
+4. Run setup.sh
+    ```
+    ./setup.sh
+    ```
+5. Install [tensorflow and ITEX](https://pypi.org/project/intel-extension-for-tensorflow/)
+6. Download weights
+    ```
+        pushd .
+        cd ./DeepLearningExamples/TensorFlow2/Segmentation/MaskRCNN
+        python scripts/download_weights.py --save_dir=./weights
+        popd 
+    ```
+7. Set environment variables for Intel® oneAPI Base Toolkit: 
+    Default installation location `{ONEAPI_ROOT}` is `/opt/intel/oneapi` for root account, `${HOME}/intel/oneapi` for other accounts
+    ```bash
+    source {ONEAPI_ROOT}/compiler/latest/env/vars.sh
+    source {ONEAPI_ROOT}/mkl/latest/env/vars.sh
+    source {ONEAPI_ROOT}/tbb/latest/env/vars.sh
+    source {ONEAPI_ROOT}/mpi/latest/env/vars.sh
+    source {ONEAPI_ROOT}/ccl/latest/env/vars.sh
+8. Setup required environment paramaters
+
+    |   **Parameter**    | **export command**                                    |
+    | :---: | :--- |
+    |  **DATASET_DIR**   | `export DATASET_DIR=/the/path/to/dataset`             |
+    |   **BATCH_SIZE** (optional)   | `export BATCH_SIZE=4`           |
+    |   **PRECISION**   | `export PRECISION=bfloat16` (float16 or fp32)           |
+    |   **GPU_TYPE**    | `export GPU_TYPE=<flex_140 or flex_170>`                 |
+7. Run `run_model.sh`
+
+## Output
+
+Output typically looks like:
+```
+2023-09-11 14:54:49,905 I dllogger        (1, 20) loss: 639.5632934570312
+2023-09-11 14:54:49,906 I dllogger        (1, 20) train_time: 23.89216899871826, train_throughput: 21.438093303125907
+2023-09-11 14:54:49,914 I dllogger        (1,) loss: 639.5632934570312
+2023-09-11 14:54:49,914 I dllogger        () loss: 639.5632934570312
+2023-09-11 14:54:49,915 I dllogger        () train_time: 23.90118169784546, train_throughput: 23.507529269636105
+```
+
+Final results of the training run can be found in `results.yaml` file.
+```
+results:
+ - key: throughput
+   value: 32.896633477338334
+   unit: images/sec
+```
diff --git a/models_v2/tensorflow/maskrcnn/inference/gpu/requirements.txt b/models_v2/tensorflow/maskrcnn/inference/gpu/requirements.txt
new file mode 100644
index 000000000..48a5735b6
--- /dev/null
+++ b/models_v2/tensorflow/maskrcnn/inference/gpu/requirements.txt
@@ -0,0 +1,5 @@
+opencv-python-headless
+pybind11
+pycocotools
+numpy<1.24
+-e "git+https://github.com/NVIDIA/dllogger#egg=dllogger"
diff --git a/models_v2/tensorflow/maskrcnn/inference/gpu/run_model.sh b/models_v2/tensorflow/maskrcnn/inference/gpu/run_model.sh
new file mode 100755
index 000000000..47fd27bf3
--- /dev/null
+++ b/models_v2/tensorflow/maskrcnn/inference/gpu/run_model.sh
@@ -0,0 +1,119 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+
+#!/bin/bash
+
+# Create an array of input directories that are expected and then verify that they exist
+declare -A input_envs
+input_envs[DATASET_DIR]=${DATASET_DIR}
+input_envs[OUTPUT_DIR]=${OUTPUT_DIR}
+input_envs[GPU_TYPE]=${GPU_TYPE}
+input_envs[PRECISION]=${PRECISION}
+
+for i in "${!input_envs[@]}"; do
+  var_name=$i
+  env_param=${input_envs[$i]}
+
+  if [[ -z $env_param ]]; then
+    echo "The required environment variable $var_name is not set" >&2
+    exit 1
+  fi
+done
+
+if [ -d "${DATASET_DIR}" ]; then
+  echo "DATASET_DIR is: "${DATASET_DIR}
+else
+  echo "Error: the path of dataset does not exist!"
+  exit 1
+fi
+
+BATCH_SIZE=${BATCH_SIZE:-16}
+
+if [ "${PRECISION}" == "float16" ]; then
+  echo "PRECISION is float16"
+  AMP="--amp"
+else
+  AMP=""
+  echo "Only float16 PRECISION is supported."
+  exit 1
+fi
+
+echo 'Running with parameters:'
+echo " DATASET_PATH: ${DATASET_DIR}"
+echo " PRECISION: ${PRECISION}"
+echo "OUTPUT_DIR: ${OUTPUT_DIR}"
+echo " BATCH_SIZE: ${BATCH_SIZE}"
+
+mkdir -p PRETRAINED_WEIGHTS
+python -u ./DeepLearningExamples/TensorFlow2/Segmentation/MaskRCNN/scripts/download_weights.py --save_dir=$PRETRAINED_WEIGHTS
+
+declare -a str
+device_id=$( lspci | grep -i display | sed -n '1p' | awk '{print $7}' )
+num_devs=$(lspci | grep -i display | awk '{print $7}' | wc -l)
+num_threads=1
+k=0
+
+cd ./DeepLearningExamples/TensorFlow2/Segmentation/MaskRCNN
+
+ if [[ ${GPU_TYPE} == flex_170 ]]; then
+    if [[ ${device_id} == "56c0" ]]; then
+      echo "Running ${PRECISION} MaskRCNN Inference on Flex 170"
+      python scripts/inference.py \
+      --data_dir=$DATASET_DIR \
+      --batch_size=$BATCH_SIZE \
+      --no_xla \
+      --weights_dir=$PRETRAINED_WEIGHTS $AMP |& tee ${OUTPUT_DIR}/Maskrcnn_inference_${PRECISION}.log
+      value=$(cat ${OUTPUT_DIR}/Maskrcnn_inference_${PRECISION}.log | grep -o "'predict_throughput': [0-9.]*" | awk -F ": " '{print $2}' | tail -1)
+    fi
+elif [[ ${GPU_TYPE} == flex_140 ]]; then
+    if [[ ${device_id} == "56c1" ]]; then
+      BATCH_SIZE=1
+      echo "Running ${PRECISION} MaskRCNN Inference with BATCH SIZE 1 on Flex 140"
+      for i in $( eval echo {0..$((num_devs-1))} )
+          do
+            for j in $( eval echo {1..$num_threads} )
+            do
+            str+=("ZE_AFFINITY_MASK="${i}" numactl -C ${k} -l \
+            python scripts/inference.py \
+            --data_dir=$DATASET_DIR \
+            --batch_size=$BATCH_SIZE \
+            --no_xla \
+            --weights_dir=$PRETRAINED_WEIGHTS $AMP ")
+            ((k=k+1))
+            done
+          done
+      parallel --lb -d, --tagstring "[{#}]" ::: "${str[@]}" 2>&1 | tee ${OUTPUT_DIR}/Maskrcnn_inference_${PRECISION}.log
+      value=$(grep -rnw "5000/Unknown" ${OUTPUT_DIR}/Maskrcnn_inference_${PRECISION}.log | grep -o "'predict_throughput': [0-9.]*" | awk -F ": " '{print $2}' | tail -2 | awk '{ sum_total += $1 } END { print sum_total }')
+    fi
+fi
+key="throughput"
+unit="images/sec"
+
+throughput=$(cat maskrcnn_inference_${PRECISION}_BS${BATCH_SIZE}.log | grep Throughput | awk -F ' ' '{print $5}')
+yaml_content=$(cat <<EOF
+results:
+ - key: throughput
+   value: $throughput
+   unit: records/sec
+EOF
+)
+
+# Write the content to a YAML file
+echo "$yaml_content" >  ${OUTPUT_DIR}/results.yaml
+echo "YAML file created, path:${OUTPUT_DIR}/results.yaml"
diff --git a/models_v2/tensorflow/maskrcnn/inference/gpu/setup.sh b/models_v2/tensorflow/maskrcnn/inference/gpu/setup.sh
new file mode 100755
index 000000000..f2bb0a482
--- /dev/null
+++ b/models_v2/tensorflow/maskrcnn/inference/gpu/setup.sh
@@ -0,0 +1,37 @@
+#!/usr/bin/env bash
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# setup.sh
+#  - install OS pkgs
+#  - should create virtual env & install pip  requirement.txt
+#  - git clones & applying patches
+
+set -e
+apt-get update && apt-get install -y python3-venv protobuf-compiler
+apt-get install -y --no-install-recommends parallel pciutils numactl
+pip install -r requirements.txt
+
+current_dir=$(pwd)
+if [ -d "DeepLearningExamples" ]; then
+  echo "Repository already exists. Skipping clone."
+else
+  git clone https://github.com/NVIDIA/DeepLearningExamples.git
+  cd DeepLearningExamples
+  git checkout 5be8a3cae21ee2d80e3935a4746827cb3367bca
+  git apply $current_dir/EnableInference.patch
+  cd -
+fi
diff --git a/models_v2/tensorflow/maskrcnn/inference/gpu/tests.yaml b/models_v2/tensorflow/maskrcnn/inference/gpu/tests.yaml
new file mode 100644
index 000000000..199920f47
--- /dev/null
+++ b/models_v2/tensorflow/maskrcnn/inference/gpu/tests.yaml
@@ -0,0 +1,15 @@
+inference-fp16:
+  cmd: bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/setup.sh ${FRAMEWORK_VERSION} ${FRAMEWORK_EXTENSION_VERSION} ${IS_LKG_DROP} ${AIKIT_VERSION}; bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/maskrcnn-inference/run_test.sh ${PRECISION} ${IS_LKG_DROP} ${DATASET_DIR}
+  env:
+    PRECISION: "fp16"
+    DATASET_DIR: "/tf_dataset/dataset/coco_dataset/COCO2017_training_data"
+inference-fp32:
+  cmd: bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/setup.sh ${FRAMEWORK_VERSION} ${FRAMEWORK_EXTENSION_VERSION} ${IS_LKG_DROP} ${AIKIT_VERSION}; bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/maskrcnn-inference/run_test.sh ${PRECISION} ${IS_LKG_DROP} ${DATASET_DIR}
+  env:
+    PRECISION: "fp32"
+    DATASET_DIR: "/tf_dataset/dataset/coco_dataset/COCO2017_training_data"  
+inference-bfloat16:
+  cmd: bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/setup.sh ${FRAMEWORK_VERSION} ${FRAMEWORK_EXTENSION_VERSION} ${IS_LKG_DROP} ${AIKIT_VERSION}; bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/maskrcnn-inference/run_test.sh ${PRECISION} ${IS_LKG_DROP} ${DATASET_DIR}
+  env:
+    PRECISION: "bfloat16"
+    DATASET_DIR: "/tf_dataset/dataset/coco_dataset/COCO2017_training_data"
diff --git a/models_v2/tensorflow/maskrcnn/training/gpu/CONTAINER.md b/models_v2/tensorflow/maskrcnn/training/gpu/CONTAINER.md
new file mode 100644
index 000000000..f1a4977ca
--- /dev/null
+++ b/models_v2/tensorflow/maskrcnn/training/gpu/CONTAINER.md
@@ -0,0 +1,72 @@
+# TensorFlow MaskRCNN Training
+
+## Description
+
+This document has instructions for running MaskRCNN training with BFloat16 precision using Intel® Extension for TensorFlow on Intel® Data Center GPU Max Series.
+
+## Datasets
+
+Download and preprocess the COCO 2017 dataset using the [instructions here](./README.md). After running the conversion script you should have a directory with the COCO 2017 dataset in the TF records format.
+
+Set the `DATASET_DIR` to point to the TF records directory when running MaskRCNN. 
+
+## Quick Start Scripts
+| Script name | Description |
+|-------------|-------------|
+| `run_model.sh` | Runs MaskRCNN BF16 training on single and two tiles |
+
+Requirements:
+* Host has [Intel® Data Center GPU Max Series](https://ark.intel.com/content/www/us/en/ark/products/series/232874/intel-data-center-gpu-max-series.html)
+* Follow instructions to install GPU-compatible [drivers](https://dgpu-docs.intel.com/driver/installation.html)
+* Docker
+
+## Docker pull command:
+
+```bash
+docker pull intel/image-segmentation:tf-max-gpu-maskrcnn-training
+```
+To MaskRCNN training container includes scripts, models and libraries needed to run BFloat16 training. To run the `run_model.sh` quickstart script using this container, you are required to volume mount the pre-processed COCO 2017 dataset to run the script. You will also need to provide an output directory to store logs.
+
+```bash
+#Optional
+export BATCH_SIZE=<provide batch size. Default is 4>
+
+#Required
+export DATASET_DIR=<path to pre-processed COCO 2017 dataset>
+export MULTI_TILE=<specify True for Multi-tile training and False for single-tile training>
+export PRECISION=bfloat16
+export OUTPUT_DIR=<path to output directory>
+
+IMAGE_NAME=intel/image-segmentation:tf-max-gpu-maskrcnn-training
+
+if [[ ${MULTI_TILE} == "False" ]]; then
+    SCRIPT="mpirun -np 1 -prepend-rank -ppn 1 bash run_model.sh"
+else 
+    SCRIPT="bash run_model.sh"
+fi
+
+DOCKER_ARGS="--rm --init -it"
+docker run \
+--device /dev/dri \
+--env BATCH_SIZE=${BATCH_SIZE} \
+--env MULTI_TILE=${MULTI_TILE} \
+--env DATASET_DIR=${DATASET_DIR} \
+--env PRECISION=${PRECISION} \
+--env OUTPUT_DIR=${OUTPUT_DIR} \
+--volume /dev/dri:/dev/dri \
+--volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
+--volume ${DATASET_DIR}:${DATASET_DIR} \
+${DOCKER_ARGS} \
+${IMAGE_NAME} \
+$SCRIPT
+```
+## Documentation and Sources
+
+[GitHub* Repository](https://github.com/IntelAI/models/tree/master/docker/max-gpu)
+
+## Support
+Support for Intel® Extension for TensorFlow* is available at [Intel® AI Analytics Toolkit](https://www.intel.com/content/www/us/en/developer/tools/oneapi/ai-analytics-toolkit.html#gs.qbretz) Additionally, the Intel® Extension for TensorFlow* team tracks both bugs and enhancement requests using [GitHub issues](https://github.com/intel/intel-extension-for-tensorflow/issues). Before submitting a suggestion or bug report, please search the GitHub issues to see if your issue has already been reported.
+
+## License Agreement
+
+LEGAL NOTICE: By accessing, downloading or using this software and any required dependent software (the “Software Package”), you agree to the terms and conditions of the software license agreements for the Software Package, which may also include notices, disclaimers, or license terms for third party software included with the Software Package. Please refer to the [license file](https://github.com/IntelAI/models/tree/master/third_party) for additional details.
diff --git a/models_v2/tensorflow/maskrcnn/training/gpu/EnableBF16.patch b/models_v2/tensorflow/maskrcnn/training/gpu/EnableBF16.patch
new file mode 100644
index 000000000..53d313e62
--- /dev/null
+++ b/models_v2/tensorflow/maskrcnn/training/gpu/EnableBF16.patch
@@ -0,0 +1,395 @@
+diff --git a/TensorFlow2/Segmentation/MaskRCNN/main.py b/TensorFlow2/Segmentation/MaskRCNN/main.py
+index 53b59d6e..6902a0ba 100644
+--- a/TensorFlow2/Segmentation/MaskRCNN/main.py
++++ b/TensorFlow2/Segmentation/MaskRCNN/main.py
+@@ -57,6 +57,20 @@ def main():
+     # setup dataset
+     dataset = Dataset(params)
+ 
++    try:
++      import horovod.tensorflow.keras as hvd
++      import tensorflow as tf
++      hvd.init()
++      params.mpi_num = hvd.size()
++      params.model_dir = params.model_dir if hvd.rank() == 0 else \
++          os.path.join(params.model_dir, str(hvd.rank()))
++      xpus = tf.config.experimental.list_physical_devices('XPU')
++      tf.config.experimental.set_visible_devices(xpus[hvd.local_rank()], 'XPU')
++      logging.info("MPI horovod is enabled, this is running in MPI mode! local_rank = {}".format(hvd.local_rank()))
++    except ImportError:
++      params.mpi_num = 0
++      logging.info("No MPI horovod support, this is running in no-MPI mode!")
++
+     if params.mode == 'train':
+         run_training(dataset, params)
+     if params.mode == 'eval':
+diff --git a/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/arguments.py b/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/arguments.py
+index 63368c09..ad04efa4 100644
+--- a/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/arguments.py
++++ b/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/arguments.py
+@@ -53,6 +53,14 @@ RUNTIME_GROUP.add_argument(
+     ]
+ )
+ 
++LOGGING_GROUP.add_argument(
++    '--mpi_num',
++    type=int,
++    default=0,
++    metavar='N',
++    help='Number of mpi ranks'
++)
++
+ RUNTIME_GROUP.add_argument(
+     '--data_dir',
+     type=str,
+@@ -72,7 +80,7 @@ RUNTIME_GROUP.add_argument(
+ RUNTIME_GROUP.add_argument(
+     '--backbone_checkpoint',
+     type=str,
+-    default='/weights/rn50_tf_amp_ckpt_v20.06.0/nvidia_rn50_tf_amp',
++    default=None,
+     metavar='FILE',
+     help='Pretrained checkpoint for resnet'
+ )
+diff --git a/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/dataset/dataset.py b/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/dataset/dataset.py
+index 29d2373a..4bd0f2b6 100644
+--- a/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/dataset/dataset.py
++++ b/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/dataset/dataset.py
+@@ -48,6 +48,32 @@ class Dataset:
+ 
+     def train_fn(self, batch_size):
+         """ Input function for training. """
++        # Features
++        if self._params.use_synthetic_data:
++            self._logger.info("Using synthetic data")
++            name_to_features = {
++                "source_ids": tf.zeros((batch_size, ), dtype=tf.float32),
++                "images": tf.zeros((batch_size, 1024, 1024, 3), dtype=tf.float32),
++                "image_info": tf.zeros((batch_size, 5,) , dtype=tf.float32),
++                "cropped_gt_masks": tf.zeros((batch_size, 100, 116, 116), dtype=tf.float32),
++                "gt_boxes": tf.zeros((batch_size, 100, 4) , dtype=tf.float32),
++                "gt_classes": tf.zeros((batch_size, 100, 1) , dtype=tf.float32),
++                "score_targets_2": tf.zeros((batch_size, 256, 256, 3), dtype=tf.float32),
++                "box_targets_2": tf.zeros((batch_size, 256, 256, 12), dtype=tf.float32),
++                "score_targets_3": tf.zeros((batch_size, 128, 128, 3), dtype=tf.float32),
++                "box_targets_3": tf.zeros((batch_size, 128, 128, 12), dtype=tf.float32),
++                "score_targets_4": tf.zeros((batch_size, 64, 64, 3), dtype=tf.float32),
++                "box_targets_4": tf.zeros((batch_size, 64, 64, 12), dtype=tf.float32),
++                "score_targets_5": tf.zeros((batch_size, 32, 32, 3), dtype=tf.float32),
++                "box_targets_5": tf.zeros((batch_size, 32, 32, 12), dtype=tf.float32),
++                "score_targets_6": tf.zeros((batch_size, 16, 16, 3), dtype=tf.float32),
++                "box_targets_6": tf.zeros((batch_size, 16, 16, 12), dtype=tf.float32),
++            }
++
++            return tf.data.Dataset.from_tensors((name_to_features, {})).repeat()
++
++        if self._params.mpi_num:
++            batch_size = batch_size * self._params.mpi_num
+         data = tf.data.TFRecordDataset(self._train_files)
+ 
+         data = data.cache()
+@@ -73,7 +99,9 @@ class Dataset:
+ 
+         data = data.prefetch(buffer_size=tf.data.experimental.AUTOTUNE)
+         data = data.with_options(self._data_options)
+-
++        if self._params.mpi_num:
++            import horovod.tensorflow.keras as hvd
++            data = data.shard(hvd.size(), hvd.rank())
+         return data
+ 
+     def eval_fn(self, batch_size):
+diff --git a/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/dataset/dataset_parser.py b/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/dataset/dataset_parser.py
+index 96da0e36..3c071b10 100644
+--- a/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/dataset/dataset_parser.py
++++ b/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/dataset/dataset_parser.py
+@@ -99,7 +99,7 @@ def dataset_parser(value, mode, params, use_instance_mask, seed=None, regenerate
+ 
+     example_decoder = create_example_decoder()
+ 
+-    with tf.xla.experimental.jit_scope(compile_ops=True):
++    with tf.xla.experimental.jit_scope(compile_ops=False):
+ 
+         with tf.name_scope('parser'):
+ 
+diff --git a/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/model/anchors.py b/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/model/anchors.py
+index ba922500..e6455b3f 100644
+--- a/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/model/anchors.py
++++ b/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/model/anchors.py
+@@ -240,7 +240,7 @@ class AnchorLabeler:
+             tf.constant(0, dtype=tf.int32, shape=match_results.shape))
+         ignore_labels = tf.fill(match_results.shape, -1)
+ 
+-        return (ignore_labels + positive_labels + negative_labels,
++        return (tf.cast(ignore_labels + positive_labels + negative_labels, tf.float32),
+                 positive_labels, negative_labels)
+ 
+     def label_anchors(self, gt_boxes, gt_labels):
+diff --git a/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/model/losses.py b/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/model/losses.py
+index f4e22985..b681d4c0 100644
+--- a/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/model/losses.py
++++ b/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/model/losses.py
+@@ -28,8 +28,8 @@ class MaskRCNNLoss(tf.keras.layers.Layer):
+     so that it doesn't expand `mask_targets`.
+     """
+ 
+-    def __init__(self):
+-        super().__init__(trainable=False, dtype=tf.float32)
++    def __init__(self, name="MaskRCNNLoss"):
++        super().__init__(name=name, trainable=False, dtype=tf.float32)
+ 
+     def call(self, inputs, **kwargs):
+         """
+@@ -82,8 +82,8 @@ class FastRCNNLoss(tf.keras.layers.Layer):
+     Reference: https://github.com/facebookresearch/Detectron/blob/master/detectron/modeling/fast_rcnn_heads.py
+     """
+ 
+-    def __init__(self, num_classes):
+-        super().__init__(trainable=False, dtype=tf.float32)
++    def __init__(self, num_classes, name="FastRCNNLoss"):
++        super().__init__(name=name, trainable=False, dtype=tf.float32)
+         self._num_classes = num_classes
+ 
+     def call(self, inputs, **kwargs):
+@@ -155,8 +155,8 @@ class RPNLoss(tf.keras.layers.Layer):
+     Computes total RPN detection loss including box and score from all levels.
+     """
+ 
+-    def __init__(self, batch_size, rpn_batch_size_per_im, min_level, max_level):
+-        super().__init__(trainable=False, dtype=tf.float32)
++    def __init__(self, batch_size, rpn_batch_size_per_im, min_level, max_level, name="RPNLoss"):
++        super().__init__(name=name, trainable=False, dtype=tf.float32)
+         self._batch_size = batch_size
+         self._rpn_batch_size_per_im = rpn_batch_size_per_im
+         self._min_level = min_level
+diff --git a/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/model/mask_rcnn.py b/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/model/mask_rcnn.py
+index 1cc1e3fd..21b9f9a7 100644
+--- a/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/model/mask_rcnn.py
++++ b/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/model/mask_rcnn.py
+@@ -251,21 +251,21 @@ class MaskRCNN(tf.keras.Model):
+         mask_rcnn_loss = self.mask_rcnn_loss(model_outputs)
+         mask_rcnn_loss *= self._params.mrcnn_weight_loss_mask
+         self.add_loss(mask_rcnn_loss)
+-        self.add_metric(mask_rcnn_loss, name='mask_rcnn_loss')
++        # self.add_metric(mask_rcnn_loss, name='mask_rcnn_loss')
+ 
+         fast_rcnn_class_loss, fast_rcnn_box_loss = self.fast_rcnn_loss(model_outputs)
+         fast_rcnn_box_loss *= self._params.fast_rcnn_box_loss_weight
+         self.add_loss(fast_rcnn_box_loss)
+-        self.add_metric(fast_rcnn_box_loss, name='fast_rcnn_box_loss')
++        # self.add_metric(fast_rcnn_box_loss, name='fast_rcnn_box_loss')
+         self.add_loss(fast_rcnn_class_loss)
+-        self.add_metric(fast_rcnn_class_loss, name='fast_rcnn_class_loss')
++        # self.add_metric(fast_rcnn_class_loss, name='fast_rcnn_class_loss')
+ 
+         rpn_score_loss, rpn_box_loss = self.rpn_loss(model_outputs)
+         rpn_box_loss *= self._params.rpn_box_loss_weight
+         self.add_loss(rpn_box_loss)
+-        self.add_metric(rpn_box_loss, name='rpn_box_loss')
++        # self.add_metric(rpn_box_loss, name='rpn_box_loss')
+         self.add_loss(rpn_score_loss)
+-        self.add_metric(rpn_score_loss, name='rpn_score_loss')
++        # self.add_metric(rpn_score_loss, name='rpn_score_loss')
+ 
+         l2_regularization_loss = tf.add_n([
+             tf.nn.l2_loss(tf.cast(v, dtype=tf.float32))
+@@ -274,7 +274,7 @@ class MaskRCNN(tf.keras.Model):
+         ])
+         l2_regularization_loss *= self._params.l2_weight_decay
+         self.add_loss(l2_regularization_loss)
+-        self.add_metric(l2_regularization_loss, name='l2_regularization_loss')
++        # self.add_metric(l2_regularization_loss, name='l2_regularization_loss')
+ 
+     def get_config(self):
+         pass
+diff --git a/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/ops/roi_ops.py b/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/ops/roi_ops.py
+index 8670753e..deaf434a 100644
+--- a/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/ops/roi_ops.py
++++ b/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/ops/roi_ops.py
+@@ -96,16 +96,16 @@ def _propose_rois_gpu(scores,
+         pre_nms_boxes = tf.cast(pre_nms_boxes, dtype=tf.float32)
+         pre_nms_scores = tf.cast(pre_nms_scores, dtype=tf.float32)
+ 
+-        with tf.device('CPU:0'):
+-            boxes, scores, _, _ = tf.image.combined_non_max_suppression(
+-                pre_nms_boxes,
+-                pre_nms_scores,
+-                max_output_size_per_class=topk_limit,
+-                max_total_size=post_nms_topk_limit,
+-                iou_threshold=rpn_nms_threshold,
+-                score_threshold=0.0,
+-                pad_per_class=False
+-            )
++        # with tf.device('CPU:0'):
++        boxes, scores, _, _ = tf.image.combined_non_max_suppression(
++            pre_nms_boxes,
++            pre_nms_scores,
++            max_output_size_per_class=topk_limit,
++            max_total_size=post_nms_topk_limit,
++            iou_threshold=rpn_nms_threshold,
++            score_threshold=0.0,
++            pad_per_class=False
++        )
+ 
+         boxes = box_utils.to_absolute_coordinates(boxes, height, width)
+ 
+diff --git a/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/runtime/callbacks.py b/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/runtime/callbacks.py
+index 4e7b56c9..20ae757c 100644
+--- a/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/runtime/callbacks.py
++++ b/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/runtime/callbacks.py
+@@ -127,12 +127,12 @@ class DLLoggerPerfCallback(KerasCallback):
+         self._dllogger.log(
+             step=step,
+             data={
++                f'{mode}_time': self._calculate_total_time(self._start_timestamps[mode], time.time()),
+                 f'{mode}_throughput': self._calculate_throughput(deltas, self._batch_sizes[mode]),
+-                f'{mode}_latency': self._calculate_latency(deltas),
+-                f'{mode}_latency_90': self._calculate_latency_confidence(deltas, 90.0),
+-                f'{mode}_latency_95': self._calculate_latency_confidence(deltas, 95.0),
+-                f'{mode}_latency_99': self._calculate_latency_confidence(deltas, 99.0),
+-                f'{mode}_time': self._calculate_total_time(self._start_timestamps[mode], time.time())
++                # f'{mode}_latency': self._calculate_latency(deltas),
++                # f'{mode}_latency_90': self._calculate_latency_confidence(deltas, 90.0),
++                # f'{mode}_latency_95': self._calculate_latency_confidence(deltas, 95.0),
++                # f'{mode}_latency_99': self._calculate_latency_confidence(deltas, 99.0),
+             }
+         )
+ 
+diff --git a/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/runtime/learning_rate.py b/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/runtime/learning_rate.py
+index ee96214e..a3566a1a 100644
+--- a/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/runtime/learning_rate.py
++++ b/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/runtime/learning_rate.py
+@@ -1,6 +1,6 @@
+ import tensorflow as tf
+ 
+-
++@tf.keras.utils.register_keras_serializable('PiecewiseConstantWithWarmupSchedule')
+ class PiecewiseConstantWithWarmupSchedule(tf.keras.optimizers.schedules.LearningRateSchedule):
+     """
+     Schedule that starts with learning rate at `init_value` and monotonically increases
+diff --git a/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/runtime/run.py b/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/runtime/run.py
+index d7b001b3..f0854fd7 100644
+--- a/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/runtime/run.py
++++ b/TensorFlow2/Segmentation/MaskRCNN/mrcnn_tf2/runtime/run.py
+@@ -14,35 +14,39 @@ from mrcnn_tf2.runtime.weights_mapping import WEIGHTS_MAPPING
+ def run_training(dataset, params):
+     setup(params)
+ 
+-    strategy = tf.distribute.MirroredStrategy()
+-    params.replicas = strategy.num_replicas_in_sync
++    params.replicas = 1
+     params.global_train_batch_size = params.train_batch_size * params.replicas
+-    logging.info(f'Distributed Strategy is activated for {params.replicas} device(s)')
+-
+-    with strategy.scope():
+-
+-        learning_rate = PiecewiseConstantWithWarmupSchedule(
+-            init_value=params.init_learning_rate,
+-            # scale boundaries from epochs to steps
+-            boundaries=[
+-                int(b * dataset.train_size / params.global_train_batch_size)
+-                for b in params.learning_rate_boundaries
+-            ],
+-            values=params.learning_rate_values,
+-            # scale only by local BS as distributed strategy later scales it by number of replicas
+-            scale=params.train_batch_size
+-        )
+ 
+-        optimizer = tf.keras.optimizers.SGD(
+-            learning_rate=learning_rate,
+-            momentum=params.momentum
+-        )
++    learning_rate_init_value = params.init_learning_rate
++    if params.mpi_num > 1:
++        learning_rate_init_value = learning_rate_init_value * params.mpi_num
++    learning_rate = PiecewiseConstantWithWarmupSchedule(
++        init_value=learning_rate_init_value,
++        # scale boundaries from epochs to steps
++        boundaries=[
++            int(b * dataset.train_size / params.global_train_batch_size)
++            for b in params.learning_rate_boundaries
++        ],
++        values=params.learning_rate_values,
++        # scale only by local BS as distributed strategy later scales it by number of replicas
++        scale=params.train_batch_size
++    )
+ 
+-        mask_rcnn_model = create_model(params)
++    optimizer = tf.keras.optimizers.legacy.SGD(
++        learning_rate=learning_rate,
++        momentum=params.momentum
++    )
++    if params.mpi_num > 1:
++        import horovod.tensorflow.keras as hvd
++        logging.info("MPI horovod is enabled, using hvd.DistributedOptimizer")
++        optimizer = hvd.DistributedOptimizer(optimizer, groups=1)
+ 
+-        mask_rcnn_model.compile(
+-            optimizer=optimizer
+-        )
++    mask_rcnn_model = create_model(params)
++
++    mask_rcnn_model.compile(
++        optimizer=optimizer,
++        experimental_run_tf_function=False
++    )
+ 
+     # distributed strategy splits data between instances so we need global BS
+     train_data = dataset.train_fn(batch_size=params.global_train_batch_size)
+@@ -112,8 +116,9 @@ def setup(params):
+         logging.info('XLA is activated')
+ 
+     if params.amp:
+-        policy = tf.keras.mixed_precision.experimental.Policy("mixed_float16", loss_scale="dynamic")
+-        tf.keras.mixed_precision.experimental.set_policy(policy)
++        # change precision from mixed_float16 to mixed_bfloat16
++        policy = tf.keras.mixed_precision.Policy("mixed_bfloat16")
++        tf.keras.mixed_precision.set_global_policy(policy)
+         logging.info('AMP is activated')
+ 
+ 
+@@ -164,13 +169,26 @@ def create_callbacks(params):
+             mapping=lambda name: WEIGHTS_MAPPING.get(name.replace(':0', ''), name)
+         )
+ 
+-    yield tf.keras.callbacks.ModelCheckpoint(
+-        filepath=os.path.join(params.model_dir, params.checkpoint_name_format),
+-        verbose=1
+-    )
++    # if params.mpi_num > 1:
++    #     import horovod.tensorflow.keras as hvd
++    #     if hvd.rank() == 0:
++    #         yield tf.keras.callbacks.ModelCheckpoint(
++    #             filepath=os.path.join(params.model_dir, params.checkpoint_name_format),
++    #             verbose=1
++    #         )
++    # else:
++    #     yield tf.keras.callbacks.ModelCheckpoint(
++    #         filepath=os.path.join(params.model_dir, params.checkpoint_name_format),
++    #         verbose=1
++    #     )
+ 
+     if params.log_tensorboard:
+         yield tf.keras.callbacks.TensorBoard(
+-            log_dir=params.log_tensorboard,
+-            update_freq='batch'
++            log_dir=params.log_tensorboard, profile_batch=0,
++            update_freq=5
+         )
++
++    if params.mpi_num > 1:
++        import horovod.tensorflow.keras as hvd
++        logging.info("MPI horovod is enabled, hvd.callbacks.BroadcastGlobalVariablesCallback")
++        yield hvd.callbacks.BroadcastGlobalVariablesCallback(0)
+diff --git a/TensorFlow2/Segmentation/MaskRCNN/scripts/train.py b/TensorFlow2/Segmentation/MaskRCNN/scripts/train.py
+index 6b61827e..e46ff590 100644
+--- a/TensorFlow2/Segmentation/MaskRCNN/scripts/train.py
++++ b/TensorFlow2/Segmentation/MaskRCNN/scripts/train.py
+@@ -62,7 +62,6 @@ if __name__ == '__main__':
+         f'python {main_path}'
+         f' train'
+         f' --data_dir "{flags.data_dir}"'
+-        f' --backbone_checkpoint "{checkpoint_path}"'
+         f' --train_batch_size {flags.batch_size}'
+     )
+     cmd_eval = (
diff --git a/models_v2/tensorflow/maskrcnn/training/gpu/README.md b/models_v2/tensorflow/maskrcnn/training/gpu/README.md
new file mode 100644
index 000000000..3e7d548b7
--- /dev/null
+++ b/models_v2/tensorflow/maskrcnn/training/gpu/README.md
@@ -0,0 +1,112 @@
+# Mask RCNN training
+
+Mask RCNN Training using Intel® Extension for TensorFlow.
+
+## Model Information
+
+| **Use Case** | **Framework** | **Model Repo** |      **Branch/Commit/Tag**      | **Optional Patch** |
+| :---: | :---: | :---: |:-------------------------------:|:--------:|
+|   training   |  TensorFlow   | [DeepLearningExamples/MaskRCNN](https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow2/Segmentation/MaskRCNN) |         master                  | [EnableBF16.patch](#bf16patch) |
+
+**Note**: Refer to [CONTAINER.md](CONTAINER.md) for MaskRCNN training instructions using docker containers.
+# Pre-Requisite
+
+* Host has Intel® Data Center GPU Max
+* Host has installed latest Intel® Data Center GPU Max Series
+  Driver https://dgpu-docs.intel.com/driver/installation.html
+* The following Intel® oneAPI Base Toolkit components are required:
+  - Intel® oneAPI DPC++ Compiler (Placeholder DPCPPROOT as its installation path)
+  - Intel® oneAPI Math Kernel Library (oneMKL) (Placeholder MKLROOT as its installation path)
+  - Intel® oneAPI MPI Library
+  - Intel® oneAPI TBB Library
+  - Intel® oneAPI CCL Library
+
+  Follow instructions at [Intel® oneAPI Base Toolkit Download page](https://www.intel.com/content/www/us/en/developer/tools/oneapi/base-toolkit-download.html?operatingsystem=linux) to setup the package manager repository.
+
+# Dataset
+
+Download & preprocess COCO 2017 dataset.
+
+```
+export DATASET_DIR=/path/to/dataset/dir
+git clone https://github.com/NVIDIA/DeepLearningExamples.git
+cd ./DeepLearningExamples/TensorFlow2/Segmentation/MaskRCNN/dataset
+bash dataset/download_and_preprocess_coco.sh $DATASET_DIR
+```
+
+## Training
+
+1. `git clone https://github.com/IntelAI/models.git`
+2. `cd models/models_v2/tensorflow/maskrcnn/training/gpu`
+3. Create virtual environment `venv` and activate it:
+    ```
+    python3 -m venv venv
+    . ./venv/bin/activate
+    ```
+4. Run setup.sh
+    ```
+    ./setup.sh
+    ```
+5. Install [tensorflow and ITEX](https://pypi.org/project/intel-extension-for-tensorflow/)
+6. If you run multi-tile (export MULTI_TILE=True), please install "intel-optimization-for-horovod" with `pip install intel-optimization-for-horovod`
+7. Set environment variables for Intel® oneAPI Base Toolkit: 
+    Default installation location `{ONEAPI_ROOT}` is `/opt/intel/oneapi` for root account, `${HOME}/intel/oneapi` for other accounts
+    ```bash
+    source {ONEAPI_ROOT}/compiler/latest/env/vars.sh
+    source {ONEAPI_ROOT}/mkl/latest/env/vars.sh
+    source {ONEAPI_ROOT}/tbb/latest/env/vars.sh
+    source {ONEAPI_ROOT}/mpi/latest/env/vars.sh
+    source {ONEAPI_ROOT}/ccl/latest/env/vars.sh
+8. Download weights
+    ```
+        pushd .
+        cd ./DeepLearningExamples/TensorFlow2/Segmentation/MaskRCNN
+        python scripts/download_weights.py --save_dir=./weights
+        popd 
+    ```
+9. Setup required environment paramaters
+
+   | **Parameter**                  |               **export command**                |
+   |:-----------------------------------------------:|:----------------------------------------------:|
+   | **DATASET_DIR**                |    `export DATASET_DIR=/the/path/to/dataset`    |
+   | **OUTPUT_DIR**  (optional)                 |   `export OUTPUT_DIR=/the/path/to/output_dir`   |
+   | **BATCH_SIZE** (optional)      |              `export BATCH_SIZE=4`              |
+   | **PRECISION**                  | `export PRECISION=bfloat16` (bfloat16 or fp32)  |
+   | **EPOCHS** (optional)          |                `export EPOCHS=1`                |
+   | **STEPS_PER_EPOCH** (optional) |           `export STEPS_PER_EPOCH=20`           |
+   | **MULTI_TILE**           |    `export MULTI_TILE=False` (False or True)    |
+
+8. Run `run_model.sh`
+
+## Output
+
+Single-tile output will typically look like:
+
+```
+2023-09-11 14:54:49,905 I dllogger        (1, 20) loss: 639.5632934570312
+2023-09-11 14:54:49,906 I dllogger        (1, 20) train_time: 23.89216899871826, train_throughput: 21.438093303125907
+2023-09-11 14:54:49,914 I dllogger        (1,) loss: 639.5632934570312
+2023-09-11 14:54:49,914 I dllogger        () loss: 639.5632934570312
+2023-09-11 14:54:49,915 I dllogger        () train_time: 23.90118169784546, train_throughput: 23.507529269636105
+```
+Multi-tile output will typically look like:
+
+```
+[1] 2023-10-28 06:00:28,866 I dllogger        (1, 20) train_time: 38.43677306175232, train_throughput: 27.085276273687406
+[0] 2023-10-28 06:00:28,866 I dllogger        (1, 20) train_time: 38.44535684585571, train_throughput: 27.104091813787576
+[1] 2023-10-28 06:00:28,870 I dllogger        (1,) loss: 694.025390625
+[1] 2023-10-28 06:00:28,870 I dllogger        () loss: 694.025390625
+[0] 2023-10-28 06:00:28,870 I dllogger        (1,) loss: 694.025390625
+[1] 2023-10-28 06:00:28,870 I dllogger        () train_time: 38.44074249267578, train_throughput: 27.125991704955165
+[0] 2023-10-28 06:00:28,870 I dllogger        () loss: 694.025390625
+[0] 2023-10-28 06:00:28,870 I dllogger        () train_time: 38.449461460113525, train_throughput: 27.118943899083977
+```
+
+Final results of the training run can be found in `results.yaml` file.
+
+```
+results:
+ - key: throughput
+   value: 23.507529269636105
+   unit: images/sec
+```
diff --git a/models_v2/tensorflow/maskrcnn/training/gpu/requirements.txt b/models_v2/tensorflow/maskrcnn/training/gpu/requirements.txt
new file mode 100644
index 000000000..ea0b5da67
--- /dev/null
+++ b/models_v2/tensorflow/maskrcnn/training/gpu/requirements.txt
@@ -0,0 +1,23 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+
+opencv-python-headless
+pybind11
+pycocotools
+-e "git+https://github.com/NVIDIA/dllogger#egg=dllogger"
diff --git a/models_v2/tensorflow/maskrcnn/training/gpu/run_model.sh b/models_v2/tensorflow/maskrcnn/training/gpu/run_model.sh
new file mode 100755
index 000000000..217fe87a4
--- /dev/null
+++ b/models_v2/tensorflow/maskrcnn/training/gpu/run_model.sh
@@ -0,0 +1,106 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+
+#!/bin/bash
+
+# Create an array of input directories that are expected and then verify that they exist
+declare -A input_envs
+input_envs[PRECISION]=${PRECISION}
+input_envs[DATASET_DIR]=${DATASET_DIR}
+input_envs[OUTPUT_DIR]=${OUTPUT_DIR}
+input_envs[MULTI_TILE]=${MULTI_TILE}
+
+for i in "${!input_envs[@]}"; do
+  var_name=$i
+  env_param=${input_envs[$i]}
+
+  if [[ -z $env_param ]]; then
+    echo "The required environment variable $var_name is not set" >&2
+    exit 1
+  fi
+done
+
+if [ -d ${DATASET_DIR} ];then
+  echo "DATASET_DIR is: "${DATASET_DIR}
+else
+  echo "Error: the path of dataset does not exist!"
+  exit 1
+fi
+
+BATCH_SIZE=${BATCH_SIZE:-4}
+EPOCHS=${EPOCHS:-1}
+STEPS_PER_EPOCH=${STEPS_PER_EPOCH:-20}
+
+if [ ${PRECISION} == "bfloat16" ];then
+  echo "PRECISION is bfloat16"
+  AMP="--amp"
+else
+  echo "PRECISION is "${PRECISION}
+  AMP=""
+fi
+
+echo 'Running with parameters:'
+echo " DATASET_PATH: ${DATASET_DIR}"
+echo " OUTPUT_DIR: ${OUTPUT_DIR}"
+echo " PRECISION: ${PRECISION}"
+echo " BATCH_SIZE: $BATCH_SIZE"
+echo " EPOCHS: $EPOCHS"
+echo " STEPS_PER_EPOCH: $STEPS_PER_EPOCH"
+echo " MULTI_TILE: $MULTI_TILE"
+
+mpi=""
+mpi_number="2"
+if [[ $MULTI_TILE == "True" ]];then
+    mpi="mpirun -np $mpi_number -prepend-rank -ppn $mpi_number "
+fi
+
+rm -fr $OUTPUT_DIR
+
+cd ./DeepLearningExamples/TensorFlow2/Segmentation/MaskRCNN
+
+
+$mpi python main.py train \
+--data_dir $DATASET_DIR \
+--model_dir=$OUTPUT_DIR \
+--train_batch_size $BATCH_SIZE \
+--seed=0 --use_synthetic_data \
+--epochs $EPOCHS --steps_per_epoch $STEPS_PER_EPOCH \
+--log_every=1 --log_warmup_steps=1 \
+$AMP |& tee maskrcnn_training_${PRECISION}_BS${BATCH_SIZE}.log
+
+if [[ $MULTI_TILE == "False" ]];then
+    result=$(cat maskrcnn_training_${PRECISION}_BS${BATCH_SIZE}.log | grep train_throughput  | tail -n 1 | awk -F ' ' '{print $9}')
+    throughput=$result
+else
+    result=$(cat maskrcnn_training_${PRECISION}_BS${BATCH_SIZE}.log | grep train_throughput  | tail -n 1 | awk -F ' ' '{print $10}')
+    throughput=$(echo "$result $mpi_number" |awk '{printf("%.2f", $1*$2)}')
+fi
+cd -
+
+yaml_content=$(cat <<EOF
+results:
+ - key: throughput
+   value: $throughput
+   unit: images/sec
+EOF
+)
+
+# Write the content to a YAML file
+echo "$yaml_content" >  ${OUTPUT_DIR}/results.yaml
+echo "YAML file created."
diff --git a/models_v2/tensorflow/maskrcnn/training/gpu/setup.sh b/models_v2/tensorflow/maskrcnn/training/gpu/setup.sh
new file mode 100755
index 000000000..031e00535
--- /dev/null
+++ b/models_v2/tensorflow/maskrcnn/training/gpu/setup.sh
@@ -0,0 +1,36 @@
+#!/usr/bin/env bash
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# setup.sh
+#  - install OS pkgs
+#  - should create virtual env & install pip  requirement.txt
+#  - git clones & applying patches
+
+set -e
+apt-get update && apt-get install -y python3-venv protobuf-compiler
+pip install -r requirements.txt
+
+current_dir=$(pwd)
+if [ -d "DeepLearningExamples" ]; then
+  echo "Repository already exists. Skipping clone."
+else
+  git clone https://github.com/NVIDIA/DeepLearningExamples.git
+  cd ./DeepLearningExamples/TensorFlow2/Segmentation/MaskRCNN
+  git checkout c481324031ecf0f70f8939516c02e16cac60446d
+  git apply  $current_dir/EnableBF16.patch
+  cd -
+fi
diff --git a/models_v2/tensorflow/maskrcnn/training/gpu/tests.yaml b/models_v2/tensorflow/maskrcnn/training/gpu/tests.yaml
new file mode 100644
index 000000000..1d92c4df8
--- /dev/null
+++ b/models_v2/tensorflow/maskrcnn/training/gpu/tests.yaml
@@ -0,0 +1,10 @@
+training-fp32:
+  cmd: bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/setup.sh ${FRAMEWORK_VERSION} ${FRAMEWORK_EXTENSION_VERSION} ${IS_LKG_DROP} ${AIKIT_VERSION}; bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/maskrcnn-training/run_test.sh ${PRECISION} ${IS_LKG_DROP} ${DATASET_DIR}
+  env:
+    PRECISION: "fp32"
+    DATASET_DIR: "/tf_dataset/dataset/coco_dataset/COCO2017_training_data"  
+training-bfloat16:
+  cmd: bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/setup.sh ${FRAMEWORK_VERSION} ${FRAMEWORK_EXTENSION_VERSION} ${IS_LKG_DROP} ${AIKIT_VERSION}; bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/maskrcnn-training/run_test.sh ${PRECISION} ${IS_LKG_DROP} ${DATASET_DIR}
+  env:
+    PRECISION: "bfloat16"
+    DATASET_DIR: "/tf_dataset/dataset/coco_dataset/COCO2017_training_data"
diff --git a/models_v2/tensorflow/resnet50v1_5/inference/gpu/CONTAINER_FLEX.md b/models_v2/tensorflow/resnet50v1_5/inference/gpu/CONTAINER_FLEX.md
new file mode 100644
index 000000000..e98b8e2ba
--- /dev/null
+++ b/models_v2/tensorflow/resnet50v1_5/inference/gpu/CONTAINER_FLEX.md
@@ -0,0 +1,87 @@
+# Running ResNet50 v1.5 Inference with Int8 on Intel® Data Center GPU Flex Series using Intel® Extension for TensorFlow*
+
+## Overview
+
+This document has instructions for running ResNet50 v1.5 inference using Intel® Extension for TensorFlow* with Intel® Data Center GPU Flex Series.
+
+
+## Requirements
+| Item | Detail |
+| ------ | ------- |
+| Host machine  | [Intel® Data Center GPU Flex Series](https://ark.intel.com/content/www/us/en/ark/products/series/230021/intel-data-center-gpu-flex-series.html) 170 or 140 |
+| Drivers | GPU-compatible drivers need to be installed: [Download Driver](https://dgpu-docs.intel.com/driver/installation.html)
+| Software | Docker* |
+
+## Get Started
+
+### Download Datasets
+
+Download and preprocess the ImageNet dataset using the [instructions here](https://github.com/IntelAI/models/blob/master/datasets/imagenet/README.md).
+After running the conversion script you should have a directory with the
+ImageNet dataset in the TF records format.
+
+Set the `DATASET_DIR` to point to the TF records directory when running ResNet50 v1.5.
+
+### Quick Start Scripts
+
+| Script name | Description |
+|:-------------:|:-------------:|
+| `run_model.sh` | Runs inference for int8 precision |
+
+## Run Using Docker
+
+### Set up Docker Image
+
+```
+docker pull intel/image-recognition:tf-flex-gpu-resnet50v1-5-inference
+```
+
+### Run Docker Image
+The ResNet50 v1-5 inference container includes scripts, model and libraries needed to run int8 inference. To run the `run_model.sh` script using this container, you'll need to provide volume mounts for the ImageNet dataset for measuring accuracy test mode. For inference test mode, dummy dataset will be used.  You will need to provide an output directory where log files will be written. 
+
+**Note:** The default batch size for Flex series 140 is 256 for batch inference and 1024 for Flex series 170. Additionally, add `--cap-add=SYS_NICE` to the `docker run` command for executing the script on Flex series 140.
+```bash
+#Optional 
+BATCH_SIZE=<provide a batch size. Otherwise default batch sizes will be used>
+
+#Required
+export PRECISION=int8
+export OUTPUT_DIR=<path to output directory>
+export DATASET_DIR=<path to the preprocessed imagenet dataset>
+export PB_FILE_PATH=/workspace/tf-flex-series-resnet50v1-5-inference/models/pretrained_models/resnet50v1_5-frozen_graph-int8-gpu.pb
+IMAGE_NAME=intel/image-recognition:tf-flex-gpu-resnet50v1-5-inference
+TEST_MODE=<provide either inference or accuracy>
+FLEX_GPU_TYPE=<provide either flex_170 or flex_140 for inference mode>
+SCRIPT=run_model.sh
+
+docker run \
+  --device=/dev/dri \
+  --ipc=host \
+  --privileged \
+  --env PRECISION=${PRECISION} \
+  --env FLEX_GPU_TYPE=${FLEX_GPU_TYPE} \
+  --env OUTPUT_DIR=${OUTPUT_DIR} \
+  --env DATASET_DIR=${DATASET_DIR} \
+  --env TEST_MODE=${TEST_MODE} \
+  --env BATCH_SIZE=${BATCH_SIZE} \
+  --env PB_FILE_PATH=${PB_FILE_PATH} \
+  --env http_proxy=${http_proxy} \
+  --env https_proxy=${https_proxy} \
+  --env no_proxy=${no_proxy} \
+  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
+  --volume ${DATASET_DIR}:${DATASET_DIR} \
+  --rm -it \
+  $IMAGE_NAME \
+  /bin/bash $SCRIPT
+```
+
+## Documentation and Sources
+
+[GitHub* Repository](https://github.com/IntelAI/models/tree/master/docker/flex-gpu)
+
+## Support
+Support for Intel® Extension for TensorFlow* is found via the [Intel® AI Analytics Toolkit.](https://www.intel.com/content/www/us/en/developer/tools/oneapi/ai-analytics-toolkit.html#gs.qbretz) Additionally, the Intel® Extension for TensorFlow* team tracks both bugs and enhancement requests using [GitHub issues](https://github.com/intel/intel-extension-for-tensorflow/issues). Before submitting a suggestion or bug report, please search the GitHub issues to see if your issue has already been reported.
+
+## License Agreement
+
+LEGAL NOTICE: By accessing, downloading or using this software and any required dependent software (the “Software Package”), you agree to the terms and conditions of the software license agreements for the Software Package, which may also include notices, disclaimers, or license terms for third party software included with the Software Package. Please refer to the [license file](https://github.com/IntelAI/models/tree/master/third_party) for additional details.
diff --git a/models_v2/tensorflow/resnet50v1_5/inference/gpu/README.md b/models_v2/tensorflow/resnet50v1_5/inference/gpu/README.md
new file mode 100644
index 000000000..460b70d47
--- /dev/null
+++ b/models_v2/tensorflow/resnet50v1_5/inference/gpu/README.md
@@ -0,0 +1,172 @@
+# ResNet50v1.5 Model Inference with Intel® Extention for TensorFlow
+Best known method of ResNet50v1.5 Inference(float32, float16, bfloat16, tensorfloat32, int8) for Intel® Extention for TensorFlow.
+
+## Model Information
+| **Use Case** |**Framework** | **Model Repo** | **Branch/Commit/Tag** | **PB**
+| :---: | :---: | :---: | :---: | :---: |
+| inference | Tensorflow |[IntelAI/Models](https://github.com/IntelAI/models) | master | [resnet50_v1.pb](https://storage.googleapis.com/intel-optimized-tensorflow/models/3_1/resnet50_v1.pb) [resnet50_v1_int8.pb](https://storage.googleapis.com/intel-optimized-tensorflow/models/3_1/resnet50_v1_int8.pb) | |
+
+**Note:** Model files were copied and modified in this directory. Refer to [CONTAINER_FLEX.md](CONTAINER_FLEX.md) for ResNet50v1.5 Inference instructions using docker containers.
+
+# Pre-Requisite
+* Host has Intel® Data Center GPU Flex Series
+* Host has installed latest Intel® Data Center GPU Flex Series Drivers https://dgpu-docs.intel.com/driver/installation.html
+* The following Intel® oneAPI Base Toolkit components are required:
+  - Intel® oneAPI DPC++ Compiler (Placeholder DPCPPROOT as its installation path)
+  - Intel® oneAPI Math Kernel Library (oneMKL) (Placeholder MKLROOT as its installation path)
+  - Intel® oneAPI MPI Library
+  - Intel® oneAPI TBB Library
+  - Intel® oneAPI CCL Library
+
+  Follow instructions at [Intel® oneAPI Base Toolkit Download page](https://www.intel.com/content/www/us/en/developer/tools/oneapi/base-toolkit-download.html?operatingsystem=linux) to setup the package manager repository.
+
+# Dataset 
+To download and preprocess the ImageNet validation
+and training images to get them into the TF records format. The scripts used
+are based on the conversion scripts from the
+[TensorFlow TPU repo](https://github.com/tensorflow/tpu) and have been adapted
+to allow for offline preprocessing with cloud storage. Note that downloading
+the original images from ImageNet requires registering for an account and
+logging in.
+
+1. Go to the [ImageNet webpage (https://image-net.org)](https://image-net.org)
+   and log in (or create an account, if you don't have one). After logging in,
+   click the link at the top to get to the "Download" page. Select "2012" from
+   the list of available datasets.
+   Download the following tar files and save it to the compute system
+   (e.g. `/home/<user>/imagenet_raw_data`), 500GB disk space is required:
+
+   * Training images (Task 1 & 2). 138GB. MD5: 1d675b47d978889d74fa0da5fadfb00e
+   * Validation images (all tasks). 6.3GB. MD5: 29b22e2961454d5413ddabcf34fc5622
+
+   After this step is done, `/home/<user>/imagenet_raw_data` should have the above two
+   tar files, one is 138GB (`ILSVRC2012_img_train.tar`) and the other 6.3GB
+   (`ILSVRC2012_img_val.tar`).
+
+2. Setup a python 3 virtual environment with TensorFlow and the other
+   dependencies specified below. Note that google-cloud-storage is a dependency
+   of the script, but these instructions will not be using cloud storage.
+   ```
+   python3 -m venv tf_env
+   source tf_env/bin/activate
+   pip install --upgrade pip==19.3.1
+   pip install intel-tensorflow
+   pip install -I urllib3
+   pip install wget
+   deactivate
+   ```
+
+3. Download and run the [imagenet_to_tfrecords.sh](imagenet_to_tfrecords.sh) script and pass
+   arguments for the directory with the ImageNet tar files that were downloaded. To pre-process the entire dataset, pass `training` flag to the following script. To pre-process only the validation dataset, pass `inference` flag to the following script as shown below. 
+   in step 1 (e.g. `/home/<user>/imagenet_raw_data`).
+
+   ```bash
+   wget https://raw.githubusercontent.com/IntelAI/models/master/datasets/imagenet/imagenet_to_tfrecords.sh
+   ./imagenet_to_tfrecords.sh <IMAGENET DIR> inference
+   ```
+   The `imagenet_to_tfrecords.sh` script extracts the ImageNet tar files, downloads and
+   then runs the [`imagenet_to_gcs.py`](imagenet_to_gcs.py) script to convert the
+   files to TF records. As the script is running you should see output like:
+   ```
+   I0911 16:23:59.174904 140581751400256 imagenet_to_gcs.py:354] Finished writing file: <IMAGENET DIR>/tf_records/train/train-00000-of-01024
+   I0911 16:23:59.199399 140581751400256 imagenet_to_gcs.py:354] Finished writing file: <IMAGENET DIR>/tf_records/train/train-00001-of-01024
+   I0911 16:23:59.221770 140581751400256 imagenet_to_gcs.py:354] Finished writing file: <IMAGENET DIR>/tf_records/train/train-00002-of-01024
+   I0911 16:23:59.251754 140581751400256 imagenet_to_gcs.py:354] Finished writing file: <IMAGENET DIR>/tf_records/train/train-00003-of-01024
+   ...
+   I0911 16:24:22.338566 140581751400256 imagenet_to_gcs.py:402] Processing the validation data.
+   I0911 16:24:23.271091 140581751400256 imagenet_to_gcs.py:354] Finished writing file: <IMAGENET DIR>/tf_records/validation/validation-00000-of-00128
+   I0911 16:24:24.260855 140581751400256 imagenet_to_gcs.py:354] Finished writing file: <IMAGENET DIR>/tf_records/validation/validation-00001-of-00128
+   I0911 16:24:25.179738 140581751400256 imagenet_to_gcs.py:354] Finished writing file: <IMAGENET DIR>/tf_records/validation/validation-00002-of-00128
+   I0911 16:24:26.097850 140581751400256 imagenet_to_gcs.py:354] Finished writing file: <IMAGENET DIR>/tf_records/validation/validation-00003-of-00128
+   I0911 16:24:27.028785 140581751400256 imagenet_to_gcs.py:354] Finished writing file: <IMAGENET DIR>/tf_records/validation/validation-00004-of-00128
+   ...
+   ```
+   After the `imagenet_to_gcs.py` script completes, the `imagenet_to_tfrecords.sh` script combines
+   the train and validation files into the `<IMAGENET DIR>/tf_records`
+   directory. The folder should contains 1024 training files and 128 validation
+   files.
+   ```
+   $ ls -1 <IMAGENET DIR>/tf_records/
+   train-00000-of-01024
+   train-00001-of-01024
+   train-00002-of-01024
+   train-00003-of-01024
+   ...
+   validation-00000-of-00128
+   validation-00001-of-00128
+   validation-00002-of-00128
+   validation-00003-of-00128
+   ...
+   ```
+## Inference
+1. `git clone https://github.com/IntelAI/models.git`
+2. `cd models/models_v2/tensorflow/resnet50v1_5/inference/gpu`
+3. create virtual environment `venv` and activate it:
+    ```
+    python3 -m venv venv
+    . ./venv/bin/activate
+    ```
+4. Run setup.sh
+    ```
+    ./setup.sh
+    ```
+5. Install [tensorflow and ITEX](https://pypi.org/project/intel-extension-for-tensorflow/)
+6. Download the PB files (pre trained models) and the set the path for PB_FILE_PATH:
+   ```
+   # For int8:
+   wget https://storage.googleapis.com/intel-optimized-tensorflow/models/3_1/resnet50_v1_int8.pb
+
+   # For float32, tensorflow32, float16 and bfloat16:
+   wget https://storage.googleapis.com/intel-optimized-tensorflow/models/3_1/resnet50_v1.pb
+   ```
+7. Set environment variables for Intel® oneAPI Base Toolkit: 
+    Default installation location `{ONEAPI_ROOT}` is `/opt/intel/oneapi` for root account, `${HOME}/intel/oneapi` for other accounts
+    ```bash
+    source {ONEAPI_ROOT}/compiler/latest/env/vars.sh
+    source {ONEAPI_ROOT}/mkl/latest/env/vars.sh
+    source {ONEAPI_ROOT}/tbb/latest/env/vars.sh
+    source {ONEAPI_ROOT}/mpi/latest/env/vars.sh
+    source {ONEAPI_ROOT}/ccl/latest/env/vars.sh
+8. Setup required environment paramaters
+   #### Set Model Parameters
+   Export those parameters to script or environment.
+   | **Parameter** | **export** |
+   | :---: | :---: |
+   | **DATASET_DIR** | `export DATASET_PATH=/the/path/to/ImageNet` (accuracy mode need) |
+   | **PB_FILE_PATH** | `export PB_FILE_PATH=/the/path/to/xx.pb` (int8:resnet50_v1_int8.pb, others:resnet50_v1.pb)|
+   | **BATCH_SIZE** | `export BATCH_SIZE=1024` (optional, default is 1024) |
+   | **TEST_MODE** | `export TEST_MODE=inference` (inference or accuracy)  |
+   | **DTYPE** | `export PRECISION=float32` (float32,tensorfloat32,float16 ,bfloat16 or int8) |    
+   | **FLEX_GPU_TYPE** |  `export FLEX_GPU_TYPE=<flex_140 or flex_170>`      |
+8. Run `run_model.sh`
+
+## Output
+
+Output typically looks like:
+#inference
+```
+Iteration 4997: 0.xxxx sec
+Iteration 4998: 0.xxxx sec
+Iteration 4999: 0.xxxx sec
+Iteration 5000: 0.xxxx sec
+Average time: 0.xxxx sec
+Batch size = 1024
+Throughput: xxxx images/sec
+```
+#accuracy
+```
+Iteration time: 0.0042 ms
+Processed 49920 images. (Top1 accuracy, Top5 accuracy) = (xxxx, xxxx)
+Iteration time: 0.0043 ms
+Processed 49952 images. (Top1 accuracy, Top5 accuracy) = (xxxx, xxxx)
+Iteration time: 0.0042 ms
+Processed 49984 images. (Top1 accuracy, Top5 accuracy) = (xxxx, xxxx)
+```
+
+Final results of the training run can be found in `results.yaml` file.
+```
+results:
+ - key: throughput
+   value: 23.507529269636105
+   unit: images/sec
+```
diff --git a/models/object_detection/tensorflow/faster_rcnn/inference/int8/__init__.py b/models_v2/tensorflow/resnet50v1_5/inference/gpu/__init__.py
similarity index 100%
rename from models/object_detection/tensorflow/faster_rcnn/inference/int8/__init__.py
rename to models_v2/tensorflow/resnet50v1_5/inference/gpu/__init__.py
diff --git a/models_v2/tensorflow/resnet50v1_5/inference/gpu/datasets.py b/models_v2/tensorflow/resnet50v1_5/inference/gpu/datasets.py
new file mode 100644
index 000000000..954a54d5c
--- /dev/null
+++ b/models_v2/tensorflow/resnet50v1_5/inference/gpu/datasets.py
@@ -0,0 +1,96 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2019 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+#
+
+# Copyright 2017 The TensorFlow Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+
+"""Benchmark dataset utilities.
+"""
+
+from abc import abstractmethod
+import os
+
+import tensorflow as tf
+
+import preprocessing
+
+IMAGENET_NUM_TRAIN_IMAGES = 1281167
+IMAGENET_NUM_VAL_IMAGES = 50000
+IMAGENET_NUM_CLASSES = 1000
+
+class Dataset(object):
+  """Abstract class for cnn benchmarks dataset."""
+
+  def __init__(self, name, data_dir=None):
+    self.name = name
+    if data_dir is None:
+      raise ValueError('Data directory not specified')
+    self.data_dir = data_dir
+
+  def tf_record_pattern(self, subset):
+    return os.path.join(self.data_dir, '%s-*-of-*' % subset)
+
+  def reader(self):
+    return tf.compat.v1.TFRecordReader()
+
+  @abstractmethod
+  def num_classes(self):
+    pass
+
+  @abstractmethod
+  def num_examples_per_epoch(self, subset):
+    pass
+
+  def __str__(self):
+    return self.name
+
+
+class ImagenetData(Dataset):
+
+  def __init__(self, data_dir=None):
+    super(ImagenetData, self).__init__('ImageNet', data_dir)
+
+  def num_classes(self):
+    return IMAGENET_NUM_CLASSES
+
+  def num_examples_per_epoch(self, subset='train'):
+    if subset == 'train':
+      return IMAGENET_NUM_TRAIN_IMAGES
+    elif subset == 'validation':
+      return IMAGENET_NUM_VAL_IMAGES
+    elif subset == 'calibrate' or subset == 'calibration':
+      return 100
+    else:
+      raise ValueError('Invalid data subset "%s"' % subset)
+
+  def get_image_preprocessor(self):
+    return preprocessing.RecordInputImagePreprocessor
\ No newline at end of file
diff --git a/models_v2/tensorflow/resnet50v1_5/inference/gpu/eval_image_classifier_inference.py b/models_v2/tensorflow/resnet50v1_5/inference/gpu/eval_image_classifier_inference.py
new file mode 100644
index 000000000..81ce52061
--- /dev/null
+++ b/models_v2/tensorflow/resnet50v1_5/inference/gpu/eval_image_classifier_inference.py
@@ -0,0 +1,305 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2019 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+#
+import os
+import time
+import sys
+from argparse import ArgumentParser
+
+import tensorflow as tf
+from tensorflow.python.tools.optimize_for_inference_lib import optimize_for_inference
+from tensorflow.python.framework import dtypes
+
+import datasets
+from optimize_for_benchmark import optimize_for_benchmark
+import numpy as np
+
+INPUTS = 'input_tensor'
+OUTPUTS = 'softmax_tensor'
+
+RESNET_IMAGE_SIZE = 224
+
+
+class eval_classifier_optimized_graph:
+  """Evaluate image classifier with optimized TensorFlow graph"""
+
+  def __init__(self):
+
+    arg_parser = ArgumentParser(description='Parse args')
+
+    arg_parser.add_argument('-b', "--batch-size",
+                            help="Specify the batch size. If this " \
+                                 "parameter is not specified or is -1, the " \
+                                 "largest ideal batch size for the model will " \
+                                 "be used.",
+                            dest="batch_size", type=int, default=-1)
+
+    arg_parser.add_argument('-e', "--num-inter-threads",
+                            help='The number of inter-thread.',
+                            dest='num_inter_threads', type=int, default=0)
+
+    arg_parser.add_argument('-a', "--num-intra-threads",
+                            help='The number of intra-thread.',
+                            dest='num_intra_threads', type=int, default=0)
+
+    arg_parser.add_argument('-m', "--model-name",
+                            help='Specify the model name to run benchmark for',
+                            dest='model_name')
+
+    arg_parser.add_argument('-g', "--input-graph",
+                            help='Specify the input graph for the transform tool',
+                            dest='input_graph')
+
+    arg_parser.add_argument('-d', "--data-location",
+                            help='Specify the location of the data. '
+                                 'If this parameter is not specified, '
+                                 'the benchmark will use random/dummy data.',
+                            dest="data_location", default=None)
+
+    arg_parser.add_argument('-r', "--accuracy-only",
+                            help='For accuracy measurement only.',
+                            dest='accuracy_only', action='store_true')
+    arg_parser.add_argument('--calibrate', dest='calibrate',
+                            help='Run accuracy with calibration data,'
+                                 'to generate min_max ranges, calibrate=[True/False]',
+                            type=bool, default=False)
+    arg_parser.add_argument("--results-file-path",
+                            help="File path for the inference results",
+                            dest="results_file_path", default=None)
+    arg_parser.add_argument("--warmup-steps", type=int, default=10,
+                            help="number of warmup steps")
+    arg_parser.add_argument("--steps", type=int, default=50,
+                            help="number of steps")
+
+    arg_parser.add_argument(
+      '--data-num-inter-threads', dest='data_num_inter_threads',
+      help='number threads across operators',
+      type=int, default=32)
+    arg_parser.add_argument(
+      '--data-num-intra-threads', dest='data_num_intra_threads',
+      help='number threads for data layer operator',
+      type=int, default=14)
+    arg_parser.add_argument(
+      '--num-cores', dest='num_cores',
+      help='number of cores',
+      type=int, default=28)
+    arg_parser.add_argument(
+      '--dtype',
+      help='data type, only support float32/bfloat16/float16/tensorfloat32/int8',
+      type=str, default="float32")  
+
+    arg_parser.add_argument("--benchmark",
+                            help='Run in benchmark mode.',
+                            dest='benchmark', action='store_true')
+
+    self.args = arg_parser.parse_args()
+    # validate the arguements
+    self.validate_args()
+    if self.args.dtype == "float16":
+      os.environ['ITEX_AUTO_MIXED_PRECISION'] = '1'
+      os.environ['ITEX_AUTO_MIXED_PRECISION_DATA_TYPE'] = 'FLOAT16'
+    elif self.args.dtype == "bfloat16":
+      os.environ['ITEX_AUTO_MIXED_PRECISION'] = '1'
+      os.environ['ITEX_AUTO_MIXED_PRECISION_DATA_TYPE'] = 'BFLOAT16'
+    elif self.args.dtype == "tensorfloat32":
+      os.environ['ITEX_FP32_MATH_MODE'] = 'TF32'
+    elif self.args.dtype == "float32":
+      print("using default data type: float32")
+    elif self.args.dtype == "int8":
+      print("using default data type: int8")
+    else:
+      print("Unknown Datatype")     
+
+  def write_results_output(self, predictions, filenames, labels):
+    # If a results_file_path is provided, write the predictions to the file
+    if self.args.results_file_path:
+      top_predictions = np.argmax(predictions, 1)
+      with open(self.args.results_file_path, "a") as fp:
+        for filename, expected_label, top_prediction in zip(filenames, labels, top_predictions):
+          fp.write("{},{},{}\n".format(filename, expected_label, top_prediction))
+
+  def run(self):
+    """run benchmark with optimized graph"""
+
+    print("Run inference")
+
+    data_config = tf.compat.v1.ConfigProto()
+    data_config.intra_op_parallelism_threads = self.args.data_num_intra_threads
+    data_config.inter_op_parallelism_threads = self.args.data_num_inter_threads
+    data_config.use_per_session_threads = 1
+
+    infer_config = tf.compat.v1.ConfigProto()
+    infer_config.intra_op_parallelism_threads = self.args.num_intra_threads
+    infer_config.inter_op_parallelism_threads = self.args.num_inter_threads
+    infer_config.use_per_session_threads = 1
+
+    data_graph = tf.Graph()
+    with data_graph.as_default():
+      if (self.args.data_location):
+        print("Inference with real data.")
+        if self.args.calibrate:
+            subset = 'calibration'
+        else:
+            subset = 'validation'
+        dataset = datasets.ImagenetData(self.args.data_location)
+        preprocessor = dataset.get_image_preprocessor()(
+            RESNET_IMAGE_SIZE, RESNET_IMAGE_SIZE, self.args.batch_size,
+            num_cores=self.args.num_cores,
+            resize_method='crop')
+
+        images, labels, filenames = preprocessor.minibatch(dataset, subset=subset)
+
+        # If a results file path is provided, then start the prediction output file
+        if self.args.results_file_path:
+          with open(self.args.results_file_path, "w+") as fp:
+            fp.write("filename,actual,prediction\n")
+      else:
+        print("Inference with dummy data.")
+        input_shape = [self.args.batch_size, RESNET_IMAGE_SIZE, RESNET_IMAGE_SIZE, 3]
+        images = tf.random.uniform(input_shape, 0.0, 255.0, dtype=tf.float32, name='synthetic_images')
+
+    infer_graph = tf.Graph()
+    with infer_graph.as_default():
+      graph_def = tf.compat.v1.GraphDef()
+      with tf.compat.v1.gfile.FastGFile(self.args.input_graph, 'rb') as input_file:
+        input_graph_content = input_file.read()
+        graph_def.ParseFromString(input_graph_content)
+
+      output_graph = optimize_for_inference(graph_def, [INPUTS], 
+                              [OUTPUTS], dtypes.float32.as_datatype_enum, False)
+      if self.args.benchmark:
+        input_shape = [self.args.batch_size, RESNET_IMAGE_SIZE, RESNET_IMAGE_SIZE, 3]
+        dummy_input = np.random.randn(*input_shape)
+        output_graph = optimize_for_benchmark(output_graph, tf.float32, dummy_input)
+
+      tf.import_graph_def(output_graph, name='')
+
+    # Definite input and output Tensors for detection_graph
+    input_tensor = infer_graph.get_tensor_by_name('input_tensor:0')
+    output_tensor = infer_graph.get_tensor_by_name('softmax_tensor:0')
+
+    data_sess  = tf.compat.v1.Session(graph=data_graph,  config=data_config)
+    infer_sess = tf.compat.v1.Session(graph=infer_graph, config=infer_config)
+
+    num_processed_images = 0
+    num_remaining_images = dataset.num_examples_per_epoch(subset=subset) - num_processed_images \
+        if self.args.data_location else self.args.batch_size * self.args.steps
+
+    if (not self.args.accuracy_only):
+      iteration = 0
+      warm_up_iteration = self.args.warmup_steps
+      total_run = self.args.steps
+      total_time = 0
+
+      while num_remaining_images >= self.args.batch_size and iteration < total_run:
+        iteration += 1
+        tf_filenames = None
+        np_labels = None
+        data_load_start = time.time()
+        if self.args.results_file_path:
+          image_np, np_labels, tf_filenames = data_sess.run([images, labels, filenames])
+        else:
+          image_np = data_sess.run(images)
+
+        data_load_time = time.time() - data_load_start
+
+        num_processed_images += self.args.batch_size
+        num_remaining_images -= self.args.batch_size
+
+        start_time = time.time()
+        if self.args.benchmark:
+          predictions = infer_sess.run(output_tensor)
+        else:
+          predictions = infer_sess.run(output_tensor, feed_dict={input_tensor: image_np})
+        time_consume = time.time() - start_time
+
+        # Write out the file name, expected label, and top prediction
+        self.write_results_output(predictions, tf_filenames, np_labels)
+
+        # only add data loading time for real data, not for dummy data
+        if self.args.data_location:
+          time_consume += data_load_time
+
+        print('Iteration %d: %.6f sec' % (iteration, time_consume))
+        sys.stdout.flush()
+        if iteration > warm_up_iteration:
+          total_time += time_consume
+
+      time_average = total_time / (iteration - warm_up_iteration)
+      print('Average time: %.6f sec' % (time_average))
+
+      print('Batch size = %d' % self.args.batch_size)
+      if (self.args.batch_size == 1):
+        print('Latency: %.3f ms' % (time_average * 1000))
+      # print throughput for both batch size 1 and 128
+      print('Throughput: %.3f images/sec' % (self.args.batch_size / time_average))
+      sys.stdout.flush()
+
+    else: # accuracy check
+      total_accuracy1, total_accuracy5 = (0.0, 0.0)
+
+      while num_remaining_images >= self.args.batch_size:
+        # Reads and preprocess data
+        tf_filenames = None
+        if self.args.results_file_path:
+          np_images, np_labels, tf_filenames = data_sess.run([images, labels, filenames])
+        else:
+          np_images, np_labels = data_sess.run([images, labels])
+        num_processed_images += self.args.batch_size
+        num_remaining_images -= self.args.batch_size
+
+        start_time = time.time()
+        # Compute inference on the preprocessed data
+        predictions = infer_sess.run(output_tensor,
+                               {input_tensor: np_images})
+        elapsed_time = time.time() - start_time
+
+        # Write out the file name, expected label, and top prediction
+        self.write_results_output(predictions, tf_filenames, np_labels)
+
+        with tf.Graph().as_default() as accu_graph:
+          accuracy1 = tf.reduce_sum(
+            input_tensor=tf.cast(tf.nn.in_top_k(predictions=tf.constant(predictions),
+                                   targets=tf.constant(np_labels), k=1), tf.float32))
+
+          accuracy5 = tf.reduce_sum(
+            input_tensor=tf.cast(tf.nn.in_top_k(predictions=tf.constant(predictions),
+                                   targets=tf.constant(np_labels), k=5), tf.float32))
+          with tf.compat.v1.Session() as accu_sess:
+            np_accuracy1, np_accuracy5 = accu_sess.run([accuracy1, accuracy5])
+
+          total_accuracy1 += np_accuracy1
+          total_accuracy5 += np_accuracy5
+
+        print("Iteration time: %0.4f ms" % elapsed_time)
+        print("Processed %d images. (Top1 accuracy, Top5 accuracy) = (%0.4f, %0.4f)" \
+                  % (num_processed_images, total_accuracy1 / num_processed_images,
+                     total_accuracy5 / num_processed_images))
+
+  def validate_args(self):
+    """validate the arguments"""
+
+    if not self.args.data_location:
+      if self.args.accuracy_only:
+        raise ValueError("You must use real data for accuracy measurement.")
+
+
+if __name__ == "__main__":
+  evaluate_opt_graph = eval_classifier_optimized_graph()
+  evaluate_opt_graph.run()
diff --git a/models_v2/tensorflow/resnet50v1_5/inference/gpu/optimize_for_benchmark.py b/models_v2/tensorflow/resnet50v1_5/inference/gpu/optimize_for_benchmark.py
new file mode 100644
index 000000000..41dc27035
--- /dev/null
+++ b/models_v2/tensorflow/resnet50v1_5/inference/gpu/optimize_for_benchmark.py
@@ -0,0 +1,38 @@
+
+from tensorflow.core.framework import graph_pb2
+from tensorflow.core.framework import node_def_pb2
+from tensorflow.python.framework import tensor_util
+from tensorflow.python.tools.optimize_for_inference_lib import ensure_graph_is_valid
+
+import numpy as np
+
+def optimize_for_benchmark(input_graph_def, const_dtype, dummy_input):
+  ensure_graph_is_valid(input_graph_def)
+  optimized_graph_def = change_placehoder_to_const(input_graph_def, const_dtype, dummy_input)
+  ensure_graph_is_valid(input_graph_def)
+  return optimized_graph_def
+
+def change_placehoder_to_const(input_graph_def, const_dtype, dummy_input):
+  result_graph_def = graph_pb2.GraphDef()
+  for node in input_graph_def.node:
+    if node.op == 'Placeholder':
+      new_const = node_def_pb2.NodeDef()
+      new_const.op = 'Const'
+      new_const.name = node.name
+      new_const.attr["dtype"].CopyFrom(node.attr["dtype"])
+      tensor_proto = tensor_util.make_tensor_proto(dummy_input,
+                                                   const_dtype,
+                                                   dummy_input.shape)
+      new_const.attr["value"].tensor.CopyFrom(tensor_proto)
+      result_graph_def.node.extend([new_const])
+    else:
+      new_node = node_def_pb2.NodeDef()
+      new_node.CopyFrom(node)
+      retained_input = []
+      for input_node in new_node.input:
+        retained_input.append(input_node)
+      new_node.input[:] = retained_input
+
+      result_graph_def.node.extend([new_node])
+
+  return result_graph_def
diff --git a/models_v2/tensorflow/resnet50v1_5/inference/gpu/preprocessing.py b/models_v2/tensorflow/resnet50v1_5/inference/gpu/preprocessing.py
new file mode 100644
index 000000000..61f43d872
--- /dev/null
+++ b/models_v2/tensorflow/resnet50v1_5/inference/gpu/preprocessing.py
@@ -0,0 +1,177 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2019 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+#
+
+# Copyright 2017 The TensorFlow Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import tensorflow as tf
+from tensorflow.python.data.experimental import parallel_interleave
+from tensorflow.python.data.experimental import map_and_batch
+from tensorflow.python.platform import gfile
+
+
+def parse_example_proto(example_serialized):
+  """Parses an Example proto containing a training example of an image.
+  """
+  # Dense features in Example proto.
+  feature_map = {
+    'image/encoded': tf.io.FixedLenFeature([], dtype=tf.string,
+                                        default_value=''),
+    'image/class/label': tf.io.FixedLenFeature([1], dtype=tf.int64,
+                                            default_value=-1),
+    'image/filename': tf.io.FixedLenFeature([], dtype=tf.string,
+                                         default_value="")
+  }
+  sparse_float32 = tf.io.VarLenFeature(dtype=tf.float32)
+  # Sparse features in Example proto.
+  feature_map.update(
+    {k: sparse_float32 for k in ['image/object/bbox/xmin',
+                                 'image/object/bbox/ymin',
+                                 'image/object/bbox/xmax',
+                                 'image/object/bbox/ymax']})
+
+  features = tf.io.parse_single_example(serialized=example_serialized, features=feature_map)
+  label = tf.cast(features['image/class/label'], dtype=tf.int32)
+  filename = tf.cast(features['image/filename'], dtype=tf.string)
+
+  return features['image/encoded'], label, filename
+
+
+def eval_image(image, height, width, resize_method,
+               central_fraction=0.875, scope=None):
+
+  with tf.compat.v1.name_scope('eval_image'):
+    if resize_method == 'crop':
+      shape = tf.shape(input=image)
+      image = tf.cond(pred=tf.less(shape[0], shape[1]),
+                      true_fn=lambda: tf.image.resize(image,
+                                                     tf.convert_to_tensor(value=[256, 256 * shape[1] / shape[0]],
+                                                                          dtype=tf.int32)),
+                      false_fn=lambda: tf.image.resize(image,
+                                                     tf.convert_to_tensor(value=[256 * shape[0] / shape[1], 256],
+                                                                          dtype=tf.int32)))
+
+      shape = tf.shape(input=image)
+      y0 = (shape[0] - height) // 2
+      x0 = (shape[1] - width) // 2
+      distorted_image = tf.image.crop_to_bounding_box(image, y0, x0, height, width)
+      distorted_image.set_shape([height, width, 3])
+      means = tf.broadcast_to([123.68, 116.78, 103.94], tf.shape(input=distorted_image))
+      return distorted_image - means
+    else:  # bilinear
+      if image.dtype != tf.float32:
+        image = tf.image.convert_image_dtype(image, dtype=tf.float32)
+      # Crop the central region of the image with an area containing 87.5% of
+      # the original image.
+      if central_fraction:
+        image = tf.image.central_crop(image, central_fraction=central_fraction)
+
+      if height and width:
+        # Resize the image to the specified height and width.
+        image = tf.expand_dims(image, 0)
+        image = tf.image.resize(image, [height, width],
+                                         method=tf.image.ResizeMethod.BILINEAR)
+        image = tf.squeeze(image, [0])
+      image = tf.subtract(image, 0.5)
+      image = tf.multiply(image, 2.0)
+      return image
+
+class RecordInputImagePreprocessor(object):
+  """Preprocessor for images with RecordInput format."""
+
+  def __init__(self,
+               height,
+               width,
+               batch_size,
+               num_cores,
+               resize_method="bilinear"):
+
+    self.height = height
+    self.width = width
+    self.batch_size = batch_size
+    self.num_cores = num_cores
+    self.resize_method = resize_method
+
+  def parse_and_preprocess(self, value):
+    # parse
+    image_buffer, label_index, filename = parse_example_proto(value)
+    # preprocess
+    image = tf.image.decode_jpeg(
+      image_buffer, channels=3, fancy_upscaling=False, dct_method='INTEGER_FAST')
+    image = eval_image(image, self.height, self.width, self.resize_method)
+    return (image, label_index, filename)
+
+  def minibatch(self, dataset, subset, cache_data=False):
+
+    with tf.compat.v1.name_scope('batch_processing'):
+
+      glob_pattern = dataset.tf_record_pattern(subset)
+      file_names = gfile.Glob(glob_pattern)
+      if not file_names:
+        raise ValueError('Found no files in --data_dir matching: {}'
+                         .format(glob_pattern))
+      ds = tf.data.TFRecordDataset.list_files(file_names)
+
+      ds = ds.apply(
+        parallel_interleave(
+          tf.data.TFRecordDataset, cycle_length=self.num_cores, block_length=5,
+          sloppy=True,
+          buffer_output_elements=10000, prefetch_input_elements=10000))
+
+      if cache_data:
+        ds = ds.take(1).cache().repeat()
+
+      ds = ds.prefetch(buffer_size=10000)
+      #ds = ds.prefetch(buffer_size=self.batch_size)
+
+      # num of parallel batches not greater than 56
+      max_num_parallel_batches = min(56, 2 * self.num_cores)
+      ds = ds.apply(
+        map_and_batch(
+          map_func=self.parse_and_preprocess,
+          batch_size=self.batch_size,
+          num_parallel_batches=max_num_parallel_batches,
+          num_parallel_calls=None))
+
+      ds = ds.prefetch(buffer_size=tf.data.experimental.AUTOTUNE)
+
+      ds_iterator = tf.compat.v1.data.make_one_shot_iterator(ds)
+      images, labels, filename = ds_iterator.get_next()
+      # reshape
+      labels = tf.reshape(labels, [self.batch_size])
+      filename = tf.reshape(filename, [self.batch_size])
+
+      return images, labels, filename
diff --git a/models_v2/tensorflow/resnet50v1_5/inference/gpu/requirements.txt b/models_v2/tensorflow/resnet50v1_5/inference/gpu/requirements.txt
new file mode 100644
index 000000000..56aed9a1c
--- /dev/null
+++ b/models_v2/tensorflow/resnet50v1_5/inference/gpu/requirements.txt
@@ -0,0 +1,2 @@
+tensorflow
+intel-extension-for-tensorflow[xpu]
\ No newline at end of file
diff --git a/models_v2/tensorflow/resnet50v1_5/inference/gpu/run_model.sh b/models_v2/tensorflow/resnet50v1_5/inference/gpu/run_model.sh
new file mode 100755
index 000000000..a2e705af0
--- /dev/null
+++ b/models_v2/tensorflow/resnet50v1_5/inference/gpu/run_model.sh
@@ -0,0 +1,152 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# SPDX-License-Identifier: EPL-2.0
+#
+
+#!/bin/bash
+
+# Create an array of input directories that are expected and then verify that they exist
+declare -A input_envs
+input_envs[PRECISION]=${PRECISION}
+input_envs[PB_FILE_PATH]=${PB_FILE_PATH}
+#input_envs[DATASET_DIR]=${DATASET_DIR} 
+#input_envs[BATCH_SIZE]=${BATCH_SIZE} #if not set batch size, the default 1024 will use
+input_envs[TEST_MODE]=${TEST_MODE}
+input_envs[OUTPUT_DIR]=${OUTPUT_DIR}
+
+for i in "${!input_envs[@]}"; do
+  var_name=$i
+  env_param=${input_envs[$i]}
+ 
+  if [[ -z $env_param ]]; then
+    echo "The required environment variable $var_name is not set" >&2
+    exit 1
+  fi
+done
+
+mkdir -p ${OUTPUT_DIR}
+
+#dataset only need for accuracy
+if [ -n "${DATASET_DIR}" ];then
+  if [ -d ${DATASET_DIR} ];then
+    echo "DATASET_DIR is "${DATASET_DIR}
+  else
+    echo "Error: the path of dataset does not exist!"
+    exit 1
+  fi
+elif [ ${TEST_MODE} == "accuracy" ];then
+  echo "Error: the path of dataset is required!"
+  exit 1
+fi
+
+BATCH_SIZE="${BATCH_SIZE:-1024}"
+echo 'Running with parameters:'
+echo " DATASET_PATH: ${DATASET_DIR}"
+echo " PB_FILE_PATH: ${PB_FILE_PATH}"
+echo " PRECISION: ${PRECISION}"
+echo " BATCH_SIZE: ${BATCH_SIZE}"
+echo " MODE: ${TEST_MODE}" 
+
+if [ ${PRECISION} == "float16" ];then
+  export ITEX_AUTO_MIXED_PRECISION=1
+  export ITEX_AUTO_MIXED_PRECISION_DATA_TYPE="FLOAT16"
+elif [ ${PRECISION} == "bfloat16" ];then
+  export ITEX_AUTO_MIXED_PRECISION=1
+  export ITEX_AUTO_MIXED_PRECISION_DATA_TYPE="BFLOAT16"
+elif [ ${PRECISION} == "tensorfloat32" ];then
+  export ITEX_FP32_MATH_MODE=TF32
+elif [ ${PRECISION} == "float32" ];then
+  echo "Using default datatype: float32"
+elif [ ${PRECISION} == "int8" ];then
+  ARGS="--benchmark"
+  echo "Using default datatype: int8"
+else
+  echo "Error: Only support float32/bfloat16/float16/tensorfloat32/int8"
+  exit 1    
+fi
+
+declare -a str
+device_id=$( lspci | grep -i display | sed -n '1p' | awk '{print $7}' )
+num_devs=$(lspci | grep -i display | awk '{print $7}' | wc -l)
+num_threads=1
+k=0
+
+if [ ${TEST_MODE} == "accuracy" ];then
+  python eval_image_classifier_inference.py --input-graph=${PB_FILE_PATH} --batch-size=${BATCH_SIZE} --data-num-inter-threads 1 --accuracy-only --data-location ${DATASET_DIR} --dtype ${PRECISION} \
+  |& tee resnet50_${TEST_MODE}_${PRECISION}.log
+elif [ ${TEST_MODE} == "inference" ];then
+    if [[ -z ${FLEX_GPU_TYPE} ]]; then
+      echo "FLEX_GPU_TYPE not set. Please set either flex_170 or flex_140"
+      exit 0
+    fi  
+  if [[ ${FLEX_GPU_TYPE} == flex_140 ]]; then 
+    if [[ ${device_id} == "56c1" ]]; then
+  	echo "Running benchmark"
+	if [[ ${BATCH_SIZE} == "1" ]]; then
+	  for i in $( eval echo {0..$((num_devs-1))} )
+    	do
+   	  for j in $( eval echo {1..$num_threads} )
+           do
+		   str+=("ZE_AFFINITY_MASK="${i}" numactl -C ${k} -l python eval_image_classifier_inference.py --input-graph=${PB_FILE_PATH} --batch-size=${BATCH_SIZE} --warmup-steps=10 --steps=5000 --dtype ${PRECISION} ${ARGS} ")
+		   ((k=k+1))
+	    done
+    done
+  else
+	    for i in $( eval echo {0..$((num_devs-1))} )
+	    do
+		    str+=("ZE_AFFINITY_MASK="${i}" python eval_image_classifier_inference.py --input-graph=${PB_FILE_PATH} --batch-size=${BATCH_SIZE} --warmup-steps=10 --steps=5000 --dtype ${PRECISION} ${ARGS} ")
+	    done
+	fi
+echo "resnet50 int8 inference on Flex series 140"
+parallel --lb -d, --tagstring "[{#}]" ::: \
+    "${str[@]}" 2>&1 | tee $OUTPUT_DIR/resnet50_${TEST_MODE}_${PRECISION}.log
+    fi
+elif [[ ${FLEX_GPU_TYPE} == flex_170 ]]; then
+    if [[ ${device_id} == "56c0" ]]; then
+	    echo "resnet50 int8 inference on Flex series 170"
+	    python eval_image_classifier_inference.py --input-graph=${PB_FILE_PATH} --batch-size=${BATCH_SIZE} --warmup-steps=10 --steps=500 --dtype ${PRECISION} ${ARGS} \
+      |& tee $OUTPUT_DIR/resnet50_${TEST_MODE}_${PRECISION}.log
+    fi
+fi
+fi
+if [ ${TEST_MODE} == "accuracy" ];then
+  value=$(cat resnet50_${TEST_MODE}_${PRECISION}.log | grep "(Top1 accuracy, Top5 accuracy) " | tail -n 1 | sed -e "s/.*(//" | sed -e "s/,.*//")
+  key="accuracy"
+  unit=""
+elif [ ${TEST_MODE} == "inference" ];then 
+    if [[ ${device_id} == "56c0" ]]; then
+      value=$(cat $OUTPUT_DIR/resnet50_${TEST_MODE}_${PRECISION}.log | grep "Throughput" | sed -e "s/.*://" | sed -e "s/ images\/sec//")
+    elif [[ ${device_id} == "56c1" ]]; then
+      value=$(cat $OUTPUT_DIR/resnet50_${TEST_MODE}_${PRECISION}.log | grep "Throughput" | sed -e "s/.*://" | sed -e "s/ images\/sec//" |  awk '{ sum_total += $1 } END { print sum_total }')
+    key="throughput"
+    unit="images/s"
+  fi
+fi
+
+yaml_content=$(cat <<EOF
+results:
+ - key: $key
+   value: $value
+   unit: $unit
+EOF
+)
+
+# Write the content to a YAML file
+
+echo "$yaml_content" >  ${OUTPUT_DIR}/results.yaml
+echo "YAML file created, path:$OUTPUT_DIR/results.yaml"
diff --git a/models_v2/tensorflow/resnet50v1_5/inference/gpu/setup.sh b/models_v2/tensorflow/resnet50v1_5/inference/gpu/setup.sh
new file mode 100755
index 000000000..e5d36e63c
--- /dev/null
+++ b/models_v2/tensorflow/resnet50v1_5/inference/gpu/setup.sh
@@ -0,0 +1,24 @@
+#!/usr/bin/env bash
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# setup.sh
+#  - install OS pkgs
+#  - should create virtual env & install pip  requirement.txt
+#  - git clones & applying patches
+
+set -e
+apt-get update && apt-get install -y python3-venv protobuf-compiler
diff --git a/models_v2/tensorflow/resnet50v1_5/inference/gpu/tests.yaml b/models_v2/tensorflow/resnet50v1_5/inference/gpu/tests.yaml
new file mode 100644
index 000000000..a7e3cbb8d
--- /dev/null
+++ b/models_v2/tensorflow/resnet50v1_5/inference/gpu/tests.yaml
@@ -0,0 +1,26 @@
+inference-int8:
+  cmd: bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/setup.sh ${FRAMEWORK_VERSION} ${FRAMEWORK_EXTENSION_VERSION} ${IS_LKG_DROP} ${AIKIT_VERSION}; bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/resnet50v1_5-inference/run_test.sh ${PRECISION} ${IS_LKG_DROP} ${DATASET_DIR}
+  env:
+    PRECISION: "int8"
+    DATASET_DIR: "/tf_dataset/dataset/TF_Imagenet_FullData/"
+inference-float32:
+  cmd: bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/setup.sh ${FRAMEWORK_VERSION} ${FRAMEWORK_EXTENSION_VERSION} ${IS_LKG_DROP} ${AIKIT_VERSION}; bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/resnet50v1_5-inference/run_test.sh ${PRECISION} ${IS_LKG_DROP} ${DATASET_DIR}
+  env:
+    PRECISION: "float32"
+    DATASET_DIR: "/tf_dataset/dataset/TF_Imagenet_FullData/"  
+inference-bfloat16:
+  cmd: bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/setup.sh ${FRAMEWORK_VERSION} ${FRAMEWORK_EXTENSION_VERSION} ${IS_LKG_DROP} ${AIKIT_VERSION}; bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/resnet50v1_5-inference/run_test.sh ${PRECISION} ${IS_LKG_DROP} ${DATASET_DIR}
+  env:
+    PRECISION: "bfloat16"
+    DATASET_DIR: "/tf_dataset/dataset/TF_Imagenet_FullData/"  
+inference-float16:
+  cmd: bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/setup.sh ${FRAMEWORK_VERSION} ${FRAMEWORK_EXTENSION_VERSION} ${IS_LKG_DROP} ${AIKIT_VERSION}; bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/resnet50v1_5-inference/run_test.sh ${PRECISION} ${IS_LKG_DROP} ${DATASET_DIR}
+  env:
+    PRECISION: "float16"
+    DATASET_DIR: "/tf_dataset/dataset/TF_Imagenet_FullData/"
+inference-tensorfloat32:
+  cmd: bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/setup.sh ${FRAMEWORK_VERSION} ${FRAMEWORK_EXTENSION_VERSION} ${IS_LKG_DROP} ${AIKIT_VERSION}; bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/resnet50v1_5-inference/run_test.sh ${PRECISION} ${IS_LKG_DROP} ${DATASET_DIR}
+  env:
+    PRECISION: "tensorfloat32"
+    DATASET_DIR: "/tf_dataset/dataset/TF_Imagenet_FullData/"
+
diff --git a/models_v2/tensorflow/resnet50v1_5/training/__init__.py b/models_v2/tensorflow/resnet50v1_5/training/__init__.py
new file mode 100644
index 000000000..ce0a27f82
--- /dev/null
+++ b/models_v2/tensorflow/resnet50v1_5/training/__init__.py
@@ -0,0 +1,20 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+#
+
diff --git a/models_v2/tensorflow/resnet50v1_5/training/gpu/CONTAINER.md b/models_v2/tensorflow/resnet50v1_5/training/gpu/CONTAINER.md
new file mode 100644
index 000000000..e945f7fca
--- /dev/null
+++ b/models_v2/tensorflow/resnet50v1_5/training/gpu/CONTAINER.md
@@ -0,0 +1,70 @@
+# TensorFlow ResNet50_v1.5 Training
+
+## Description
+
+This document has instructions for running ResNet50 v1.5 training with BFloat16 precision using Intel® Extension for TensorFlow on Intel® Data Center GPU Max Series.
+
+## Datasets
+
+Download and preprocess the ImageNet dataset using the [instructions here](https://github.com/IntelAI/models/blob/master/datasets/imagenet/README.md). After running the conversion script you should have a directory with the ImageNet dataset in the TF records format.
+
+Set the `DATASET_DIR` to point to the TF records directory when running ResNet50 v1.5.
+
+## Quick Start Scripts
+| Script name | Description |
+|-------------|-------------|
+| `run_model.sh` | Runs ResNet50 v1.5 BF16 training on single and two tiles |
+
+Requirements:
+* Host has [Intel® Data Center GPU Max Series](https://ark.intel.com/content/www/us/en/ark/products/series/232874/intel-data-center-gpu-max-series.html)
+* Follow instructions to install GPU-compatible [drivers](https://dgpu-docs.intel.com/driver/installation.html)
+* Docker
+
+### Docker pull command:
+```
+docker pull intel/image-recognition:tf-max-gpu-resnet50v1-5-training
+```
+The ResNet50 v1.5 training container includes scripts, models and libraries needed to run BFloat16 training. The script `run_model.sh` requires configuration files as input. For single-tile training, specify the configuration files present in `configure` folder as the `CONFIG_FILE` environment variable. For Multi-tile training, specify the configuration files in `hvd_configure` folder. You will also need to provide an output directory to store logs. To use Imagenet dataset, you will need to volume mount the dataset. For dummy data, use corresponding configuration files and volume mount of ImageNet dataset is not required and hence `DATASET_DIR` volume mounts are not required.
+
+The following example shows how to run BF16 training using real data. 
+
+```bash
+#Optional
+export DATASET_DIR=<path to pre-processed ImageNet datasets>
+
+#Required
+export CONFIG_FILE=/workspace/tf-max-series-resnet50v1-5-training/models/hvd_configure/itex_bf16_lars.yaml
+export MULTI_TILE=<specify True for Multi-tile training and False for single-tile training>
+export SCRIPT=run_model.sh
+export OUTPUT_DIR=<provide path to output logs directory>
+
+DOCKER_ARGS="--rm --init -it"
+IMAGE_NAME=intel/image-recognition:tf-max-gpu-resnet50v1-5-training
+
+docker run \
+  --device=/dev/dri \
+  --ipc=host \
+  --env DATASET_DIR=${DATASET_DIR} \
+  --env OUTPUT_DIR=${OUTPUT_DIR} \
+  --env MULTI_TILE=${MULTI_TILE} \
+  --env CONFIG_FILE=${CONFIG_FILE} \
+  --env http_proxy=${http_proxy} \
+  --env https_proxy=${https_proxy} \
+  --env no_proxy=${no_proxy} \
+  --volume ${DATASET_DIR}:${DATASET_DIR} \
+  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
+  --volume /dev/dri:/dev/dri \
+  ${DOCKER_ARGS} \
+  $IMAGE_NAME \
+  /bin/bash $SCRIPT
+  ```
+## Documentation and Sources
+
+[GitHub* Repository](https://github.com/IntelAI/models/tree/master/docker/max-gpu)
+
+## Support
+Support for Intel® Extension for TensorFlow* is available at [Intel® AI Analytics Toolkit](https://www.intel.com/content/www/us/en/developer/tools/oneapi/ai-analytics-toolkit.html#gs.qbretz) Additionally, the Intel® Extension for TensorFlow* team tracks both bugs and enhancement requests using [GitHub issues](https://github.com/intel/intel-extension-for-tensorflow/issues). Before submitting a suggestion or bug report, please search the GitHub issues to see if your issue has already been reported.
+
+## License Agreement
+
+LEGAL NOTICE: By accessing, downloading or using this software and any required dependent software (the “Software Package”), you agree to the terms and conditions of the software license agreements for the Software Package, which may also include notices, disclaimers, or license terms for third party software included with the Software Package. Please refer to the [license file](https://github.com/IntelAI/models/tree/master/third_party) for additional details.
diff --git a/models_v2/tensorflow/resnet50v1_5/training/gpu/README.md b/models_v2/tensorflow/resnet50v1_5/training/gpu/README.md
new file mode 100644
index 000000000..bb04158e3
--- /dev/null
+++ b/models_v2/tensorflow/resnet50v1_5/training/gpu/README.md
@@ -0,0 +1,82 @@
+# ResNet50 Model Training Convergence for Intel® Extention for TensorFlow
+
+## Model Information
+| **Case** |**Framework** | **Model Repo** | **Tag** 
+| :---: | :---: | :---: | :---: |
+| Training | Tensorflow | [Tensorflow-Models](https://github.com/tensorflow/models) | v2.8.0 |
+
+# Pre-Requisite
+* Host has Intel® Data Center GPU Max
+* Host has installed latest Intel® Data Center GPU Max Series Drivers https://dgpu-docs.intel.com/driver/installation.html
+* The following Intel® oneAPI Base Toolkit components are required:
+  - Intel® oneAPI DPC++ Compiler (Placeholder DPCPPROOT as its installation path)
+  - Intel® oneAPI Math Kernel Library (oneMKL) (Placeholder MKLROOT as its installation path)
+  - Intel® oneAPI MPI Library
+  - Intel® oneAPI TBB Library
+  - Intel® oneAPI CCL Library
+
+  Follow instructions at [Intel® oneAPI Base Toolkit Download page](https://www.intel.com/content/www/us/en/developer/tools/oneapi/base-toolkit-download.html?operatingsystem=linux) to setup the package manager repository.
+
+# Dataset 
+Using TensorFlow Datasets.
+`classifier_trainer.py`` supports ImageNet with [TensorFlow Datasets(TFDS)](https://www.tensorflow.org/datasets/overview) .
+
+Please see the following [example snippet](https://github.com/tensorflow/datasets/blob/master/tensorflow_datasets/scripts/download_and_prepare.py) for more information on how to use TFDS to download and prepare datasets, and specifically the [TFDS ImageNet readme](https://github.com/tensorflow/datasets/blob/master/docs/catalog/imagenet2012.md) for manual download instructions.
+
+Legacy TFRecords
+Download the ImageNet dataset and convert it to TFRecord format. The following [script](https://github.com/tensorflow/tpu/blob/master/tools/datasets/imagenet_to_gcs.py) and [README](https://github.com/tensorflow/tpu/tree/master/tools/datasets#imagenet_to_gcspy) provide a few options.
+
+> Note that the legacy ResNet runners, e.g. [resnet/resnet_ctl_imagenet_main.py](https://github.com/tensorflow/models/blob/v2.8.0/official/vision/image_classification/resnet/resnet_ctl_imagenet_main.py) require TFRecords whereas `classifier_trainer.py` can use both by setting the builder to 'records' or 'tfds' in the configurations.
+
+## Training
+1. `git clone https://github.com/IntelAI/models.git`
+2. `cd models/models_v2/tensorflow/resnet50v1_5/training/gpu`
+3. create virtual environment `venv` and activate it:
+    ```
+    python3 -m venv venv
+    . ./venv/bin/activate
+    ```
+4. Run setup.sh
+    ```
+    ./setup.sh
+    ```
+5. Install [tensorflow and ITEX](https://pypi.org/project/intel-extension-for-tensorflow/)
+6. If you run multi-tile (export MULTI_TILE=True), please install "intel-optimization-for-horovod" with `pip install intel-optimization-for-horovod`
+7. Set environment variables for Intel® oneAPI Base Toolkit: 
+    Default installation location `{ONEAPI_ROOT}` is `/opt/intel/oneapi` for root account, `${HOME}/intel/oneapi` for other accounts
+    ```bash
+    source {ONEAPI_ROOT}/compiler/latest/env/vars.sh
+    source {ONEAPI_ROOT}/mkl/latest/env/vars.sh
+    source {ONEAPI_ROOT}/tbb/latest/env/vars.sh
+    source {ONEAPI_ROOT}/mpi/latest/env/vars.sh
+    source {ONEAPI_ROOT}/ccl/latest/env/vars.sh
+8. Setup required environment paramaters
+  There are several config yaml files in configure and hvd_configure folder. Set one of them as CONFIG_FILE, then model would correspondly run with `real data` or `dummy data`. Single-tile please use yaml file in configure folder. Distribute training please use yaml file in hvd_configure folder, `itex_bf16_lars.yaml`/`itex_fp32_lars.yaml` for HVD real data and `itex_dummy_bf16_lars.yaml`/`itex_dummy_fp32_lars.yaml` for HVD dummy data.
+Export those parameters to script or environment.
+
+    |   **Parameter**    | **export command**                                    |
+    | :---: | :--- |
+    |  **DATASET_DIR**   | `export DATASET_DIR=/the/path/to/dataset`    (if you choose dummy data, you can igore this parameter)          |
+    |   **OUTPUT_DIR**   | `export OUTPUT_DIR=/the/path/to/output_dir`           |
+    |   **MULTI_TILE**   | `export MULTI_TILE=False (False or True)`           |
+    |   **CONFIG_FILE**   | `path/to/itex_xx.yaml  (dataset type and precision) ` |
+6. Run `run_model.sh`
+
+## Output
+
+Output will typically look like:
+```
+I1101 12:22:02.439692 139875177744192 keras_utils.py:145] TimeHistory: 57.00 seconds, xxx examples/second between steps 0 and 100
+I1101 12:22:51.165375 139875177744192 keras_utils.py:145] TimeHistory: 48.71 seconds, xxx examples/second between steps 100 and 200
+I1101 12:23:39.856714 139875177744192 keras_utils.py:145] TimeHistory: 48.69 seconds, xxx examples/second between steps 200 and 300
+I1101 12:24:28.548917 139875177744192 keras_utils.py:145] TimeHistory: 48.69 seconds, xxx examples/second between steps 300 and 400
+
+```
+
+Final results of the training run can be found in `results.yaml` file.
+```
+results:
+ - key: throughput
+   value: xxx
+   unit: images/sec
+```
diff --git a/models_v2/tensorflow/resnet50v1_5/training/gpu/configure/itex_bf16.yaml b/models_v2/tensorflow/resnet50v1_5/training/gpu/configure/itex_bf16.yaml
new file mode 100644
index 000000000..1fa6d26be
--- /dev/null
+++ b/models_v2/tensorflow/resnet50v1_5/training/gpu/configure/itex_bf16.yaml
@@ -0,0 +1,74 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+
+# Training configuration for ResNet trained on ImageNet on TPUs.
+# Takes ~4 minutes, 30 seconds seconds per epoch for a v3-32.
+# Reaches > 76.1% within 90 epochs.
+# Note: This configuration uses a scaled per-replica batch size based on the number of devices.
+runtime:
+  distribution_strategy: 'mirrored'
+  num_gpus: 1
+train_dataset:
+  name: 'imagenet2012'
+  data_dir: None
+  builder: 'records'
+  split: 'train'
+  one_hot: False
+  image_size: 224
+  num_classes: 1000
+  num_examples: 1281167
+  batch_size: 256
+  use_per_replica_batch_size: True
+  mean_subtract: False
+  standardize: False
+  dtype: 'bfloat16'
+validation_dataset:
+  name: 'imagenet2012'
+  data_dir: None
+  builder: 'records'
+  split: 'validation'
+  one_hot: False
+  image_size: 224
+  num_classes: 1000
+  num_examples: 50000
+  batch_size: 256
+  use_per_replica_batch_size: True
+  mean_subtract: False
+  standardize: False
+  dtype: 'bfloat16'
+model:
+  name: 'resnet'
+  model_params:
+    rescale_inputs: True
+  optimizer:
+    name: 'momentum'
+    momentum: 0.9
+    decay: 0.9
+    epsilon: 0.001
+    moving_average_decay: 0.
+    lookahead: False
+  loss:
+    label_smoothing: 0.1
+train:
+  callbacks:
+    enable_checkpoint_and_export: True
+  resume_checkpoint: True
+  epochs: 90
+evaluation:
+  epochs_between_evals: 1
diff --git a/models_v2/tensorflow/resnet50v1_5/training/gpu/configure/itex_dummy_bf16.yaml b/models_v2/tensorflow/resnet50v1_5/training/gpu/configure/itex_dummy_bf16.yaml
new file mode 100644
index 000000000..958c55d76
--- /dev/null
+++ b/models_v2/tensorflow/resnet50v1_5/training/gpu/configure/itex_dummy_bf16.yaml
@@ -0,0 +1,74 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+
+# Training configuration for ResNet trained on ImageNet on TPUs.
+# Takes ~4 minutes, 30 seconds seconds per epoch for a v3-32.
+# Reaches > 76.1% within 90 epochs.
+# Note: This configuration uses a scaled per-replica batch size based on the number of devices.
+runtime:
+  distribution_strategy: 'mirrored'
+  num_gpus: 1
+train_dataset:
+  name: 'imagenet2012'
+  data_dir: None
+  builder: 'synthetic'
+  split: 'train'
+  one_hot: False
+  image_size: 224
+  num_classes: 1000
+  num_examples: 1281167
+  batch_size: 256
+  use_per_replica_batch_size: True
+  mean_subtract: False
+  standardize: False
+  dtype: 'bfloat16'
+validation_dataset:
+  name: 'imagenet2012'
+  data_dir: None
+  builder: 'synthetic'
+  split: 'validation'
+  one_hot: False
+  image_size: 224
+  num_classes: 1000
+  num_examples: 50000
+  batch_size: 256
+  use_per_replica_batch_size: True
+  mean_subtract: False
+  standardize: False
+  dtype: 'bfloat16'
+model:
+  name: 'resnet'
+  model_params:
+    rescale_inputs: True
+  optimizer:
+    name: 'momentum'
+    momentum: 0.9
+    decay: 0.9
+    epsilon: 0.001
+    moving_average_decay: 0.
+    lookahead: False
+  loss:
+    label_smoothing: 0.1
+train:
+  callbacks:
+    enable_checkpoint_and_export: True
+  resume_checkpoint: True
+  epochs: 1
+evaluation:
+  epochs_between_evals: 1
diff --git a/models_v2/tensorflow/resnet50v1_5/training/gpu/configure/itex_dummy_fp32.yaml b/models_v2/tensorflow/resnet50v1_5/training/gpu/configure/itex_dummy_fp32.yaml
new file mode 100644
index 000000000..46f988c6c
--- /dev/null
+++ b/models_v2/tensorflow/resnet50v1_5/training/gpu/configure/itex_dummy_fp32.yaml
@@ -0,0 +1,74 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+
+# Training configuration for ResNet trained on ImageNet on TPUs.
+# Takes ~4 minutes, 30 seconds seconds per epoch for a v3-32.
+# Reaches > 76.1% within 90 epochs.
+# Note: This configuration uses a scaled per-replica batch size based on the number of devices.
+runtime:
+  distribution_strategy: 'mirrored'
+  num_gpus: 1
+train_dataset:
+  name: 'imagenet2012'
+  data_dir: None
+  builder: 'synthetic'
+  split: 'train'
+  one_hot: False
+  image_size: 224
+  num_classes: 1000
+  num_examples: 1281167
+  batch_size: 256
+  use_per_replica_batch_size: True
+  mean_subtract: False
+  standardize: False
+  dtype: 'float32'
+validation_dataset:
+  name: 'imagenet2012'
+  data_dir: None
+  builder: 'synthetic'
+  split: 'validation'
+  one_hot: False
+  image_size: 224
+  num_classes: 1000
+  num_examples: 50000
+  batch_size: 256
+  use_per_replica_batch_size: True
+  mean_subtract: False
+  standardize: False
+  dtype: 'float32'
+model:
+  name: 'resnet'
+  model_params:
+    rescale_inputs: True
+  optimizer:
+    name: 'momentum'
+    momentum: 0.9
+    decay: 0.9
+    epsilon: 0.001
+    moving_average_decay: 0.
+    lookahead: False
+  loss:
+    label_smoothing: 0.1
+train:
+  callbacks:
+    enable_checkpoint_and_export: True
+  resume_checkpoint: True
+  epochs: 1
+evaluation:
+  epochs_between_evals: 1
diff --git a/models_v2/tensorflow/resnet50v1_5/training/gpu/configure/itex_fp32.yaml b/models_v2/tensorflow/resnet50v1_5/training/gpu/configure/itex_fp32.yaml
new file mode 100644
index 000000000..df2b7cbb7
--- /dev/null
+++ b/models_v2/tensorflow/resnet50v1_5/training/gpu/configure/itex_fp32.yaml
@@ -0,0 +1,74 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+
+# Training configuration for ResNet trained on ImageNet on TPUs.
+# Takes ~4 minutes, 30 seconds seconds per epoch for a v3-32.
+# Reaches > 76.1% within 90 epochs.
+# Note: This configuration uses a scaled per-replica batch size based on the number of devices.
+runtime:
+  distribution_strategy: 'mirrored'
+  num_gpus: 1
+train_dataset:
+  name: 'imagenet2012'
+  data_dir: None
+  builder: 'records'
+  split: 'train'
+  one_hot: False
+  image_size: 224
+  num_classes: 1000
+  num_examples: 1281167
+  batch_size: 256
+  use_per_replica_batch_size: True
+  mean_subtract: False
+  standardize: False
+  dtype: 'float32'
+validation_dataset:
+  name: 'imagenet2012'
+  data_dir: None
+  builder: 'records'
+  split: 'validation'
+  one_hot: False
+  image_size: 224
+  num_classes: 1000
+  num_examples: 50000
+  batch_size: 256
+  use_per_replica_batch_size: True
+  mean_subtract: False
+  standardize: False
+  dtype: 'float32'
+model:
+  name: 'resnet'
+  model_params:
+    rescale_inputs: True
+  optimizer:
+    name: 'momentum'
+    momentum: 0.9
+    decay: 0.9
+    epsilon: 0.001
+    moving_average_decay: 0.
+    lookahead: False
+  loss:
+    label_smoothing: 0.1
+train:
+  callbacks:
+    enable_checkpoint_and_export: True
+  resume_checkpoint: True
+  epochs: 90
+evaluation:
+  epochs_between_evals: 1
diff --git a/models_v2/tensorflow/resnet50v1_5/training/gpu/hvd_configure/itex_bf16_lars.yaml b/models_v2/tensorflow/resnet50v1_5/training/gpu/hvd_configure/itex_bf16_lars.yaml
new file mode 100644
index 000000000..ba5bb8f20
--- /dev/null
+++ b/models_v2/tensorflow/resnet50v1_5/training/gpu/hvd_configure/itex_bf16_lars.yaml
@@ -0,0 +1,74 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+
+evaluation:
+  epochs_between_evals: 1
+model:
+  learning_rate:
+    initial_lr: 10.8
+    name: polynomial
+    warmup_epochs: 5
+  loss:
+    label_smoothing: 0.1
+  model_params:
+    rescale_inputs: true
+  name: resnet
+  optimizer:
+    epsilon: 0
+    lookahead: false
+    momentum: 0.9
+    name: lars
+    weight_decay: 1.25e-05
+model_dir: models
+runtime:
+  distribution_strategy: mirrored
+  num_gpus: 1
+train:
+  callbacks:
+    enable_checkpoint_and_export: true
+  epochs: 42
+  resume_checkpoint: true
+train_dataset:
+  batch_size: 512
+  builder: records
+  data_dir: None
+  dtype: bfloat16
+  image_size: 224
+  mean_subtract: false
+  name: imagenet2012
+  num_classes: 1000
+  num_examples: 1281167
+  one_hot: false
+  split: train
+  standardize: false
+  use_per_replica_batch_size: true
+validation_dataset:
+  batch_size: 512
+  builder: records
+  data_dir: None
+  dtype: bfloat16
+  image_size: 224
+  mean_subtract: false
+  name: imagenet2012
+  num_classes: 1000
+  num_examples: 50000
+  one_hot: false
+  split: validation
+  standardize: false
+  use_per_replica_batch_size: true
diff --git a/models_v2/tensorflow/resnet50v1_5/training/gpu/hvd_configure/itex_dummy_bf16_lars.yaml b/models_v2/tensorflow/resnet50v1_5/training/gpu/hvd_configure/itex_dummy_bf16_lars.yaml
new file mode 100644
index 000000000..eefd2651b
--- /dev/null
+++ b/models_v2/tensorflow/resnet50v1_5/training/gpu/hvd_configure/itex_dummy_bf16_lars.yaml
@@ -0,0 +1,73 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+evaluation:
+  epochs_between_evals: 1
+model:
+  learning_rate:
+    initial_lr: 10.8
+    name: polynomial
+    warmup_epochs: 5
+  loss:
+    label_smoothing: 0.1
+  model_params:
+    rescale_inputs: true
+  name: resnet
+  optimizer:
+    epsilon: 0
+    lookahead: false
+    momentum: 0.9
+    name: lars
+    weight_decay: 1.25e-05
+model_dir: models
+runtime:
+  distribution_strategy: mirrored
+  num_gpus: 1
+train:
+  callbacks:
+    enable_checkpoint_and_export: true
+  epochs: 1
+  resume_checkpoint: true
+train_dataset:
+  batch_size: 512
+  builder: synthetic
+  data_dir: None
+  dtype: bfloat16
+  image_size: 224
+  mean_subtract: false
+  name: imagenet2012
+  num_classes: 1000
+  num_examples: 1281167
+  one_hot: false
+  split: train
+  standardize: false
+  use_per_replica_batch_size: true
+validation_dataset:
+  batch_size: 512
+  builder: synthetic
+  data_dir: None
+  dtype: bfloat16
+  image_size: 224
+  mean_subtract: false
+  name: imagenet2012
+  num_classes: 1000
+  num_examples: 50000
+  one_hot: false
+  split: validation
+  standardize: false
+  use_per_replica_batch_size: true
diff --git a/models_v2/tensorflow/resnet50v1_5/training/gpu/hvd_configure/itex_dummy_fp32_lars.yaml b/models_v2/tensorflow/resnet50v1_5/training/gpu/hvd_configure/itex_dummy_fp32_lars.yaml
new file mode 100644
index 000000000..56dab5571
--- /dev/null
+++ b/models_v2/tensorflow/resnet50v1_5/training/gpu/hvd_configure/itex_dummy_fp32_lars.yaml
@@ -0,0 +1,73 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+evaluation:
+  epochs_between_evals: 2
+model:
+  learning_rate:
+    initial_lr: 9.0
+    name: polynomial
+    warmup_epochs: 3
+  loss:
+    label_smoothing: 0.1
+  model_params:
+    rescale_inputs: true
+  name: resnet
+  optimizer:
+    epsilon: 0
+    lookahead: false
+    momentum: 0.9
+    name: lars
+    weight_decay: 1.25e-05
+model_dir: models
+runtime:
+  distribution_strategy: mirrored
+  num_gpus: 1
+train:
+  callbacks:
+    enable_checkpoint_and_export: true
+  epochs: 1
+  resume_checkpoint: true
+train_dataset:
+  batch_size: 256
+  builder: synthetic
+  data_dir: None
+  dtype: float32
+  image_size: 224
+  mean_subtract: false
+  name: imagenet2012
+  num_classes: 1000
+  num_examples: 1281167
+  one_hot: false
+  split: train
+  standardize: false
+  use_per_replica_batch_size: true
+validation_dataset:
+  batch_size: 256
+  builder: synthetic
+  data_dir: None
+  dtype: float32
+  image_size: 224
+  mean_subtract: false
+  name: imagenet2012
+  num_classes: 1000
+  num_examples: 50000
+  one_hot: false
+  split: validation
+  standardize: false
+  use_per_replica_batch_size: true
diff --git a/models_v2/tensorflow/resnet50v1_5/training/gpu/hvd_configure/itex_fp32_lars.yaml b/models_v2/tensorflow/resnet50v1_5/training/gpu/hvd_configure/itex_fp32_lars.yaml
new file mode 100644
index 000000000..dc51a3870
--- /dev/null
+++ b/models_v2/tensorflow/resnet50v1_5/training/gpu/hvd_configure/itex_fp32_lars.yaml
@@ -0,0 +1,74 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+
+evaluation:
+  epochs_between_evals: 2
+model:
+  learning_rate:
+    initial_lr: 9.0
+    name: polynomial
+    warmup_epochs: 3
+  loss:
+    label_smoothing: 0.1
+  model_params:
+    rescale_inputs: true
+  name: resnet
+  optimizer:
+    epsilon: 0
+    lookahead: false
+    momentum: 0.9
+    name: lars
+    weight_decay: 1.25e-05
+model_dir: models
+runtime:
+  distribution_strategy: mirrored
+  num_gpus: 1
+train:
+  callbacks:
+    enable_checkpoint_and_export: true
+  epochs: 39
+  resume_checkpoint: true
+train_dataset:
+  batch_size: 256
+  builder: records
+  data_dir: None
+  dtype: float32
+  image_size: 224
+  mean_subtract: false
+  name: imagenet2012
+  num_classes: 1000
+  num_examples: 1281167
+  one_hot: false
+  split: train
+  standardize: false
+  use_per_replica_batch_size: true
+validation_dataset:
+  batch_size: 512
+  builder: records
+  data_dir: None
+  dtype: float32
+  image_size: 224
+  mean_subtract: false
+  name: imagenet2012
+  num_classes: 1000
+  num_examples: 50000
+  one_hot: false
+  split: validation
+  standardize: false
+  use_per_replica_batch_size: true
diff --git a/models_v2/tensorflow/resnet50v1_5/training/gpu/hvd_support.patch b/models_v2/tensorflow/resnet50v1_5/training/gpu/hvd_support.patch
new file mode 100644
index 000000000..860d2adfc
--- /dev/null
+++ b/models_v2/tensorflow/resnet50v1_5/training/gpu/hvd_support.patch
@@ -0,0 +1,865 @@
+diff --git a/official/legacy/image_classification/callbacks.py b/official/legacy/image_classification/callbacks.py
+index a4934ed88..a7eaad9fb 100644
+--- a/official/legacy/image_classification/callbacks.py
++++ b/official/legacy/image_classification/callbacks.py
+@@ -149,8 +149,8 @@ class CustomTensorBoard(tf.keras.callbacks.TensorBoard):
+   def _calculate_metrics(self) -> MutableMapping[str, Any]:
+     logs = {}
+     # TODO(b/149030439): disable LR reporting.
+-    # if self._track_lr:
+-    #   logs['learning_rate'] = self._calculate_lr()
++    if self._track_lr:
++      logs['learning_rate'] = self._calculate_lr()
+     return logs
+ 
+   def _calculate_lr(self) -> int:
+diff --git a/official/vision/image_classification/augment.py b/official/vision/image_classification/augment.py
+index f322d31da..e668ff731 100644
+--- a/official/vision/image_classification/augment.py
++++ b/official/vision/image_classification/augment.py
+@@ -25,7 +25,10 @@ from __future__ import print_function
+ import math
+ from typing import Any, Dict, List, Optional, Text, Tuple
+ 
+-from keras.layers.preprocessing import image_preprocessing as image_ops
++try:
++  from keras.src.layers.preprocessing import image_preprocessing as image_ops
++except ImportError:
++  from keras.layers.preprocessing import image_preprocessing as image_ops
+ import tensorflow as tf
+ 
+ 
+diff --git a/official/vision/image_classification/callbacks.py b/official/vision/image_classification/callbacks.py
+index a4934ed88..7eaaf0bea 100644
+--- a/official/vision/image_classification/callbacks.py
++++ b/official/vision/image_classification/callbacks.py
+@@ -78,6 +78,7 @@ def get_callbacks(
+             save_weights_only=True,
+             verbose=1))
+     callbacks.append(MovingAverageCallback())
++  #callbacks.append(ThresholdStopping(monitor = "val_accuracy", threshold = 0.759))
+   return callbacks
+ 
+ 
+@@ -254,3 +255,38 @@ class AverageModelCheckpoint(tf.keras.callbacks.ModelCheckpoint):
+       result = super()._save_model(epoch, logs)  # pytype: disable=attribute-error  # typed-keras
+       self.model.set_weights(non_avg_weights)
+       return result
++
++class ThresholdStopping(tf.keras.callbacks.Callback):
++  def __init__(self,
++               monitor="val_accuracy",
++               threshold=1
++              ):
++    super().__init__()
++
++    self.monitor = monitor
++    self.threshold = threshold
++    self.stopped_epoch = 0
++
++  def on_train_begin(self, logs=None):
++    self.stopped_epoch = 0
++
++  def on_epoch_end(self, epoch, logs=None):
++    current = self.get_monitor_value(logs)
++    if current is None:
++      return
++    if current >= self.threshold:
++      self.stopped_epoch = epoch
++      self.model.stop_training = True
++
++  def get_monitor_value(self, logs):
++    logs = logs or {}
++    monitor_value = logs.get(self.monitor)
++    if monitor_value is None:
++      logging.warning(
++        "Early stopping conditioned on metric `%s` "
++        "which is not available. Available metrics are: %s",
++        self.monitor,
++        ",".join(list(logs.keys())),
++        )
++    return monitor_value
++
+diff --git a/official/vision/image_classification/classifier_trainer.py b/official/vision/image_classification/classifier_trainer.py
+index ab6fbaea9..4655da449 100644
+--- a/official/vision/image_classification/classifier_trainer.py
++++ b/official/vision/image_classification/classifier_trainer.py
+@@ -36,7 +36,16 @@ from official.vision.image_classification.configs import configs
+ from official.vision.image_classification.efficientnet import efficientnet_model
+ from official.vision.image_classification.resnet import common
+ from official.vision.image_classification.resnet import resnet_model
++import time
+ 
++global is_mpi
++try:
++    import horovod.tensorflow.keras as hvd
++    hvd.init()
++    is_mpi = hvd.size()
++except ImportError:
++    is_mpi = 0
++    print("No MPI horovod support, this is running in no-MPI mode!")
+ 
+ def get_models() -> Mapping[str, tf.keras.Model]:
+   """Returns the mapping from model type name to Keras model."""
+@@ -271,7 +280,7 @@ def define_classifier_flags():
+       help='The name of the dataset, e.g. ImageNet, etc.')
+   flags.DEFINE_integer(
+       'log_steps',
+-      default=100,
++      default=200,
+       help='The interval of steps between logging of batch level stats.')
+ 
+ 
+@@ -289,6 +298,12 @@ def train_and_eval(
+   """Runs the train and eval path using compile/fit."""
+   logging.info('Running train and eval.')
+ 
++  if is_mpi:
++    gpus = tf.config.experimental.list_physical_devices('XPU')
++    for gpu in gpus:
++      tf.config.experimental.set_memory_growth(gpu, True)
++    tf.config.experimental.set_visible_devices(gpus[hvd.local_rank()], 'XPU')
++
+   distribute_utils.configure_cluster(params.runtime.worker_hosts,
+                                      params.runtime.task_index)
+ 
+@@ -299,7 +314,7 @@ def train_and_eval(
+       num_gpus=params.runtime.num_gpus,
+       tpu_address=params.runtime.tpu)
+ 
+-  strategy_scope = distribute_utils.get_strategy_scope(strategy)
++  #strategy_scope = distribute_utils.get_strategy_scope(strategy)
+ 
+   logging.info('Detected %d devices.',
+                strategy.num_replicas_in_sync if strategy else 1)
+@@ -324,56 +339,74 @@ def train_and_eval(
+ 
+   logging.info('Global batch size: %d', train_builder.global_batch_size)
+ 
+-  with strategy_scope:
+-    model_params = params.model.model_params.as_dict()
+-    model = get_models()[params.model.name](**model_params)
+-    learning_rate = optimizer_factory.build_learning_rate(
+-        params=params.model.learning_rate,
+-        batch_size=train_builder.global_batch_size,
+-        train_epochs=train_epochs,
+-        train_steps=train_steps)
+-    optimizer = optimizer_factory.build_optimizer(
+-        optimizer_name=params.model.optimizer.name,
+-        base_learning_rate=learning_rate,
+-        params=params.model.optimizer.as_dict(),
+-        model=model)
+-    optimizer = performance.configure_optimizer(
+-        optimizer,
+-        use_float16=train_builder.dtype == 'float16',
+-        loss_scale=get_loss_scale(params))
+-
+-    metrics_map = _get_metrics(one_hot)
+-    metrics = [metrics_map[metric] for metric in params.train.metrics]
+-    steps_per_loop = train_steps if params.train.set_epoch_loop else 1
+-
+-    if one_hot:
+-      loss_obj = tf.keras.losses.CategoricalCrossentropy(
+-          label_smoothing=params.model.loss.label_smoothing)
+-    else:
+-      loss_obj = tf.keras.losses.SparseCategoricalCrossentropy()
+-    model.compile(
+-        optimizer=optimizer,
+-        loss=loss_obj,
+-        metrics=metrics,
+-        steps_per_execution=steps_per_loop)
+-
+-    initial_epoch = 0
+-    if params.train.resume_checkpoint:
+-      initial_epoch = resume_from_checkpoint(
+-          model=model, model_dir=params.model_dir, train_steps=train_steps)
++  model_params = params.model.model_params.as_dict()
++  model = get_models()[params.model.name](**model_params)
++  learning_rate = optimizer_factory.build_learning_rate(
++    params=params.model.learning_rate,
++    batch_size=train_builder.global_batch_size * hvd.size(),
++    train_epochs=train_epochs,
++    train_steps=train_steps)
++  optimizer = optimizer_factory.build_optimizer(
++    optimizer_name=params.model.optimizer.name,
++    base_learning_rate=learning_rate,
++    params=params.model.optimizer.as_dict(),
++    model=model)
++  optimizer = performance.configure_optimizer(
++    optimizer,
++    use_float16=train_builder.dtype == 'float16',
++    loss_scale=get_loss_scale(params))
++
++  metrics_map = _get_metrics(one_hot)
++  metrics = [metrics_map[metric] for metric in params.train.metrics]
++  steps_per_loop = train_steps if params.train.set_epoch_loop else 1
+ 
+-    callbacks = custom_callbacks.get_callbacks(
+-        model_checkpoint=params.train.callbacks.enable_checkpoint_and_export,
+-        include_tensorboard=params.train.callbacks.enable_tensorboard,
+-        time_history=params.train.callbacks.enable_time_history,
+-        track_lr=params.train.tensorboard.track_lr,
+-        write_model_weights=params.train.tensorboard.write_model_weights,
+-        initial_step=initial_epoch * train_steps,
+-        batch_size=train_builder.global_batch_size,
+-        log_steps=params.train.time_history.log_steps,
+-        model_dir=params.model_dir,
+-        backup_and_restore=params.train.callbacks.enable_backup_and_restore)
++  if one_hot:
++    loss_obj = tf.keras.losses.CategoricalCrossentropy(
++      label_smoothing=params.model.loss.label_smoothing)
++  else:
++    loss_obj = tf.keras.losses.SparseCategoricalCrossentropy()
++
++  hvd_optimizer = hvd.DistributedOptimizer(optimizer, num_groups=1)
++  model.compile(
++    optimizer=hvd_optimizer,
++    loss=loss_obj,
++    metrics=metrics,
++    steps_per_execution=steps_per_loop)
+ 
++  initial_epoch = 0
++  if params.train.resume_checkpoint:
++    initial_epoch = resume_from_checkpoint(
++      model=model, model_dir=params.model_dir, train_steps=train_steps)
++
++  # Add broadcast callback for rank0
++  callbacks = []
++
++  if hvd.local_rank() == 0:
++    callbacks = custom_callbacks.get_callbacks(
++      model_checkpoint=params.train.callbacks.enable_checkpoint_and_export,
++      include_tensorboard=params.train.callbacks.enable_tensorboard,
++      time_history=params.train.callbacks.enable_time_history,
++      track_lr=params.train.tensorboard.track_lr,
++      write_model_weights=params.train.tensorboard.write_model_weights,
++      initial_step=initial_epoch * train_steps,
++      batch_size=train_builder.global_batch_size,
++      log_steps=params.train.time_history.log_steps,
++      model_dir=params.model_dir,
++      backup_and_restore=params.train.callbacks.enable_backup_and_restore)
++  else:
++    callbacks = custom_callbacks.get_callbacks(
++      model_checkpoint=False,
++      include_tensorboard=params.train.callbacks.enable_tensorboard,
++      time_history=params.train.callbacks.enable_time_history,
++      track_lr=params.train.tensorboard.track_lr,
++      write_model_weights=params.train.tensorboard.write_model_weights,
++      initial_step=initial_epoch * train_steps,
++      batch_size=train_builder.global_batch_size,
++      log_steps=params.train.time_history.log_steps,
++      model_dir=params.model_dir,
++      backup_and_restore=params.train.callbacks.enable_backup_and_restore)
++
++  callbacks.append(hvd.callbacks.BroadcastGlobalVariablesCallback(0))
+   serialize_config(params=params, model_dir=params.model_dir)
+ 
+   if params.evaluation.skip_eval:
+@@ -384,7 +417,9 @@ def train_and_eval(
+         'validation_steps': validation_steps,
+         'validation_freq': params.evaluation.epochs_between_evals,
+     }
+-
++  print('[info] Training steps = ', train_steps)
++  print('[info] Validation steps = ', validation_steps)
++  global_start_time = time.time()
+   history = model.fit(
+       train_dataset,
+       epochs=train_epochs,
+@@ -394,6 +429,11 @@ def train_and_eval(
+       verbose=2,
+       **validation_kwargs)
+ 
++  global_end_time = time.time()
++  print('[info] Global start time = ', time.asctime(time.localtime(global_start_time)))
++  print('[info] Global end time = ', time.asctime(time.localtime(global_end_time)))
++  print('[info] Global consume time = ', ((global_end_time - global_start_time) / (60.0)), ' mins')
++
+   validation_output = None
+   if not params.evaluation.skip_eval:
+     validation_output = model.evaluate(
+diff --git a/official/vision/image_classification/configs/base_configs.py b/official/vision/image_classification/configs/base_configs.py
+index 760b3dce0..3939249b6 100644
+--- a/official/vision/image_classification/configs/base_configs.py
++++ b/official/vision/image_classification/configs/base_configs.py
+@@ -180,6 +180,7 @@ class OptimizerConfig(hyperparams.Config):
+   beta_1: float = None
+   beta_2: float = None
+   epsilon: float = None
++  weight_decay: float = None
+ 
+ 
+ @dataclasses.dataclass
+diff --git a/official/vision/image_classification/dataset_factory.py b/official/vision/image_classification/dataset_factory.py
+index a0458eccc..275c6d5fb 100644
+--- a/official/vision/image_classification/dataset_factory.py
++++ b/official/vision/image_classification/dataset_factory.py
+@@ -29,6 +29,7 @@ import tensorflow_datasets as tfds
+ from official.modeling.hyperparams import base_config
+ from official.vision.image_classification import augment
+ from official.vision.image_classification import preprocessing
++import horovod.tensorflow as hvd
+ 
+ AUGMENTERS = {
+     'autoaugment': augment.AutoAugment,
+@@ -207,7 +208,16 @@ class DatasetBuilder:
+   def num_steps(self) -> int:
+     """The number of steps (batches) to exhaust this dataset."""
+     # Always divide by the global batch size to get the correct # of steps
+-    return self.num_examples // self.global_batch_size
++    distributed_size = 1
++    if self.config.split == 'train':
++      distributed_size = hvd.size()
++    divide_steps = self.num_examples // (self.global_batch_size * distributed_size)
++    remain_steps = self.num_examples % (self.global_batch_size * distributed_size)
++    if remain_steps == 0:
++      return divide_steps
++    else:
++      return divide_steps + 1
++    #return self.num_examples // (self.global_batch_size * hvd.size())
+ 
+   @property
+   def dtype(self) -> tf.dtypes.DType:
+@@ -403,14 +413,10 @@ class DatasetBuilder:
+     Returns:
+       A TensorFlow dataset outputting batched images and labels.
+     """
+-    if (self.config.builder != 'tfds' and self.input_context and
+-        self.input_context.num_input_pipelines > 1):
+-      dataset = dataset.shard(self.input_context.num_input_pipelines,
+-                              self.input_context.input_pipeline_id)
++    if self.is_training:
++      dataset = dataset.shard(hvd.size(), hvd.rank())
+       logging.info(
+-          'Sharding the dataset: input_pipeline_id=%d '
+-          'num_input_pipelines=%d', self.input_context.num_input_pipelines,
+-          self.input_context.input_pipeline_id)
++          'Sharding the dataset: total size: %d ', hvd.size(), " local rank: %d ", hvd.rank())
+ 
+     if self.is_training and self.config.builder == 'records':
+       # Shuffle the input files.
+@@ -455,10 +461,10 @@ class DatasetBuilder:
+       # replicas automatically when strategy.distribute_datasets_from_function
+       # is called, so we use local batch size here.
+       dataset = dataset.batch(
+-          self.local_batch_size, drop_remainder=self.is_training)
++          self.local_batch_size, drop_remainder=False)
+     else:
+       dataset = dataset.batch(
+-          self.global_batch_size, drop_remainder=self.is_training)
++          self.global_batch_size, drop_remainder=False)
+ 
+     # Prefetch overlaps in-feed with training
+     dataset = dataset.prefetch(tf.data.experimental.AUTOTUNE)
+diff --git a/official/vision/image_classification/lars_optimizer.py b/official/vision/image_classification/lars_optimizer.py
+new file mode 100644
+index 000000000..029ae654e
+--- /dev/null
++++ b/official/vision/image_classification/lars_optimizer.py
+@@ -0,0 +1,248 @@
++# Copyright 2018 The TensorFlow Authors. All Rights Reserved.
++#
++# Licensed under the Apache License, Version 2.0 (the "License");
++# you may not use this file except in compliance with the License.
++# You may obtain a copy of the License at
++#
++#     http://www.apache.org/licenses/LICENSE-2.0
++#
++# Unless required by applicable law or agreed to in writing, software
++# distributed under the License is distributed on an "AS IS" BASIS,
++# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
++# See the License for the specific language governing permissions and
++# limitations under the License.
++# ==============================================================================
++"""Layer-wise Adaptive Rate Scaling optimizer for large-batch training."""
++
++from __future__ import absolute_import
++from __future__ import division
++from __future__ import print_function
++
++import tensorflow as tf
++# from tf2_common.training import optimizer_v2modified
++from tensorflow.python.framework import ops
++
++from tensorflow.python.ops import array_ops
++from tensorflow.python.ops import linalg_ops
++from tensorflow.python.ops import math_ops
++from tensorflow.python.training import training_ops
++from tensorflow.python.ops import state_ops
++
++# from tensorflow.python.keras import backend_config
++# from tensorflow.python.keras.optimizer_v2 import optimizer_v2
++tf_minor = int(tf.__version__.split('.')[1])
++if tf_minor >=13:
++  from keras.src.optimizers.legacy import optimizer_v2
++  from keras.src import backend_config
++elif tf_minor >= 12:
++  from keras.optimizers.legacy import optimizer_v2
++  from keras import backend_config
++elif tf_minor >= 9:
++  from keras.optimizers.optimizer_v2 import optimizer_v2
++  from keras import backend_config
++elif tf_minor >= 6:
++  from keras.optimizer_v2 import optimizer_v2
++  from keras import backend_config
++else:
++  from tensorflow.python.keras.optimizer_v2 import optimizer_v2
++  from tensorflow.python.keras import backend_config
++
++# class LARSOptimizer(optimizer_v2modified.OptimizerV2Modified):
++class LARSOptimizer(optimizer_v2.OptimizerV2):
++  """Layer-wise Adaptive Rate Scaling for large batch training.
++
++  Introduced by "Large Batch Training of Convolutional Networks" by Y. You,
++  I. Gitman, and B. Ginsburg. (https://arxiv.org/abs/1708.03888)
++
++  Implements the LARS learning rate scheme presented in the paper above. This
++  optimizer is useful when scaling the batch size to up to 32K without
++  significant performance degradation. It is recommended to use the optimizer
++  in conjunction with:
++      - Gradual learning rate warm-up
++      - Linear learning rate scaling
++      - Poly rule learning rate decay
++
++  Note, LARS scaling is currently only enabled for dense tensors. Sparse tensors
++  use the default momentum optimizer.
++  """
++
++  def __init__(
++      self,
++      learning_rate,
++      momentum=0.9,
++      weight_decay=0.0001,
++      # The LARS coefficient is a hyperparameter
++      eeta=0.001,
++      epsilon=0.0,
++      name="LARSOptimizer",
++      # Enable skipping variables from LARS scaling.
++      # TODO(sameerkm): Enable a direct mechanism to pass a
++      # subset of variables to the optimizer.
++      skip_list=None,
++      use_nesterov=False,
++      **kwargs):
++    """Construct a new LARS Optimizer.
++
++    Args:
++      learning_rate: A `Tensor`, floating point value, or a schedule that is a
++        `tf.keras.optimizers.schedules.LearningRateSchedule`, or a callable
++        that takes no arguments and returns the actual value to use. The
++        learning rate.
++      momentum: A floating point value. Momentum hyperparameter.
++      weight_decay: A floating point value. Weight decay hyperparameter.
++      eeta: LARS coefficient as used in the paper. Dfault set to LARS
++        coefficient from the paper. (eeta / weight_decay) determines the highest
++        scaling factor in LARS.
++      epsilon: Optional epsilon parameter to be set in models that have very
++        small gradients. Default set to 0.0.
++      name: Optional name prefix for variables and ops created by LARSOptimizer.
++      skip_list: List of strings to enable skipping variables from LARS scaling.
++        If any of the strings in skip_list is a subset of var.name, variable
++        'var' is skipped from LARS scaling. For a typical classification model
++        with batch normalization, the skip_list is ['batch_normalization',
++        'bias']
++      use_nesterov: when set to True, nesterov momentum will be enabled
++      **kwargs: keyword arguments.
++
++    Raises:
++      ValueError: If a hyperparameter is set to a non-sensical value.
++    """
++    if momentum < 0.0:
++      raise ValueError("momentum should be positive: %s" % momentum)
++    if weight_decay < 0.0:
++      raise ValueError("weight_decay should be positive: %s" % weight_decay)
++    super(LARSOptimizer, self).__init__(name=name, **kwargs)
++
++    self._set_hyper("learning_rate", learning_rate)
++
++    # When directly using class members, instead of
++    # _set_hyper and _get_hyper (such as learning_rate above),
++    # the values are fixed after __init(), and not being
++    # updated during the training process.
++    # This provides better performance but less flexibility.
++    self.momentum = momentum
++    self.weight_decay = weight_decay
++    self.eeta = eeta
++    self.epsilon = epsilon or backend_config.epsilon()
++    self._skip_list = skip_list
++    self.use_nesterov = use_nesterov
++
++  def _prepare_local(self, var_device, var_dtype, apply_state):
++    lr_t = self._get_hyper("learning_rate", var_dtype)
++    local_step = math_ops.cast(self.iterations, var_dtype)
++    lr_t = math_ops.cast(lr_t(local_step), var_dtype)
++    learning_rate_t = array_ops.identity(lr_t)
++
++    apply_state[(var_device, var_dtype)].update(
++        dict(
++            learning_rate=learning_rate_t,
++            ))
++
++  def _create_slots(self, var_list):
++    for v in var_list:
++      self.add_slot(v, "momentum")
++
++  def compute_lr(self, grad, var, coefficients):
++    scaled_lr = coefficients["learning_rate"]
++    if self._skip_list is None or not any(v in var.name
++                                          for v in self._skip_list):
++      w_norm = linalg_ops.norm(var, ord=2)
++      g_norm = linalg_ops.norm(grad, ord=2)
++      trust_ratio = array_ops.where(
++          math_ops.greater(w_norm, 0),
++          array_ops.where(
++              math_ops.greater(g_norm, 0),
++              (self.eeta * w_norm /
++               (g_norm + self.weight_decay * w_norm + self.epsilon)), 1.0), 1.0)
++
++      scaled_lr = coefficients["learning_rate"] * trust_ratio
++      # Add the weight regularization gradient
++      grad = grad + self.weight_decay * var
++    return scaled_lr, grad
++
++  def _apply_dense(self, grad, var, apply_state=None):
++    var_device, var_dtype = var.device, var.dtype.base_dtype
++    coefficients = ((apply_state or {}).get((var_device, var_dtype))
++                    or self._fallback_apply_state(var_device, var_dtype))
++
++    scaled_lr, grad = self.compute_lr(grad, var, coefficients)
++    mom = self.get_slot(var, "momentum")
++    return training_ops.apply_momentum(
++        var,
++        mom,
++        math_ops.cast(1.0, var.dtype.base_dtype),
++        grad * scaled_lr,
++        self.momentum,
++        use_locking=False,
++        use_nesterov=self.use_nesterov)
++
++  def _resource_apply_dense(self, grad, var, apply_state=None):
++    var_device, var_dtype = var.device, var.dtype.base_dtype
++    coefficients = ((apply_state or {}).get((var_device, var_dtype))
++                    or self._fallback_apply_state(var_device, var_dtype))
++
++    scaled_lr, grad = self.compute_lr(grad, var, coefficients)
++    mom = self.get_slot(var, "momentum")
++    # Use ApplyKerasMomentum instead of ApplyMomentum
++    # training_ops.resource_apply_keras_momentum(
++    #     var.handle,
++    #     mom.handle,
++    #     scaled_lr,
++    #     grad,
++    #     coefficients["momentum"],
++    #     use_locking=False,
++    #     use_nesterov=self.use_nesterov)
++
++    mom_t = mom * self.momentum - grad * scaled_lr
++    mom_t = state_ops.assign(mom, mom_t, use_locking=False)
++    if self.use_nesterov:
++      var_t = var + mom_t * self.momentum - grad * scaled_lr
++    else:
++      var_t = var + mom_t
++    return state_ops.assign(var, var_t, use_locking=False).op
++
++  # Fallback to momentum optimizer for sparse tensors
++  def _apply_sparse(self, grad, var, apply_state=None):
++    var_device, var_dtype = var.device, var.dtype.base_dtype
++    coefficients = ((apply_state or {}).get((var_device, var_dtype))
++                    or self._fallback_apply_state(var_device, var_dtype))
++
++    mom = self.get_slot(var, "momentum")
++    return training_ops.sparse_apply_momentum(
++        var,
++        mom,
++        coefficients["learning_rate"],
++        grad.values,
++        grad.indices,
++        self.momentum,
++        use_locking=False,
++        use_nesterov=self.use_nesterov)
++
++  def _resource_apply_sparse(self, grad, var, indices, apply_state=None):
++    var_device, var_dtype = var.device, var.dtype.base_dtype
++    coefficients = ((apply_state or {}).get((var_device, var_dtype))
++                    or self._fallback_apply_state(var_device, var_dtype))
++
++    mom = self.get_slot(var, "momentum")
++    return training_ops.resource_sparse_apply_keras_momentum(
++        var.handle,
++        mom.handle,
++        coefficients["learning_rate"],
++        grad,
++        indices,
++        self.momentum,
++        use_locking=False,
++        use_nesterov=self.use_nesterov)
++
++  def get_config(self):
++    config = super(LARSOptimizer, self).get_config()
++    config.update({
++        "learning_rate": self._serialize_hyperparameter("learning_rate"),
++        "momentum": self.momentum,
++        "weight_decay": self.weight_decay,
++        "eeta": self.eeta,
++        "epsilon": self.epsilon,
++        "use_nesterov": self.use_nesterov,
++        "skip_list": self._skip_list,
++    })
++    return config
+diff --git a/official/vision/image_classification/learning_rate.py b/official/vision/image_classification/learning_rate.py
+index 72f7e9518..19207291d 100644
+--- a/official/vision/image_classification/learning_rate.py
++++ b/official/vision/image_classification/learning_rate.py
+@@ -22,10 +22,14 @@ from typing import Any, Mapping, Optional
+ 
+ import numpy as np
+ import tensorflow as tf
++from tensorflow.python.util.tf_export import keras_export
++from tensorflow.python.framework import ops
++from tensorflow.python.ops import math_ops
+ 
+ BASE_LEARNING_RATE = 0.1
+ 
+ 
++@tf.keras.utils.register_keras_serializable(package='Custom', name='WarmupDeacySchedule')
+ class WarmupDecaySchedule(tf.keras.optimizers.schedules.LearningRateSchedule):
+   """A wrapper for LearningRateSchedule that includes warmup steps."""
+ 
+@@ -66,10 +70,11 @@ class WarmupDecaySchedule(tf.keras.optimizers.schedules.LearningRateSchedule):
+     return lr
+ 
+   def get_config(self) -> Mapping[str, Any]:
+-    config = self._lr_schedule.get_config()
++    config = {}
+     config.update({
+         "warmup_steps": self._warmup_steps,
+         "warmup_lr": self._warmup_lr,
++        "lr_schedule": self._lr_schedule,
+     })
+     return config
+ 
+@@ -115,3 +120,104 @@ class CosineDecayWithWarmup(tf.keras.optimizers.schedules.LearningRateSchedule):
+         "warmup_steps": self._warmup_steps,
+         "init_learning_rate": self._init_learning_rate,
+     }
++
++@tf.keras.utils.register_keras_serializable(package='Custom', name='PolynomialDeacyWithWarmup')
++class PolynomialDecayWithWarmup(
++tf.keras.optimizers.schedules.LearningRateSchedule):
++  """A LearningRateSchedule that uses a polynomial decay with warmup."""
++  def __init__(
++        self,
++        batch_size,
++        steps_per_epoch,
++        train_steps,
++        initial_learning_rate=None,
++        end_learning_rate=None,
++        warmup_epochs=None,
++        compute_lr_on_cpu=False,
++        name=None):
++    """Applies a polynomial decay to the learning rate with warmup."""
++    super(PolynomialDecayWithWarmup, self).__init__()
++
++    self.batch_size = batch_size
++    self.steps_per_epoch = steps_per_epoch
++    self.train_steps = train_steps
++    self.name = name
++    self.learning_rate_ops_cache = {}
++    self.compute_lr_on_cpu = compute_lr_on_cpu
++
++    if batch_size < 16384:
++        self.initial_learning_rate = 10.0
++        warmup_epochs_ = 5
++    elif batch_size < 32768:
++        self.initial_learning_rate = 25.0
++        warmup_epochs_ = 5
++    else:
++        self.initial_learning_rate = 31.2
++        warmup_epochs_ = 25
++
++    # Override default poly learning rate and warmup epochs
++    if initial_learning_rate:
++        self.initial_learning_rate = initial_learning_rate
++
++    if end_learning_rate:
++        self.end_learning_rate = end_learning_rate
++    else:
++        self.end_learning_rate = 0.0001
++
++    if warmup_epochs is not None:
++        warmup_epochs_ = warmup_epochs
++    self.warmup_epochs = warmup_epochs_
++
++    warmup_steps = warmup_epochs_ * steps_per_epoch
++    self.warmup_steps = tf.cast(warmup_steps, tf.float32)
++    self.decay_steps = train_steps - warmup_steps + 1
++    self.poly_rate_scheduler = tf.keras.optimizers.schedules.PolynomialDecay(
++        initial_learning_rate=self.initial_learning_rate,
++        decay_steps=self.decay_steps,
++        end_learning_rate=self.end_learning_rate,
++        power=2.0)
++
++  def __call__(self, step):
++    if tf.executing_eagerly():
++      return self._get_learning_rate(step)
++
++    # In an eager function or graph, the current implementation of optimizer
++    # repeatedly call and thus create ops for the learning rate schedule. To
++    # avoid this, we cache the ops if not executing eagerly.
++    graph = tf.compat.v1.get_default_graph()
++    if graph not in self.learning_rate_ops_cache:
++      if self.compute_lr_on_cpu:
++        with tf.device('/device:CPU:0'):
++          self.learning_rate_ops_cache[graph] = self._get_learning_rate(step)
++      else:
++        self.learning_rate_ops_cache[graph] = self._get_learning_rate(step)
++    return self.learning_rate_ops_cache[graph]
++
++  def _get_learning_rate(self, step):
++    with ops.name_scope_v2(self.name or 'PolynomialDecayWithWarmup') as name:
++      initial_learning_rate = ops.convert_to_tensor(
++        self.initial_learning_rate, name='initial_learning_rate')
++      warmup_steps = ops.convert_to_tensor(
++        self.warmup_steps, name='warmup_steps')
++      step = tf.cast(step, tf.float32)
++      warmup_rate = (
++            initial_learning_rate * step / warmup_steps)
++
++      poly_steps = math_ops.subtract(step, warmup_steps)
++      poly_rate = self.poly_rate_scheduler(poly_steps)
++
++      decay_rate = tf.where(step <= warmup_steps,
++                          warmup_rate, poly_rate, name=name)
++      return decay_rate
++
++  def get_config(self):
++    return {
++      'batch_size': self.batch_size,
++      'steps_per_epoch': self.steps_per_epoch,
++      'train_steps': self.train_steps,
++      'initial_learning_rate': self.initial_learning_rate,
++      'end_learning_rate': self.end_learning_rate,
++      'warmup_epochs': self.warmup_epochs,
++      'name': self.name,
++    }
++
+diff --git a/official/vision/image_classification/optimizer_factory.py b/official/vision/image_classification/optimizer_factory.py
+index 48a4512ee..0ed4cb914 100644
+--- a/official/vision/image_classification/optimizer_factory.py
++++ b/official/vision/image_classification/optimizer_factory.py
+@@ -26,6 +26,7 @@ import tensorflow_addons as tfa
+ from official.modeling import optimization
+ from official.vision.image_classification import learning_rate
+ from official.vision.image_classification.configs import base_configs
++import official.vision.image_classification.lars_optimizer as lars_optimizer
+ 
+ # pylint: disable=protected-access
+ 
+@@ -61,12 +62,12 @@ def build_optimizer(
+   if optimizer_name == 'sgd':
+     logging.info('Using SGD optimizer')
+     nesterov = params.get('nesterov', False)
+-    optimizer = tf.keras.optimizers.SGD(
++    optimizer = tf.keras.optimizers.legacy.SGD(
+         learning_rate=base_learning_rate, nesterov=nesterov)
+   elif optimizer_name == 'momentum':
+     logging.info('Using momentum optimizer')
+     nesterov = params.get('nesterov', False)
+-    optimizer = tf.keras.optimizers.SGD(
++    optimizer = tf.keras.optimizers.legacy.SGD(
+         learning_rate=base_learning_rate,
+         momentum=params['momentum'],
+         nesterov=nesterov)
+@@ -75,7 +76,7 @@ def build_optimizer(
+     rho = params.get('decay', None) or params.get('rho', 0.9)
+     momentum = params.get('momentum', 0.9)
+     epsilon = params.get('epsilon', 1e-07)
+-    optimizer = tf.keras.optimizers.RMSprop(
++    optimizer = tf.keras.optimizers.legacy.RMSprop(
+         learning_rate=base_learning_rate,
+         rho=rho,
+         momentum=momentum,
+@@ -85,7 +86,7 @@ def build_optimizer(
+     beta_1 = params.get('beta_1', 0.9)
+     beta_2 = params.get('beta_2', 0.999)
+     epsilon = params.get('epsilon', 1e-07)
+-    optimizer = tf.keras.optimizers.Adam(
++    optimizer = tf.keras.optimizers.legacy.Adam(
+         learning_rate=base_learning_rate,
+         beta_1=beta_1,
+         beta_2=beta_2,
+@@ -102,6 +103,16 @@ def build_optimizer(
+         beta_1=beta_1,
+         beta_2=beta_2,
+         epsilon=epsilon)
++  elif optimizer_name == 'lars':
++    logging.info('Using Lars')
++    weight_decay = params.get('weight_decay', 0.0002)
++    epsilon = params.get('epsilon', 0)
++    optimizer = lars_optimizer.LARSOptimizer(
++          learning_rate=base_learning_rate,
++          momentum=params.get('momentum', 0.9),
++          weight_decay=weight_decay,
++          skip_list=['batch_normalization', 'bias', 'bn'],
++          epsilon=epsilon)
+   else:
+     raise ValueError('Unknown optimizer %s' % optimizer_name)
+ 
+@@ -139,7 +150,7 @@ def build_learning_rate(params: base_configs.LearningRateConfig,
+   else:
+     warmup_steps = 0
+ 
+-  lr_multiplier = params.scale_by_batch_size
++  lr_multiplier = 0 #params.scale_by_batch_size
+ 
+   if lr_multiplier and lr_multiplier > 0:
+     # Scale the learning rate based on the batch size and a multiplier
+@@ -172,6 +183,14 @@ def build_learning_rate(params: base_configs.LearningRateConfig,
+         batch_size=batch_size,
+         total_steps=train_epochs * train_steps,
+         warmup_steps=warmup_steps)
++  elif decay_type == 'polynomial':
++    lr = learning_rate.PolynomialDecayWithWarmup(
++        batch_size=batch_size,
++        steps_per_epoch=train_steps,
++        train_steps=train_epochs * train_steps,
++        initial_learning_rate=base_lr,
++        warmup_epochs=params.warmup_epochs)
++    return lr
+   if warmup_steps > 0:
+     if decay_type not in ['cosine_with_warmup']:
+       logging.info('Applying %d warmup steps to the learning rate',
+diff --git a/official/vision/image_classification/resnet/imagenet_preprocessing.py b/official/vision/image_classification/resnet/imagenet_preprocessing.py
+index 86ba3ed98..298a4a412 100644
+--- a/official/vision/image_classification/resnet/imagenet_preprocessing.py
++++ b/official/vision/image_classification/resnet/imagenet_preprocessing.py
+@@ -113,7 +113,7 @@ def process_record_dataset(dataset,
+   dataset = dataset.map(
+       lambda value: parse_record_fn(value, is_training, dtype),
+       num_parallel_calls=tf.data.experimental.AUTOTUNE)
+-  dataset = dataset.batch(batch_size, drop_remainder=drop_remainder)
++  dataset = dataset.batch(batch_size, drop_remainder=False)
+ 
+   # Operations between the final prefetch and the get_next call to the iterator
+   # will happen synchronously during run time. We prefetch here again to
+@@ -350,7 +350,7 @@ def input_fn(is_training,
+       parse_record_fn=parse_record_fn,
+       dtype=dtype,
+       datasets_num_private_threads=datasets_num_private_threads,
+-      drop_remainder=drop_remainder,
++      drop_remainder=False,
+       tf_data_experimental_slack=tf_data_experimental_slack,
+   )
+ 
+diff --git a/official/vision/image_classification/resnet/resnet_runnable.py b/official/vision/image_classification/resnet/resnet_runnable.py
+index fe3059f77..c521e5992 100644
+--- a/official/vision/image_classification/resnet/resnet_runnable.py
++++ b/official/vision/image_classification/resnet/resnet_runnable.py
+@@ -100,7 +100,7 @@ class ResnetRunnable(orbit.StandardTrainer, orbit.StandardEvaluator):
+         datasets_num_private_threads=self.flags_obj
+         .datasets_num_private_threads,
+         dtype=self.dtype,
+-        drop_remainder=True)
++        drop_remainder=False)
+     orbit.StandardTrainer.__init__(
+         self,
+         train_dataset,
diff --git a/models_v2/tensorflow/resnet50v1_5/training/gpu/requirements.txt b/models_v2/tensorflow/resnet50v1_5/training/gpu/requirements.txt
new file mode 100644
index 000000000..534afed48
--- /dev/null
+++ b/models_v2/tensorflow/resnet50v1_5/training/gpu/requirements.txt
@@ -0,0 +1,7 @@
+gin 
+gin-config 
+tfa-nightly 
+tensorflow-model-optimization 
+tensorflow-datasets
+protobuf==3.20.3
+pyyaml
\ No newline at end of file
diff --git a/models_v2/tensorflow/resnet50v1_5/training/gpu/resnet50.patch b/models_v2/tensorflow/resnet50v1_5/training/gpu/resnet50.patch
new file mode 100644
index 000000000..d5f946aad
--- /dev/null
+++ b/models_v2/tensorflow/resnet50v1_5/training/gpu/resnet50.patch
@@ -0,0 +1,43 @@
+diff --git a/official/vision/image_classification/augment.py b/official/vision/image_classification/augment.py
+index f322d31da..294fcd543 100644
+--- a/official/vision/image_classification/augment.py
++++ b/official/vision/image_classification/augment.py
+@@ -25,7 +25,10 @@ from __future__ import print_function
+ import math
+ from typing import Any, Dict, List, Optional, Text, Tuple
+ 
+-from keras.layers.preprocessing import image_preprocessing as image_ops
++try:
++    from keras.src.layers.preprocessing import image_preprocessing as image_ops
++except ImportError:
++    from keras.layers.preprocessing import image_preprocessing as image_ops
+ import tensorflow as tf
+ 
+ 
+diff --git a/official/vision/image_classification/classifier_trainer.py b/official/vision/image_classification/classifier_trainer.py
+index ab6fbaea9..0347bc8c9 100644
+--- a/official/vision/image_classification/classifier_trainer.py
++++ b/official/vision/image_classification/classifier_trainer.py
+@@ -283,6 +283,14 @@ def serialize_config(params: base_configs.ExperimentConfig, model_dir: str):
+   hyperparams.save_params_dict_to_yaml(params, params_save_path)
+ 
+ 
++class dummy_context:
++  def __init__(self):
++    pass
++  def __enter__(self):
++    pass
++  def __exit__(self, exc_type, exc_value, traceback):
++    pass
++
+ def train_and_eval(
+     params: base_configs.ExperimentConfig,
+     strategy_override: tf.distribute.Strategy) -> Mapping[str, Any]:
+@@ -323,6 +331,7 @@ def train_and_eval(
+   initialize(params, train_builder)
+ 
+   logging.info('Global batch size: %d', train_builder.global_batch_size)
++  strategy_scope = dummy_context()
+ 
+   with strategy_scope:
+     model_params = params.model.model_params.as_dict()
diff --git a/models_v2/tensorflow/resnet50v1_5/training/gpu/run_model.sh b/models_v2/tensorflow/resnet50v1_5/training/gpu/run_model.sh
new file mode 100755
index 000000000..2563046c3
--- /dev/null
+++ b/models_v2/tensorflow/resnet50v1_5/training/gpu/run_model.sh
@@ -0,0 +1,113 @@
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+
+#!/bin/bash
+
+# Create an array of input directories that are expected and then verify that they exist
+declare -A input_envs
+input_envs[CONFIG_FILE]=${CONFIG_FILE}
+input_envs[OUTPUT_DIR]=${OUTPUT_DIR}
+#input_envs[DATASET_DIR]=${DATASET_DIR}
+input_envs[MULTI_TILE]=${MULTI_TILE}
+
+for i in "${!input_envs[@]}"; do
+  var_name=$i
+  env_param=${input_envs[$i]}
+ 
+  if [[ -z $env_param ]]; then
+    echo "The required environment variable $var_name is not set" >&2
+    exit 1
+  fi
+done
+
+#BATCH_SIZE=${BATCH_SIZE:-1}
+OUTPUT_DIR=${OUTPUT_DIR:-$PWD}
+script_path="$(realpath "$0")"
+script_directory=$(dirname "$script_path")
+
+echo 'Running with parameters:'
+echo " DATASET_PATH: ${DATASET_DIR}"
+echo " OUTPUT_DIR: ${OUTPUT_DIR}"
+echo " CONFIG_FILE: ${CONFIG_FILE}"
+echo " MULTI_TILE: $MULTI_TILE"
+
+if [ ! -d "$OUTPUT_DIR" ]; then
+    mkdir -p $OUTPUT_DIR
+else
+    rm -rf $OUTPUT_DIR && mkdir -p $OUTPUT_DIR                         
+fi
+
+if [ $MULTI_TILE == "True" ];then
+  current_dir=$(pwd)
+  if [ -d "tensorflow-models" ]; then
+    echo "Repository already exists. Skipping clone."
+  else
+    mkdir $current_dir/resnet50_hvd/ && cd $current_dir/resnet50_hvd/
+    git clone -b v2.8.0 https://github.com/tensorflow/models.git tensorflow-models
+    cd tensorflow-models
+    git apply $current_dir/hvd_support.patch
+  fi
+  export PYTHONPATH=$script_directory/resnet50_hvd/tensorflow-models
+  mpirun -np 2 -prepend-rank -ppn 2 \
+  python ${PYTHONPATH}/official/vision/image_classification/classifier_trainer.py \
+  --mode=train_and_eval \
+  --model_type=resnet \
+  --dataset=imagenet \
+  --model_dir=$OUTPUT_DIR \
+  --data_dir=$DATASET_DIR \
+  --config_file=$CONFIG_FILE |& tee Resnet50_training_${MULTI_TILE}.log
+  value0=$(cat ./Resnet50_training_${MULTI_TILE}.log | grep examples/second | grep '\[0\]' | tail -1 | awk -F 'examples/second' '{print $1}' | awk -F ',' '{print $2}')
+  value1=$(cat ./Resnet50_training_${MULTI_TILE}.log | grep examples/second | grep '\[1\]' | tail -1 | awk -F 'examples/second' '{print $1}' | awk -F ',' '{print $2}')
+  value=$(echo "$value1 + $value0" )
+else
+  current_dir=$(pwd)
+  if [ -d "tensorflow-models" ]; then
+    echo "Repository already exists. Skipping clone."
+  else
+    mkdir $current_dir/resnet50/ && cd $current_dir/resnet50/
+    git clone -b v2.8.0 https://github.com/tensorflow/models.git tensorflow-models
+    cd tensorflow-models
+    git apply $current_dir/resnet50.patch
+    cd $current_dir
+  fi
+  export PYTHONPATH=$script_directory/resnet50/tensorflow-models
+  python ${PYTHONPATH}/official/vision/image_classification/classifier_trainer.py \
+  --mode=train_and_eval \
+  --model_type=resnet \
+  --dataset=imagenet \
+  --model_dir=$OUTPUT_DIR \
+  --data_dir=$DATASET_DIR \
+  --config_file=$CONFIG_FILE |& tee Resnet50_training_${MULTI_TILE}.log
+  value=$(cat ./Resnet50_training_${MULTI_TILE}.log | grep 'examples/second' | tail -1 | awk -F 'examples/second' '{print $1}' | awk -F ',' '{print $2}')
+fi
+
+key="throughput"
+unit="images/sec"
+
+yaml_content=$(cat <<EOF
+results:
+ - key: $key
+   value: $value
+   unit: $unit
+EOF
+)
+
+# Write the content to a YAML file
+
+echo "$yaml_content" > $OUTPUT_DIR/results.yaml
+echo "YAML file created, path: $OUTPUT_DIR/results.yaml"
diff --git a/models_v2/tensorflow/resnet50v1_5/training/gpu/setup.sh b/models_v2/tensorflow/resnet50v1_5/training/gpu/setup.sh
new file mode 100755
index 000000000..43d400e3a
--- /dev/null
+++ b/models_v2/tensorflow/resnet50v1_5/training/gpu/setup.sh
@@ -0,0 +1,25 @@
+#!/usr/bin/env bash
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# setup.sh
+#  - install OS pkgs
+#  - should create virtual env & install pip  requirement.txt
+#  - git clones & applying patches
+
+set -e
+apt-get update && apt-get install -y python3-venv
+pip install -r requirements.txt
diff --git a/models_v2/tensorflow/resnet50v1_5/training/gpu/tests.yaml b/models_v2/tensorflow/resnet50v1_5/training/gpu/tests.yaml
new file mode 100644
index 000000000..ceecd0be8
--- /dev/null
+++ b/models_v2/tensorflow/resnet50v1_5/training/gpu/tests.yaml
@@ -0,0 +1,24 @@
+inference-bf16:
+  cmd: bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/setup.sh ${FRAMEWORK_VERSION} ${FRAMEWORK_EXTENSION_VERSION} ${IS_LKG_DROP} ${AIKIT_VERSION}; bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/resnet50v1_5-training/run_test.sh ${CONFIG_FILE} ${IS_LKG_DROP} ${DATASET_DIR} ${MULTI_TILE}
+  env:
+    MULTI_TILE: "False"
+    CONFIG_FILE: "configure/itex_bf16.yaml"
+    DATASET_DIR: "/tf_dataset/dataset/TF_Imagenet_FullData/"
+inference-fp32:
+  cmd: bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/setup.sh ${FRAMEWORK_VERSION} ${FRAMEWORK_EXTENSION_VERSION} ${IS_LKG_DROP} ${AIKIT_VERSION}; bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/resnet50v1_5-training/run_test.sh ${CONFIG_FILE} ${IS_LKG_DROP} ${DATASET_DIR} ${MULTI_TILE}
+  env:
+    MULTI_TILE: "False"
+    CONFIG_FILE: "configure/itex_fp32.yaml"
+    DATASET_DIR: "/tf_dataset/dataset/TF_Imagenet_FullData/" 
+inference-bf16:
+  cmd: bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/setup.sh ${FRAMEWORK_VERSION} ${FRAMEWORK_EXTENSION_VERSION} ${IS_LKG_DROP} ${AIKIT_VERSION}; bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/resnet50v1_5-training/run_test.sh ${CONFIG_FILE} ${IS_LKG_DROP} ${DATASET_DIR} ${MULTI_TILE}
+  env:
+    MULTI_TILE: "True"
+    CONFIG_FILE: "hvd_configure/itex_bf16_lars.yaml"
+    DATASET_DIR: "/tf_dataset/dataset/TF_Imagenet_FullData/"
+inference-fp32:
+  cmd: bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/setup.sh ${FRAMEWORK_VERSION} ${FRAMEWORK_EXTENSION_VERSION} ${IS_LKG_DROP} ${AIKIT_VERSION}; bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/resnet50v1_5-training/run_test.sh ${CONFIG_FILE} ${IS_LKG_DROP} ${DATASET_DIR} ${MULTI_TILE}
+  env:
+    MULTI_TILE: "True"
+    CONFIG_FILE: "hvd_configure/itex_fp32_lars.yaml"
+    DATASET_DIR: "/tf_dataset/dataset/TF_Imagenet_FullData/" 
diff --git a/models_v2/tensorflow/stable_diffusion/inference/gpu/CONTAINER.md b/models_v2/tensorflow/stable_diffusion/inference/gpu/CONTAINER.md
new file mode 100644
index 000000000..a62442c98
--- /dev/null
+++ b/models_v2/tensorflow/stable_diffusion/inference/gpu/CONTAINER.md
@@ -0,0 +1,65 @@
+# Running Stable Diffusion Inference with FP32 and FP16 on Intel® Data Center GPU Flex Series using Intel® Extension for TensorFlow*
+
+## Overview
+
+This document has instructions for running Stable Diffusion inference using Intel® Extension for TensorFlow* with Intel® Data Center GPU Flex Series.
+
+## Requirements
+| Item | Detail |
+| ------ | ------- |
+| Host machine  | [Intel® Data Center GPU Flex Series](https://ark.intel.com/content/www/us/en/ark/products/series/230021/intel-data-center-gpu-flex-series.html) 170 |
+| Drivers | GPU-compatible drivers need to be installed: [Download Driver](https://dgpu-docs.intel.com/driver/installation.html)
+| Software | Docker* |
+
+## Get Started
+
+### Quick Start Scripts
+
+| Script name | Description |
+|:-------------:|:-------------:|
+| `run_model.sh` | Runs online inference for FP32,FP16 precisions on Flex series 170 |
+
+## Run Using Docker
+
+### Set up Docker Image
+
+```bash
+docker pull intel/generative-ai:tf-flex-gpu-stable-diffusion-inference
+```
+
+### Run Docker Image
+
+The Stable Diffusion inference container contains scripts, models and libraries needed to run FP32 and FP16 inference. You will need to provide an output directory where log files will be written.
+
+```bash
+export PRECISION=<provide fp32 or fp16 as precision input>
+export OUTPUT_DIR=<path to output directory>
+
+IMAGE_NAME=intel/generative-ai:tf-flex-gpu-stable-diffusion-inference
+SCRIPT=run_model.sh
+
+docker run \
+  --device=/dev/dri \
+  --ipc=host \
+  --privileged \
+  --env PRECISION=${PRECISION} \
+  --env OUTPUT_DIR=${OUTPUT_DIR} \
+  --env http_proxy=${http_proxy} \
+  --env https_proxy=${https_proxy} \
+  --env no_proxy=${no_proxy} \
+  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
+  --rm -it \
+  $IMAGE_NAME \
+  /bin/bash $SCRIPT
+```
+
+## Documentation and Sources
+
+[GitHub* Repository](https://github.com/IntelAI/models/tree/master/docker/flex-gpu)
+
+## Support
+Support for Intel® Extension for TensorFlow* is found via the [Intel® AI Analytics Toolkit.](https://www.intel.com/content/www/us/en/developer/tools/oneapi/ai-analytics-toolkit.html#gs.qbretz) Additionally, the Intel® Extension for TensorFlow* team tracks both bugs and enhancement requests using [GitHub issues](https://github.com/intel/intel-extension-for-tensorflow/issues). Before submitting a suggestion or bug report, please search the GitHub issues to see if your issue has already been reported.
+
+## License Agreement
+
+LEGAL NOTICE: By accessing, downloading or using this software and any required dependent software (the “Software Package”), you agree to the terms and conditions of the software license agreements for the Software Package, which may also include notices, disclaimers, or license terms for third party software included with the Software Package. Please refer to the [license file](https://github.com/IntelAI/models/tree/master/third_party) for additional details.
diff --git a/models_v2/tensorflow/stable_diffusion/inference/gpu/README.md b/models_v2/tensorflow/stable_diffusion/inference/gpu/README.md
new file mode 100644
index 000000000..ecb56810d
--- /dev/null
+++ b/models_v2/tensorflow/stable_diffusion/inference/gpu/README.md
@@ -0,0 +1,54 @@
+# Stable Diffusion Inference
+
+Stable Diffusion Inference using Intel® Extension for TensorFlow.
+
+## Model Information
+
+| **Use Case** | **Framework** | **Model Repo** |          **Branch/Commit/Tag**           |  **Optional Patch** |
+|:---:| :---: | :---: |:----------------------------------------:| :---: |
+|  Inference   |  Tensorflow   | [keras-cv](https://github.com/keras-team/keras-cv.git) | 66fa74b6a2a0bb1e563ae8bce66496b118b95200 |  [patch](#patch) |
+
+**Note**: Refer to [CONTAINER.md](CONTAINER.md) for Stable Diffusion Inference instructions using docker containers.
+
+# Pre-Requisite
+* Host has Intel® Data Center GPU Flex Series
+* Host has installed latest Intel® Data Center GPU Flex Series Driver https://dgpu-docs.intel.com/driver/installation.html
+* Install [Intel® Extension for TensorFlow](https://pypi.org/project/intel-extension-for-tensorflow/)
+
+
+## Inference
+1. `git clone https://github.com/IntelAI/models.git`
+2. `cd models/models_v2/tensorflow/stable_diffusion/inference/gpu`
+3. Create virtual environment `venv` and activate it:
+    ```
+    python3 -m venv venv
+    . ./venv/bin/activate
+    ```
+4. Run setup.sh
+    ```
+    ./setup.sh
+    ```
+5. Install [tensorflow and ITEX](https://pypi.org/project/intel-extension-for-tensorflow/)
+6. Setup required environment paramaters
+
+    |   **Parameter**    |                   **export command**                    |
+    |:-------------------------------------------------------:| :---: |
+    |   **PRECISION**   |     `export PRECISION=fp16` (fp32 or fp16)      |
+7. Run `run_model.sh`
+
+## Output
+
+Output typically looks like:
+```
+50/50 [==============================] - 8s 150ms/step
+latency 153.37058544158936 ms, throughput 6.520155068331838 it/s
+Start plotting the generated images to ./images/fp16_imgs_50steps.png
+```
+
+Final results of the training run can be found in `results.yaml` file.
+```
+results:
+ - key: throughput
+   value: 6.520155068331838
+   unit: it/s
+```
diff --git a/models_v2/tensorflow/stable_diffusion/inference/gpu/keras_fid.py b/models_v2/tensorflow/stable_diffusion/inference/gpu/keras_fid.py
new file mode 100644
index 000000000..22e2891e8
--- /dev/null
+++ b/models_v2/tensorflow/stable_diffusion/inference/gpu/keras_fid.py
@@ -0,0 +1,74 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+#
+
+# example of calculating the frechet inception distance in Keras
+import numpy as np
+from numpy import cov
+from numpy import trace
+from numpy import iscomplexobj
+from numpy import asarray
+from numpy.random import randint
+from scipy.linalg import sqrtm
+from keras.applications.inception_v3 import InceptionV3
+from keras.applications.inception_v3 import preprocess_input
+from skimage.transform import resize
+
+
+# scale an array of images to a new size
+def scale_images(images, new_shape):
+    images_list = list()
+    for image in images:
+        # resize with nearest neighbor interpolation
+        new_image = resize(image, new_shape, 0)
+        # store
+        images_list.append(new_image)
+    return asarray(images_list)
+
+
+def calculate_fid(model, images1, images2):
+    act1 = model.predict(images1)
+    act2 = model.predict(images2)
+    mu1, sigma1 = act1.mean(axis=0), cov(act1, rowvar=False)
+    mu2, sigma2 = act2.mean(axis=0), cov(act2, rowvar=False)
+    ssdiff = np.sum((mu1 - mu2) ** 2.0)
+    covmean = sqrtm(sigma1.dot(sigma2))
+    if iscomplexobj(covmean):
+        covmean = covmean.real
+    fid = ssdiff + trace(sigma1 + sigma2 - 2.0 * covmean)
+    return fid
+
+
+def fid(images1, images2):
+    if images1.shape[0] == 1:
+        images1 = np.repeat(images1, 2, axis=0)
+    if images2.shape[0] == 1:
+        images2 = np.repeat(images2, 2, axis=0)
+    assert (images1.shape == images2.shape)
+    # convert integer to floating point values
+    images1 = images1.astype("float32")
+    images2 = images2.astype("float32")
+    H, W, C = images1.shape[1], images1.shape[2], images1.shape[3]
+    # pre-process images
+    images1 = preprocess_input(images1)
+    images2 = preprocess_input(images2)
+    # fid between images1 and images1
+    model = InceptionV3(include_top=False, pooling="avg", input_shape=(H, W, C))
+    fid = calculate_fid(model, images1, images2)
+    return fid
diff --git a/models_v2/tensorflow/stable_diffusion/inference/gpu/patch b/models_v2/tensorflow/stable_diffusion/inference/gpu/patch
new file mode 100644
index 000000000..e805f26a1
--- /dev/null
+++ b/models_v2/tensorflow/stable_diffusion/inference/gpu/patch
@@ -0,0 +1,309 @@
+diff --git a/keras_cv/models/stable_diffusion/__internal__/layers/attention_block.py b/keras_cv/models/stable_diffusion/__internal__/layers/attention_block.py
+index e7e1896..21fba0c 100644
+--- a/keras_cv/models/stable_diffusion/__internal__/layers/attention_block.py
++++ b/keras_cv/models/stable_diffusion/__internal__/layers/attention_block.py
+@@ -14,6 +14,12 @@
+
+ import tensorflow as tf
+ from tensorflow import keras
++try:
++    import intel_extension_for_tensorflow as itex
++    keras.layers.GroupNormalization = itex.ops.GroupNormalization
++except:
++    pass
++
+
+ from keras_cv.models.stable_diffusion.__internal__.layers.padded_conv2d import (
+     PaddedConv2D,
+diff --git a/keras_cv/models/stable_diffusion/__internal__/layers/resnet_block.py b/keras_cv/models/stable_diffusion/__internal__/layers/resnet_block.py
+index 29aeaaa..11d4be6 100644
+--- a/keras_cv/models/stable_diffusion/__internal__/layers/resnet_block.py
++++ b/keras_cv/models/stable_diffusion/__internal__/layers/resnet_block.py
+@@ -13,6 +13,13 @@
+ # limitations under the License.
+
+ from tensorflow import keras
++try:
++    import intel_extension_for_tensorflow as itex
++    keras.layers.GroupNormalization = itex.ops.GroupNormalization
++except:
++    pass
++
++
+
+ from keras_cv.models.stable_diffusion.__internal__.layers.padded_conv2d import (
+     PaddedConv2D,
+diff --git a/keras_cv/models/stable_diffusion/decoder.py b/keras_cv/models/stable_diffusion/decoder.py
+index fe619d3..ccc4b5b 100644
+--- a/keras_cv/models/stable_diffusion/decoder.py
++++ b/keras_cv/models/stable_diffusion/decoder.py
+@@ -13,6 +13,12 @@
+ # limitations under the License.
+
+ from tensorflow import keras
++try:
++    import intel_extension_for_tensorflow as itex
++    keras.layers.GroupNormalization = itex.ops.GroupNormalization
++except:
++    pass
++
+
+ from keras_cv.models.stable_diffusion.__internal__.layers.attention_block import (  # noqa: E501
+     AttentionBlock,
+diff --git a/keras_cv/models/stable_diffusion/diffusion_model.py b/keras_cv/models/stable_diffusion/diffusion_model.py
+index 25b5241..dafdcbc 100644
+--- a/keras_cv/models/stable_diffusion/diffusion_model.py
++++ b/keras_cv/models/stable_diffusion/diffusion_model.py
+@@ -14,7 +14,14 @@
+
+ import tensorflow as tf
+ from tensorflow import keras
+-
++try:
++    import intel_extension_for_tensorflow as itex
++    keras.layers.GroupNormalization = itex.ops.GroupNormalization
++    keras.layers.LayerNormalization = itex.ops.LayerNormalization
++except:
++    pass
++
++
+ from keras_cv.models.stable_diffusion.__internal__.layers.padded_conv2d import (
+     PaddedConv2D,
+ )
+@@ -302,6 +309,26 @@ class CrossAttention(keras.layers.Layer):
+         self.num_heads = num_heads
+         self.head_size = head_size
+         self.out_proj = keras.layers.Dense(num_heads * head_size)
++
++    def naive_scaled_dot_product_attention(self, query, key, value):
++        i_dtype = query.dtype
++        atten_scores = tf.matmul(query, key, transpose_b=True)
++        atten_scores = tf.multiply(atten_scores, tf.cast(self.scale, i_dtype))
++        atten_probs = tf.nn.softmax(atten_scores)
++        # `atten_output` = [B, N, F, H]
++        atten_output = tf.matmul(atten_probs, value)
++        # `atten_output` = [B, F, N, H]
++        atten_output  = tf.transpose(a=atten_output, perm=[0, 2, 1, 3])
++        return atten_output
++
++
++    def sdp(self, q, k, v):
++        try:
++            from intel_extension_for_tensorflow.python.ops.multi_head_attention import scaled_dot_product_attention
++            output = scaled_dot_product_attention(q, k, v, use_fast_attention=True)
++        except ImportError:
++            output = self.naive_scaled_dot_product_attention(q, k, v)
++        return output
+
+     def call(self, inputs):
+         inputs, context = inputs
+@@ -316,17 +343,10 @@ class CrossAttention(keras.layers.Layer):
+         )
+
+         q = tf.transpose(q, (0, 2, 1, 3))  # (bs, num_heads, time, head_size)
+-        k = tf.transpose(k, (0, 2, 3, 1))  # (bs, num_heads, head_size, time)
++        k = tf.transpose(k, (0, 2, 1, 3))  # (bs, num_heads, head_size, time)
+         v = tf.transpose(v, (0, 2, 1, 3))  # (bs, num_heads, time, head_size)
+
+-        score = td_dot(q, k) * self.scale
+-        weights = keras.activations.softmax(
+-            score
+-        )  # (bs, num_heads, time, time)
+-        attn = td_dot(weights, v)
+-        attn = tf.transpose(
+-            attn, (0, 2, 1, 3)
+-        )  # (bs, time, num_heads, head_size)
++        attn = self.sdp(q, k, v)
+         out = tf.reshape(
+             attn, (-1, inputs.shape[1], self.num_heads * self.head_size)
+         )
+@@ -352,10 +372,11 @@ class GEGLU(keras.layers.Layer):
+     def call(self, inputs):
+         x = self.dense(inputs)
+         x, gate = x[..., : self.output_dim], x[..., self.output_dim :]
+-        tanh_res = keras.activations.tanh(
+-            gate * 0.7978845608 * (1 + 0.044715 * (gate**2))
+-        )
+-        return x * 0.5 * gate * (1 + tanh_res)
++        # tanh_res = keras.activations.tanh(
++        #     gate * 0.7978845608 * (1 + 0.044715 * (gate**2))
++        # )
++        # return x * 0.5 * gate * (1 + tanh_res)
++        return x * tf.keras.activations.gelu(gate, approximate=True)
+
+
+ def td_dot(a, b):
+diff --git a/keras_cv/models/stable_diffusion/image_encoder.py b/keras_cv/models/stable_diffusion/image_encoder.py
+index 614b11d..8212389 100644
+--- a/keras_cv/models/stable_diffusion/image_encoder.py
++++ b/keras_cv/models/stable_diffusion/image_encoder.py
+@@ -13,6 +13,12 @@
+ # limitations under the License.
+
+ from tensorflow import keras
++try:
++    import intel_extension_for_tensorflow as itex
++    keras.layers.GroupNormalization = itex.ops.GroupNormalization
++except:
++    pass
++
+
+ from keras_cv.models.stable_diffusion.__internal__.layers.attention_block import (  # noqa: E501
+     AttentionBlock,
+diff --git a/keras_cv/models/stable_diffusion/stable_diffusion.py b/keras_cv/models/stable_diffusion/stable_diffusion.py
+index 5ef7471..93d619d 100644
+--- a/keras_cv/models/stable_diffusion/stable_diffusion.py
++++ b/keras_cv/models/stable_diffusion/stable_diffusion.py
+@@ -29,7 +29,8 @@ import math
+ import numpy as np
+ import tensorflow as tf
+ from tensorflow import keras
+-
++from keras import backend as K
++import os
+ from keras_cv.models.stable_diffusion.clip_tokenizer import SimpleTokenizer
+ from keras_cv.models.stable_diffusion.constants import _ALPHAS_CUMPROD
+ from keras_cv.models.stable_diffusion.constants import _UNCONDITIONAL_TOKENS
+@@ -51,6 +52,7 @@ class StableDiffusionBase:
+         img_height=512,
+         img_width=512,
+         jit_compile=False,
++        precision="fp32",
+     ):
+         # UNet requires multiples of 2**7 = 128
+         img_height = round(img_height / 128) * 128
+@@ -66,6 +68,7 @@ class StableDiffusionBase:
+         self._tokenizer = None
+
+         self.jit_compile = jit_compile
++        self.to_fp16 = (precision == "fp16")
+
+     def text_to_image(
+         self,
+@@ -76,6 +79,9 @@ class StableDiffusionBase:
+         unconditional_guidance_scale=7.5,
+         seed=None,
+     ):
++        if self.to_fp16:
++            K.set_floatx('float16')
++
+         encoded_text = self.encode_text(prompt)
+
+         return self.generate_image(
+@@ -207,18 +213,21 @@ class StableDiffusionBase:
+
+         # Iterative reverse diffusion stage
+         timesteps = tf.range(1, 1000, 1000 // num_steps)
++        t_embs_lst = self._get_timesteps_embedding(timesteps, batch_size)
++        contexts = tf.concat((unconditional_context, context), 0)
++
+         alphas, alphas_prev = self._get_initial_alphas(timesteps)
+         progbar = keras.utils.Progbar(len(timesteps))
+         iteration = 0
+         for index, timestep in list(enumerate(timesteps))[::-1]:
+             latent_prev = latent  # Set aside the previous latent vector
+-            t_emb = self._get_timestep_embedding(timestep, batch_size)
+-            unconditional_latent = self.diffusion_model.predict_on_batch(
+-                [latent, t_emb, unconditional_context]
+-            )
+-            latent = self.diffusion_model.predict_on_batch(
+-                [latent, t_emb, context]
+-            )
++            latents = tf.concat((latent, latent), 0)
++            t_embs = t_embs_lst[index]
++
++            pred_latent = self.diffusion_model.predict_on_batch(
++                [latents, t_embs, contexts])
++            unconditional_latent, latent = tf.split(pred_latent, 2)
++
+             latent = unconditional_latent + unconditional_guidance_scale * (
+                 latent - unconditional_latent
+             )
+@@ -304,6 +313,23 @@ class StableDiffusionBase:
+             self._tokenizer = SimpleTokenizer()
+         return self._tokenizer
+
++    def _get_timesteps_embedding(
++        self, timesteps, batch_size, dim=320, max_period=10000
++    ):
++        half = dim // 2
++        freqs = tf.math.exp(
++            -math.log(max_period) * tf.range(0, half, dtype=tf.float32) / half
++        )
++        # timesteps shape: [num_steps]
++        args = tf.cast(tf.reshape(timesteps, [-1, 1]), dtype=tf.float32) * freqs
++        # embeddings shape:(steps, half)
++        embeddings = tf.concat([tf.math.cos(args), tf.math.sin(args)], axis=1)
++        embeddings = tf.expand_dims(embeddings, axis=1)
++        if self.to_fp16:
++            embeddings = tf.cast(embeddings, tf.float16)
++        #  2 is to concatenate the embedding of two forward pass
++        return tf.repeat(embeddings, batch_size * 2, axis=1)
++
+     def _get_timestep_embedding(
+         self, timestep, batch_size, dim=320, max_period=10000
+     ):
+@@ -314,6 +340,8 @@ class StableDiffusionBase:
+         args = tf.convert_to_tensor([timestep], dtype=tf.float32) * freqs
+         embedding = tf.concat([tf.math.cos(args), tf.math.sin(args)], 0)
+         embedding = tf.reshape(embedding, [1, -1])
++        if self.to_fp16:
++            embedding = tf.cast(embedding, tf.float16)
+         return tf.repeat(embedding, batch_size, axis=0)
+
+     def _get_initial_alphas(self, timesteps):
+@@ -324,14 +352,25 @@ class StableDiffusionBase:
+
+     def _get_initial_diffusion_noise(self, batch_size, seed):
+         if seed is not None:
+-            return tf.random.stateless_normal(
+-                (batch_size, self.img_height // 8, self.img_width // 8, 4),
+-                seed=[seed, seed],
+-            )
++            if self.to_fp16:
++                return tf.random.stateless_normal(
++                    (batch_size, self.img_height // 8, self.img_width // 8, 4),
++                    seed=[seed, seed], dtype=tf.float16
++                )
++            else:
++                return tf.random.stateless_normal(
++                    (batch_size, self.img_height // 8, self.img_width // 8, 4),
++                    seed=[seed, seed],
++                )
+         else:
+-            return tf.random.normal(
+-                (batch_size, self.img_height // 8, self.img_width // 8, 4)
+-            )
++            if self.to_fp16:
++                return tf.random.normal(
++                        (batch_size, self.img_height // 8, self.img_width // 8, 4), dtype=tf.float16
++                    )
++            else:
++                return tf.random.normal(
++                    (batch_size, self.img_height // 8, self.img_width // 8, 4)
++                )
+
+     @staticmethod
+     def _get_pos_ids():
+@@ -390,8 +429,9 @@ class StableDiffusion(StableDiffusionBase):
+         img_height=512,
+         img_width=512,
+         jit_compile=False,
++        precision="fp32",
+     ):
+-        super().__init__(img_height, img_width, jit_compile)
++        super().__init__(img_height, img_width, jit_compile, precision)
+         print(
+             "By using this model checkpoint, you acknowledge that its usage is "
+             "subject to the terms of the CreativeML Open RAIL-M license at "
+@@ -475,8 +515,9 @@ class StableDiffusionV2(StableDiffusionBase):
+         img_height=512,
+         img_width=512,
+         jit_compile=False,
++        precision="fp32",
+     ):
+-        super().__init__(img_height, img_width, jit_compile)
++        super().__init__(img_height, img_width, jit_compile, precision)
+         print(
+             "By using this model checkpoint, you acknowledge that its usage is "
+             "subject to the terms of the CreativeML Open RAIL++-M license at "
diff --git a/models_v2/tensorflow/stable_diffusion/inference/gpu/requirements.txt b/models_v2/tensorflow/stable_diffusion/inference/gpu/requirements.txt
new file mode 100644
index 000000000..fab85b927
--- /dev/null
+++ b/models_v2/tensorflow/stable_diffusion/inference/gpu/requirements.txt
@@ -0,0 +1,23 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+#
+
+scikit-image
+matplotlib
+scipy==1.11.1
\ No newline at end of file
diff --git a/models_v2/tensorflow/stable_diffusion/inference/gpu/run_model.sh b/models_v2/tensorflow/stable_diffusion/inference/gpu/run_model.sh
new file mode 100755
index 000000000..0d528e9d6
--- /dev/null
+++ b/models_v2/tensorflow/stable_diffusion/inference/gpu/run_model.sh
@@ -0,0 +1,53 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+
+#!/bin/bash
+
+# Create an array of input directories that are expected and then verify that they exist
+declare -A input_envs
+input_envs[PRECISION]=${PRECISION}
+input_envs[OUTPUT_DIR]=${OUTPUT_DIR}
+
+for i in "${!input_envs[@]}"; do
+  var_name=$i
+  env_param=${input_envs[$i]}
+ 
+  if [[ -z $env_param ]]; then
+    echo "The required environment variable $var_name is not set" >&2
+    exit 1
+  fi
+done
+
+
+echo 'Running with parameters:'
+echo " PRECISION: ${PRECISION}"
+
+python stable_diffusion_inference.py --precision $PRECISION |& tee stable_diffusion_inference.log
+throughput=$(cat stable_diffusion_inference.log | grep throughput | sed -e "s/.*,//" | awk -F ' ' '{print $2}')
+yaml_content=$(cat <<EOF
+results:
+ - key: throughput
+   value: $throughput
+   unit: it/s
+EOF
+)
+
+# Write the content to a YAML file
+echo "$yaml_content" >  ${OUTPUT_DIR}/results.yaml
+echo "YAML file created."
diff --git a/models_v2/tensorflow/stable_diffusion/inference/gpu/setup.sh b/models_v2/tensorflow/stable_diffusion/inference/gpu/setup.sh
new file mode 100755
index 000000000..057a4e0ce
--- /dev/null
+++ b/models_v2/tensorflow/stable_diffusion/inference/gpu/setup.sh
@@ -0,0 +1,32 @@
+#!/usr/bin/env bash
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+set -e
+apt-get update && apt-get install -y python3-venv
+pip install -r requirements.txt
+
+current_dir=$(pwd)
+if [ -d "keras-cv" ]; then
+  echo "Repository already exists. Skipping clone."
+else
+  git clone https://github.com/keras-team/keras-cv.git keras-cv
+  cd ./keras-cv
+  git reset --hard 66fa74b6a2a0bb1e563ae8bce66496b118b95200
+  git apply $current_dir/patch
+  pip install .
+  cd -
+fi
diff --git a/models_v2/tensorflow/stable_diffusion/inference/gpu/stable_diffusion_inference.py b/models_v2/tensorflow/stable_diffusion/inference/gpu/stable_diffusion_inference.py
new file mode 100644
index 000000000..fe5266b72
--- /dev/null
+++ b/models_v2/tensorflow/stable_diffusion/inference/gpu/stable_diffusion_inference.py
@@ -0,0 +1,106 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+#
+
+import time
+from keras_cv.models.stable_diffusion import StableDiffusion, StableDiffusionV2
+from tensorflow import keras
+import matplotlib.pyplot as plt
+import tensorflow as tf
+import argparse
+from keras_fid import fid
+import os
+import numpy as np
+from numpy import cov
+from numpy import trace
+from numpy import iscomplexobj
+from numpy.random import random
+from scipy.linalg import sqrtm
+
+
+parser = argparse.ArgumentParser("Stable Diffusion inference with TensorFlow")
+parser.add_argument(
+    "--batch_size", default=1, type=int, help="number of images generated at one time"
+)
+parser.add_argument(
+    "--use_xla",
+    action="store_true",
+    default=False,
+    help="whether to enable XLA compilation",
+)
+parser.add_argument(
+    "--num_steps", default=50, type=int, help="number of diffusion steps"
+)
+parser.add_argument(
+    "--prompt",
+    default="a photo of an astronaut riding a horse on mars",
+    type=str,
+    help="the text prompt list to put into the text encoder",
+)
+parser.add_argument(
+    "--precision", default="fp32", type=str, help="precision, only support(fp32, fp16)"
+)
+parser.add_argument("--iterations", type=int, default=2, help="number of iterations")
+args = parser.parse_args()
+
+
+def text2image():
+    model = StableDiffusion(
+        img_width=512,
+        img_height=512,
+        jit_compile=args.use_xla,
+        precision=args.precision,
+    )
+    seed = 65537
+    print("Start Warmup")
+    model.text_to_image(
+        "warming up the model", batch_size=args.batch_size, num_steps=args.num_steps
+    )
+    # Start inference
+    print("Start running inference and generating images")
+    start = time.time()
+    for i in range(args.iterations):
+        images = model.text_to_image(prompt=args.prompt, batch_size=args.batch_size, seed=seed)
+    end = time.time()
+    latency = (end - start) / args.iterations / args.num_steps
+    throughput = 1 / latency
+    print("latency {} ms, throughput {} it/s".format(latency * 1000, throughput))
+    return images
+
+
+def plot_images(images):
+    path = "./images"
+    if not os.path.isdir(path):
+        os.mkdir(path)
+    png_name = "{}/{}_imgs_{}steps.png".format(path,
+        args.precision, args.num_steps
+    )
+    print("Start plotting the generated images to %s" % (png_name))
+    plt.figure(figsize=(20, 20))
+    for i in range(len(images)):
+        ax = plt.subplot(1, len(images), i + 1)
+        plt.imshow(images[i])
+        plt.axis("off")
+        plt.savefig(png_name)
+
+
+
+if __name__ == "__main__":
+    images = text2image()
+    plot_images(images)
\ No newline at end of file
diff --git a/models_v2/tensorflow/stable_diffusion/inference/gpu/tests.yaml b/models_v2/tensorflow/stable_diffusion/inference/gpu/tests.yaml
new file mode 100644
index 000000000..8569fcba1
--- /dev/null
+++ b/models_v2/tensorflow/stable_diffusion/inference/gpu/tests.yaml
@@ -0,0 +1,8 @@
+inference-fp16:
+  cmd: bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/setup.sh ${FRAMEWORK_VERSION} ${FRAMEWORK_EXTENSION_VERSION} ${IS_LKG_DROP} ${AIKIT_VERSION}; bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/stable_diffusion-inference/run_test.sh ${PRECISION} ${IS_LKG_DROP}
+  env:
+    PRECISION: "fp16"
+inference-fp32:
+  cmd: bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/setup.sh ${FRAMEWORK_VERSION} ${FRAMEWORK_EXTENSION_VERSION} ${IS_LKG_DROP} ${AIKIT_VERSION}; bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/stable_diffusion-inference/run_test.sh ${PRECISION} ${IS_LKG_DROP}
+  env:
+    PRECISION: "fp32"
diff --git a/models_v2/tensorflow/wide_deep_large_ds/inference/gpu/CONTAINER.md b/models_v2/tensorflow/wide_deep_large_ds/inference/gpu/CONTAINER.md
new file mode 100644
index 000000000..09302d92a
--- /dev/null
+++ b/models_v2/tensorflow/wide_deep_large_ds/inference/gpu/CONTAINER.md
@@ -0,0 +1,89 @@
+# Running Wide and Deep Inference with FP16 on Intel® Data Center GPU Flex Series using Intel® Extension for TensorFlow*
+
+## Overview
+
+This document has instructions for running Wide and Deep model inference using Intel® Extension for TensorFlow* with Intel® Data Center GPU Flex Series.
+
+## Requirements
+| Item | Detail |
+| ------ | ------- |
+| Host machine  | [Intel® Data Center GPU Flex Series](https://ark.intel.com/content/www/us/en/ark/products/series/230021/intel-data-center-gpu-flex-series.html) 170 or 140 |
+| Drivers | GPU-compatible drivers need to be installed: [Download Driver](https://dgpu-docs.intel.com/driver/installation.html)
+| Software | Docker* |
+
+## Get Started
+
+### Download Datasets
+
+Follow [instructions](https://github.com/IntelAI/models/tree/master/datasets/large_kaggle_advertising_challenge/README.md) to download and preprocess the Large Kaggle Display Advertising Challenge Dataset.
+
+Set the `DATASET_PATH` to point to the TF records directory.
+
+### Pre-trained Model
+
+Get the pre-trained model as follows. 
+```bash
+wget https://storage.googleapis.com/intel-optimized-tensorflow/models/3_0/wide_deep_fp16_pretrained_model.pb
+```
+Set the `PB_FILE_PATH` to point to the downloaded model. 
+
+### Quick Start Scripts
+
+| Script name | Description |
+|:-------------:|:-------------:|
+| `run_model.sh` | Runs batch inference for fp16 precision on Flex 170 and Flex 140 |
+
+## Run Using Docker
+
+### Set up Docker Image
+
+```bash
+docker pull intel/recommendation:tf-flex-gpu-wide-and-deep-inference
+```
+
+### Run Docker Image
+
+The Wide and Deep inference container includes scripts, model and libraries needed to run FP16 inference. To run the `run_model.sh` script using this container, you'll need to provide volume mounts for the dataset and pre-trained model. You will also need to provide an output directory where log files will be written. 
+
+```bash
+#Optional
+BATCH_SIZE=<provide batch size. Default is 10000>
+
+#Required
+export DATASET_PATH=<path to processed dataset directory>
+export PB_FILE_PATH=<path to pre-trained model>
+export OUTPUT_DIR=<path to output logs directory>
+
+GPU_TYPE=<provide either flex_170 or flex_140>
+IMAGE_NAME=intel/recommendation:tf-flex-gpu-wide-and-deep-inference
+
+docker run \
+  --device=/dev/dri \
+  --ipc=host \
+  --privileged \
+  --env BATCH_SIZE=${BATCH_SIZE} \
+  --env GPU_TYPE=${GPU_TYPE} \
+  --env OUTPUT_DIR=${OUTPUT_DIR} \
+  --env DATASET_PATH=${DATASET_PATH} \
+  --env PB_FILE_PATH=${PB_FILE_PATH} \
+  --env http_proxy=${http_proxy} \
+  --env https_proxy=${https_proxy} \
+  --env no_proxy=${no_proxy} \
+  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
+  --volume ${DATASET_PATH}:${DATASET_PATH} \
+  --volume ${PB_FILE_PATH}:${PB_FILE_PATH} \
+  --rm -it \
+  $IMAGE_NAME \
+  /bin/bash run_model.sh
+```
+**Note:** Add `--cap-add=SYS_NICE` to the `docker run` command for executing the script on Flex series 140.
+## Documentation and Sources
+
+[GitHub* Repository](https://github.com/IntelAI/models/tree/master/docker/flex-gpu)
+
+## Support
+Support for Intel® Extension for TensorFlow* is found via the [Intel® AI Analytics Toolkit.](https://www.intel.com/content/www/us/en/developer/tools/oneapi/ai-analytics-toolkit.html#gs.qbretz) Additionally, the Intel® Extension for TensorFlow* team tracks both bugs and enhancement requests using [GitHub issues](https://github.com/intel/intel-extension-for-tensorflow/issues). Before submitting a suggestion or bug report, please search the GitHub issues to see if your issue has already been reported.
+
+## License Agreement
+
+LEGAL NOTICE: By accessing, downloading or using this software and any required dependent software (the “Software Package”), you agree to the terms and conditions of the software license agreements for the Software Package, which may also include notices, disclaimers, or license terms for third party software included with the Software Package. Please refer to the [license file](https://github.com/IntelAI/models/tree/master/third_party) for additional details.
diff --git a/models_v2/tensorflow/wide_deep_large_ds/inference/gpu/README.md b/models_v2/tensorflow/wide_deep_large_ds/inference/gpu/README.md
new file mode 100644
index 000000000..c0b127475
--- /dev/null
+++ b/models_v2/tensorflow/wide_deep_large_ds/inference/gpu/README.md
@@ -0,0 +1,67 @@
+# Wide and Deep Large Model Inference
+
+Wide and Deep Large Model Inference Intel® Extension for TensorFlow.
+
+## Model Information
+
+| **Use Case** | **Framework** | **Model Repo** | **Branch/Commit/Tag** | **Optional Patch** |
+|:---:| :---: |:--------------:|:---------------------:|:------------------:|
+|  Inference   |  TensorFlow   |       -        |           -           |         -          |
+
+**Note**: Refer to [CONTAINER.md](CONTAINER.md) for Wide and Deep Inference instructions using docker containers.
+
+# Pre-Requisite
+
+* Host has Intel® Data Center GPU Flex Series
+* Host has installed latest Intel® Data Center GPU Flex Series Driver https://dgpu-docs.intel.com/driver/installation.html
+
+# Dataset and pretrained model
+* Follow [instructions](https://github.com/IntelAI/models/tree/master/datasets/large_kaggle_advertising_challenge/README.md) to download and preprocess the Large Kaggle Display Advertising Challenge Dataset.
+
+## Inference
+
+1. `git clone https://github.com/IntelAI/models.git`
+2. `cd models/models_v2/tensorflow/wide_deep_large_ds/inference/gpu`
+3. Create virtual environment `venv` and activate it:
+    ```
+    python3 -m venv venv
+    . ./venv/bin/activate
+    ```
+4. Run setup.sh
+    ```
+    ./setup.sh
+    ```
+5. Install [tensorflow and ITEX](https://pypi.org/project/intel-extension-for-tensorflow/)
+6. Get pretrained model and set the path for `PB_FILE_PATH`: `wget https://storage.googleapis.com/intel-optimized-tensorflow/models/3_0/wide_deep_fp16_pretrained_model.pb`
+7. Setup required environment paramaters
+ 
+|      **Parameter**       |                      **export command**                       |
+|:------------------------:|:-------------------------------------------------------------:|
+|     **DATASET_PATH**     |          `export DATASET_PATH=/the/path/to/dataset`           |
+|     **PB_FILE_PATH**     |             `export PB_FILE_PATH=/the/path/to/pb`             |
+|     **GPU_TYPE**         |             `export GPU_TYPE=<flex_170 or flex_140>`          |
+| **BATCH_SIZE**(optional) |          `export BATCH_SIZE=10000`                            |
+
+## Output
+
+Output typically looks like:
+
+```
+--------------------------------------------------
+Total test records           :  20000000
+Batch size is                :  10000
+Number of batches            :  2000
+Inference duration (seconds) :  7.5929
+Average Latency (ms/batch)   :  3.9963
+Throughput is (records/sec)  :  2502326.933
+--------------------------------------------------
+```
+
+Final results of the training run can be found in `results.yaml` file.
+
+```
+results:
+ - key: throughput
+   value: 2502326.933
+   unit: records/sec
+```
diff --git a/models_v2/tensorflow/wide_deep_large_ds/inference/gpu/inference.py b/models_v2/tensorflow/wide_deep_large_ds/inference/gpu/inference.py
new file mode 100644
index 000000000..d33dcc5a0
--- /dev/null
+++ b/models_v2/tensorflow/wide_deep_large_ds/inference/gpu/inference.py
@@ -0,0 +1,211 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2018 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+import sys
+import os
+import numpy as np
+import argparse
+import collections
+import time
+import math
+import json
+import datetime
+
+import tensorflow as tf
+from tensorflow.python.framework import graph_util
+from tensorflow.python.framework import ops
+from tensorflow.core.framework import graph_pb2
+from google.protobuf import text_format
+
+
+def str2bool(v):
+    if v.lower() in ('true'):
+        return True
+    else:
+        return False
+
+
+parser = argparse.ArgumentParser()
+parser.add_argument('--input_graph', type=str,
+                    help='file name for graph',
+                    dest='input_graph',
+                    required=True)
+parser.add_argument('--data_location', type=str,
+                    help='full path of data file',
+                    dest='data_location',
+                    required=True)
+parser.add_argument('--batch_size', type=int,
+                    help='batch size for inference.Default is 512',
+                    default=512,
+                    dest='batch_size')
+parser.add_argument('--iterations', type=int,
+                    help='number of iterations to run in benchmark mode',
+                    default=2000,
+                    dest='iterations')
+parser.add_argument('--num_intra_threads', type=int,
+                    help='number of threads for an operator',
+                    required=False,
+                    default=28,
+                    dest='num_intra_threads')
+parser.add_argument('--num_inter_threads', type=int,
+                    help='number of threads across operators',
+                    required=False,
+                    default=2,
+                    dest='num_inter_threads')
+parser.add_argument('--num_omp_threads', type=str,
+                    help='number of threads to use',
+                    required=False,
+                    default=None,
+                    dest='num_omp_threads')
+parser.add_argument("--accuracy_only", type=str2bool,
+                    nargs='?', const=True, default=False,
+                    dest='compute_accuracy', required=False,
+                    help="Enable accuracy calculation")
+
+args = parser.parse_args()
+if args.num_omp_threads:
+    os.environ["OMP_NUM_THREADS"] = args.num_omp_threads
+
+output_probabilities_node = 'import/import/head/predictions/probabilities'
+probabilities_node = 'import/' + output_probabilities_node + ':0'
+placeholder_name = 'import/new_numeric_placeholder'
+categorical_placeholder = 'import/new_categorical_placeholder'
+
+config = tf.compat.v1.ConfigProto(log_device_placement=False,
+                                  inter_op_parallelism_threads=args.num_inter_threads,
+                                  intra_op_parallelism_threads=args.num_intra_threads)
+graph = ops.Graph()
+graph_def = graph_pb2.GraphDef()
+
+filename, file_ext = os.path.splitext(args.input_graph)
+
+batch_size = args.batch_size
+with open(args.input_graph, "rb") as f:
+    if file_ext == ".pbtxt":
+        text_format.Merge(f.read(), graph_def)
+    else:
+        graph_def.ParseFromString(f.read())
+with graph.as_default():
+    tf.import_graph_def(graph_def)
+numeric_feature_names = ["numeric_1"]
+string_feature_names = ["string_1"]
+if args.compute_accuracy:
+    full_features_names = numeric_feature_names + string_feature_names + ["label"]
+    feature_datatypes = [tf.io.FixedLenSequenceFeature([], tf.float32, default_value=0.0, allow_missing=True)] + [
+        tf.io.FixedLenSequenceFeature(
+            [], tf.int64, default_value=0, allow_missing=True)] + [
+                            tf.io.FixedLenSequenceFeature([], tf.int64, default_value=0, allow_missing=True)]
+else:
+    full_features_names = numeric_feature_names + string_feature_names
+    feature_datatypes = [tf.io.FixedLenSequenceFeature([], tf.float32, default_value=0.0, allow_missing=True)] + [
+        tf.io.FixedLenSequenceFeature(
+            [], tf.int64, default_value=0, allow_missing=True)]
+
+
+def input_fn(data_file, num_epochs, shuffle, batch_size):
+    """Generate an input function for the Estimator."""
+
+    def _parse_function(proto):
+        f = collections.OrderedDict(
+            zip(full_features_names, feature_datatypes))
+        parsed_features = tf.io.parse_example(proto, f)
+        parsed_feature_vals_num = [tf.reshape(
+            parsed_features["numeric_1"], shape=[-1, 13])]
+        parsed_feature_vals_str = [tf.reshape(
+            parsed_features["string_1"], shape=[-1, 2]) for i in string_feature_names]
+        parsed_feature_vals = parsed_feature_vals_num + parsed_feature_vals_str
+        if args.compute_accuracy:
+            parsed_feature_vals_label = [tf.reshape(parsed_features[i], shape=[-1]) for i in ["label"]]
+            parsed_feature_vals = parsed_feature_vals + parsed_feature_vals_label
+        return parsed_feature_vals
+
+    # Extract lines from input files using the Dataset API.
+    dataset = tf.data.TFRecordDataset([data_file])
+    if shuffle:
+        dataset = dataset.shuffle(buffer_size=20000)
+    dataset = dataset.batch(batch_size)
+    dataset = dataset.map(_parse_function, num_parallel_calls=28)
+    dataset = dataset.prefetch(batch_size * 10)
+    return dataset
+
+
+data_file = args.data_location
+no_of_test_samples = sum(1 for _ in tf.compat.v1.python_io.tf_record_iterator(data_file))
+no_of_batches = math.ceil(float(no_of_test_samples) / batch_size)
+placeholder_list = ['import/new_numeric_placeholder:0', 'import/new_categorical_placeholder:0']
+input_tensor = [graph.get_tensor_by_name(name) for name in placeholder_list]
+output_name = "import/head/predictions/probabilities"
+output_tensor = graph.get_tensor_by_name("import/" + output_name + ":0")
+correctly_predicted = 0
+total_infer_consume = 0.0
+warm_iter = 100
+features_list = []
+with tf.compat.v1.Session(config=config, graph=graph) as sess:
+    res_dataset = input_fn(data_file, 1, False, batch_size)
+    iterator = tf.compat.v1.data.make_one_shot_iterator(res_dataset)
+    next_element = iterator.get_next()
+    for i in range(int(no_of_batches)):
+        batch = sess.run(next_element)
+        features = batch[0:3]
+        features_list.append(features)
+
+with tf.compat.v1.Session(config=config, graph=graph) as sess1:
+    i = 0
+    if not args.compute_accuracy:
+        no_of_batches = args.iterations
+        no_of_test_samples = no_of_batches * batch_size
+    while True:
+        if i >= no_of_batches:
+            break
+        if i > warm_iter:
+            inference_start = time.time()
+        logistic = None
+        if args.compute_accuracy:
+            logistic = sess1.run(output_tensor, dict(zip(input_tensor, features_list[i][0:2])))
+        else:
+            logistic = sess1.run(output_tensor, dict(zip(input_tensor, features_list[0][0:2])))
+        if i > warm_iter:
+            infer_time = time.time() - inference_start
+            total_infer_consume += infer_time
+        if args.compute_accuracy:
+            predicted_labels = np.argmax(logistic, 1)
+            correctly_predicted = correctly_predicted + np.sum(features_list[i][2] == predicted_labels)
+
+        i = i + 1
+    inference_end = time.time()
+if args.compute_accuracy:
+    accuracy = (
+            float(correctly_predicted) / float(no_of_test_samples))
+evaluate_duration = total_infer_consume
+latency = (1000 * batch_size * float(evaluate_duration) / float(no_of_test_samples - warm_iter * batch_size))
+throughput = (no_of_test_samples - warm_iter * batch_size) / evaluate_duration
+
+print('--------------------------------------------------')
+print('Total test records           : ', no_of_test_samples)
+print('Batch size is                : ', batch_size)
+print('Number of batches            : ', int(no_of_batches))
+if args.compute_accuracy:
+    print('Classification accuracy (%)  : ', round((accuracy * 100), 4))
+    print('No of correct predictions    : ', int(correctly_predicted))
+print('Inference duration (seconds) : ', round(evaluate_duration, 4))
+print('Average Latency (ms/batch)   : ', round(latency, 4))
+print('Throughput is (records/sec)  : ', round(throughput, 3))
+print('--------------------------------------------------')
\ No newline at end of file
diff --git a/models_v2/tensorflow/wide_deep_large_ds/inference/gpu/run_model.sh b/models_v2/tensorflow/wide_deep_large_ds/inference/gpu/run_model.sh
new file mode 100755
index 000000000..1f98cd458
--- /dev/null
+++ b/models_v2/tensorflow/wide_deep_large_ds/inference/gpu/run_model.sh
@@ -0,0 +1,85 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+
+#!/bin/bash
+
+# Create an array of input directories that are expected and then verify that they exist
+declare -A input_envs
+input_envs[DATASET_PATH]=${DATASET_PATH}
+input_envs[PB_FILE_PATH]=${PB_FILE_PATH}
+input_envs[OUTPUT_DIR]=${OUTPUT_DIR}
+input_envs[GPU_TYPE]=${GPU_TYPE}
+
+for i in "${!input_envs[@]}"; do
+  var_name=$i
+  env_param=${input_envs[$i]}
+ 
+  if [[ -z $env_param ]]; then
+    echo "The required environment variable $var_name is not set" >&2
+    exit 1
+  fi
+done
+
+BATCH_SIZE=${BATCH_SIZE:-10000}
+
+echo 'Running with parameters:'
+echo " DATASET_PATH: ${DATASET_PATH}"
+echo " PB_FILE_PATH: ${PB_FILE_PATH}"
+echo " BATCH_SIZE: ${BATCH_SIZE}"
+echo "OUTPUT_DIR: ${OUTPUT_DIR}"
+echo " BATCH_SIZE: ${BATCH_SIZE}"
+
+declare -a str
+device_id=$( lspci | grep -i display | sed -n '1p' | awk '{print $7}' )
+num_devs=$(lspci | grep -i display | awk '{print $7}' | wc -l)
+
+if [[ ${GPU_TYPE} == flex_170 ]]; then 
+    if [[ ${device_id} == "56c0" ]]; then
+      echo "Running FP16 Wide and Deep Large Inference on Flex 170"
+      python inference.py  --input_graph=$PB_FILE_PATH \
+             --data_location=$DATASET_PATH --batch_size=$BATCH_SIZE \
+            --num_inter_threads=1 |& tee wide-deep-large-ds_inference_BS${BATCH_SIZE}.log
+      throughput=$(cat wide-deep-large-ds_inference_BS${BATCH_SIZE}.log | grep Throughput | awk -F ' ' '{print $12}')
+    fi
+elif [[ ${GPU_TYPE} == flex_140 ]]; then 
+    if [[ ${device_id} == "56c1" ]]; then
+      echo "Running FP16 Wide and Deep Large Inference on Flex 140"
+      for i in $( eval echo {0..$((num_devs-1))} )
+        do
+          str+=("ZE_AFFINITY_MASK="${i}" python inference.py  --input_graph=$PB_FILE_PATH \
+             --data_location=$DATASET_PATH --batch_size=$BATCH_SIZE \
+            --num_inter_threads=1 ")
+        done
+      parallel --lb -d, --tagstring "[{#}]" ::: "${str[@]}" |& tee wide-deep-large-ds_inference_BS${BATCH_SIZE}.log
+      throughput=$(cat wide-deep-large-ds_inference_BS${BATCH_SIZE}.log | grep Throughput | awk -F ' ' '{print $13}' | awk '{ sum_total += $1 } END { print sum_total }' )
+    fi
+fi
+ 
+
+yaml_content=$(cat <<EOF
+results:
+ - key: throughput
+   value: $throughput
+   unit: records/sec
+EOF
+)
+
+# Write the content to a YAML file
+echo "$yaml_content" >  ${OUTPUT_DIR}/results.yaml
+echo "YAML file created."
diff --git a/models_v2/tensorflow/wide_deep_large_ds/inference/gpu/setup.sh b/models_v2/tensorflow/wide_deep_large_ds/inference/gpu/setup.sh
new file mode 100755
index 000000000..20b62faff
--- /dev/null
+++ b/models_v2/tensorflow/wide_deep_large_ds/inference/gpu/setup.sh
@@ -0,0 +1,21 @@
+#!/usr/bin/env bash
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+set -e
+apt-get update && apt-get install -y python3-venv protobuf-compiler
+apt-get install -y --no-install-recommends parallel pciutils numactl
+
diff --git a/models_v2/tensorflow/wide_deep_large_ds/inference/gpu/tests.yaml b/models_v2/tensorflow/wide_deep_large_ds/inference/gpu/tests.yaml
new file mode 100644
index 000000000..86a658471
--- /dev/null
+++ b/models_v2/tensorflow/wide_deep_large_ds/inference/gpu/tests.yaml
@@ -0,0 +1,5 @@
+inference-fp16:
+  cmd: bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/setup.sh ${FRAMEWORK_VERSION} ${FRAMEWORK_EXTENSION_VERSION} ${IS_LKG_DROP} ${AIKIT_VERSION}; bash $GITHUB_WORKSPACE/tests/cicd/ITEX-XPU/wide_deep_large_ds-inference/run_test.sh ${DATASET_PATH} ${IS_LKG_DROP}
+  env:
+    DATASET_PATH: "/tf_dataset/dataset/wide_deep_kaggle/eval_preprocessed_eval.tfrecords"
+
diff --git a/quickstart/diffusion/pytorch/stable_diffusion/inference/cpu/README.md b/quickstart/diffusion/pytorch/stable_diffusion/inference/cpu/README.md
new file mode 100644
index 000000000..eb36dc513
--- /dev/null
+++ b/quickstart/diffusion/pytorch/stable_diffusion/inference/cpu/README.md
@@ -0,0 +1,71 @@
+<!--- 0. Title -->
+# PyTorch Stable Diffusion inference
+
+<!-- 10. Description -->
+## Description
+
+This document has instructions for running [Stable Diffusion, which is a text-to-image latent diffusion model created by the researchers and engineers from CompVis, Stability AI, LAION and RunwayML.](https://huggingface.co/stabilityai/stable-diffusion-2-1) inference using Intel-optimized PyTorch.
+
+## Bare Metal
+### General setup
+
+Follow [link](/docs/general/pytorch/BareMetalSetup.md) to install Conda and build Pytorch, IPEX, TorchVison Jemalloc and TCMalloc.
+
+### Install dependencies
+```
+pip install torchmetrics pycocotools transformers==4.35.2 peft==0.6.2
+pip install torch-fidelity --no-deps
+```
+
+### install model
+```
+git clone https://github.com/huggingface/diffusers.git
+cd diffusers
+git checkout v0.23.1
+git apply ../diffusers.patch
+python setup.py install
+```
+
+* [optional] Compile model with PyTorch Inductor backend
+```shell
+export TORCH_INDUCTOR=1
+```
+
+# Quick Start Scripts
+
+|  DataType   | Throughput  |  Latency    |   Accuracy  |
+| ----------- | ----------- | ----------- | ----------- |
+| FP32        | bash inference_throughput.sh fp32 ipex-jit | bash inference_realtime.sh fp32 ipex-jit | bash accuracy.sh fp32 ipex-jit |
+| BF16        | bash inference_throughput.sh bf16 ipex-jit | bash inference_realtime.sh bf16 ipex-jit | bash accuracy.sh bf16 ipex-jit |
+| BF32        | bash inference_throughput.sh bf32 ipex-jit | bash inference_realtime.sh bf32 ipex-jit | bash accuracy.sh bf32 ipex-jit |
+| FP16        | bash inference_throughput.sh fp16 ipex-jit | bash inference_realtime.sh fp16 ipex-jit | bash accuracy.sh fp16 ipex-jit |
+| INT8-BF16        | bash inference_throughput.sh int8-bf16 ipex-jit | bash inference_realtime.sh int8-bf16 ipex-jit | bash accuracy.sh int8-bf16 ipex-jit |
+| INT8-FP32        | bash inference_throughput.sh int8-fp32 ipex-jit | bash inference_realtime.sh int8-fp32 ipex-jit | bash accuracy.sh int8-fp32 ipex-jit |
+
+**NOTE**: You should get a quant_model.pt before run INT8-BF16 model or INT8-FP32 model. Please refer the [link](https://github.com/intel-innersource/frameworks.ai.nlp-toolkit.intel-nlp-toolkit/blob/develop/examples/huggingface/pytorch/text-to-image/quantization/qat/README.md).
+
+## Run the model
+
+Follow the instructions above to setup your bare metal environment, download and
+preprocess the dataset, and do the model specific setup. Once all the setup is done,
+the Model Zoo can be used to run a [quickstart script](#quick-start-scripts).
+Ensure that you have an enviornment variable set to point to an output directory.
+
+```bash
+# Clone the model zoo repo and set the MODEL_DIR
+git clone https://github.com/IntelAI/models.git
+cd models
+export MODEL_DIR=$(pwd)
+cd quickstart/diffusion/pytorch/stable_diffusion/inference/cpu/
+
+# Env vars
+export DATASET_DIR=<path to the dataset>
+export OUTPUT_DIR=<path to an output directory>
+
+# Run a quickstart script (for example, FP32 multi-instance realtime inference)
+bash inference_realtime.sh fp32 ipex-jit
+```
+
+<!--- 80. License -->
+## License
+[LICENSE](https://github.com/IntelAI/models/blob/master/LICENSE)
diff --git a/quickstart/diffusion/pytorch/stable_diffusion/inference/cpu/accuracy.sh b/quickstart/diffusion/pytorch/stable_diffusion/inference/cpu/accuracy.sh
new file mode 100644
index 000000000..e701460ee
--- /dev/null
+++ b/quickstart/diffusion/pytorch/stable_diffusion/inference/cpu/accuracy.sh
@@ -0,0 +1,110 @@
+#!/usr/bin/env bash
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+MODEL_DIR=${MODEL_DIR-$PWD}
+
+if [ ! -e "${MODEL_DIR}/models/diffusion/pytorch/stable_diffusion/inference.py" ]; then
+  echo "Could not find the script of inference.py. Please set environment variable '\${MODEL_DIR}'."
+  echo "From which the inference.py exist at the: \${MODEL_DIR}/models/diffusion/pytorch/stable_diffusion/inference.py"
+  exit 1
+fi
+
+if [ ! -d "${DATASET_DIR}" ]; then
+  echo "The DATASET_DIR \${DATASET_DIR} does not exist"
+  exit 1
+fi
+
+if [ ! -d "${OUTPUT_DIR}" ]; then
+  echo "The OUTPUT_DIR '${OUTPUT_DIR}' does not exist"
+  exit 1
+fi
+
+if [[ "$1" == *"avx"* ]]; then
+    unset DNNL_MAX_CPU_ISA
+fi
+
+ARGS=""
+if [ "$1" == "bf16" ]; then
+    ARGS="$ARGS --precision=bf16"
+    echo "### running bf16 datatype"
+elif [ "$1" == "fp16" ]; then
+    ARGS="$ARGS --precision=fp16"
+    echo "### running fp16 datatype"
+elif [ "$1" == "int8-bf16" ]; then
+    ARGS="$ARGS --precision=int8-bf16"
+    echo "### running int8-bf16 datatype"
+elif [ "$1" == "int8-fp32" ]; then
+    ARGS="$ARGS --precision=int8-fp32"
+    echo "### running int8-fp32 datatype"
+elif [ "$1" == "bf32" ]; then
+    ARGS="$ARGS --precision=bf32"
+    echo "### running bf32 datatype"
+elif [ "$1" == "fp32" ]; then
+    echo "### running fp32 datatype"
+else
+    echo "The specified precision '$1' is unsupported."
+    echo "Supported precisions are: fp32, bf32, fp16, bf16, int8-bf16, int8-fp32"
+    exit 1
+fi
+
+TORCH_INDUCTOR=${TORCH_INDUCTOR:-"0"}
+if [ "$2" == "eager" ]; then
+    echo "### running eager mode"
+elif [ "$2" == "ipex-jit" ]; then
+    ARGS="$ARGS --ipex --jit"
+    echo "### running IPEX JIT mode"
+elif [ "$2" == "compile-ipex" ]; then
+    ARGS="$ARGS --compile_ipex"
+    echo "### running torch.compile with ipex backend"
+elif [[ "$2" == "compile-inductor" || "1" == ${TORCH_INDUCTOR} ]]; then
+    export TORCHINDUCTOR_FREEZING=1
+    ARGS="$ARGS --compile_inductor"
+    echo "### running torch.compile with inductor backend"
+else
+    echo "The specified mode '$2' is unsupported."
+    echo "Supported mode are: eager, ipex-jit, compile-ipex, compile-inductor"
+    exit 1
+fi
+
+export DNNL_PRIMITIVE_CACHE_CAPACITY=1024
+export MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+export KMP_BLOCKTIME=1
+export KMP_AFFINITY=granularity=fine,compact,1,0
+
+PRECISION=$1
+
+rm -rf ${OUTPUT_DIR}/stable_diffusion_${PRECISION}_inference_accuracy*
+rm -rf ${PRECISION}_results
+
+python -m intel_extension_for_pytorch.cpu.launch \
+    --memory-allocator jemalloc \
+    --ninstance 1 \
+    --nodes-list=0 \
+    --log-dir=${OUTPUT_DIR} \
+    --log_file_prefix stable_diffusion_${PRECISION}_inference_accuracy \
+    ${MODEL_DIR}/models/diffusion/pytorch/stable_diffusion/inference.py \
+    --dataset_path=${DATASET_DIR} \
+    --accuracy \
+    -i=10 \
+    --output_dir="${PRECISION}_results" \
+    $ARGS
+
+# For the summary of results
+wait
+
+accuracy=$(grep 'FID:' ${OUTPUT_DIR}/stable_diffusion_${PRECISION}_inference_accuracy* |sed -e 's/.*FID//;s/[^0-9.]//g')
+echo ""stable_diffusion";"FID";$1;${accuracy}" | tee -a ${OUTPUT_DIR}/summary.log
diff --git a/quickstart/diffusion/pytorch/stable_diffusion/inference/cpu/accuracy_dist.sh b/quickstart/diffusion/pytorch/stable_diffusion/inference/cpu/accuracy_dist.sh
new file mode 100644
index 000000000..ab52adb4a
--- /dev/null
+++ b/quickstart/diffusion/pytorch/stable_diffusion/inference/cpu/accuracy_dist.sh
@@ -0,0 +1,163 @@
+#!/usr/bin/env bash
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+MODEL_DIR=${MODEL_DIR-$PWD}
+
+if [ ! -e "${MODEL_DIR}/models/diffusion/pytorch/stable_diffusion/inference.py" ]; then
+  echo "Could not find the script of inference.py. Please set environment variable '\${MODEL_DIR}'."
+  echo "From which the inference.py exist at the: \${MODEL_DIR}/models/diffusion/pytorch/stable_diffusion/inference.py"
+  exit 1
+fi
+
+if [ ! -d "${DATASET_DIR}" ]; then
+  echo "The DATASET_DIR \${DATASET_DIR} does not exist"
+  exit 1
+fi
+
+if [ ! -d "${OUTPUT_DIR}" ]; then
+  echo "The OUTPUT_DIR '${OUTPUT_DIR}' does not exist"
+  exit 1
+fi
+
+if [[ "$1" == *"avx"* ]]; then
+    unset DNNL_MAX_CPU_ISA
+fi
+
+ARGS=""
+if [ "$1" == "bf16" ]; then
+    ARGS="$ARGS --precision=bf16"
+    echo "### running bf16 datatype"
+elif [ "$1" == "fp16" ]; then
+    ARGS="$ARGS --precision=fp16"
+    echo "### running fp16 datatype"
+elif [ "$1" == "int8-bf16" ]; then
+    ARGS="$ARGS --precision=int8-bf16"
+    echo "### running int8-bf16 datatype"
+elif [ "$1" == "int8-fp32" ]; then
+    ARGS="$ARGS --precision=int8-fp32"
+    echo "### running int8-fp32 datatype"
+elif [ "$1" == "bf32" ]; then
+    ARGS="$ARGS --precision=bf32"
+    echo "### running bf32 datatype"
+elif [ "$1" == "fp32" ]; then
+    echo "### running fp32 datatype"
+else
+    echo "The specified precision '$1' is unsupported."
+    echo "Supported precisions are: fp32, bf32, fp16, bf16, int8-bf16, int8-fp32"
+    exit 1
+fi
+
+if [ "$2" == "eager" ]; then
+    echo "### running eager mode"
+elif [ "$2" == "ipex-jit" ]; then
+    ARGS="$ARGS --ipex --jit"
+    echo "### running IPEX JIT mode"
+elif [ "$2" == "compile-ipex" ]; then
+    ARGS="$ARGS --compile_ipex"
+    echo "### running torch.compile with ipex backend"
+elif [ "$2" == "compile-inductor" ]; then
+    export TORCHINDUCTOR_FREEZING=1
+    ARGS="$ARGS --compile_inductor"
+    echo "### running torch.compile with inductor backend"
+else
+    echo "The specified mode '$2' is unsupported."
+    echo "Supported mode are: eager, ipex-jit, compile-ipex, compile-inductor"
+    exit 1
+fi
+
+CORES=`lscpu | grep Core | awk '{print $4}'`
+SOCKETS=`lscpu | grep Socket | awk '{print $2}'`
+TOTAL_CORES=`expr $CORES \* $SOCKETS`
+NNODES=${NNODES:-1}
+HOSTFILE=${HOSTFILE:-./hostfile}
+NUM_RANKS=$(( NNODES * SOCKETS ))
+
+if [ ${LOCAL_BATCH_SIZE} ]; then
+    GLOBAL_BATCH_SIZE=$(( LOCAL_BATCH_SIZE * NNODES * SOCKETS ))
+fi
+
+CORES_PER_INSTANCE=$CORES
+
+export DNNL_PRIMITIVE_CACHE_CAPACITY=1024
+export KMP_BLOCKTIME=1
+export KMP_AFFINITY=granularity=fine,compact,1,0
+
+PRECISION=$1
+
+<< EOF
+#oneCCL settings
+export CCL_WORKER_COUNT=8
+export CCL_LOG_LEVEL=info
+export CCL_BF16=avx512bf
+export CCL_ATL_TRANSPORT=ofi
+export CCL_MNIC_COUNT=2
+export CCL_MNIC=local
+export CCL_MNIC_NAME=irdma1,irdma5
+export CCL_ALLREDUCE=ring
+export CCL_WORKER_COUNT=8
+
+for (( i = $SOCKETS; i < 2*$SOCKETS; i++ )); do  # pin CCL workers to HT
+  START_CORE=$(( i * CORES ))
+  for (( j = 0; j < $CCL_WORKER_COUNT; j++)); do
+   CCL_WORKER_AFFINITY="${CCL_WORKER_AFFINITY} $((START_CORE + j))"
+  done
+done
+
+export CCL_WORKER_AFFINITY=`echo ${CCL_WORKER_AFFINITY} | tr " " ","`
+
+
+#DDP settings
+export TORCH_CPP_LOG_LEVEL=INFO
+export TORCH_DISTRIBUTED_DEBUG=INFO
+export MASTER_ADDR=`head -1 hostfile`
+
+# Fabric settings
+export FI_PROVIDER=psm3
+export PSM3_IDENTIFY=1
+export PSM3_ALLOW_ROUTERS=1
+export PSM3_RDMA=1
+export PSM3_PRINT_STATS=0
+export PSM3_RV_MR_CACHE_SIZE=8192
+export PSM3_KASSIST_MODE=none
+#export PSM3_NIC='irdma*
+export FI_PSM3_CONN_TIMEOUT=100
+EOF
+
+rm -rf ${OUTPUT_DIR}/stable_diffusion_${PRECISION}_dist_inference_accuracy*
+
+oneccl_bindings_for_pytorch_path=$(python -c "import torch; import oneccl_bindings_for_pytorch; import os;  print(os.path.abspath(os.path.dirname(oneccl_bindings_for_pytorch.__file__)))")
+source $oneccl_bindings_for_pytorch_path/env/setvars.sh
+
+export FI_PROVIDER_PATH=$oneccl_bindings_for_pytorch_path/lib/prov
+
+python -m intel_extension_for_pytorch.cpu.launch \
+    --memory-allocator tcmalloc \
+    --distributed \
+    --nnodes ${NNODES} \
+    --hostfile ${HOSTFILE} \
+    --logical-cores-for-ccl --ccl_worker_count 8 \
+    ${MODEL_DIR}/models/diffusion/pytorch/stable_diffusion/inference.py \
+    --dataset_path=${DATASET_DIR} \
+    --dist-backend ccl \
+    --accuracy \
+    $ARGS 2>&1 | tee ${OUTPUT_DIR}/stable_diffusion_${PRECISION}_dist_inference_accuracy.log
+
+# For the summary of results
+wait
+
+accuracy=$(grep 'FID:' ${OUTPUT_DIR}/stable_diffusion_${PRECISION}_dist_inference_accuracy* |sed -e 's/.*FID//;s/[^0-9.]//g')
+echo ""stable_diffusion";"FID";$1;${accuracy}" | tee -a ${OUTPUT_DIR}/summary.log
diff --git a/quickstart/diffusion/pytorch/stable_diffusion/inference/cpu/diffusers.patch b/quickstart/diffusion/pytorch/stable_diffusion/inference/cpu/diffusers.patch
new file mode 100644
index 000000000..b80888588
--- /dev/null
+++ b/quickstart/diffusion/pytorch/stable_diffusion/inference/cpu/diffusers.patch
@@ -0,0 +1,114 @@
+diff --git a/src/diffusers/models/unet_2d_condition.py b/src/diffusers/models/unet_2d_condition.py
+index f248b243..27d4802d 100644
+--- a/src/diffusers/models/unet_2d_condition.py
++++ b/src/diffusers/models/unet_2d_condition.py
+@@ -799,8 +799,8 @@ class UNet2DConditionModel(ModelMixin, ConfigMixin, UNet2DConditionLoadersMixin)
+         sample: torch.FloatTensor,
+         timestep: Union[torch.Tensor, float, int],
+         encoder_hidden_states: torch.Tensor,
+-        class_labels: Optional[torch.Tensor] = None,
+         timestep_cond: Optional[torch.Tensor] = None,
++        class_labels: Optional[torch.Tensor] = None,
+         attention_mask: Optional[torch.Tensor] = None,
+         cross_attention_kwargs: Optional[Dict[str, Any]] = None,
+         added_cond_kwargs: Optional[Dict[str, torch.Tensor]] = None,
+diff --git a/src/diffusers/pipelines/latent_consistency_models/pipeline_latent_consistency_text2img.py b/src/diffusers/pipelines/latent_consistency_models/pipeline_latent_consistency_text2img.py
+index ff5eea2d..f5545648 100644
+--- a/src/diffusers/pipelines/latent_consistency_models/pipeline_latent_consistency_text2img.py
++++ b/src/diffusers/pipelines/latent_consistency_models/pipeline_latent_consistency_text2img.py
+@@ -701,17 +701,24 @@ class LatentConsistencyModelPipeline(
+         self._num_timesteps = len(timesteps)
+         with self.progress_bar(total=num_inference_steps) as progress_bar:
+             for i, t in enumerate(timesteps):
++                # with torch.profiler.profile(activities=[torch.profiler.ProfilerActivity.CPU], record_shapes=True) as p:
+                 latents = latents.to(prompt_embeds.dtype)
+ 
+                 # model prediction (v-prediction, eps, x)
+-                model_pred = self.unet(
+-                    latents,
+-                    t,
+-                    timestep_cond=w_embedding,
+-                    encoder_hidden_states=prompt_embeds,
+-                    cross_attention_kwargs=self.cross_attention_kwargs,
+-                    return_dict=False,
+-                )[0]
++                if hasattr(self, 'traced_unet'):
++                    model_pred = self.traced_unet(
++                        latents.to(memory_format=torch.channels_last).to(dtype=self.precision),
++                        t,
++                        encoder_hidden_states=prompt_embeds.to(dtype=self.precision),
++                        timestep_cond=w_embedding.to(dtype=self.precision)
++                    )['sample']
++                else:
++                    model_pred = self.unet(
++                        latents.to(memory_format=torch.channels_last).to(dtype=self.precision),
++                        t,
++                        encoder_hidden_states=prompt_embeds.to(dtype=self.precision),
++                        timestep_cond=w_embedding.to(dtype=self.precision)
++                    )['sample']
+ 
+                 # compute the previous noisy sample x_t -> x_t-1
+                 latents, denoised = self.scheduler.step(model_pred, t, latents, **extra_step_kwargs, return_dict=False)
+@@ -733,6 +740,9 @@ class LatentConsistencyModelPipeline(
+                         step_idx = i // getattr(self.scheduler, "order", 1)
+                         callback(step_idx, t, latents)
+ 
++                # output = p.key_averages().table(sort_by="self_cpu_time_total")
++                # print(output)
++
+         denoised = denoised.to(prompt_embeds.dtype)
+         if not output_type == "latent":
+             image = self.vae.decode(denoised / self.vae.config.scaling_factor, return_dict=False)[0]
+diff --git a/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py b/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py
+index 9911cbe7..a354405a 100644
+--- a/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py
++++ b/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py
+@@ -832,19 +832,30 @@ class StableDiffusionPipeline(DiffusionPipeline, TextualInversionLoaderMixin, Lo
+         self._num_timesteps = len(timesteps)
+         with self.progress_bar(total=num_inference_steps) as progress_bar:
+             for i, t in enumerate(timesteps):
++                # with torch.profiler.profile(activities=[torch.profiler.ProfilerActivity.CPU], record_shapes=True) as p:
+                 # expand the latents if we are doing classifier free guidance
+                 latent_model_input = torch.cat([latents] * 2) if self.do_classifier_free_guidance else latents
+                 latent_model_input = self.scheduler.scale_model_input(latent_model_input, t)
+ 
+                 # predict the noise residual
+-                noise_pred = self.unet(
+-                    latent_model_input,
+-                    t,
+-                    encoder_hidden_states=prompt_embeds,
+-                    timestep_cond=timestep_cond,
+-                    cross_attention_kwargs=self.cross_attention_kwargs,
+-                    return_dict=False,
+-                )[0]
++                if hasattr(self, 'HIGH_PRECISION_STEPS') and (i < self.HIGH_PRECISION_STEPS or i > len(timesteps)-1 - self.HIGH_PRECISION_STEPS):
++                    noise_pred = self.unet_highprecision(
++                        latent_model_input.to(memory_format=torch.channels_last).to(dtype=self.precision),
++                        t,
++                        encoder_hidden_states=prompt_embeds.to(dtype=self.precision)
++                    )['sample']
++                elif hasattr(self, 'traced_unet'):
++                    noise_pred = self.traced_unet(
++                        latent_model_input.to(memory_format=torch.channels_last).to(dtype=self.precision),
++                        t,
++                        encoder_hidden_states=prompt_embeds.to(dtype=self.precision)
++                    )['sample']
++                else:
++                    noise_pred = self.unet(
++                        latent_model_input.to(memory_format=torch.channels_last).to(dtype=self.precision),
++                        t,
++                        encoder_hidden_states=prompt_embeds.to(dtype=self.precision)
++                    )['sample']
+ 
+                 # perform guidance
+                 if self.do_classifier_free_guidance:
+@@ -875,6 +886,9 @@ class StableDiffusionPipeline(DiffusionPipeline, TextualInversionLoaderMixin, Lo
+                         step_idx = i // getattr(self.scheduler, "order", 1)
+                         callback(step_idx, t, latents)
+ 
++                # output = p.key_averages().table(sort_by="self_cpu_time_total")
++                # print(output)
++
+         if not output_type == "latent":
+             image = self.vae.decode(latents / self.vae.config.scaling_factor, return_dict=False, generator=generator)[
+                 0
diff --git a/quickstart/diffusion/pytorch/stable_diffusion/inference/cpu/hostfile b/quickstart/diffusion/pytorch/stable_diffusion/inference/cpu/hostfile
new file mode 100644
index 000000000..7b9ad531d
--- /dev/null
+++ b/quickstart/diffusion/pytorch/stable_diffusion/inference/cpu/hostfile
@@ -0,0 +1 @@
+127.0.0.1
diff --git a/quickstart/diffusion/pytorch/stable_diffusion/inference/cpu/inference_realtime.sh b/quickstart/diffusion/pytorch/stable_diffusion/inference/cpu/inference_realtime.sh
new file mode 100644
index 000000000..f215d904f
--- /dev/null
+++ b/quickstart/diffusion/pytorch/stable_diffusion/inference/cpu/inference_realtime.sh
@@ -0,0 +1,135 @@
+#!/usr/bin/env bash
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+MODEL_DIR=${MODEL_DIR-$PWD}
+
+if [ ! -e "${MODEL_DIR}/models/diffusion/pytorch/stable_diffusion/inference.py" ]; then
+  echo "Could not find the script of inference.py. Please set environment variable '\${MODEL_DIR}'."
+  echo "From which the inference.py exist at the: \${MODEL_DIR}/models/diffusion/pytorch/stable_diffusion/inference.py"
+  exit 1
+fi
+
+if [ ! -d "${DATASET_DIR}" ]; then
+  echo "The DATASET_DIR \${DATASET_DIR} does not exist"
+  exit 1
+fi
+
+if [ ! -d "${OUTPUT_DIR}" ]; then
+  echo "The OUTPUT_DIR '${OUTPUT_DIR}' does not exist"
+  exit 1
+fi
+
+if [[ "$1" == *"avx"* ]]; then
+    unset DNNL_MAX_CPU_ISA
+fi
+
+ARGS=""
+if [ "$1" == "bf16" ]; then
+    ARGS="$ARGS --precision=bf16"
+    echo "### running bf16 datatype"
+elif [ "$1" == "fp16" ]; then
+    ARGS="$ARGS --precision=fp16"
+    echo "### running fp16 datatype"
+elif [ "$1" == "int8-bf16" ]; then
+    ARGS="$ARGS --precision=int8-bf16"
+    echo "### running int8-bf16 datatype"
+elif [ "$1" == "int8-fp32" ]; then
+    ARGS="$ARGS --precision=int8-fp32"
+    echo "### running int8-fp32 datatype"
+elif [ "$1" == "bf32" ]; then
+    ARGS="$ARGS --precision=bf32"
+    echo "### running bf32 datatype"
+elif [ "$1" == "fp32" ]; then
+    echo "### running fp32 datatype"
+else
+    echo "The specified precision '$1' is unsupported."
+    echo "Supported precisions are: fp32, bf32, fp16, bf16, int8-bf16, int8-fp32"
+    exit 1
+fi
+
+TORCH_INDUCTOR=${TORCH_INDUCTOR:-"0"}
+if [ "$2" == "eager" ]; then
+    echo "### running eager mode"
+elif [ "$2" == "ipex-jit" ]; then
+    ARGS="$ARGS --ipex --jit"
+    echo "### running IPEX JIT mode"
+elif [ "$2" == "compile-ipex" ]; then
+    ARGS="$ARGS --compile_ipex"
+    echo "### running torch.compile with ipex backend"
+elif [[ "$2" == "compile-inductor" || "1" == ${TORCH_INDUCTOR} ]]; then
+    export TORCHINDUCTOR_FREEZING=1
+    ARGS="$ARGS --compile_inductor"
+    echo "### running torch.compile with inductor backend"
+else
+    echo "The specified mode '$2' is unsupported."
+    echo "Supported mode are: eager, ipex-jit, compile-ipex, compile-inductor"
+    exit 1
+fi
+
+CORES=`lscpu | grep Core | awk '{print $4}'`
+SOCKETS=`lscpu | grep Socket | awk '{print $2}'`
+NUMAS=`lscpu | grep 'NUMA node(s)' | awk '{print $3}'`
+CORES_PER_NUMA=`expr $CORES \* $SOCKETS / $NUMAS`
+
+CORES_PER_INSTANCE=4
+
+export DNNL_PRIMITIVE_CACHE_CAPACITY=1024
+export MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+export KMP_BLOCKTIME=1
+export KMP_AFFINITY=granularity=fine,compact,1,0
+export OMP_NUM_THREADS=$CORES_PER_INSTANCE
+
+NUMBER_INSTANCE=`expr $CORES_PER_NUMA / $CORES_PER_INSTANCE`
+
+PRECISION=$1
+
+rm -rf ${OUTPUT_DIR}/stable_diffusion_${PRECISION}_inference_realtime*
+
+python -m intel_extension_for_pytorch.cpu.launch \
+    --memory-allocator jemalloc \
+    --ninstances $NUMAS \
+    --log-dir ${OUTPUT_DIR} \
+    --log_file_prefix stable_diffusion_${PRECISION}_inference_realtime \
+    ${MODEL_DIR}/models/diffusion/pytorch/stable_diffusion/inference.py \
+    --dataset_path=${DATASET_DIR} \
+    --benchmark \
+    -w 1 -i 1 \
+    --weight-sharing \
+    --number-instance $NUMBER_INSTANCE \
+    $ARGS
+
+# For the summary of results
+wait
+
+TOTAL_CORES=`expr $CORES \* $SOCKETS`
+INSTANCES=`expr $TOTAL_CORES / $CORES_PER_INSTANCE`
+INSTANCES_PER_SOCKET=`expr $INSTANCES / $SOCKETS`
+
+throughput=$(grep 'Throughput:' ${OUTPUT_DIR}/stable_diffusion_${PRECISION}_inference_realtime* |sed -e 's/.*Throughput//;s/[^0-9.]//g' |awk -v INSTANCES_PER_SOCKET=$INSTANCES_PER_SOCKET '
+BEGIN {
+        sum = 0;
+        i = 0;
+      }
+      {
+        sum = sum + $1;
+        i++;
+      }
+END   {
+        sum = sum / i * INSTANCES_PER_SOCKET;
+        printf("%.3f", sum);
+}')
+echo ""stable_diffusion";"latency";$1;${throughput}" | tee -a ${OUTPUT_DIR}/summary.log
diff --git a/quickstart/diffusion/pytorch/stable_diffusion/inference/cpu/inference_throughput.sh b/quickstart/diffusion/pytorch/stable_diffusion/inference/cpu/inference_throughput.sh
new file mode 100644
index 000000000..10fe29e25
--- /dev/null
+++ b/quickstart/diffusion/pytorch/stable_diffusion/inference/cpu/inference_throughput.sh
@@ -0,0 +1,119 @@
+#!/usr/bin/env bash
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+MODEL_DIR=${MODEL_DIR-$PWD}
+
+if [ ! -e "${MODEL_DIR}/models/diffusion/pytorch/stable_diffusion/inference.py" ]; then
+  echo "Could not find the script of inference.py. Please set environment variable '\${MODEL_DIR}'."
+  echo "From which the inference.py exist at the: \${MODEL_DIR}/models/diffusion/pytorch/stable_diffusion/inference.py"
+  exit 1
+fi
+
+if [ ! -d "${DATASET_DIR}" ]; then
+  echo "The DATASET_DIR \${DATASET_DIR} does not exist"
+  exit 1
+fi
+
+if [ ! -d "${OUTPUT_DIR}" ]; then
+  echo "The OUTPUT_DIR '${OUTPUT_DIR}' does not exist"
+  exit 1
+fi
+
+if [[ "$1" == *"avx"* ]]; then
+    unset DNNL_MAX_CPU_ISA
+fi
+
+ARGS=""
+if [ "$1" == "bf16" ]; then
+    ARGS="$ARGS --precision=bf16"
+    echo "### running bf16 datatype"
+elif [ "$1" == "fp16" ]; then
+    ARGS="$ARGS --precision=fp16"
+    echo "### running fp16 datatype"
+elif [ "$1" == "int8-bf16" ]; then
+    ARGS="$ARGS --precision=int8-bf16"
+    echo "### running int8-bf16 datatype"
+elif [ "$1" == "int8-fp32" ]; then
+    ARGS="$ARGS --precision=int8-fp32"
+    echo "### running int8-fp32 datatype"
+elif [ "$1" == "bf32" ]; then
+    ARGS="$ARGS --precision=bf32"
+    echo "### running bf32 datatype"
+elif [ "$1" == "fp32" ]; then
+    echo "### running fp32 datatype"
+else
+    echo "The specified precision '$1' is unsupported."
+    echo "Supported precisions are: fp32, bf32, fp16, bf16, int8-bf16, int8-fp32"
+    exit 1
+fi
+
+TORCH_INDUCTOR=${TORCH_INDUCTOR:-"0"}
+if [ "$2" == "eager" ]; then
+    echo "### running eager mode"
+elif [ "$2" == "ipex-jit" ]; then
+    ARGS="$ARGS --ipex --jit"
+    echo "### running IPEX JIT mode"
+elif [ "$2" == "compile-ipex" ]; then
+    ARGS="$ARGS --compile_ipex"
+    echo "### running torch.compile with ipex backend"
+elif [[ "$2" == "compile-inductor" || "1" == ${TORCH_INDUCTOR} ]]; then
+    export TORCHINDUCTOR_FREEZING=1
+    ARGS="$ARGS --compile_inductor"
+    echo "### running torch.compile with inductor backend"
+else
+    echo "The specified mode '$2' is unsupported."
+    echo "Supported mode are: eager, ipex-jit, compile-ipex, compile-inductor"
+    exit 1
+fi
+
+export DNNL_PRIMITIVE_CACHE_CAPACITY=1024
+export MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+export KMP_BLOCKTIME=1
+export KMP_AFFINITY=granularity=fine,compact,1,0
+
+PRECISION=$1
+
+rm -rf ${OUTPUT_DIR}/stable_diffusion_${PRECISION}_inference_throughput*
+
+python -m intel_extension_for_pytorch.cpu.launch \
+    --memory-allocator jemalloc \
+    --throughput_mode \
+    --log-dir ${OUTPUT_DIR} \
+    --log_file_prefix stable_diffusion_${PRECISION}_inference_throughput \
+    ${MODEL_DIR}/models/diffusion/pytorch/stable_diffusion/inference.py \
+    --dataset_path=${DATASET_DIR} \
+    --benchmark \
+    -w 1 -i 10 \
+    $ARGS
+
+# For the summary of results
+wait
+
+throughput=$(grep 'Throughput:' ${OUTPUT_DIR}/stable_diffusion_${PRECISION}_inference_throughput* |sed -e 's/.*Throughput//;s/[^0-9.]//g' |awk '
+BEGIN {
+        sum = 0;
+        i = 0;
+      }
+      {
+        sum = sum + $1;
+        i++;
+      }
+END   {
+        sum = sum / i;
+        printf("%.3f", sum);
+}')
+echo ""stable_diffusion";"throughput";$1;${throughput}" | tee -a ${OUTPUT_DIR}/summary.log
diff --git a/quickstart/diffusion/pytorch/stable_diffusion/training/cpu/README.md b/quickstart/diffusion/pytorch/stable_diffusion/training/cpu/README.md
new file mode 100644
index 000000000..b45ffa6ac
--- /dev/null
+++ b/quickstart/diffusion/pytorch/stable_diffusion/training/cpu/README.md
@@ -0,0 +1,94 @@
+<!--- 0. Title -->
+# PyTorch Stable Diffusion fine-tuning
+
+<!-- 10. Description -->
+## Description
+
+This document has instructions for running [Stable Diffusion, which is a text-to-image latent diffusion model created by the researchers and engineers from CompVis, Stability AI, LAION and RunwayML.](https://huggingface.co/stabilityai/stable-diffusion-2-1) fine-tuning using Intel-optimized PyTorch.
+
+## Bare Metal
+### General setup
+
+Follow [link](/docs/general/pytorch/BareMetalSetup.md) to install Conda and build Pytorch, IPEX, TorchVison Jemalloc and TCMalloc.
+
+### Install dependencies
+```bash
+pip install torchmetrics pycocotools transformers==4.35.2 peft==0.6.2
+pip install torch-fidelity --no-deps
+```
+
+### install model
+```bash
+git clone https://github.com/huggingface/diffusers.git
+cd diffusers
+git checkout v0.23.1
+python setup.py install
+```
+
+### download dataset
+We will use some cat images: https://huggingface.co/datasets/diffusers/cat_toy_example .
+Let's first download it locally:
+```bash
+python download_dataset.py
+```
+
+### Install Stable Diffusion fine-tuning dependencies
+```bash
+pip install -r requirements.txt
+```
+
+### Initialize an [🤗Accelerate](https://github.com/huggingface/accelerate/) environment with:
+```bash
+export ACCELERATE_USE_IPEX=False
+```
+
+### Model Specific Setup
+* Set Tcmalloc Preload for better performance
+The tcmalloc should be built from the [General setup](#general-setup) section.
+```bash
+    export LD_PRELOAD="path/lib/libtcmalloc.so":$LD_PRELOAD
+```
+
+* Set IOMP preload for better performance
+IOMP should be installed in your conda env from the [General setup](#general-setup) section.
+```bash
+    export LD_PRELOAD=path/lib/libiomp5.so:$LD_PRELOAD
+```
+
+* Set ENV to use fp16 AMX if you are using a supported platform
+```bash
+    export DNNL_MAX_CPU_ISA=AVX512_CORE_AMX_FP16
+```
+
+## Quick Start Scripts
+
+| Script name | Description |
+|-------------|-------------|
+| `finetune.sh` | Fine-tuning using one socket for the specified precision (fp32, bf16, bf32 or fp16). |
+| `finetune_dist.sh` | Distributed fine-tuning for the specified precision (fp32, bf16, bf32 or fp16). |
+
+
+## Run the model
+
+Follow the instructions above to setup your bare metal environment, download and
+preprocess the dataset, and do the model specific setup. Once all the setup is done,
+the Model Zoo can be used to run a [quickstart script](#quick-start-scripts).
+Ensure that you have an enviornment variable set to point to an output directory.
+
+```bash
+# Clone the model zoo repo and set the MODEL_DIR
+git clone https://github.com/IntelAI/models.git
+cd models
+export MODEL_DIR=$(pwd)
+cd quickstart/diffusion/pytorch/stable_diffusion/training/cpu/
+
+# Env vars
+export OUTPUT_DIR=<path to an output directory>
+
+# Run fine-tuning
+bash finetune.sh
+```
+
+<!--- 80. License -->
+## License
+[LICENSE](https://github.com/IntelAI/models/blob/master/LICENSE)
diff --git a/quickstart/diffusion/pytorch/stable_diffusion/training/cpu/download_dataset.py b/quickstart/diffusion/pytorch/stable_diffusion/training/cpu/download_dataset.py
new file mode 100644
index 000000000..f93c8afff
--- /dev/null
+++ b/quickstart/diffusion/pytorch/stable_diffusion/training/cpu/download_dataset.py
@@ -0,0 +1,22 @@
+#
+# -*- coding: utf-8 -*-
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from huggingface_hub import snapshot_download
+
+local_dir = "./cat"
+snapshot_download("diffusers/cat_toy_example", local_dir=local_dir, repo_type="dataset", ignore_patterns=".gitattributes")
diff --git a/quickstart/diffusion/pytorch/stable_diffusion/training/cpu/finetune.sh b/quickstart/diffusion/pytorch/stable_diffusion/training/cpu/finetune.sh
new file mode 100644
index 000000000..5931a49a2
--- /dev/null
+++ b/quickstart/diffusion/pytorch/stable_diffusion/training/cpu/finetune.sh
@@ -0,0 +1,91 @@
+#!/usr/bin/env bash
+#
+# Copyright (c) 2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+if [ ! -e "${MODEL_DIR}/models/diffusion/pytorch/stable_diffusion/textual_inversion.py"  ]; then
+    echo "Could not find the script of textual_inversion.py. Please set environment variable '\${MODEL_DIR}'."
+    exit 1
+fi
+
+if [ -z "${OUTPUT_DIR}" ]; then
+  echo "The required environment variable OUTPUT_DIR has not been set"
+  exit 1
+fi
+
+# Create the output directory in case it doesn't already exist
+mkdir -p ${OUTPUT_DIR}
+
+ARGS=""
+if [ "$1" == "bf16" ]; then
+    ARGS="$ARGS --precision=bf16"
+    echo "### running bf16 datatype"
+elif [ "$1" == "fp16" ]; then
+    ARGS="$ARGS --precision=fp16"
+    echo "### running fp16 datatype"
+elif [ "$1" == "bf32" ]; then
+    ARGS="$ARGS --precision=bf32"
+    echo "### running bf32 datatype"
+elif [ "$1" == "fp32" ]; then
+    ARGS="$ARGS --precision=fp32"
+    echo "### running fp32 datatype"
+else
+    echo "The specified precision '$1' is unsupported."
+    echo "Supported precisions are: fp32, bf32, fp16, bf16"
+    exit 1
+fi
+
+
+CORES=`lscpu | grep Core | awk '{print $4}'`
+SOCKETS=`lscpu | grep Socket | awk '{print $2}'`
+TOTAL_CORES=`expr $CORES \* $SOCKETS`
+
+export DNNL_PRIMITIVE_CACHE_CAPACITY=1024
+export KMP_BLOCKTIME=1
+export KMP_AFFINITY=granularity=fine,compact,1,0
+
+PRECISION=$1
+
+export MODEL_NAME="stabilityai/stable-diffusion-2-1"
+export DATA_DIR="./cat"
+
+rm -rf ${OUTPUT_DIR}/stable_diffusion_finetune_log_${PRECISION}*
+
+python -m intel_extension_for_pytorch.cpu.launch \
+    --memory-allocator tcmalloc \
+    --ninstance 1 \
+    --nodes-list=0 \
+    --log_dir=${OUTPUT_DIR} \
+    --log_file_prefix="./stable_diffusion_finetune_log_${PRECISION}" \
+    ${MODEL_DIR}/models/diffusion/pytorch/stable_diffusion/textual_inversion.py \
+    --pretrained_model_name_or_path=$MODEL_NAME \
+    --train_data_dir=$DATA_DIR \
+    --learnable_property="object" \
+    --placeholder_token="\"<cat-toy>\"" --initializer_token="toy" \
+    --resolution=512 \
+    --train_batch_size=1 \
+    --gradient_accumulation_steps=1 \
+    -w 10 --max_train_steps=20 \
+    --learning_rate=2.0e-03 --scale_lr \
+    --lr_scheduler="constant" \
+    --lr_warmup_steps=0 \
+    --train-no-eval \
+    --ipex $ARGS
+
+# For the summary of results
+wait
+
+throughput=$(grep 'Throughput:' ${OUTPUT_DIR}/stable_diffusion_finetune_log_${PRECISION}* |sed -e 's/.*Throughput//;s/[^0-9.]//g')
+echo ""stable_diffusion";"finetune";"throughput";$1;${throughput}" | tee -a ${OUTPUT_DIR}/summary.log
diff --git a/quickstart/diffusion/pytorch/stable_diffusion/training/cpu/finetune_dist.sh b/quickstart/diffusion/pytorch/stable_diffusion/training/cpu/finetune_dist.sh
new file mode 100644
index 000000000..8604080ce
--- /dev/null
+++ b/quickstart/diffusion/pytorch/stable_diffusion/training/cpu/finetune_dist.sh
@@ -0,0 +1,137 @@
+#!/usr/bin/env bash
+#
+# Copyright (c) 2021 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+if [ ! -e "${MODEL_DIR}/models/diffusion/pytorch/stable_diffusion/textual_inversion.py"  ]; then
+    echo "Could not find the script of textual_inversion.py. Please set environment variable '\${MODEL_DIR}'."
+    exit 1
+fi
+
+if [ -z "${OUTPUT_DIR}" ]; then
+  echo "The required environment variable OUTPUT_DIR has not been set"
+  exit 1
+fi
+
+# Create the output directory in case it doesn't already exist
+mkdir -p ${OUTPUT_DIR}
+
+ARGS=""
+if [ "$1" == "bf16" ]; then
+    ARGS="$ARGS --precision=bf16"
+    echo "### running bf16 datatype"
+elif [ "$1" == "fp16" ]; then
+    ARGS="$ARGS --precision=fp16"
+    echo "### running fp16 datatype"
+elif [ "$1" == "bf32" ]; then
+    ARGS="$ARGS --precision=bf32"
+    echo "### running bf32 datatype"
+elif [ "$1" == "fp32" ]; then
+    ARGS="$ARGS --precision=fp32"
+    echo "### running fp32 datatype"
+else
+    echo "The specified precision '$1' is unsupported."
+    echo "Supported precisions are: fp32, bf32, fp16, bf16"
+    exit 1
+fi
+
+
+CORES=`lscpu | grep Core | awk '{print $4}'`
+SOCKETS=`lscpu | grep Socket | awk '{print $2}'`
+TOTAL_CORES=`expr $CORES \* $SOCKETS`
+NNODES=${NNODES:-1}
+HOSTFILE=${HOSTFILE:-./hostfile}
+NUM_RANKS=$(( NNODES * SOCKETS ))
+
+
+CORES_PER_INSTANCE=$CORES
+
+export DNNL_PRIMITIVE_CACHE_CAPACITY=1024
+export KMP_BLOCKTIME=1
+export KMP_AFFINITY=granularity=fine,compact,1,0
+
+PRECISION=$1
+
+export MODEL_NAME="stabilityai/stable-diffusion-2-1"
+export DATA_DIR="./cat"
+
+<< EOF
+#oneCCL settings
+export CCL_WORKER_COUNT=8
+export CCL_LOG_LEVEL=info
+export CCL_BF16=avx512bf
+export CCL_ATL_TRANSPORT=ofi
+export CCL_MNIC_COUNT=2
+export CCL_MNIC=local
+export CCL_MNIC_NAME=irdma1,irdma5
+export CCL_ALLREDUCE=ring
+export CCL_WORKER_COUNT=8
+
+for (( i = $SOCKETS; i < 2*$SOCKETS; i++ )); do  # pin CCL workers to HT
+  START_CORE=$(( i * CORES ))
+  for (( j = 0; j < $CCL_WORKER_COUNT; j++)); do
+   CCL_WORKER_AFFINITY="${CCL_WORKER_AFFINITY} $((START_CORE + j))"
+  done
+done
+
+export CCL_WORKER_AFFINITY=`echo ${CCL_WORKER_AFFINITY} | tr " " ","`
+
+
+#DDP settings
+export TORCH_CPP_LOG_LEVEL=INFO
+export TORCH_DISTRIBUTED_DEBUG=INFO
+export MASTER_ADDR=`head -1 hostfile`
+
+# Fabric settings
+export FI_PROVIDER=psm3
+export PSM3_IDENTIFY=1
+export PSM3_ALLOW_ROUTERS=1
+export PSM3_RDMA=1
+export PSM3_PRINT_STATS=0
+export PSM3_RV_MR_CACHE_SIZE=8192
+export PSM3_KASSIST_MODE=none
+#export PSM3_NIC='irdma*
+export FI_PSM3_CONN_TIMEOUT=100
+EOF
+
+rm -rf ${OUTPUT_DIR}/stable_diffusion_dist_finetune_log_${PRECISION}*
+
+oneccl_bindings_for_pytorch_path=$(python -c "import torch; import oneccl_bindings_for_pytorch; import os;  print(os.path.abspath(os.path.dirname(oneccl_bindings_for_pytorch.__file__)))")
+source $oneccl_bindings_for_pytorch_path/env/setvars.sh
+
+export FI_PROVIDER_PATH=$oneccl_bindings_for_pytorch_path/lib/prov
+
+python -m intel_extension_for_pytorch.cpu.launch \
+    --memory-allocator tcmalloc \
+    --distributed \
+    --nnodes ${NNODES} \
+    --hostfile ${HOSTFILE} \
+    --logical-cores-for-ccl --ccl_worker_count 8 \
+    ${MODEL_DIR}/models/diffusion/pytorch/stable_diffusion/textual_inversion.py \
+    --pretrained_model_name_or_path=$MODEL_NAME \
+    --train_data_dir=$DATA_DIR \
+    --learnable_property="object" \
+    --placeholder_token="\"<cat-toy>\"" --initializer_token="toy" \
+    --resolution=512 \
+    --train_batch_size=1 \
+    --gradient_accumulation_steps=1 \
+    --max_train_steps=200 \
+    --learning_rate=2.0e-03 --scale_lr \
+    --lr_scheduler="constant" \
+    --lr_warmup_steps=0 \
+    --save_as_full_pipeline \
+    --no_safe_serialization \
+    --output_dir="textual_inversion_${PRECISION}" \
+    --ipex $ARGS 2>&1 | tee ${OUTPUT_DIR}/stable_diffusion_dist_finetune_log_${PRECISION}.log
diff --git a/quickstart/diffusion/pytorch/stable_diffusion/training/cpu/hostfile b/quickstart/diffusion/pytorch/stable_diffusion/training/cpu/hostfile
new file mode 100644
index 000000000..7b9ad531d
--- /dev/null
+++ b/quickstart/diffusion/pytorch/stable_diffusion/training/cpu/hostfile
@@ -0,0 +1 @@
+127.0.0.1
diff --git a/quickstart/diffusion/pytorch/stable_diffusion/training/cpu/requirements.txt b/quickstart/diffusion/pytorch/stable_diffusion/training/cpu/requirements.txt
new file mode 100644
index 000000000..ac1aae2bd
--- /dev/null
+++ b/quickstart/diffusion/pytorch/stable_diffusion/training/cpu/requirements.txt
@@ -0,0 +1,3 @@
+ftfy
+tensorboard
+Jinja2
diff --git a/quickstart/diffusion/tensorflow/stable_diffusion/inference/cpu/accuracy.sh b/quickstart/diffusion/tensorflow/stable_diffusion/inference/cpu/accuracy.sh
new file mode 100755
index 000000000..13c987807
--- /dev/null
+++ b/quickstart/diffusion/tensorflow/stable_diffusion/inference/cpu/accuracy.sh
@@ -0,0 +1,102 @@
+#!/usr/bin/env bash
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+MODELS=${MODELS-$PWD}
+
+# Setting environment variables
+if [ -z "${TF_PATTERN_ALLOW_CTRL_DEPENDENCIES}" ]; then
+  echo "TF_PATTERN_ALLOW_CTRL_DEPENDENCIES is not set. By default, setting TF_PATTERN_ALLOW_CTRL_DEPENDENCIES=1 to allow control dependencies for fusions"
+  export TF_PATTERN_ALLOW_CTRL_DEPENDENCIES=1
+fi
+if [ -z "${TF_USE_LEGACY_KERAS}" ]; then
+  echo "TF_USE_LEGACY_KERAS is not set. By default, setting TF_USE_LEGACY_KERAS=1 to use Keras 2"
+  export TF_USE_LEGACY_KERAS=1
+fi
+if [ -z "${TF_USE_ADVANCED_CPU_OPS}" ]; then
+  echo "TF_USE_ADVANCED_CPU_OPS is not set. By default, setting TF_USE_ADVANCED_CPU_OPS=1 to enhace the overall performance"
+  export TF_USE_ADVANCED_CPU_OPS=1
+fi
+if [ -z "${TF_ONEDNN_ASSUME_FROZEN_WEIGHTS}" ]; then
+  echo "TF_ONEDNN_ASSUME_FROZEN_WEIGHTS is not set. By default, setting TF_ONEDNN_ASSUME_FROZEN_WEIGHTS=1 to perform weight caching as we're using a SavedModel"
+  export TF_ONEDNN_ASSUME_FROZEN_WEIGHTS=1
+fi
+
+if [ -z "${OUTPUT_DIR}" ]; then
+  echo "The required environment variable OUTPUT_DIR has not been set"
+  exit 1
+fi
+
+# Create the output directory in case it doesn't already exist
+mkdir -p ${OUTPUT_DIR}
+
+if [ -z "${PRECISION}" ]; then
+  echo "The required environment variable PRECISION has not been set"
+  echo "Please set PRECISION to either fp32, bfloat16, or fp16."
+  exit 1
+fi
+if [ $PRECISION != "fp32" ] && [ $PRECISION != "bfloat16" ] &&
+   [ $PRECISION != "fp16" ]; then
+  echo "The specified precision '${PRECISION}' is unsupported."
+  echo "Supported precisions is: fp32, bfloat16, fp16"
+  exit 1
+fi
+if [ $PRECISION == "bfloat16" ]; then
+  echo "Moving Gelu op to INFERLIST as we're using bfloat16 precision"
+  export TF_AUTO_MIXED_PRECISION_GRAPH_REWRITE_INFERLIST_ADD=Gelu
+fi
+if [ $PRECISION == "fp16" ]; then
+  echo "Adding Gelu,Mean,Sum,SquaredDifference op to INFERLIST as we're using fp16 precision"
+  export TF_AUTO_MIXED_PRECISION_GRAPH_REWRITE_INFERLIST_ADD=Gelu,Mean,Sum,SquaredDifference
+  export TF_AUTO_MIXED_PRECISION_GRAPH_REWRITE_DENYLIST_REMOVE=Mean,Sum,SquaredDifference
+fi
+
+if [ -z "${DATASET_DIR}" ]; then
+  echo "The required environment variable DATASET_DIR has not been set"
+  exit 1
+elif [ ! -d "${DATASET_DIR}" ]; then
+  echo "The DATASET_DIR '${DATASET_DIR}' does not exist"
+  exit 1
+fi
+
+MODE="inference"
+
+# If batch size env is not mentioned, then the workload will run with the default batch size.
+BATCH_SIZE="${BATCH_SIZE:-"1"}"
+
+# If number of steps is not mentioned, then the workload will run with the default value.
+NUM_STEPS="${NUM_STEPS:-"50"}"
+
+source "${MODELS}/quickstart/common/utils.sh"
+_command python ${MODELS}/benchmarks/launch_benchmark.py \
+  --model-name=stable_diffusion \
+  --precision ${PRECISION} \
+  --mode=${MODE} \
+  --framework tensorflow \
+  --output-dir ${OUTPUT_DIR} \
+  --batch-size ${BATCH_SIZE} \
+  --steps=${NUM_STEPS} \
+  --data-location ${DATASET_DIR} \
+  --accuracy-only \
+  $@ 2>&1 | tee ${OUTPUT_DIR}/stable_diffusion_${PRECISION}_${MODE}_bs${BATCH_SIZE}_accuracy.log
+
+if [[ $? == 0 ]]; then
+  echo "FID summary:"
+  cat ${OUTPUT_DIR}/stable_diffusion_${PRECISION}_${MODE}_bs${BATCH_SIZE}_accuracy.log | grep "FID score between original and generated images:" | sed -e "s/.* = //"
+  exit 0
+else
+  exit 1
+fi
diff --git a/quickstart/diffusion/tensorflow/stable_diffusion/inference/cpu/inference.sh b/quickstart/diffusion/tensorflow/stable_diffusion/inference/cpu/inference.sh
new file mode 100755
index 000000000..2c3f3f3f2
--- /dev/null
+++ b/quickstart/diffusion/tensorflow/stable_diffusion/inference/cpu/inference.sh
@@ -0,0 +1,53 @@
+#!/usr/bin/env bash
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+MODELS=${MODELS-$PWD}
+
+if [ -z "${OUTPUT_DIR}" ]; then
+  echo "The required environment variable OUTPUT_DIR has not been set"
+  exit 1
+fi
+
+# Create the output directory in case it doesn't already exist
+mkdir -p ${OUTPUT_DIR}
+
+if [ -z "${PRECISION}" ]; then
+  echo "The required environment variable PRECISION has not been set"
+  echo "Please set PRECISION to either fp32, bfloat16, or fp16."
+  exit 1
+fi
+if [ $PRECISION != "fp32" ] && [ $PRECISION != "bfloat16" ] &&
+   [ $PRECISION != "fp16" ]; then
+  echo "The specified precision '${PRECISION}' is unsupported."
+  echo "Supported precisions is: fp32, bfloat16, fp16"
+  exit 1
+fi
+
+MODE="inference"
+
+# If batch size env is not mentioned, then the workload will run with the default batch size.
+BATCH_SIZE="${BATCH_SIZE:-"1"}"
+
+source "${MODELS}/quickstart/common/utils.sh"
+_command python ${MODELS}/benchmarks/launch_benchmark.py \
+  --model-name=stable_diffusion \
+  --precision ${PRECISION} \
+  --mode=${MODE} \
+  --framework tensorflow \
+  --output-dir ${OUTPUT_DIR} \
+  --batch-size ${BATCH_SIZE} \
+  $@ \
diff --git a/quickstart/diffusion/tensorflow/stable_diffusion/inference/cpu/inference_realtime_multi_instance.sh b/quickstart/diffusion/tensorflow/stable_diffusion/inference/cpu/inference_realtime_multi_instance.sh
new file mode 100755
index 000000000..f27bd378a
--- /dev/null
+++ b/quickstart/diffusion/tensorflow/stable_diffusion/inference/cpu/inference_realtime_multi_instance.sh
@@ -0,0 +1,75 @@
+#!/usr/bin/env bash
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+MODELS=${MODELS-$PWD}
+
+if [ -z "${OUTPUT_DIR}" ]; then
+  echo "The required environment variable OUTPUT_DIR has not been set"
+  exit 1
+fi
+
+# Create the output directory in case it doesn't already exist
+mkdir -p ${OUTPUT_DIR}
+
+if [ -z "${PRECISION}" ]; then
+  echo "The required environment variable PRECISION has not been set"
+  echo "Please set PRECISION to either fp32, bfloat16, or fp16."
+  exit 1
+fi
+if [ $PRECISION != "fp32" ] && [ $PRECISION != "bfloat16" ] &&
+   [ $PRECISION != "fp16" ]; then
+  echo "The specified precision '${PRECISION}' is unsupported."
+  echo "Supported precisions is: fp32, bfloat16, fp16"
+  exit 1
+fi
+
+# If cores per instance env is not mentioned, then the workload will run with the default value.
+if [ -z "${CORES_PER_INSTANCE}" ]; then
+  CORES_PER_INSTANCE=4
+  echo "Runs an instance per ${CORES_PER_INSTANCE}"
+fi
+
+MODE="inference"
+
+# If batch size env is not mentioned, then the workload will run with the default batch size.
+BATCH_SIZE="${BATCH_SIZE:-"1"}"
+
+source "${MODELS}/quickstart/common/utils.sh"
+_command python ${MODELS}/benchmarks/launch_benchmark.py \
+  --model-name=stable_diffusion \
+  --precision ${PRECISION} \
+  --mode=${MODE} \
+  --framework tensorflow \
+  --output-dir ${OUTPUT_DIR} \
+  --batch-size ${BATCH_SIZE} \
+  --numa-cores-per-instance=${CORES_PER_INSTANCE} \
+  $@ \
+
+if [[ $? == 0 ]]; then
+  printf "Time taken by different instances:\n"
+  cat ${OUTPUT_DIR}/stable_diffusion_${PRECISION}_${MODE}_bs${BATCH_SIZE}_cores*_all_instances.log | grep 'Latency:' | sed -e s"/.*: //"
+  echo "Latency (min time):"
+  cat ${OUTPUT_DIR}/stable_diffusion_${PRECISION}_${MODE}_bs${BATCH_SIZE}_cores*_all_instances.log | grep 'Latency:' | sed -e s"/.*: //" | sort -n | head -1
+  printf "\nThroughput for different instances:\n"
+  cat ${OUTPUT_DIR}/stable_diffusion_${PRECISION}_${MODE}_bs${BATCH_SIZE}_cores*_all_instances.log | grep 'Avg Throughput:' | sed -e s"/.*: //"
+  echo "Throughput (total):"
+  grep 'Avg Throughput' ${OUTPUT_DIR}/stable_diffusion_${PRECISION}_${MODE}_bs${BATCH_SIZE}_cores*_all_instances.log | awk -F' ' '{sum+=$3;} END{print sum} '
+  exit 0
+else
+  exit 1
+fi
+
diff --git a/quickstart/diffusion/tensorflow/stable_diffusion/inference/cpu/inference_throughput_multi_instance.sh b/quickstart/diffusion/tensorflow/stable_diffusion/inference/cpu/inference_throughput_multi_instance.sh
new file mode 100755
index 000000000..f6ab3c31b
--- /dev/null
+++ b/quickstart/diffusion/tensorflow/stable_diffusion/inference/cpu/inference_throughput_multi_instance.sh
@@ -0,0 +1,122 @@
+#!/usr/bin/env bash
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+MODELS=${MODELS-$PWD}
+
+# Setting environment variables
+if [ -z "${TF_PATTERN_ALLOW_CTRL_DEPENDENCIES}" ]; then
+  echo "TF_PATTERN_ALLOW_CTRL_DEPENDENCIES is not set. By default, setting TF_PATTERN_ALLOW_CTRL_DEPENDENCIES=1 to allow control dependencies for fusions"
+  export TF_PATTERN_ALLOW_CTRL_DEPENDENCIES=1
+fi
+if [ -z "${TF_USE_LEGACY_KERAS}" ]; then
+  echo "TF_USE_LEGACY_KERAS is not set. By default, setting TF_USE_LEGACY_KERAS=1 to use Keras 2"
+  export TF_USE_LEGACY_KERAS=1
+fi
+if [ -z "${TF_USE_ADVANCED_CPU_OPS}" ]; then
+  echo "TF_USE_ADVANCED_CPU_OPS is not set. By default, setting TF_USE_ADVANCED_CPU_OPS=1 to enhace the overall performance"
+  export TF_USE_ADVANCED_CPU_OPS=1
+fi
+if [ -z "${TF_ONEDNN_ASSUME_FROZEN_WEIGHTS}" ]; then
+  echo "TF_ONEDNN_ASSUME_FROZEN_WEIGHTS is not set. By default, setting TF_ONEDNN_ASSUME_FROZEN_WEIGHTS=1 to perform weight caching as we're using a SavedModel"
+  export TF_ONEDNN_ASSUME_FROZEN_WEIGHTS=1
+fi
+
+if [ -z "${OUTPUT_DIR}" ]; then
+  echo "The required environment variable OUTPUT_DIR has not been set"
+  exit 1
+fi
+
+# Create the output directory in case it doesn't already exist
+mkdir -p ${OUTPUT_DIR}
+
+if [ -z "${PRECISION}" ]; then
+  echo "The required environment variable PRECISION has not been set"
+  echo "Please set PRECISION to either fp32, bfloat16, or fp16."
+  exit 1
+fi
+if [ $PRECISION != "fp32" ] && [ $PRECISION != "bfloat16" ] &&
+   [ $PRECISION != "fp16" ]; then
+  echo "The specified precision '${PRECISION}' is unsupported."
+  echo "Supported precisions is: fp32, bfloat16, fp16"
+  exit 1
+fi
+if [ $PRECISION == "bfloat16" ]; then
+  echo "Moving Gelu op to INFERLIST as we're using bfloat16 precision"
+  export TF_AUTO_MIXED_PRECISION_GRAPH_REWRITE_INFERLIST_ADD=Gelu
+fi
+if [ $PRECISION == "fp16" ]; then
+  echo "Adding Gelu,Mean,Sum,SquaredDifference op to INFERLIST as we're using fp16 precision"
+  export TF_AUTO_MIXED_PRECISION_GRAPH_REWRITE_INFERLIST_ADD=Gelu,Mean,Sum,SquaredDifference
+  export TF_AUTO_MIXED_PRECISION_GRAPH_REWRITE_DENYLIST_REMOVE=Mean,Sum,SquaredDifference
+fi
+
+MODE="inference"
+
+# If batch size env is not mentioned, then the workload will run with the default batch size.
+BATCH_SIZE="${BATCH_SIZE:-"1"}"
+
+# If number of steps is not mentioned, then the workload will run with the default value.
+NUM_STEPS="${NUM_STEPS:-"50"}"
+
+# If cores per instance env is not mentioned, then the workload will run with the default value.
+if [ -z "${CORES_PER_INSTANCE}" ]; then
+  # Get number of cores per instance
+  CORES_PER_SOCKET=`lscpu | grep 'Core(s) per socket' | awk '{print $4}'`
+  SOCKETS=`lscpu | grep Socket | awk '{print $2}'`
+  NUMAS=`lscpu | grep 'NUMA node(s)' | awk '{print $3}'`
+  CORES_PER_INSTANCE=`expr $CORES_PER_SOCKET \* $SOCKETS / $NUMAS`
+  NUM_INSTANCES=`expr $cores_per_socket / $CORES_PER_NUMA`
+
+  echo "CORES_PER_SOCKET: $CORES_PER_SOCKET"
+  echo "SOCKETS: $SOCKETS"
+  echo "NUMAS: $NUMAS"
+  echo "CORES_PER_INSTANCE: $CORES_PER_INSTANCE"
+fi
+
+if [ -z "${TF_THREAD_PINNING_MODE}" ]; then
+  echo "TF_THREAD_PINNING_MODE is not set. Default configuration of thread pinning and spinning settings"
+  export TF_THREAD_PINNING_MODE=none,$(($CORES_PER_INSTANCE-1)),400
+  echo "TF_THREAD_PINNING_MODE: $TF_THREAD_PINNING_MODE"
+fi
+
+source "${MODELS}/quickstart/common/utils.sh"
+_command python ${MODELS}/benchmarks/launch_benchmark.py \
+  --model-name=stable_diffusion \
+  --precision ${PRECISION} \
+  --mode=${MODE} \
+  --framework tensorflow \
+  --output-dir ${OUTPUT_DIR} \
+  --output-dir ${OUTPUT_DIR} \
+  --batch-size ${BATCH_SIZE} \
+  --steps=${NUM_STEPS} \
+  --numa-cores-per-instance=${CORES_PER_INSTANCE} \
+  $@ \
+
+if [[ $? == 0 ]]; then
+  printf "Time taken by different instances:\n"
+  cat ${OUTPUT_DIR}/stable_diffusion_${PRECISION}_${MODE}_bs${BATCH_SIZE}_cores*_all_instances.log | grep 'Latency:' | sed -e s"/.*: //"
+  echo "Latency (min time):"
+  cat ${OUTPUT_DIR}/stable_diffusion_${PRECISION}_${MODE}_bs${BATCH_SIZE}_cores*_all_instances.log | grep 'Latency:' | sed -e s"/.*: //" | sort -n | head -1
+  printf "\nThroughput for different instances:\n"
+  cat ${OUTPUT_DIR}/stable_diffusion_${PRECISION}_${MODE}_bs${BATCH_SIZE}_cores*_all_instances.log | grep 'Avg Throughput:' | sed -e s"/.*: //"
+  echo "Throughput (total):"
+  grep 'Avg Throughput' ${OUTPUT_DIR}/stable_diffusion_${PRECISION}_${MODE}_bs${BATCH_SIZE}_cores*_all_instances.log | awk -F' ' '{sum+=$3;} END{print sum} '
+  exit 0
+else
+  exit 1
+fi
+
diff --git a/quickstart/generative-ai/pytorch/stable_diffusion/inference/gpu/WSL2_ARC_BASE.md b/quickstart/generative-ai/pytorch/stable_diffusion/inference/gpu/WSL2_ARC_BASE.md
new file mode 100644
index 000000000..484d698bc
--- /dev/null
+++ b/quickstart/generative-ai/pytorch/stable_diffusion/inference/gpu/WSL2_ARC_BASE.md
@@ -0,0 +1,96 @@
+# Using Intel® Extension for PyTorch* Docker on Intel® Arc Series GPU 
+
+## Overview
+
+This document has instructions for running Intel® Extension for PyTorch Docker container on Intel® Arc Series GPU on Windows 11 with WSL2.
+
+## Requirements
+| Item | Detail |
+| ------ | ------- |
+| Host machine  | Intel® Arc Series A770 GPU |
+| Operating System | Windows 11 with [WSL2](https://learn.microsoft.com/en-us/windows/wsl/install) | 
+| Drivers | [GPU drivers](https://www.intel.com/content/www/us/en/products/docs/discrete-gpus/arc/software/drivers.html) |
+| Software | [Docker Desktop](https://docs.docker.com/desktop/wsl) |
+
+Here are additional pre-requisites to set ssh in order to access WSL2 remotely. 
+After having WSL2 installed on Windows environment, with a default user account on WSL2 distro,
+
+* On WSL, set up SSH Server:
+```bash
+sudo apt-get update -y && sudo apt-get upgrade -y
+sudo apt-get install openssh-server -y
+#set some port to be the SSH port for WSL:
+# you may need to remove any default port from the config (usually Port 22) for example as follows,
+echo "Port 2222" | sudo tee -a /etc/ssh/sshd_config
+# start server
+sudo service ssh start
+```
+* From Windows, open a port proxy:
+```bash
+# from administrator CMD:
+# get IP of WSL2 instance
+wsl hostname -I
+# open port proxy
+netsh interface portproxy add v4tov4 listenaddress=0.0.0.0 listenport=2222 connectaddress=<IP of WSL> connectport=2222
+# add firewall rule for the proxy port:
+netsh advfirewall firewall add rule name="Open Port 2222 for WSL2" dir=in action=allow protocol=TCP localport=2222
+#You should now be able to connect directly to the WSL instance using the Windows host IP address and the proxy port:
+ssh <Windows host IP>  -p 2222 -l <default WSL user>
+```
+**Notes:**
+* The IP address for WSL changes each time it is restarted. The proxy port will need to be updated.
+
+* In the case of multiple WSL distros installed, define the desired distro as the default before trying to connect
+## Run Using Docker
+
+### Set up Docker Image
+```bash
+docker pull intel/intel-extension-for-pytorch:2.1.10-xpu
+```
+### Run Docker Image
+
+Use the following command to start Intel® Extension for PyTorch* GPU container. You can use -v option to mount your local directory into the container.
+
+```bash
+IMAGE_NAME=intel/intel-extension-for-pytorch:2.1.10-xpu
+docker run \
+        -it --rm \
+        --device /dev/dxg \
+        --volume /usr/lib/wsl:/usr/lib/wsl \
+        IMAGE_NAME 
+```
+#### Verify if XPU is accessible from PyTorch:
+You are inside the container now. Run the following command to verify XPU is visible to PyTorch:
+```bash
+python -c "import torch;print(torch.device('xpu'))"
+```
+Sample output looks like below:
+```
+xpu
+```
+Then, verify that the XPU device is available to Intel® Extension for PyTorch\*:
+```bash
+python -c "import intel_extension_for_pytorch as ipex;print(ipex.xpu.is_available())"
+```
+Sample output looks like below:
+```
+True
+```
+Use the following command to check whether MKL is enabled as default:
+```bash
+python -c "import intel_extension_for_pytorch as ipex;print(ipex.xpu.has_onemkl())"
+```
+Sample output looks like below:
+```
+True
+```
+Finally, use the following command to show detailed information of detected device:
+```bash
+python -c "import torch; import intel_extension_for_pytorch as ipex; print(torch.__version__); print(ipex.__version__); [print(f'[{i}]: {ipex.xpu.get_device_properties(i)}') for i in range(ipex.xpu.device_count())];"
+```
+Sample output looks like below:
+```bash
+2.1.0a0+cxx11.abi
+2.1.10+xpu
+[0]: _DeviceProperties(name='Intel(R) Graphics [0x56a0]', platform_name='Intel(R) Level-Zero', dev_type='gpu, support_fp64=0, total_memory=13004MB, max_compute_units=512, gpu_eu_count=512)
+```
diff --git a/quickstart/generative-ai/pytorch/stable_diffusion/inference/gpu/WSL2_ARC_STABLE_DIFFUSION.md b/quickstart/generative-ai/pytorch/stable_diffusion/inference/gpu/WSL2_ARC_STABLE_DIFFUSION.md
new file mode 100644
index 000000000..9d7fa9f93
--- /dev/null
+++ b/quickstart/generative-ai/pytorch/stable_diffusion/inference/gpu/WSL2_ARC_STABLE_DIFFUSION.md
@@ -0,0 +1,64 @@
+# Running Stable Diffusion inference on Intel® Arc Series GPU using Intel® Extension for PyTorch*
+
+## Overview
+
+This document has instructions for running Stable Diffusion inference using Intel® Extension for PyTorch on Intel® Arc Series GPU on Windows 11 with WSL2.
+
+## Requirements
+| Item | Detail |
+| ------ | ------- |
+| Host machine  | Intel® Arc Series A770 GPU |
+| Operating System | Windows 11 with [WSL2](https://learn.microsoft.com/en-us/windows/wsl/install) | 
+| Drivers | [GPU drivers](https://www.intel.com/content/www/us/en/products/docs/discrete-gpus/arc/software/drivers.html) |
+| Software | [Docker Desktop](https://docs.docker.com/desktop/wsl) |
+
+Follow addtional pre-requisite steps mentioned [here](WSL2_ARC_BASE.md) when accessing WSL2 remotely. 
+
+## Quick Start Scripts
+
+| Script name | Description |
+|-------------|-------------|
+| `online_inference.sh` | Inference for specified precision(FP32 or FP16) with batch size 1 on Arc Series A770  |
+
+## Run Using Docker
+
+### Set up Docker Image
+```bash
+docker pull intel/generative-ai:pytorch-flex-gpu-stable-diffusion-inference
+```
+
+### Run Docker Image
+The stable diffusion inference container includes scripts,model and libraries need to run FP32 and FP16 inference. To run the `online_inference.sh` quickstart script using this container, you will need to provide an output directory where log files will be written. 
+
+```bash
+export IMAGE_NAME=intel/generative-ai:pytorch-flex-gpu-stable-diffusion-inference
+export PRECISION=<provide either fp32 or fp16>
+export OUTPUT_DIR=<path to output directory>
+export SCRIPT=quickstart/online_inference.sh
+
+DOCKER_ARGS="--rm -it"
+
+docker run \
+  --device /dev/dxg \
+  --env PRECISION=${PRECISION} \
+  --env OUTPUT_DIR=${OUTPUT_DIR} \
+  --env http_proxy=${http_proxy} \
+  --env https_proxy=${https_proxy} \
+  --env no_proxy=${no_proxy} \
+  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
+  --volume /usr/lib/wsl:/usr/lib/wsl  \
+  ${DOCKER_ARGS} \
+  ${IMAGE_NAME} \
+  /bin/bash $SCRIPT
+  ```
+  
+## Documentation and Sources
+
+[GitHub* Repository](https://github.com/IntelAI/models/tree/master/docker/flex-gpu)
+
+## Support
+Support for Intel® Extension for PyTorch* is found via the [Intel® AI Analytics Toolkit.](https://www.intel.com/content/www/us/en/developer/tools/oneapi/ai-analytics-toolkit.html#gs.qbretz) Additionally, the Intel® Extension for PyTorch* team tracks both bugs and enhancement requests using [GitHub issues](https://github.com/intel/intel-extension-for-pytorch/issues). Before submitting a suggestion or bug report, please search the GitHub issues to see if your issue has already been reported.
+
+## License Agreement
+
+LEGAL NOTICE: By accessing, downloading or using this software and any required dependent software (the “Software Package”), you agree to the terms and conditions of the software license agreements for the Software Package, which may also include notices, disclaimers, or license terms for third party software included with the Software Package. Please refer to the [license file](https://github.com/IntelAI/models/tree/master/third_party) for additional details.
diff --git a/quickstart/image_recognition/pytorch/resnet50/inference/cpu/DEVCATALOG.md b/quickstart/image_recognition/pytorch/resnet50/inference/cpu/DEVCATALOG.md
new file mode 100644
index 000000000..280152807
--- /dev/null
+++ b/quickstart/image_recognition/pytorch/resnet50/inference/cpu/DEVCATALOG.md
@@ -0,0 +1,103 @@
+# PyTorch ResNet50 inference
+
+## Description 
+This document has instructions for running ResNet50 inference using Intel Extension for PyTorch.
+
+## Pull Command
+
+Docker image based on CentOS Stream8
+```
+docker pull intel/image-recognition:centos-pytorch-cpu-resnet50-inference
+```
+Docker image based on Ubuntu 22.04
+```
+docker pull intel/image-recognition:ubuntu-pytorch-cpu-resnet50-inference
+```
+
+## Quick Start Scripts
+| Script name | Description |
+|-------------|-------------|
+| `inference_realtime.sh` | Runs multi instance realtime inference using 4 cores per instance with synthetic data for the specified precision (fp32, avx-fp32, int8, avx-int8, bf32 or bf16). |
+| `inference_throughput.sh` | Runs multi instance batch inference using 1 instance per socket with synthetic data for the specified precision (fp32, avx-fp32, int8, avx-int8, bf32 or bf16). |
+| `accuracy.sh` | Measures the inference accuracy (providing a `DATASET_DIR` environment variable is required) for the specified precision (fp32, avx-fp32, int8, avx-int8, bf32 or bf16). |
+
+**Note:** The `avx-int8` and `avx-fp32` precisions run the same scripts as `int8` and `fp32`,except that the `DNNL_MAX_CPU_ISA` environment variable is unset. The environment variable is otherwise set to `DNNL_MAX_CPU_ISA=AVX512_CORE_AMX`.
+
+## Datasets
+The [ImageNet](http://www.image-net.org/) validation dataset is used.
+
+Download and extract the ImageNet2012 dataset from http://www.image-net.org/, then move validation images to labeled subfolders, using [the valprep.sh shell script](https://raw.githubusercontent.com/soumith/imagenetloader.torch/master/valprep.sh)
+
+A after running the data prep script, your folder structure should look something like this:
+
+```
+imagenet
+└── val
+    ├── ILSVRC2012_img_val.tar
+    ├── n01440764
+    │   ├── ILSVRC2012_val_00000293.JPEG
+    │   ├── ILSVRC2012_val_00002138.JPEG
+    │   ├── ILSVRC2012_val_00003014.JPEG
+    │   ├── ILSVRC2012_val_00006697.JPEG
+    │   └── ...
+    └── ...
+```
+The folder that contains the `val` directory should be set as the `DATASET_DIR` (for example: `export DATASET_DIR=/home/<user>/imagenet`).
+
+## Docker Run
+(Optional) Export related proxy into docker environment.
+```bash
+export DOCKER_RUN_ENVS="-e ftp_proxy=${ftp_proxy} \
+  -e FTP_PROXY=${FTP_PROXY} -e http_proxy=${http_proxy} \
+  -e HTTP_PROXY=${HTTP_PROXY} -e https_proxy=${https_proxy} \
+  -e HTTPS_PROXY=${HTTPS_PROXY} -e no_proxy=${no_proxy} \
+  -e NO_PROXY=${NO_PROXY} -e socks_proxy=${socks_proxy} \
+  -e SOCKS_PROXY=${SOCKS_PROXY}"
+```
+To run ResNet inference, set environment variables to specify the dataset directory, precision and an output directory. 
+```bash
+export OS=<provide either centos or ubuntu>
+export DATASET_DIR=<path to the dataset>
+export OUTPUT_DIR=<directory where log files will be written>
+export PRECISION=<specify the precision>
+export SCRIPT=quickstart/<specify the quickstart script>
+IMAGE_NAME=intel/image-recognition:${OS}-pytorch-cpu-resnet50-inference
+DOCKER_ARGS="--privileged --init -it"
+WORKDIR=/workspace/pytorch-resnet50-inference
+
+docker run --rm \
+  --env PRECISION=${PRECISION} \
+  --env OUTPUT_DIR=${OUTPUT_DIR} \
+  --env DATASET_DIR=${DATASET_DIR} \
+  --env http_proxy=${http_proxy} \
+  --env https_proxy=${https_proxy} \
+  --env no_proxy=${no_proxy} \
+  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
+  --volume ${DATASET_DIR}:${DATASET_DIR} \
+  ${DOCKER_RUN_ENVS} \
+  --shm-size 8G \
+  -w ${WORKDIR} \
+  ${DOCKER_ARGS} \
+  $IMAGE_NAME \
+  /bin/bash $SCRIPT
+  ```
+
+## Documentation and Sources
+#### Get Started​
+[Docker* Repository](https://hub.docker.com/r/intel/image-recognition)
+
+[Main GitHub*](https://github.com/IntelAI/models)
+
+[Release Notes](https://github.com/IntelAI/models/releases)
+
+[Get Started Guide](https://github.com/IntelAI/models/blob/master/quickstart/quickstart/image_recognition/pytorch/resnet50/inference/cpu/DEVCATALOG.md)
+
+#### Code Sources
+[Dockerfile](https://github.com/IntelAI/models/tree/master/docker/pyt-cpu)
+
+[Report Issue](https://community.intel.com/t5/Intel-Optimized-AI-Frameworks/bd-p/optimized-ai-frameworks)
+
+## License Agreement
+LEGAL NOTICE: By accessing, downloading or using this software and any required dependent software (the “Software Package”), you agree to the terms and conditions of the software license agreements for the Software Package, which may also include notices, disclaimers, or license terms for third party software included with the Software Package. Please refer to the [license](https://github.com/IntelAI/models/tree/master/third_party) file for additional details.
+
+[View All Containers and Solutions 🡢](https://www.intel.com/content/www/us/en/developer/tools/software-catalog/containers.html?s=Newest)
diff --git a/quickstart/image_recognition/pytorch/resnet50/inference/cpu/README.md b/quickstart/image_recognition/pytorch/resnet50/inference/cpu/README.md
index 1d0b34b05..51689ce49 100644
--- a/quickstart/image_recognition/pytorch/resnet50/inference/cpu/README.md
+++ b/quickstart/image_recognition/pytorch/resnet50/inference/cpu/README.md
@@ -1,141 +1,109 @@
-<!--- 0. Title -->
 # PyTorch ResNet50 inference
 
-<!-- 10. Description -->
 ## Description
 
 This document has instructions for running ResNet50 inference using
 Intel-optimized PyTorch.
 
-## Bare Metal
+## Dataset
 
-### General setup
+The [ImageNet](http://www.image-net.org/) validation dataset is used when
+testing accuracy. The inference scripts use synthetic data, so no dataset
+is needed.
 
-Follow [link](/docs/general/pytorch/BareMetalSetup.md) to install Miniconda and build Pytorch, IPEX, TorchVison and Jemalloc.
-
-### Model Specific Setup
-
-* [Datasets](#dataset)
-
-```bash
-    export DATASET_DIR=#Where_to_save_Dataset
-```
-
-* Setup the Output dir to store the log
-
-```bash
-    export OUTPUT_DIR=$Where_to_save_log
-```
-
-* Setup runnning precision
-
-```bash
-    export PRECISION=$Data_type(fp32, int8, avx-int8, bf16, bf32 or fp16)
-```
-
-* Set Jemalloc Preload for better performance
-
-The jemalloc should be built from the [General setup](#general-setup) section.
-
-```bash
-    export LD_PRELOAD="path/lib/libjemalloc.so":$LD_PRELOAD
-    export MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
-```
-
-* Set IOMP preload for better performance
-
-IOMP should be installed in your conda env from the [General setup](#general-setup) section.
-
-```bash
-    export LD_PRELOAD=path/lib/libiomp5.so:$LD_PRELOAD
-```
-
-* Set ENV to use fp16 AMX if you are using a supported platform
-
-```bash
-    export DNNL_MAX_CPU_ISA=AVX512_CORE_AMX_FP16
-```
-
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| `inference_realtime.sh` | Runs multi instance realtime inference using 4 cores per instance with synthetic data for the specified precision (fp32, avx-fp32, int8, avx-int8, bf16, bf32, or fp16). |
-| `inference_throughput.sh` | Runs multi instance batch inference using 1 instance per socket with synthetic data for the specified precision (fp32, avx-fp32, int8, avx-int8, bf16,  bf32, or fp16). |
-| `accuracy.sh` | Measures the inference accuracy (providing a `DATASET_DIR` environment variable is required) for the specified precision (fp32, avx-fp32, int8, avx-int8, bf16,  bf32, or fp16). |
-
-> Note: The `avx-int8` and `avx-fp32` precisions run the same scripts as `int8` and `fp32`, except that the
-> `DNNL_MAX_CPU_ISA` environment variable is unset. The environment variable is
-> otherwise set to `DNNL_MAX_CPU_ISA=AVX512_CORE_AMX`.
-
-## Calibration
-If user needs to re-do the calibration, here is the reference command:
-```
-DATASET_DIR=${DATASET_DIR} PRECISION=int8 CONFIG_FILE=${INT8_CONFIG_FILE} bash calibration.sh
-```
-
-## Datasets
-
-### ImageNet
-
-The [ImageNet](http://www.image-net.org/) validation dataset is used to run ResNet50
-accuracy tests.
-
-Download and extract the ImageNet2012 dataset from [http://www.image-net.org/](http://www.image-net.org/),
+Download and extract the ImageNet2012 dataset from http://www.image-net.org/,
 then move validation images to labeled subfolders, using
 [the valprep.sh shell script](https://raw.githubusercontent.com/soumith/imagenetloader.torch/master/valprep.sh)
 
-A after running the data prep script, your folder structure should look something like this:
+The accuracy script looks for a folder named `val`, so after running the
+data prep script, your folder structure should look something like this:
 
-```txt
+```
 imagenet
 └── val
     ├── ILSVRC2012_img_val.tar
     ├── n01440764
-    │   ├── ILSVRC2012_val_00000293.JPEG
-    │   ├── ILSVRC2012_val_00002138.JPEG
-    │   ├── ILSVRC2012_val_00003014.JPEG
-    │   ├── ILSVRC2012_val_00006697.JPEG
-    │   └── ...
+    │   ├── ILSVRC2012_val_00000293.JPEG
+    │   ├── ILSVRC2012_val_00002138.JPEG
+    │   ├── ILSVRC2012_val_00003014.JPEG
+    │   ├── ILSVRC2012_val_00006697.JPEG
+    │   └── ...
     └── ...
 ```
-
 The folder that contains the `val` directory should be set as the
-`DATASET_DIR` (for example: `export DATASET_DIR=/home/<user>/imagenet`).
+`DATASET_DIR` when running accuracy
+(for example: `export DATASET_DIR=/home/<user>/imagenet`).
 
-**Note:** If you don't run `accuracy.sh`, you don't need to set the environment variable `DATASET_DIR`. Performance is measured with dummy data.
+## Quick Start Scripts
 
-## Docker
+| Script name | Description |
+|-------------|-------------|
+| [`inference_realtime.sh`](/quickstart/image_recognition/pytorch/resnet50/inference/cpu/inference_realtime.sh) | Runs inference using synthetic data (batch_size=1) for the specified precision (fp32, avx-fp32, int8, avx-int8, bf16, or bf32). |
+| [`inference_throughput.sh`](/quickstart/image_recognition/pytorch/resnet50/inference/cpu/inference_throughput.sh) | Runs inference to get the throughput using synthetic data for the specified precision (fp32, avx-fp32, int8, avx-int8, bf16, or bf32). |
+| [`accuracy.sh`](/quickstart/image_recognition/pytorch/resnet50/inference/cpu/accuracy.sh) | Measures the model accuracy (batch_size=128) for the specified precision (fp32, avx-fp32, int8, avx-int8, bf16, or bf32). |
 
-Make sure, you have all the requirements pre-setup in your Container as the [Bare Metal](#bare-metal) Setup section.
+> Note: The `avx-int8` and `avx-fp32` precisions run the same scripts as `int8` and `fp32`, except that the
+> `DNNL_MAX_CPU_ISA` environment variable is unset. The environment variable is
+> otherwise set to `DNNL_MAX_CPU_ISA=AVX512_CORE_AMX`.
+* Set ENV to use AMX:
+  ```
+  export DNNL_MAX_CPU_ISA=AVX512_CORE_AMX
+  ```
+
+## Run the Model on Bare Metal
 
-### Download dataset
+Follow [link](/docs/general/pytorch/BareMetalSetup.md) to install Miniconda and build Pytorch, IPEX, TorchVison and Jemalloc.
 
-Refer to the corresponding Bare Mental Section to download the dataset.
+* Set Jemalloc Preload for better performance
 
-### Running CMD
+  After [Jemalloc setup](/docs/general/pytorch/BareMetalSetup.md#build-jemalloc), set the following environment variables.
+  ```
+  export LD_PRELOAD="<path to the jemalloc directory>/lib/libjemalloc.so":$LD_PRELOAD
+  export MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+  ```
 
-```bash
-DATASET_DIR=$dir/imagenet
-OUTPUT_DIR=$Where_to_save_the_log
-docker run \
-  --env DATASET_DIR=${DATASET_DIR} \
-  --env OUTPUT_DIR=${OUTPUT_DIR} \
-  --env http_proxy=${http_proxy} \
-  --env https_proxy=${https_proxy} \
-  --volume ${DATASET_DIR}:${DATASET_DIR} \
-  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
-  --privileged --init -t \
-  intel/image_recognition:pytorch-latest-resnet50-inference \
-  /bin/bash quickstart/<script name>.sh <data_type>
-```
+* Set IOMP preload for better performance
 
-If you are new to docker and are running into issues with the container,
-see [this document](https://github.com/IntelAI/models/tree/master/docs/general/docker.md)
-for troubleshooting tips.
+  IOMP should be installed in your conda env. Set the following environment variables.
+  ```
+  pip install packaging intel-openmp
+  export LD_PRELOAD=<path to the intel-openmp directory>/lib/libiomp5.so:$LD_PRELOAD
+  ```
+
+* Follow the instructions to setup your bare metal environment on either Linux or Windows systems. Once all the setup is done,
+  the Intel® AI Reference Models can be used to run a [quickstart script](#quick-start-scripts).
+  Ensure that you have a clone of the [Intel® AI Reference Models Github repository](https://github.com/IntelAI/models) and navigate to models directory.
+  ```
+  git clone https://github.com/IntelAI/models.git
+  cd models
+  ```
+
+* Run the model:
+    
+    #### Set the environment variables
+    ```bash
+    export OUTPUT_DIR=<path to the directory where log files will be written>
+    export PRECISION=<select one precision: fp32, avx-fp32, int8, bf32, avx-int8, or bf16>
+
+    # Optional environemnt variables:
+    export BATCH_SIZE=<set a value for batch size, else it will run with default batch size>
+
+    # [optional] Compile model with PyTorch Inductor backend
+    export TORCH_INDUCTOR=1
+    ```
+
+    #### Run quickstart script:
+    ```
+    NOTE: `inference_realtime.sh` and `inference_throughput.sh` runs with synthetic data
+    ./quickstart/image_recognition/pytorch/resnet50/inference/cpu/<script.sh>
+    ```
+
+    #### `accuracy.sh` script runs with the Imagenet Dataset:
+    ```
+    export DATASET_DIR=<path to the Imagenet Dataset>
+    ./quickstart/image_recognition/pytorch/resnet50/inference/cpu/accuracy.sh
+    ```
 
-<!--- 80. License -->
 ## License
 
 [LICENSE](/LICENSE)
diff --git a/quickstart/image_recognition/pytorch/resnet50/inference/cpu/accuracy.sh b/quickstart/image_recognition/pytorch/resnet50/inference/cpu/accuracy.sh
index 12aeb2f0e..9a98e659d 100755
--- a/quickstart/image_recognition/pytorch/resnet50/inference/cpu/accuracy.sh
+++ b/quickstart/image_recognition/pytorch/resnet50/inference/cpu/accuracy.sh
@@ -42,7 +42,7 @@ fi
 
 if [ -z "${PRECISION}" ]; then
   echo "The required environment variable PRECISION has not been set"
-  echo "Please set PRECISION to fp32, avx-fp32, int8, avx-int8, or bf16."
+  echo "Please set PRECISION to fp32, avx-fp32, int8, bf32, avx-int8, or bf16."
   exit 1
 fi
 
@@ -83,7 +83,7 @@ elif [[ $PRECISION == "fp32" || $PRECISION == "avx-fp32" ]]; then
     echo "running fp32 path"
 else
     echo "The specified precision '${PRECISION}' is unsupported."
-    echo "Supported precisions are: fp32, avx-fp32, bf16, int8, and avx-int8"
+    echo "Supported precisions are: fp32, avx-fp32, bf16, bf32, int8, and avx-int8"
     exit 1
 fi
 
@@ -93,6 +93,7 @@ if [ ${WEIGHT_SHAREING} ]; then
   weight_sharing=true
 fi
 
+TORCH_INDUCTOR=${TORCH_INDUCTOR:-"0"}
 if [ "$weight_sharing" = true ]; then
     CORES=`lscpu | grep Core | awk '{print $4}'`
     SOCKETS=`lscpu | grep Socket | awk '{print $2}'`
@@ -125,9 +126,8 @@ if [ "$weight_sharing" = true ]; then
         --use-multi-stream-module \
         --instance-number 0
 
-else
-
-  python -m intel_extension_for_pytorch.cpu.launch \
+elif [[ "0" == ${TORCH_INDUCTOR} ]];then
+    python -m intel_extension_for_pytorch.cpu.launch \
       --memory-allocator jemalloc \
       ${MODEL_DIR}/models/image_recognition/pytorch/common/main.py \
       $ARGS \
@@ -135,6 +135,16 @@ else
       --pretrained \
       -j 0 \
       -b $BATCH_SIZE 2>&1 | tee ${OUTPUT_DIR}/resnet50_accuracy_log_${PRECISION}.log
+else
+    echo "Running RN50 inference with torch.compile inductor backend."
+    python -m intel_extension_for_pytorch.cpu.launch \
+      --memory-allocator jemalloc \
+      ${MODEL_DIR}/models/image_recognition/pytorch/common/main.py \
+      $ARGS \
+      --inductor \
+      --pretrained \
+      -j 0 \
+      -b $BATCH_SIZE 2>&1 | tee ${OUTPUT_DIR}/resnet50_accuracy_log_${PRECISION}.log
 fi
 wait
 
diff --git a/quickstart/image_recognition/pytorch/resnet50/inference/cpu/bfloat16/.docs/baremetal.md b/quickstart/image_recognition/pytorch/resnet50/inference/cpu/bfloat16/.docs/baremetal.md
deleted file mode 100644
index 2a2e7bc0b..000000000
--- a/quickstart/image_recognition/pytorch/resnet50/inference/cpu/bfloat16/.docs/baremetal.md
+++ /dev/null
@@ -1,44 +0,0 @@
-<!--- 50. Bare Metal -->
-## Bare Metal
-
-Follow the instructions to setup your bare metal environment on either Linux or Windows systems, download and
-preprocess the dataset, and do the model specific setup. Once all the setup is done,
-the Model Zoo can be used to run a [quickstart script](#quick-start-scripts).
-Ensure that you have enviornment variables set to point to the dataset directory
-and an output directory.
-
-Follow the instructions below to clone the Model Zoo repo before running the model on Linux or Windows:
-```
-# Clone the Model Zoo repo 
-git clone https://github.com/IntelAI/models.git
-cd models
-```
-
-### Run on Linux
-Install the following prerequisites in your environment:
-* Python 3
-* [intel-extension-for-pytorch](https://github.com/intel/intel-extension-for-pytorch)
-* [torchvision==v0.6.1](https://github.com/pytorch/vision/tree/v0.6.1)
-* numactl
-
-```
-# Env vars
-export DATASET_DIR=<path to the Imagenet Dataset>
-export OUTPUT_DIR=<path to the directory where log files will be written>
-
-# Run a quickstart script (for example, bfloat16 batch inference)
-bash quickstart/image_recognition/pytorch/resnet50/inference/cpu/bfloat16/fp32_batch_inference.sh 
-```
-
-### Run the model on Windows
-If not already setup, please follow instructions for [environment setup on Windows](/docs/general/Windows.md).
-
-Using Windows CMD.exe, run:
-```
-# Env vars
-set DATASET_DIR=<path to the Imagenet Dataset>
-set OUTPUT_DIR=<path to the directory where log files will be written>
-
-#Run a quickstart script for bfloat16 precision(for example, BFloat16 batch inference)
-bash quickstart\image_recognition\pytorch\resnet50\inference\cpu\bfloat16\fp32_batch_inference.sh 
-```
diff --git a/quickstart/image_recognition/pytorch/resnet50/inference/cpu/bfloat16/.docs/datasets.md b/quickstart/image_recognition/pytorch/resnet50/inference/cpu/bfloat16/.docs/datasets.md
deleted file mode 100644
index 06e37c796..000000000
--- a/quickstart/image_recognition/pytorch/resnet50/inference/cpu/bfloat16/.docs/datasets.md
+++ /dev/null
@@ -1,29 +0,0 @@
-<!--- 30. Datasets -->
-## Datasets
-
-The [ImageNet](http://www.image-net.org/) validation dataset is used when
-testing accuracy. The inference scripts use synthetic data, so no dataset
-is needed.
-
-Download and extract the ImageNet2012 dataset from http://www.image-net.org/,
-then move validation images to labeled subfolders, using
-[the valprep.sh shell script](https://raw.githubusercontent.com/soumith/imagenetloader.torch/master/valprep.sh)
-
-The accuracy script looks for a folder named `val`, so after running the
-data prep script, your folder structure should look something like this:
-
-```
-imagenet
-└── val
-    ├── ILSVRC2012_img_val.tar
-    ├── n01440764
-    │   ├── ILSVRC2012_val_00000293.JPEG
-    │   ├── ILSVRC2012_val_00002138.JPEG
-    │   ├── ILSVRC2012_val_00003014.JPEG
-    │   ├── ILSVRC2012_val_00006697.JPEG
-    │   └── ...
-    └── ...
-```
-The folder that contains the `val` directory should be set as the
-`DATASET_DIR` when running accuracy
-(for example: `export DATASET_DIR=/home/<user>/imagenet`).
diff --git a/quickstart/image_recognition/pytorch/resnet50/inference/cpu/bfloat16/.docs/description.md b/quickstart/image_recognition/pytorch/resnet50/inference/cpu/bfloat16/.docs/description.md
deleted file mode 100644
index 7efc83bbd..000000000
--- a/quickstart/image_recognition/pytorch/resnet50/inference/cpu/bfloat16/.docs/description.md
+++ /dev/null
@@ -1,4 +0,0 @@
-<!-- 10. Description -->
-## Description
-
-This document has instructions for running <model name> <precision> <mode>.
\ No newline at end of file
diff --git a/quickstart/image_recognition/pytorch/resnet50/inference/cpu/bfloat16/.docs/license.md b/quickstart/image_recognition/pytorch/resnet50/inference/cpu/bfloat16/.docs/license.md
deleted file mode 100644
index a60f169f4..000000000
--- a/quickstart/image_recognition/pytorch/resnet50/inference/cpu/bfloat16/.docs/license.md
+++ /dev/null
@@ -1,4 +0,0 @@
-<!--- 80. License -->
-## License
-
-[LICENSE](/LICENSE)
diff --git a/quickstart/image_recognition/pytorch/resnet50/inference/cpu/bfloat16/.docs/quickstart.md b/quickstart/image_recognition/pytorch/resnet50/inference/cpu/bfloat16/.docs/quickstart.md
deleted file mode 100644
index 70f4cad05..000000000
--- a/quickstart/image_recognition/pytorch/resnet50/inference/cpu/bfloat16/.docs/quickstart.md
+++ /dev/null
@@ -1,9 +0,0 @@
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| [`bf16_online_inference.sh`](bf16_online_inference.sh) | Runs online inference using synthetic data (batch_size=1). |
-| [`bf16_batch_inference.sh`](bf16_batch_inference.sh) | Runs batch inference using synthetic data (batch_size=128). |
-| [`bf16_accuracy.sh`](bf16_accuracy.sh) | Measures the model accuracy (batch_size=128). |
-
diff --git a/quickstart/image_recognition/pytorch/resnet50/inference/cpu/bfloat16/.docs/title.md b/quickstart/image_recognition/pytorch/resnet50/inference/cpu/bfloat16/.docs/title.md
deleted file mode 100644
index 6416ff7ff..000000000
--- a/quickstart/image_recognition/pytorch/resnet50/inference/cpu/bfloat16/.docs/title.md
+++ /dev/null
@@ -1,2 +0,0 @@
-<!--- 0. Title -->
-# <model name> <precision> <mode>
diff --git a/quickstart/image_recognition/pytorch/resnet50/inference/cpu/bfloat16/README.md b/quickstart/image_recognition/pytorch/resnet50/inference/cpu/bfloat16/README.md
deleted file mode 100644
index 8188a00c5..000000000
--- a/quickstart/image_recognition/pytorch/resnet50/inference/cpu/bfloat16/README.md
+++ /dev/null
@@ -1,99 +0,0 @@
-<!--- 0. Title -->
-# ResNet50 BFloat16 inference
-
-<!-- 10. Description -->
-## Description
-
-This document has instructions for running ResNet50 BFloat16 inference.
-
-<!--- 30. Datasets -->
-## Datasets
-
-The [ImageNet](http://www.image-net.org/) validation dataset is used when
-testing accuracy. The inference scripts use synthetic data, so no dataset
-is needed.
-
-Download and extract the ImageNet2012 dataset from http://www.image-net.org/,
-then move validation images to labeled subfolders, using
-[the valprep.sh shell script](https://raw.githubusercontent.com/soumith/imagenetloader.torch/master/valprep.sh)
-
-The accuracy script looks for a folder named `val`, so after running the
-data prep script, your folder structure should look something like this:
-
-```
-imagenet
-└── val
-    ├── ILSVRC2012_img_val.tar
-    ├── n01440764
-    │   ├── ILSVRC2012_val_00000293.JPEG
-    │   ├── ILSVRC2012_val_00002138.JPEG
-    │   ├── ILSVRC2012_val_00003014.JPEG
-    │   ├── ILSVRC2012_val_00006697.JPEG
-    │   └── ...
-    └── ...
-```
-The folder that contains the `val` directory should be set as the
-`DATASET_DIR` when running accuracy
-(for example: `export DATASET_DIR=/home/<user>/imagenet`).
-
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| [`bf16_online_inference.sh`](bf16_online_inference.sh) | Runs online inference using synthetic data (batch_size=1). |
-| [`bf16_batch_inference.sh`](bf16_batch_inference.sh) | Runs batch inference using synthetic data (batch_size=128). |
-| [`bf16_accuracy.sh`](bf16_accuracy.sh) | Measures the model accuracy (batch_size=128). |
-
-
-<!--- 50. Bare Metal -->
-## Bare Metal
-
-Follow the instructions to setup your bare metal environment on either Linux or Windows systems, download and
-preprocess the dataset, and do the model specific setup. Once all the setup is done,
-the Model Zoo can be used to run a [quickstart script](#quick-start-scripts).
-Ensure that you have enviornment variables set to point to the dataset directory
-and an output directory.
-
-Follow the instructions below to clone the Model Zoo repo before running the model on Linux or Windows:
-```
-# Clone the Model Zoo repo 
-git clone https://github.com/IntelAI/models.git
-cd models
-```
-
-### Run on Linux
-Install the following prerequisites in your environment:
-* Python 3
-* [intel-extension-for-pytorch](https://github.com/intel/intel-extension-for-pytorch)
-* [torchvision==v0.6.1](https://github.com/pytorch/vision/tree/v0.6.1)
-* numactl
-* Run the model:
-
-  ```
-  # Env vars
-  export DATASET_DIR=<path to the Imagenet Dataset>
-  export OUTPUT_DIR=<path to the directory where log files will be written>
-
-  # Run a quickstart script (for example, bfloat16 batch inference)
-  bash quickstart/image_recognition/pytorch/resnet50/inference/cpu/bfloat16/fp32_batch_inference.sh 
-  ```
-
-### Run on Windows
-If not already setup, please follow instructions for [environment setup on Windows](/docs/general/Windows.md).
-
-Using Windows CMD.exe, run:
-```
-# Env vars
-set DATASET_DIR=<path to the Imagenet Dataset>
-set OUTPUT_DIR=<path to the directory where log files will be written>
-
-#Run a quickstart script for bfloat16 precision(for example, BFloat16 batch inference)
-bash quickstart\image_recognition\pytorch\resnet50\inference\cpu\bfloat16\fp32_batch_inference.sh 
-```
-
-<!--- 80. License -->
-## License
-
-[LICENSE](/LICENSE)
-
diff --git a/quickstart/image_recognition/pytorch/resnet50/inference/cpu/bfloat16/bf16_accuracy.sh b/quickstart/image_recognition/pytorch/resnet50/inference/cpu/bfloat16/bf16_accuracy.sh
deleted file mode 100755
index 86752c8eb..000000000
--- a/quickstart/image_recognition/pytorch/resnet50/inference/cpu/bfloat16/bf16_accuracy.sh
+++ /dev/null
@@ -1,54 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2020 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-MODEL_DIR=${MODEL_DIR-$PWD}
-
-source "${MODEL_DIR}/quickstart/common/utils.sh"
-_get_platform_type
-TCMALLOC_ARGS=""
-ARGS=""
-
-if [[ ${PLATFORM} == "linux" ]]; then
-  pip list | grep intel-extension-for-pytorch
-  if [[ "$?" == 0 ]]; then
-    TCMALLOC_ARGS=" -m intel_pytorch_extension.cpu.launch --enable_tcmalloc"
-    # in case IPEX is used we set ipex and jit path args
-    ARGS="--ipex --jit"
-    echo "Running using ${ARGS} args ..."
-  fi
-fi
-
-if [ -z "${DATASET_DIR}" ]; then
-  echo "The required environment variable DATASET_DIR has not been set"
-  exit 1
-fi
-
-if [ ! -d "${DATASET_DIR}" ]; then
-  echo "The DATASET_DIR '${DATASET_DIR}' does not exist"
-  exit 1
-fi
-
-python ${TCMALLOC_ARGS} \
-    models/image_recognition/pytorch/common/main.py \
-    --arch resnet50 ${DATASET_DIR} \
-    --evaluate \
-    --pretrained \
-    --jit \
-    ${ARGS} \
-    --bf16 \
-    --workers 0 \
-    --batch-size 128
diff --git a/quickstart/image_recognition/pytorch/resnet50/inference/cpu/bfloat16/bf16_batch_inference.sh b/quickstart/image_recognition/pytorch/resnet50/inference/cpu/bfloat16/bf16_batch_inference.sh
deleted file mode 100755
index aa7dd0858..000000000
--- a/quickstart/image_recognition/pytorch/resnet50/inference/cpu/bfloat16/bf16_batch_inference.sh
+++ /dev/null
@@ -1,47 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2020 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-MODEL_DIR=${MODEL_DIR-$PWD}
-
-source "${MODEL_DIR}/quickstart/common/utils.sh"
-_get_platform_type
-MULTI_INSTANCE_ARGS=""
-ARGS=""
-
-if [[ ${PLATFORM} == "linux" ]]; then
-    pip list | grep intel-extension-for-pytorch
-    if [[ "$?" == 0 ]]; then
-        CORES=`lscpu | grep Core | awk '{print $4}'`
-        SOCKETS=`lscpu | grep Socket | awk '{print $2}'`
-        MULTI_INSTANCE_ARGS=" -m intel_extension_for_pytorch.cpu.launch --enable_tcmalloc --ninstances ${SOCKETS} --ncore_per_instance ${CORES}"
-        
-        echo "Running '${SOCKETS}' instance"
-        # in case IPEX is used we set ipex and jit path args
-        ARGS="--ipex --jit"
-        echo "Running using ${ARGS} args ..."
-    fi
-fi
-
-python ${MULTI_INSTANCE_ARGS} \
-    models/image_recognition/pytorch/common/main.py \
-    --arch resnet50 ../ \
-    --evaluate \
-    --jit \
-    ${ARGS} \
-    --bf16 \
-    --batch-size 128 \
-    --dummy
diff --git a/quickstart/image_recognition/pytorch/resnet50/inference/cpu/bfloat16/bf16_online_inference.sh b/quickstart/image_recognition/pytorch/resnet50/inference/cpu/bfloat16/bf16_online_inference.sh
deleted file mode 100755
index 6e4178878..000000000
--- a/quickstart/image_recognition/pytorch/resnet50/inference/cpu/bfloat16/bf16_online_inference.sh
+++ /dev/null
@@ -1,49 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2020 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-MODEL_DIR=${MODEL_DIR-$PWD}
-
-source "${MODEL_DIR}/quickstart/common/utils.sh"
-_get_platform_type
-MULTI_INSTANCE_ARGS=""
-ARGS=""
-
-if [[ ${PLATFORM} == "linux" ]] && [[ ! -z "${ARGS}" ]]; then
-    pip list | grep intel-extension-for-pytorch
-    if [[ "$?" == 0 ]]; then
-        CORES=`lscpu | grep Core | awk '{print $4}'`
-        SOCKETS=`lscpu | grep Socket | awk '{print $2}'`
-        TOTAL_CORES=`expr $CORES \* $SOCKETS`
-        CORES_PER_INSTANCE=4
-        INSTANCES=`expr $TOTAL_CORES / $CORES_PER_INSTANCE`
-        MULTI_INSTANCE_ARGS=" -m intel_extension_for_pytorch.cpu.launch --enable_tcmalloc --ninstances ${INSTANCES} --ncore_per_instance ${CORES_PER_INSTANCE}"
-
-        echo "Running '${INSTANCES}' instance"
-        # in case IPEX is used we set ipex and jit path args
-        ARGS="--ipex --jit"
-        echo "Running using ${ARGS} args ..."
-    fi
-fi
-
-python ${MULTI_INSTANCE_ARGS} \
-    models/image_recognition/pytorch/common/main.py \
-    --arch resnet50 ../ \
-    --evaluate \
-    ${ARGS} \
-    --bf16 \
-    --batch-size 1 \
-    --dummy
diff --git a/quickstart/image_recognition/pytorch/resnet50/inference/cpu/fp32/.docs/baremetal.md b/quickstart/image_recognition/pytorch/resnet50/inference/cpu/fp32/.docs/baremetal.md
deleted file mode 100644
index 89347dd67..000000000
--- a/quickstart/image_recognition/pytorch/resnet50/inference/cpu/fp32/.docs/baremetal.md
+++ /dev/null
@@ -1,45 +0,0 @@
-<!--- 50. Bare Metal -->
-## Bare Metal
-
-Follow the instructions to setup your bare metal environment on Linux or Windows systems, download and
-preprocess the dataset, and do the model specific setup. Once all the setup is done,
-the Model Zoo can be used to run a [quickstart script](#quick-start-scripts).
-Ensure that you have enviornment variables set to point to the dataset directory
-and an output directory.
-
-Follow the instructions below to clone the Model Zoo repo before running the model on Linux or Windows:
-```
-# Clone the Model Zoo repo 
-git clone https://github.com/IntelAI/models.git
-cd models
-```
-
-### Run on Linux
-Install the following prerequisites in your environment:
-* Python 3
-* [intel-extension-for-pytorch](https://github.com/intel/intel-extension-for-pytorch)
-* [torchvision==v0.6.1](https://github.com/pytorch/vision/tree/v0.6.1)
-* numactl
-
-```
-# Env vars
-export DATASET_DIR=<path to the Imagenet Dataset>
-export OUTPUT_DIR=<path to the directory where log files will be written>
-
-# Run a quickstart script (for example, FP32 batch inference)
-bash quickstart/image_recognition/pytorch/resnet50/inference/cpu/fp32/fp32_batch_inference.sh 
-```
-
-### Run on Windows
-If not already setup, please follow instructions for [environment setup on Windows](/docs/general/Windows.md).
-
-Using Windows CMD.exe, run:
-```
-# Env vars
-set DATASET_DIR=<path to the Imagenet Dataset>
-set OUTPUT_DIR=<path to the directory where log files will be written>
-
-#Run a quickstart script for fp32 precision(for example, FP32 batch inference)
-bash quickstart\image_recognition\pytorch\resnet50\inference\cpu\fp32\fp32_batch_inference.sh 
-```
-
diff --git a/quickstart/image_recognition/pytorch/resnet50/inference/cpu/fp32/.docs/datasets.md b/quickstart/image_recognition/pytorch/resnet50/inference/cpu/fp32/.docs/datasets.md
deleted file mode 100644
index 06e37c796..000000000
--- a/quickstart/image_recognition/pytorch/resnet50/inference/cpu/fp32/.docs/datasets.md
+++ /dev/null
@@ -1,29 +0,0 @@
-<!--- 30. Datasets -->
-## Datasets
-
-The [ImageNet](http://www.image-net.org/) validation dataset is used when
-testing accuracy. The inference scripts use synthetic data, so no dataset
-is needed.
-
-Download and extract the ImageNet2012 dataset from http://www.image-net.org/,
-then move validation images to labeled subfolders, using
-[the valprep.sh shell script](https://raw.githubusercontent.com/soumith/imagenetloader.torch/master/valprep.sh)
-
-The accuracy script looks for a folder named `val`, so after running the
-data prep script, your folder structure should look something like this:
-
-```
-imagenet
-└── val
-    ├── ILSVRC2012_img_val.tar
-    ├── n01440764
-    │   ├── ILSVRC2012_val_00000293.JPEG
-    │   ├── ILSVRC2012_val_00002138.JPEG
-    │   ├── ILSVRC2012_val_00003014.JPEG
-    │   ├── ILSVRC2012_val_00006697.JPEG
-    │   └── ...
-    └── ...
-```
-The folder that contains the `val` directory should be set as the
-`DATASET_DIR` when running accuracy
-(for example: `export DATASET_DIR=/home/<user>/imagenet`).
diff --git a/quickstart/image_recognition/pytorch/resnet50/inference/cpu/fp32/.docs/description.md b/quickstart/image_recognition/pytorch/resnet50/inference/cpu/fp32/.docs/description.md
deleted file mode 100644
index 1537bba84..000000000
--- a/quickstart/image_recognition/pytorch/resnet50/inference/cpu/fp32/.docs/description.md
+++ /dev/null
@@ -1,4 +0,0 @@
-<!-- 10. Description -->
-## Description
-
-This document has instructions for running <model name> <precision> <mode>.
diff --git a/quickstart/image_recognition/pytorch/resnet50/inference/cpu/fp32/.docs/license.md b/quickstart/image_recognition/pytorch/resnet50/inference/cpu/fp32/.docs/license.md
deleted file mode 100644
index a60f169f4..000000000
--- a/quickstart/image_recognition/pytorch/resnet50/inference/cpu/fp32/.docs/license.md
+++ /dev/null
@@ -1,4 +0,0 @@
-<!--- 80. License -->
-## License
-
-[LICENSE](/LICENSE)
diff --git a/quickstart/image_recognition/pytorch/resnet50/inference/cpu/fp32/.docs/quickstart.md b/quickstart/image_recognition/pytorch/resnet50/inference/cpu/fp32/.docs/quickstart.md
deleted file mode 100644
index af17a9557..000000000
--- a/quickstart/image_recognition/pytorch/resnet50/inference/cpu/fp32/.docs/quickstart.md
+++ /dev/null
@@ -1,8 +0,0 @@
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| [`fp32_online_inference.sh`](fp32_online_inference.sh) | Runs online inference using synthetic data (batch_size=1). |
-| [`fp32_batch_inference.sh`](fp32_batch_inference.sh) | Runs batch inference using synthetic data (batch_size=128). |
-| [`fp32_accuracy.sh`](fp32_accuracy.sh) | Measures the model accuracy (batch_size=128). |
diff --git a/quickstart/image_recognition/pytorch/resnet50/inference/cpu/fp32/.docs/title.md b/quickstart/image_recognition/pytorch/resnet50/inference/cpu/fp32/.docs/title.md
deleted file mode 100644
index 6416ff7ff..000000000
--- a/quickstart/image_recognition/pytorch/resnet50/inference/cpu/fp32/.docs/title.md
+++ /dev/null
@@ -1,2 +0,0 @@
-<!--- 0. Title -->
-# <model name> <precision> <mode>
diff --git a/quickstart/image_recognition/pytorch/resnet50/inference/cpu/fp32/README.md b/quickstart/image_recognition/pytorch/resnet50/inference/cpu/fp32/README.md
deleted file mode 100644
index d452dc6a3..000000000
--- a/quickstart/image_recognition/pytorch/resnet50/inference/cpu/fp32/README.md
+++ /dev/null
@@ -1,96 +0,0 @@
-<!--- 0. Title -->
-# ResNet50 FP32 inference
-
-<!-- 10. Description -->
-## Description
-
-This document has instructions for running ResNet50 FP32 inference.
-
-<!--- 30. Datasets -->
-## Datasets
-
-The [ImageNet](http://www.image-net.org/) validation dataset is used when
-testing accuracy. The inference scripts use synthetic data, so no dataset
-is needed.
-
-Download and extract the ImageNet2012 dataset from http://www.image-net.org/,
-then move validation images to labeled subfolders, using
-[the valprep.sh shell script](https://raw.githubusercontent.com/soumith/imagenetloader.torch/master/valprep.sh)
-
-The accuracy script looks for a folder named `val`, so after running the
-data prep script, your folder structure should look something like this:
-
-```
-imagenet
-└── val
-    ├── ILSVRC2012_img_val.tar
-    ├── n01440764
-    │   ├── ILSVRC2012_val_00000293.JPEG
-    │   ├── ILSVRC2012_val_00002138.JPEG
-    │   ├── ILSVRC2012_val_00003014.JPEG
-    │   ├── ILSVRC2012_val_00006697.JPEG
-    │   └── ...
-    └── ...
-```
-The folder that contains the `val` directory should be set as the
-`DATASET_DIR` when running accuracy
-(for example: `export DATASET_DIR=/home/<user>/imagenet`).
-
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| [`fp32_online_inference.sh`](fp32_online_inference.sh) | Runs online inference using synthetic data (batch_size=1). |
-| [`fp32_batch_inference.sh`](fp32_batch_inference.sh) | Runs batch inference using synthetic data (batch_size=128). |
-| [`fp32_accuracy.sh`](fp32_accuracy.sh) | Measures the model accuracy (batch_size=128). |
-
-<!--- 50. Bare Metal -->
-## Bare Metal
-
-Follow the instructions to setup your bare metal environment on Linux or Windows systems, download and
-preprocess the dataset, and do the model specific setup. Once all the setup is done,
-the Model Zoo can be used to run a [quickstart script](#quick-start-scripts).
-Ensure that you have enviornment variables set to point to the dataset directory
-and an output directory.
-
-Follow the instructions below to clone the Model Zoo repo before running the model on Linux or Windows:
-```
-# Clone the Model Zoo repo 
-git clone https://github.com/IntelAI/models.git
-cd models
-```
-
-### Run on Linux
-Install the following prerequisites in your environment:
-* Python 3
-* [intel-extension-for-pytorch](https://github.com/intel/intel-extension-for-pytorch)
-* [torchvision==v0.6.1](https://github.com/pytorch/vision/tree/v0.6.1)
-* numactl
-* Run the model:
-    ```
-    # Env vars
-    export DATASET_DIR=<path to the Imagenet Dataset>
-    export OUTPUT_DIR=<path to the directory where log files will be written>
-
-    # Run a quickstart script (for example, FP32 batch inference)
-    bash quickstart/image_recognition/pytorch/resnet50/inference/cpu/fp32/fp32_batch_inference.sh 
-    ```
-
-### Run on Windows
-If not already setup, please follow instructions for [environment setup on Windows](/docs/general/Windows.md).
-
-Using Windows CMD.exe, run:
-```
-# Env vars
-set DATASET_DIR=<path to the Imagenet Dataset>
-set OUTPUT_DIR=<path to the directory where log files will be written>
-
-#Run a quickstart script for fp32 precision(for example, FP32 batch inference)
-bash quickstart\image_recognition\pytorch\resnet50\inference\cpu\fp32\fp32_batch_inference.sh 
-```
-
-<!--- 80. License -->
-## License
-
-[LICENSE](/LICENSE)
diff --git a/quickstart/image_recognition/pytorch/resnet50/inference/cpu/fp32/fp32_accuracy.sh b/quickstart/image_recognition/pytorch/resnet50/inference/cpu/fp32/fp32_accuracy.sh
deleted file mode 100755
index b479150c4..000000000
--- a/quickstart/image_recognition/pytorch/resnet50/inference/cpu/fp32/fp32_accuracy.sh
+++ /dev/null
@@ -1,52 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2020 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-MODEL_DIR=${MODEL_DIR-$PWD}
-
-source "${MODEL_DIR}/quickstart/common/utils.sh"
-_get_platform_type
-TCMALLOC_ARGS=""
-ARGS=""
-
-if [[ ${PLATFORM} == "linux" ]]; then
-  pip list | grep intel-extension-for-pytorch
-  if [[ "$?" == 0 ]]; then
-    TCMALLOC_ARGS=" -m intel_extension_for_pytorch.cpu.launch --enable_tcmalloc"
-    # in case IPEX is used we set ipex and jit path args
-    ARGS="--ipex --jit"
-    echo "Running using ${ARGS} args ..."
-  fi
-fi
-
-if [ -z "${DATASET_DIR}" ]; then
-  echo "The required environment variable DATASET_DIR has not been set"
-  exit 1
-fi
-
-if [ ! -d "${DATASET_DIR}" ]; then
-  echo "The DATASET_DIR '${DATASET_DIR}' does not exist"
-  exit 1
-fi
-
-python ${TCMALLOC_ARGS} \
-    models/image_recognition/pytorch/common/main.py \
-    --arch resnet50 ${DATASET_DIR} \
-    --evaluate \
-    --pretrained \
-    ${ARGS} \
-    --workers 0 \
-    --batch-size 128
diff --git a/quickstart/image_recognition/pytorch/resnet50/inference/cpu/fp32/fp32_batch_inference.sh b/quickstart/image_recognition/pytorch/resnet50/inference/cpu/fp32/fp32_batch_inference.sh
deleted file mode 100755
index 61c1befcf..000000000
--- a/quickstart/image_recognition/pytorch/resnet50/inference/cpu/fp32/fp32_batch_inference.sh
+++ /dev/null
@@ -1,46 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2020 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-MODEL_DIR=${MODEL_DIR-$PWD}
-
-source "${MODEL_DIR}/quickstart/common/utils.sh"
-_get_platform_type
-MULTI_INSTANCE_ARGS=""
-ARGS=""
-
-# Run IPEX and multi instance only on Linux systems, 
-# otherwise run regular torch on Linux or Windows.
-if [[ ${PLATFORM} == "linux" ]]; then
-    pip list | grep intel-extension-for-pytorch
-    if [[ "$?" == 0 ]]; then
-        CORES=`lscpu | grep Core | awk '{print $4}'`
-        SOCKETS=`lscpu | grep Socket | awk '{print $2}'`
-        MULTI_INSTANCE_ARGS=" -m intel_extension_for_pytorch.cpu.launch --enable_tcmalloc --ninstances ${SOCKETS} --ncore_per_instance ${CORES}"
-        
-        echo "Running '${SOCKETS}' instance"
-        # in case IPEX is used we set ipex and jit path args
-        ARGS="--ipex --jit"
-        echo "Running using ${ARGS} args ..."
-    fi
-fi
-
-python ${MULTI_INSTANCE_ARGS} \
-    models/image_recognition/pytorch/common/main.py \
-    --arch resnet50 ../ \
-    --evaluate \
-    ${ARGS} \
-    --batch-size 128 \
-    --dummy 2>&1 | tee ${OUTPUT_DIR}/fp32_batch_inference.log
diff --git a/quickstart/image_recognition/pytorch/resnet50/inference/cpu/fp32/fp32_online_inference.sh b/quickstart/image_recognition/pytorch/resnet50/inference/cpu/fp32/fp32_online_inference.sh
deleted file mode 100755
index 8ba75758e..000000000
--- a/quickstart/image_recognition/pytorch/resnet50/inference/cpu/fp32/fp32_online_inference.sh
+++ /dev/null
@@ -1,48 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2020 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-MODEL_DIR=${MODEL_DIR-$PWD}
-
-source "${MODEL_DIR}/quickstart/common/utils.sh"
-_get_platform_type
-MULTI_INSTANCE_ARGS=""
-ARGS=""
-
-if [[ ${PLATFORM} == "linux" ]]; then
-    pip list | grep intel-extension-for-pytorch
-    if [[ "$?" == 0 ]]; then
-        CORES=`lscpu | grep Core | awk '{print $4}'`
-        SOCKETS=`lscpu | grep Socket | awk '{print $2}'`
-        TOTAL_CORES=`expr $CORES \* $SOCKETS`
-        CORES_PER_INSTANCE=4
-        INSTANCES=`expr $TOTAL_CORES / $CORES_PER_INSTANCE`
-        MULTI_INSTANCE_ARGS=" -m intel_extension_for_pytorch.cpu.launch --enable_tcmalloc --ninstances ${INSTANCES} --ncore_per_instance ${CORES_PER_INSTANCE}"
-
-        echo "Running '${INSTANCES}' instance"
-        # in case IPEX is used we set ipex and jit path args
-        ARGS="--ipex --jit"
-        echo "Running using ${ARGS} args ..."
-    fi
-fi
-
-python ${MULTI_INSTANCE_ARGS} \
-    models/image_recognition/pytorch/common/main.py \
-    --arch resnet50 ../ \
-    --evaluate \
-    ${ARGS} \
-    --batch-size 1 \
-    --dummy
diff --git a/quickstart/image_recognition/pytorch/resnet50/inference/cpu/inference_realtime.sh b/quickstart/image_recognition/pytorch/resnet50/inference/cpu/inference_realtime.sh
index f2be1adc2..614f56ef7 100755
--- a/quickstart/image_recognition/pytorch/resnet50/inference/cpu/inference_realtime.sh
+++ b/quickstart/image_recognition/pytorch/resnet50/inference/cpu/inference_realtime.sh
@@ -32,7 +32,7 @@ mkdir -p ${OUTPUT_DIR}
 
 if [ -z "${PRECISION}" ]; then
   echo "The required environment variable PRECISION has not been set"
-  echo "Please set PRECISION to fp32, avx-fp32, int8, avx-int8, or bf16."
+  echo "Please set PRECISION to fp32, avx-fp32, int8, bf32, avx-int8, or bf16."
   exit 1
 fi
 
@@ -64,7 +64,7 @@ elif [[ $PRECISION == "fp32" || $PRECISION == "avx-fp32" ]]; then
     echo "running fp32 path"
 else
     echo "The specified precision '${PRECISION}' is unsupported."
-    echo "Supported precisions are: fp32, avx-fp32, bf16, int8, and avx-int8"
+    echo "Supported precisions are: fp32, avx-fp32, bf16, int8, bf32 and avx-int8"
     exit 1
 fi
 
@@ -82,18 +82,35 @@ export OMP_NUM_THREADS=$CORES_PER_INSTANCE
 
 NUMBER_INSTANCE=`expr $CORES_PER_NUMA / $CORES_PER_INSTANCE`
 
-python -m intel_extension_for_pytorch.cpu.launch \
-    --memory-allocator jemalloc \
-    --ninstances $NUMAS \
-    --log_path=${OUTPUT_DIR} \
-    --log_file_prefix="./resnet50_latency_log_${PRECISION}" \
-    ${MODEL_DIR}/models/image_recognition/pytorch/common/main.py \
-    $ARGS \
-    --ipex \
-    -j 0 \
-    -b $BATCH_SIZE \
-    --weight-sharing \
-    --number-instance $NUMBER_INSTANCE
+TORCH_INDUCTOR=${TORCH_INDUCTOR:-"0"}
+if [[ "0" == ${TORCH_INDUCTOR} ]];then
+    python -m intel_extension_for_pytorch.cpu.launch \
+        --memory-allocator jemalloc \
+        --ninstances $NUMAS \
+        --log_path=${OUTPUT_DIR} \
+        --log_file_prefix="./resnet50_latency_log_${PRECISION}" \
+        ${MODEL_DIR}/models/image_recognition/pytorch/common/main.py \
+        $ARGS \
+        --ipex \
+        -j 0 \
+        -b $BATCH_SIZE \
+        --weight-sharing \
+        --number-instance $NUMBER_INSTANCE
+else
+    echo "Running RN50 inference with torch.compile inductor backend."
+    python -m intel_extension_for_pytorch.cpu.launch \
+        --memory-allocator jemalloc \
+        --ninstance ${SOCKETS} \
+        --log_path=${OUTPUT_DIR} \
+        --log_file_prefix="./resnet50_latency_log_${PRECISION}" \
+        ${MODEL_DIR}/models/image_recognition/pytorch/common/main.py \
+        $ARGS \
+        --inductor \
+        -j 0 \
+        -b $BATCH_SIZE \
+	--weight-sharing \
+        --number-instance $NUMBER_INSTANCE
+fi
 
 wait
 
diff --git a/quickstart/image_recognition/pytorch/resnet50/inference/cpu/inference_throughput.sh b/quickstart/image_recognition/pytorch/resnet50/inference/cpu/inference_throughput.sh
index 451bb0ee5..540fa9e52 100755
--- a/quickstart/image_recognition/pytorch/resnet50/inference/cpu/inference_throughput.sh
+++ b/quickstart/image_recognition/pytorch/resnet50/inference/cpu/inference_throughput.sh
@@ -32,7 +32,7 @@ mkdir -p ${OUTPUT_DIR}
 
 if [ -z "${PRECISION}" ]; then
   echo "The required environment variable PRECISION has not been set"
-  echo "Please set PRECISION to fp32, avx-fp32, int8, avx-int8, or bf16."
+  echo "Please set PRECISION to fp32, avx-fp32, int8, bf32, avx-int8, or bf16."
   exit 1
 fi
 
@@ -46,7 +46,7 @@ ARGS=""
 ARGS="$ARGS -e -a resnet50 ../ --dummy"
 
 # default value, you can fine-tune it to get perfect performance.
-BATCH_SIZE=112
+BATCH_SIZE=${BATCH_SIZE:-112}
 
 CORES=`lscpu | grep Core | awk '{print $4}'`
 SOCKETS=`lscpu | grep Socket | awk '{print $2}'`
@@ -57,27 +57,27 @@ if [[ $PRECISION == "int8" || $PRECISION == "avx-int8" ]]; then
     echo "running int8 path"
     NUMA_NODES=`lscpu | grep 'NUMA node(s)' | awk '{print $3}'`
     CORES_PER_NODE=`expr $TOTAL_CORES / $NUMA_NODES`
-    BATCH_SIZE=`expr $CORES_PER_NODE \* 8`
+    BATCH_SIZE=${BATCH_SIZE:-`expr $CORES_PER_NODE \* 8`}
     ARGS="$ARGS --int8 --configure-dir ${MODEL_DIR}/models/image_recognition/pytorch/common/resnet50_configure_sym.json"
 elif [[ $PRECISION == "bf16" ]]; then
-    BATCH_SIZE=68
+    BATCH_SIZE=${BATCH_SIZE:-68}
     ARGS="$ARGS --bf16 --jit"
     echo "running bf16 path"
 elif [[ $PRECISION == "bf32" ]]; then
-    BATCH_SIZE=68
+    BATCH_SIZE=${BATCH_SIZE:-68}
     ARGS="$ARGS --bf32 --jit"
     echo "running bf32 path"
 elif [[ $PRECISION == "fp16" ]]; then
-    BATCH_SIZE=68
+    BATCH_SIZE=${BATCH_SIZE:-68}
     ARGS="$ARGS --fp16 --jit"
     echo "running fp16 path"
 elif [[ $PRECISION == "fp32" || $PRECISION == "avx-fp32" ]]; then
-    BATCH_SIZE=64
+    BATCH_SIZE=${BATCH_SIZE:-64}
     ARGS="$ARGS --jit"
     echo "running fp32 path"
 else
     echo "The specified precision '${PRECISION}' is unsupported."
-    echo "Supported precisions are: fp32, avx-fp32, bf16, int8, and avx-int8"
+    echo "Supported precisions are: fp32, avx-fp32, bf16, int8, bf32, and avx-int8"
     exit 1
 fi
 
@@ -91,6 +91,7 @@ if [ ${WEIGHT_SHAREING} ]; then
   weight_sharing=true
 fi
 
+TORCH_INDUCTOR=${TORCH_INDUCTOR:-"0"}
 if [ "$weight_sharing" = true ]; then
     CORES=`lscpu | grep Core | awk '{print $4}'`
     SOCKETS=`lscpu | grep Socket | awk '{print $2}'`
@@ -126,7 +127,7 @@ if [ "$weight_sharing" = true ]; then
     done
     wait
 
-else
+elif [[ "0" == ${TORCH_INDUCTOR} ]];then
     python -m intel_extension_for_pytorch.cpu.launch \
 	--memory-allocator jemalloc \
 	--throughput_mode \
@@ -138,6 +139,19 @@ else
         --seed 2020 \
         -j 0 \
         -b $BATCH_SIZE
+else
+    echo "Running RN50 inference with torch.compile inductor backend."
+    python -m intel_extension_for_pytorch.cpu.launch \
+	--memory-allocator jemalloc \
+	--throughput_mode \
+        --log_path=${OUTPUT_DIR} \
+        --log_file_prefix="./resnet50_throughput_log_${PRECISION}" \
+        ${MODEL_DIR}/models/image_recognition/pytorch/common/main.py \
+        $ARGS \
+        --inductor \
+        --seed 2020 \
+        -j 0 \
+        -b $BATCH_SIZE
 fi
 wait
 
diff --git a/quickstart/image_recognition/pytorch/resnet50/training/cpu/DEVCATALOG.md b/quickstart/image_recognition/pytorch/resnet50/training/cpu/DEVCATALOG.md
new file mode 100644
index 000000000..f8066dfd2
--- /dev/null
+++ b/quickstart/image_recognition/pytorch/resnet50/training/cpu/DEVCATALOG.md
@@ -0,0 +1,109 @@
+# PyTorch ResNet50 training
+
+## Description 
+This document has instructions for running ResNet50 training using Intel Extension for PyTorch.
+
+## Pull Command
+
+Docker image based on CentOS Stream8
+```
+docker pull intel/image-recognition:centos-pytorch-cpu-resnet50-training
+```
+
+Docker image based on Ubuntu 22.04
+```
+docker pull intel/image-recognition:ubuntu-pytorch-cpu-resnet50-training
+```
+
+## Quick Start Scripts
+| Script name | Description |
+|-------------|-------------|
+| `training.sh` | Trains using one node for one epoch for the specified precision (fp32, avx-fp32,bf32 or bf16). |
+
+**Note:** The `avx-fp32` precision runs the same scripts as `fp32`, except that the `DNNL_MAX_CPU_ISA` environment variable is unset. The environment variable is otherwise set to `DNNL_MAX_CPU_ISA=AVX512_CORE_AMX`.
+
+## Datasets
+Download and extract the ImageNet2012 training and validation dataset from [http://www.image-net.org/](http://www.image-net.org/), then move validation images to labeled subfolders, using [the valprep.sh shell script](https://raw.githubusercontent.com/soumith/imagenetloader.torch/master/valprep.sh)
+
+After running the data prep script, your folder structure should look something like this:
+```
+imagenet
+├── train
+│   ├── n02085620
+│   │   ├── n02085620_10074.JPEG
+│   │   ├── n02085620_10131.JPEG
+│   │   ├── n02085620_10621.JPEG
+│   │   └── ...
+│   └── ...
+└── val
+    ├── n01440764
+    │   ├── ILSVRC2012_val_00000293.JPEG
+    │   ├── ILSVRC2012_val_00002138.JPEG
+    │   ├── ILSVRC2012_val_00003014.JPEG
+    │   └── ...
+    └── ...
+```
+The folder that contains the `val` and `train` directories should be set as the`DATASET_DIR` (for example: `export DATASET_DIR=/home/<user>/imagenet`).
+
+## Docker Run
+(Optional) Export related proxy into docker environment.
+```bash
+export DOCKER_RUN_ENVS="-e ftp_proxy=${ftp_proxy} \
+  -e FTP_PROXY=${FTP_PROXY} -e http_proxy=${http_proxy} \
+  -e HTTP_PROXY=${HTTP_PROXY} -e https_proxy=${https_proxy} \
+  -e HTTPS_PROXY=${HTTPS_PROXY} -e no_proxy=${no_proxy} \
+  -e NO_PROXY=${NO_PROXY} -e socks_proxy=${socks_proxy} \
+  -e SOCKS_PROXY=${SOCKS_PROXY}"
+```
+
+To run ResNet50 training, set environment variables to specify the dataset directory, precision to run, and an output directory. 
+
+```bash
+export OS=<provide either centos or ubuntu>
+export DATASET_DIR=<path to the dataset>
+export OUTPUT_DIR=<directory where log files will be written>
+export PRECISION=<specify the precision to run>
+export SCRIPT=quickstart/training.sh 
+
+IMAGE_NAME=intel/image-recognition:${OS}-pytorch-cpu-resnet50-training
+WORKDIR=/workspace/pytorch-resnet50-training
+DOCKER_ARGS="--privileged --init -it"
+TRAINING_EPOCHS=1
+
+docker run --rm \
+  --env DATASET_DIR=${DATASET_DIR} \
+  --env PRECISION=${PRECISION} \
+  --env OUTPUT_DIR=${OUTPUT_DIR} \
+  --env TRAINING_EPOCHS=${TRAINING_EPOCHS} \
+  --env http_proxy=${http_proxy} \
+  --env https_proxy=${https_proxy} \
+  --env no_proxy=${no_proxy} \
+  --volume ${DATASET_DIR}:${DATASET_DIR} \
+  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
+  ${DOCKER_RUN_ENVS} \
+  --shm-size 8G \
+  -w ${WORKDIR} \
+  ${DOCKER_ARGS} \
+  $IMAGE_NAME \
+  /bin/bash $SCRIPT
+```
+
+## Documentation and Sources
+#### Get Started​
+[Docker* Repository](https://hub.docker.com/r/intel/image-recognition)
+
+[Main GitHub*](https://github.com/IntelAI/models)
+
+[Release Notes](https://github.com/IntelAI/models/releases)
+
+[Get Started Guide](https://github.com/IntelAI/models/blob/master/quickstart/quickstart/image_recognition/pytorch/resnet50/training/cpu/DEVCATALOG.md)
+
+#### Code Sources
+[Dockerfile](https://github.com/IntelAI/models/tree/master/docker/pyt-cpu)
+
+[Report Issue](https://community.intel.com/t5/Intel-Optimized-AI-Frameworks/bd-p/optimized-ai-frameworks)
+
+## License Agreement
+LEGAL NOTICE: By accessing, downloading or using this software and any required dependent software (the “Software Package”), you agree to the terms and conditions of the software license agreements for the Software Package, which may also include notices, disclaimers, or license terms for third party software included with the Software Package. Please refer to the [license](https://github.com/IntelAI/models/tree/master/third_party) file for additional details.
+
+[View All Containers and Solutions 🡢](https://www.intel.com/content/www/us/en/developer/tools/software-catalog/containers.html?s=Newest)
diff --git a/quickstart/image_recognition/pytorch/resnet50/training/cpu/README.md b/quickstart/image_recognition/pytorch/resnet50/training/cpu/README.md
index 75e687c17..e897d9ef4 100644
--- a/quickstart/image_recognition/pytorch/resnet50/training/cpu/README.md
+++ b/quickstart/image_recognition/pytorch/resnet50/training/cpu/README.md
@@ -1,12 +1,56 @@
-<!--- 0. Title -->
 # PyTorch ResNet50 training
 
-<!-- 10. Description -->
 ## Description
 
 This document has instructions for running ResNet50 training using
 Intel-optimized PyTorch.
 
+
+## Dataset
+
+The [ImageNet](http://www.image-net.org/) validation dataset is used when
+testing accuracy. The inference scripts use synthetic data, so no dataset
+is needed.
+
+Download and extract the ImageNet2012 dataset from http://www.image-net.org/,
+then move validation images to labeled subfolders, using
+[the valprep.sh shell script](https://raw.githubusercontent.com/soumith/imagenetloader.torch/master/valprep.sh)
+
+The accuracy script looks for a folder named `val`, so after running the
+data prep script, your folder structure should look something like this:
+
+```
+imagenet
+└── val
+    ├── ILSVRC2012_img_val.tar
+    ├── n01440764
+    │   ├── ILSVRC2012_val_00000293.JPEG
+    │   ├── ILSVRC2012_val_00002138.JPEG
+    │   ├── ILSVRC2012_val_00003014.JPEG
+    │   ├── ILSVRC2012_val_00006697.JPEG
+    │   └── ...
+    └── ...
+```
+The folder that contains the `val` directory should be set as the
+`DATASET_DIR` when running accuracy
+(for example: `export DATASET_DIR=/home/<user>/imagenet`).
+
+## Quick Start Scripts
+
+| Script name | Description |
+|-------------|-------------|
+| `training.sh` | Trains using one node for one epoch for the specified precision (fp32, avx-fp32, bf16, bf32 or fp16). |
+| `training_dist.sh` | Distributed trains using one node for one epoch for the specified precision (fp32, avx-fp32, bf16, bf32 or fp16). |
+| `training_single_socket.sh` | Trains on a single socket for one epoch for the specified precision (fp32, avx-fp32, bf16, bf32 or fp16). |
+
+> Note: The `avx-int8` and `avx-fp32` precisions run the same scripts as `int8` and `fp32`, except that the
+> `DNNL_MAX_CPU_ISA` environment variable is unset. The environment variable is
+> otherwise set to `DNNL_MAX_CPU_ISA=AVX512_CORE_AMX`.
+* Set ENV to use AMX:
+  ```
+  export DNNL_MAX_CPU_ISA=AVX512_CORE_AMX
+  ```
+
 ## Bare Metal
 
 ### General setup
@@ -33,6 +77,7 @@ The tcmalloc should be built from the [General setup](#general-setup) section.
 IOMP should be installed in your conda env from the [General setup](#general-setup) section.
 
 ```bash
+    pip install packaging intel-openmp
     export LD_PRELOAD=path/lib/libiomp5.so:$LD_PRELOAD
 ```
 
@@ -41,7 +86,6 @@ IOMP should be installed in your conda env from the [General setup](#general-set
 ```bash
     export DNNL_MAX_CPU_ISA=AVX512_CORE_AMX_FP16
 ```
-
 * Set ENV to use multi-node distributed training (no need for single-node multi-sockets)
 
   In this case, we use data-parallel distributed training and every rank will hold same model replica. The NNODES is the number of ip in the HOSTFILE. To use multi-nodes distributed training you should firstly setup the passwordless login (you can refer to [link](https://linuxize.com/post/how-to-setup-passwordless-ssh-login/)) between these nodes.
@@ -53,56 +97,16 @@ IOMP should be installed in your conda env from the [General setup](#general-set
     export MASTER_ADDR=#your_master_addr
 ```
 
-## Datasets
-
-### ImageNet
-
-Download and extract the ImageNet2012 training and validation dataset from
-[http://www.image-net.org/](http://www.image-net.org/),
-then move validation images to labeled subfolders, using
-[the valprep.sh shell script](https://raw.githubusercontent.com/soumith/imagenetloader.torch/master/valprep.sh)
-
-A after running the data prep script, your folder structure should look something like this:
-
-```txt
-imagenet
-├── train
-│   ├── n02085620
-│   │   ├── n02085620_10074.JPEG
-│   │   ├── n02085620_10131.JPEG
-│   │   ├── n02085620_10621.JPEG
-│   │   └── ...
-│   └── ...
-└── val
-    ├── n01440764
-    │   ├── ILSVRC2012_val_00000293.JPEG
-    │   ├── ILSVRC2012_val_00002138.JPEG
-    │   ├── ILSVRC2012_val_00003014.JPEG
-    │   └── ...
-    └── ...
-```
-
-The folder that contains the `val` and `train` directories should be set as the
-`DATASET_DIR` (for example: `export DATASET_DIR=/home/<user>/imagenet`).
-
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| `training.sh` | Trains using one node for one epoch for the specified precision (fp32, avx-fp32, bf16, bf32 or fp16). |
-| `training_dist.sh` | Distributed trains using one node for one epoch for the specified precision (fp32, avx-fp32, bf16, bf32 or fp16). |
-
 ## Run the model
 
 Follow the instructions above to setup your bare metal environment, download and
 preprocess the dataset, and do the model specific setup. Once all the setup is done,
-the Model Zoo can be used to run a [quickstart script](#quick-start-scripts).
+the Intel® AI Reference Models can be used to run a [quickstart script](#quick-start-scripts).
 Ensure that you have enviornment variables set to point to the dataset directory,
 an output directory, precision, and the number of training epochs.
 
 ```bash
-# Clone the model zoo repo and set the MODEL_DIR
+# Clone the Intel® AI Reference Models repo and set the MODEL_DIR
 git clone https://github.com/IntelAI/models.git
 cd models
 export MODEL_DIR=$(pwd)
@@ -113,8 +117,11 @@ export OUTPUT_DIR=<Where_to_save_log>
 export PRECISION=<precision to run (fp32, avx-fp32, bf16, bf32, or fp16)>
 export TRAINING_EPOCHS=<epoch_number(90 or other number)>
 
+# [optional] Compile model with PyTorch Inductor backend
+export TORCH_INDUCTOR=1
+
 # Run the training quickstart script
-cd ${MODEL_DIR}/quickstart/image_recognition/pytorch/resnet50/training/cpu
+./quickstart/image_recognition/pytorch/resnet50/training/cpu/training.sh
 bash training.sh
 
 # Run the distributed training quickstart script
diff --git a/quickstart/image_recognition/pytorch/resnet50/training/cpu/training.sh b/quickstart/image_recognition/pytorch/resnet50/training/cpu/training.sh
index 40dce74e5..d55222295 100755
--- a/quickstart/image_recognition/pytorch/resnet50/training/cpu/training.sh
+++ b/quickstart/image_recognition/pytorch/resnet50/training/cpu/training.sh
@@ -47,7 +47,7 @@ fi
 
 if [ -z "${PRECISION}" ]; then
   echo "The required environment variable PRECISION has not been set"
-  echo "Please set PRECISION to fp32, avx-fp32, or bf16."
+  echo "Please set PRECISION to fp32, avx-fp32, bf32 or bf16."
   exit 1
 fi
 
@@ -89,25 +89,44 @@ export USE_IPEX=1
 export KMP_BLOCKTIME=1
 export KMP_AFFINITY=granularity=fine,compact,1,0
 
-BATCH_SIZE=128
+BATCH_SIZE=${BATCH_SIZE:-128}
 
 rm -rf ./resnet50_training_log_*
 
-python -m intel_extension_for_pytorch.cpu.launch \
-    --memory-allocator jemalloc \
-    --ninstances 1 \
-    --ncore_per_instance ${CORES_PER_INSTANCE} \
-    --log_path=${OUTPUT_DIR} \
-    --log_file_prefix="./resnet50_training_log_${PRECISION}" \
-    ${MODEL_DIR}/models/image_recognition/pytorch/common/main.py \
-    $ARGS \
-    --ipex \
-    -j 0 \
-    --seed 2020 \
-    --epochs $TRAINING_EPOCHS \
-    --train-no-eval \
-    -w 50 \
-    -b $BATCH_SIZE
+TORCH_INDUCTOR=${TORCH_INDUCTOR:-"0"}
+if [[ "0" == ${TORCH_INDUCTOR} ]];then
+    python -m intel_extension_for_pytorch.cpu.launch \
+        --memory-allocator jemalloc \
+        --ninstances 1 \
+        --ncore_per_instance ${CORES_PER_INSTANCE} \
+        --log_path=${OUTPUT_DIR} \
+        --log_file_prefix="./resnet50_training_log_${PRECISION}" \
+        ${MODEL_DIR}/models/image_recognition/pytorch/common/main.py \
+        $ARGS \
+        --ipex \
+        -j 0 \
+        --seed 2020 \
+        --epochs $TRAINING_EPOCHS \
+        --train-no-eval \
+        -w 50 \
+        -b $BATCH_SIZE
+else
+    python -m intel_extension_for_pytorch.cpu.launch \
+        --memory-allocator jemalloc \
+        --ninstances 1 \
+        --ncore_per_instance ${CORES_PER_INSTANCE} \
+        --log_path=${OUTPUT_DIR} \
+        --log_file_prefix="./resnet50_training_log_${PRECISION}" \
+        ${MODEL_DIR}/models/image_recognition/pytorch/common/main.py \
+        $ARGS \
+        --inductor \
+        -j 0 \
+        --seed 2020 \
+        --epochs $TRAINING_EPOCHS \
+        --train-no-eval \
+        -w 50 \
+        -b $BATCH_SIZE
+fi
 
 # For the summary of results
 wait
diff --git a/quickstart/image_recognition/pytorch/resnet50/training/cpu/training_dist.sh b/quickstart/image_recognition/pytorch/resnet50/training/cpu/training_dist.sh
index e43421281..e64d7b3fa 100755
--- a/quickstart/image_recognition/pytorch/resnet50/training/cpu/training_dist.sh
+++ b/quickstart/image_recognition/pytorch/resnet50/training/cpu/training_dist.sh
@@ -146,24 +146,48 @@ source $oneccl_bindings_for_pytorch_path/env/setvars.sh
 
 export FI_PROVIDER_PATH=$oneccl_bindings_for_pytorch_path/lib/prov
 
-python -m intel_extension_for_pytorch.cpu.launch \
-    --memory-allocator tcmalloc \
-    --distributed \
-    --nnodes ${NNODES} \
-    --hostfile ${HOSTFILE} \
-    --logical_cores_for_ccl --ccl_worker_count 8 \
-    ${MODEL_DIR}/models/image_recognition/pytorch/common/train.py \
-    $ARGS \
-    --epochs $TRAINING_EPOCHS \
-    --warmup-epochs 2  \
-    --ipex \
-    -j 0 \
-    -b $LOCAL_BATCH_SIZE \
-    --seed 2020 \
-    --dist-backend ccl \
-    --base-op=LARS \
-    --base-lr 10.5 \
-    --weight-decay 0.00005 2>&1 | tee ${OUTPUT_DIR}/resnet50_dist_training_log_${PRECISION}.log
+TORCH_INDUCTOR=${TORCH_INDUCTOR:-"0"}
+if [[ "0" == ${TORCH_INDUCTOR} ]];then
+  python -m intel_extension_for_pytorch.cpu.launch \
+      --memory-allocator tcmalloc \
+      --distributed \
+      --nnodes ${NNODES} \
+      --hostfile ${HOSTFILE} \
+      --logical_cores_for_ccl --ccl_worker_count 8 \
+      ${MODEL_DIR}/models/image_recognition/pytorch/common/train.py \
+      $ARGS \
+      --epochs $TRAINING_EPOCHS \
+      --warmup-epochs 2  \
+      --ipex \
+      -j 0 \
+      -b $LOCAL_BATCH_SIZE \
+      --seed 2020 \
+      --dist-backend ccl \
+      --base-op=LARS \
+      --base-lr 10.5 \
+      --weight-decay 0.00005 2>&1 | tee ${OUTPUT_DIR}/resnet50_dist_training_log_${PRECISION}.log
+else
+  export TORCHINDUCTOR_FREEZING=1
+  python -m intel_extension_for_pytorch.cpu.launch \
+      --memory-allocator tcmalloc \
+      --distributed \
+      --nnodes ${NNODES} \
+      --hostfile ${HOSTFILE} \
+      --logical_cores_for_ccl --ccl_worker_count 8 \
+      ${MODEL_DIR}/models/image_recognition/pytorch/common/train.py \
+      $ARGS \
+      --epochs $TRAINING_EPOCHS \
+      --warmup-epochs 2  \
+      --inductor \
+      -j 0 \
+      -b $LOCAL_BATCH_SIZE \
+      --seed 2020 \
+      --dist-backend ccl \
+      --base-op=LARS \
+      --base-lr 10.5 \
+      --weight-decay 0.00005 2>&1 | tee ${OUTPUT_DIR}/resnet50_dist_training_log_${PRECISION}.log
+fi
+
 # For the summary of results
 wait
 
diff --git a/quickstart/image_recognition/pytorch/resnext-32x16d/inference/cpu/DEVCATALOG.md b/quickstart/image_recognition/pytorch/resnext-32x16d/inference/cpu/DEVCATALOG.md
new file mode 100644
index 000000000..32ccda081
--- /dev/null
+++ b/quickstart/image_recognition/pytorch/resnext-32x16d/inference/cpu/DEVCATALOG.md
@@ -0,0 +1,104 @@
+# PyTorch ResNext 32x16d inference
+
+## Description
+This document has instructions for running ResNext 32x16d inference using Intel Extension for PyTorch
+
+## Pull Command
+
+Docker image based on CentOS Stream8
+```
+docker pull intel/image-recognition:centos-pytorch-cpu-resnext-32x16d-inference
+```
+Docker image based on Ubuntu 22.04
+```
+docker pull intel/image-recognition:ubuntu-pytorch-cpu-resnext-32x16d-inference
+```
+
+## Quick Start Scripts
+| Script name | Description |
+|-------------|-------------|
+| `inference_realtime.sh` | Runs multi instance realtime inference using 4 cores per instance with synthetic data for the specified precision (fp32, avx-fp32, int8, avx-int8, bf32 or bf16). |
+| `inference_throughput.sh` | Runs multi instance batch inference using 1 instance per socket with synthetic data for the specified precision (fp32, avx-fp32, int8, avx-int8, bf32 or bf16). |
+| `accuracy.sh` | Measures the inference accuracy (providing a `DATASET_DIR` environment variable is required) for the specified precision (fp32, avx-fp32, int8, avx-int8, bf32 or bf16). |
+
+**Note:** The `avx-int8` and `avx-fp32` precisions run the same scripts as `int8` and `fp32`, except that the `DNNL_MAX_CPU_ISA` environment variable is unset. The environment variable is otherwise set to `DNNL_MAX_CPU_ISA=AVX512_CORE_AMX`.
+
+### ImageNet
+The [ImageNet](http://www.image-net.org/) validation dataset is used to run the ResNext 32x16d accuracy script. The realtime and throughput inference scripts use synthetic data.
+
+Download and extract the ImageNet2012 dataset from [http://www.image-net.org/](http://www.image-net.org/),then move validation images to labeled subfolders, using [the valprep.sh shell script](https://raw.githubusercontent.com/soumith/imagenetloader.torch/master/valprep.sh)
+
+After running the data prep script, your folder structure should look something like this:
+```
+imagenet
+└── val
+    ├── ILSVRC2012_img_val.tar
+    ├── n01440764
+    │   ├── ILSVRC2012_val_00000293.JPEG
+    │   ├── ILSVRC2012_val_00002138.JPEG
+    │   ├── ILSVRC2012_val_00003014.JPEG
+    │   ├── ILSVRC2012_val_00006697.JPEG
+    │   └── ...
+    └── ...
+```
+The folder that contains the `val` directory should be set as the `DATASET_DIR` (for example: `export DATASET_DIR=/home/<user>/imagenet`).
+
+## Docker Run
+(Optional) Export related proxy into docker environment.
+```bash
+export DOCKER_RUN_ENVS="-e ftp_proxy=${ftp_proxy} \
+  -e FTP_PROXY=${FTP_PROXY} -e http_proxy=${http_proxy} \
+  -e HTTP_PROXY=${HTTP_PROXY} -e https_proxy=${https_proxy} \
+  -e HTTPS_PROXY=${HTTPS_PROXY} -e no_proxy=${no_proxy} \
+  -e NO_PROXY=${NO_PROXY} -e socks_proxy=${socks_proxy} \
+  -e SOCKS_PROXY=${SOCKS_PROXY}"
+```
+
+To run ResNext32x16d inference, set environment variables to specify the dataset directory, precision to run, and an output directory.
+
+```bash
+export OS=<provide either centos or ubuntu>
+export DATASET_DIR=<path to the dataset>
+export OUTPUT_DIR=<directory where log files will be written>
+export PRECISION=<specify the precision to run>
+export SCRIPT=quickstart/<specify the quickstart script>
+
+IMAGE_NAME=intel/image-recognition:${OS}-pytorch-cpu-resnext-32x16d-inference
+DOCKER_ARGS="--privileged --init -it"
+WORKDIR=/workspace/pytorch-resnext-32x16d-inference
+
+docker run --rm \
+  --env PRECISION=${PRECISION} \
+  --env OUTPUT_DIR=${OUTPUT_DIR} \
+  --env DATASET_DIR=${DATASET_DIR} \
+  --env http_proxy=${http_proxy} \
+  --env https_proxy=${https_proxy} \
+  --env no_proxy=${no_proxy} \
+  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
+  --volume ${DATASET_DIR}:${DATASET_DIR} \
+  ${DOCKER_RUN_ENVS} \
+  --shm-size 8G \
+  -w ${WORKDIR} \
+  ${DOCKER_ARGS} \
+  $IMAGE_NAME \
+  /bin/bash $SCRIPT
+```
+## Documentation and Sources
+#### Get Started​
+[Docker* Repository](https://hub.docker.com/r/intel/image-recognition)
+
+[Main GitHub*](https://github.com/IntelAI/models)
+
+[Release Notes](https://github.com/IntelAI/models/releases)
+
+[Get Started Guide](https://github.com/IntelAI/models/blob/master/quickstart/quickstart/image_recognition/pytorch/resnext-32x16d/inference/cpu/DEVCATALOG.md)
+
+#### Code Sources
+[Dockerfile](https://github.com/IntelAI/models/tree/master/docker/pyt-cpu)
+
+[Report Issue](https://community.intel.com/t5/Intel-Optimized-AI-Frameworks/bd-p/optimized-ai-frameworks)
+
+## License Agreement
+LEGAL NOTICE: By accessing, downloading or using this software and any required dependent software (the “Software Package”), you agree to the terms and conditions of the software license agreements for the Software Package, which may also include notices, disclaimers, or license terms for third party software included with the Software Package. Please refer to the [license](https://github.com/IntelAI/models/tree/master/third_party) file for additional details.
+
+[View All Containers and Solutions 🡢](https://www.intel.com/content/www/us/en/developer/tools/software-catalog/containers.html?s=Newest)
diff --git a/quickstart/image_recognition/pytorch/resnext-32x16d/inference/cpu/README.md b/quickstart/image_recognition/pytorch/resnext-32x16d/inference/cpu/README.md
index ef8e04c32..4337d17bf 100644
--- a/quickstart/image_recognition/pytorch/resnext-32x16d/inference/cpu/README.md
+++ b/quickstart/image_recognition/pytorch/resnext-32x16d/inference/cpu/README.md
@@ -47,13 +47,10 @@ The folder that contains the `val` directory should be set as the
 > Note: The `avx-int8` and `avx-fp32` precisions run the same scripts as `int8` and `fp32`, except that the
 > `DNNL_MAX_CPU_ISA` environment variable is unset. The environment variable is
 > otherwise set to `DNNL_MAX_CPU_ISA=AVX512_CORE_AMX`.
-
-Follow the instructions to setup your bare metal environment on either Linux or Windows systems. Once all the setup is done,
-the Model Zoo can be used to run a [quickstart script](#quick-start-scripts).
-Ensure that you have a clone of the [Model Zoo Github repository](https://github.com/IntelAI/models).
-```
-git clone https://github.com/IntelAI/models.git
-```
+* Set ENV to use AMX:
+  ```
+  export DNNL_MAX_CPU_ISA=AVX512_CORE_AMX
+  ```
 
 ## Run on Linux
 
@@ -71,12 +68,15 @@ Follow [link](/docs/general/pytorch/BareMetalSetup.md) to install Miniconda and
 
   IOMP should be installed in your conda env. Set the following environment variables.
   ```
+  pip install packaging intel-openmp
   export LD_PRELOAD=<path to the intel-openmp directory>/lib/libiomp5.so:$LD_PRELOAD
   ```
 
-* Set ENV to use AMX if you are using SPR
+* Follow the instructions to setup your bare metal environment on either Linux or Windows systems. Once all the setup is done,
+  the Intel® AI Reference Models can be used to run a [quickstart script](#quick-start-scripts).
+  Ensure that you have a clone of the [Intel® AI Reference Models Github repository](https://github.com/IntelAI/models).
   ```
-  export DNNL_MAX_CPU_ISA=AVX512_CORE_AMX
+  git clone https://github.com/IntelAI/models.git
   ```
 
 * Run the model:
@@ -84,12 +84,19 @@ Follow [link](/docs/general/pytorch/BareMetalSetup.md) to install Miniconda and
   cd models
 
   # Set environment variables
-  export DATASET_DIR=<path_to_Imagenet_Dataset>
   export OUTPUT_DIR=<path to the directory where log files will be written>
-  export PRECISION=<precision to run>
+  export PRECISION=<select precision to run: fp32, avx-fp32, int8, avx-int8, bf32 or bf16>
+
+  # Optional environment variable:
+  export BATCH_SIZE=<set a value for batch size else it will run with the default value>
 
   # Run a quickstart script 
-  bash quickstart/image_recognition/pytorch/resnext-32x16d/inference/cpu/inference_realtime.sh
+  ./quickstart/image_recognition/pytorch/resnext-32x16d/inference/cpu/<quickstart_script.sh>
+
+  # Set `DATASET_DIR` only for running the `accuracy.sh` script:
+  set DATASET_DIR=<path to the Imagenet Dataset>
+
+  ./quickstart/image_recognition/pytorch/resnext-32x16d/inference/cpu/accuracy.sh
   ```
 
 ## Run on Windows
@@ -99,13 +106,20 @@ Using Windows CMD.exe, run:
 ```
 cd models
 
-# Env vars
+# Set environment variables
+export OUTPUT_DIR=<path to the directory where log files will be written>
+export PRECISION=<select precision to run: fp32, avx-fp32, int8, avx-int8, bf32 or bf16>
+
+# Optional environment variable:
+export BATCH_SIZE=<set a value for batch size else it will run with the default value>
+
+# Run a quickstart script 
+bash quickstart\image_recognition\pytorch\resnext-32x16d\inference\cpu\<quickstart_script.sh>
+
+# Set `DATASET_DIR` only for running the `accuracy.sh` script:
 set DATASET_DIR=<path to the Imagenet Dataset>
-set OUTPUT_DIR=<path to the directory where log files will be written>
-set PRECISION=<precision to run>
 
-#Run a quickstart script for fp32 precision(FP32 inference_realtime or inference_throughput or accuracy)
-bash quickstart\image_recognition\pytorch\resnext-32x16d\inference\cpu\inference_realtime.sh
+bash quickstart\image_recognition\pytorch\resnext-32x16d\inference\cpu\accuracy.sh
 ```
 
 <!--- 80. License -->
diff --git a/quickstart/image_recognition/pytorch/resnext-32x16d/inference/cpu/accuracy.sh b/quickstart/image_recognition/pytorch/resnext-32x16d/inference/cpu/accuracy.sh
index 843fb896f..298ac0fbb 100755
--- a/quickstart/image_recognition/pytorch/resnext-32x16d/inference/cpu/accuracy.sh
+++ b/quickstart/image_recognition/pytorch/resnext-32x16d/inference/cpu/accuracy.sh
@@ -42,7 +42,7 @@ fi
 
 if [ -z "${PRECISION}" ]; then
   echo "The required environment variable PRECISION has not been set"
-  echo "Please set PRECISION to fp32, avx-fp32, int8, avx-int8, or bf16."
+  echo "Please set PRECISION to fp32, avx-fp32, int8, avx-int8, bf32, or bf16."
   exit 1
 fi
 
@@ -50,7 +50,8 @@ export DNNL_PRIMITIVE_CACHE_CAPACITY=1024
 export KMP_BLOCKTIME=1
 export KMP_AFFINITY=granularity=fine,compact,1,0
 
-BATCH_SIZE=128
+# If BATCH_SIZE is not set, it will use default batch size
+BATCH_SIZE=${BATCH_SIZE:-128}
 
 rm -rf ${OUTPUT_DIR}/resnext101_accuracy_log*
 
@@ -79,7 +80,7 @@ elif [[ $PRECISION == "fp32" || $PRECISION == "avx-fp32" ]]; then
     echo "running fp32 path"
 else
     echo "The specified precision '${PRECISION}' is unsupported."
-    echo "Supported precisions are: fp32, avx-fp32, bf16, int8, and avx-int8"
+    echo "Supported precisions are: fp32, avx-fp32, bf16, int8, bf32 and avx-int8"
     exit 1
 fi
 
diff --git a/quickstart/image_recognition/pytorch/resnext-32x16d/inference/cpu/inference_realtime.sh b/quickstart/image_recognition/pytorch/resnext-32x16d/inference/cpu/inference_realtime.sh
index 606a2e3ba..5040418c6 100755
--- a/quickstart/image_recognition/pytorch/resnext-32x16d/inference/cpu/inference_realtime.sh
+++ b/quickstart/image_recognition/pytorch/resnext-32x16d/inference/cpu/inference_realtime.sh
@@ -32,11 +32,12 @@ mkdir -p ${OUTPUT_DIR}
 
 if [ -z "${PRECISION}" ]; then
   echo "The required environment variable PRECISION has not been set"
-  echo "Please set PRECISION to fp32, avx-fp32, int8, avx-int8, or bf16."
+  echo "Please set PRECISION to fp32, avx-fp32, int8, avx-int8, bf32 or bf16."
   exit 1
 fi
 
-BATCH_SIZE=1
+# If BATCH_SIZE is not set, it will use default batch size
+BATCH_SIZE=${BATCH_SIZE-1}
 
 rm -rf ${OUTPUT_DIR}/resnext101_latency_log*
 
@@ -61,7 +62,7 @@ elif [[ $PRECISION == "fp32" || $PRECISION == "avx-fp32" ]]; then
     echo "running fp32 path"
 else
     echo "The specified precision '${PRECISION}' is unsupported."
-    echo "Supported precisions are: fp32, avx-fp32, bf16, int8, and avx-int8"
+    echo "Supported precisions are: fp32, avx-fp32, bf16, int8, and avx-int8, bf32"
     exit 1
 fi
 
@@ -109,7 +110,6 @@ if [[ ${PLATFORM} == "linux" ]]; then
     TOTAL_CORES=`expr $CORES \* $SOCKETS`
     INSTANCES=`expr $TOTAL_CORES / $CORES_PER_INSTANCE`
     INSTANCES_PER_SOCKET=`expr $INSTANCES / $SOCKETS`
-
     throughput=$(grep 'Throughput:' ${OUTPUT_DIR}/resnext101_latency_log* |sed -e 's/.*Throughput//;s/[^0-9.]//g' |awk -v INSTANCES_PER_SOCKET=$INSTANCES_PER_SOCKET '
     BEGIN {
             sum = 0;
diff --git a/quickstart/image_recognition/pytorch/resnext-32x16d/inference/cpu/inference_throughput.sh b/quickstart/image_recognition/pytorch/resnext-32x16d/inference/cpu/inference_throughput.sh
index 7450486fd..1474fbc89 100755
--- a/quickstart/image_recognition/pytorch/resnext-32x16d/inference/cpu/inference_throughput.sh
+++ b/quickstart/image_recognition/pytorch/resnext-32x16d/inference/cpu/inference_throughput.sh
@@ -32,7 +32,7 @@ mkdir -p ${OUTPUT_DIR}
 
 if [ -z "${PRECISION}" ]; then
   echo "The required environment variable PRECISION has not been set"
-  echo "Please set PRECISION to fp32, avx-fp32, int8, avx-int8, or bf16."
+  echo "Please set PRECISION to fp32, avx-fp32, int8, avx-int8, bf32 or bf16."
   exit 1
 fi
 
@@ -59,7 +59,7 @@ elif [[ $PRECISION == "fp32" || $PRECISION == "avx-fp32" ]]; then
     echo "running fp32 path"
 else
     echo "The specified precision '${PRECISION}' is unsupported."
-    echo "Supported precisions are: fp32, avx-fp32, bf16, int8, and avx-int8"
+    echo "Supported precisions are: fp32, avx-fp32, bf16, int8, bf32 and avx-int8"
     exit 1
 fi
 
@@ -92,7 +92,8 @@ export DNNL_PRIMITIVE_CACHE_CAPACITY=1024
 export KMP_BLOCKTIME=1
 export KMP_AFFINITY=granularity=fine,compact,1,0
 
-BATCH_SIZE=116
+# If BATCH_SIZE is not set, it will use default batch size
+BATCH_SIZE=${BATCH_SIZE:-116}
 
 python ${MULTI_INSTANCE_ARGS} \
     ${MODEL_DIR}/models/image_recognition/pytorch/common/main.py \
diff --git a/quickstart/image_recognition/tensorflow/densenet169/inference/cpu/.docs/docker.md b/quickstart/image_recognition/tensorflow/densenet169/inference/cpu/.docs/docker.md
index 83b9028b0..ed0865637 100644
--- a/quickstart/image_recognition/tensorflow/densenet169/inference/cpu/.docs/docker.md
+++ b/quickstart/image_recognition/tensorflow/densenet169/inference/cpu/.docs/docker.md
@@ -8,7 +8,11 @@ and an output directory.
 
 ```
 DATASET_DIR=<path to the dataset>
+<<<<<<<< HEAD:quickstart/image_recognition/tensorflow/resnet101/inference/cpu/.docs/docker.md
+PRECISION=<set the precision "int8" or "fp32">
+========
 PRECISION=fp32
+>>>>>>>> r3.1:quickstart/image_recognition/tensorflow/densenet169/inference/cpu/.docs/docker.md
 OUTPUT_DIR=<directory where log files will be written>
 # For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
 export BATCH_SIZE=<customized batch size value>
diff --git a/quickstart/image_recognition/tensorflow/inceptionv3/inference/cpu/.docs/docker.md b/quickstart/image_recognition/tensorflow/inceptionv3/inference/cpu/.docs/docker.md
index b20adb462..388854a5d 100644
--- a/quickstart/image_recognition/tensorflow/inceptionv3/inference/cpu/.docs/docker.md
+++ b/quickstart/image_recognition/tensorflow/inceptionv3/inference/cpu/.docs/docker.md
@@ -8,7 +8,11 @@ and an output directory.
 
 ```
 DATASET_DIR=<path to the dataset>
+<<<<<<<< HEAD:quickstart/image_recognition/tensorflow/densenet169/inference/cpu/.docs/docker.md
+PRECISION=fp32
+========
 PRECISION=<set the precision to "int8" or "fp32">
+>>>>>>>> r3.1:quickstart/image_recognition/tensorflow/inceptionv3/inference/cpu/.docs/docker.md
 OUTPUT_DIR=<directory where log files will be written>
 # For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
 export BATCH_SIZE=<customized batch size value>
diff --git a/quickstart/image_recognition/tensorflow/inceptionv4/inference/cpu/.docs/baremetal.md b/quickstart/image_recognition/tensorflow/inceptionv4/inference/cpu/.docs/baremetal.md
deleted file mode 100644
index 7d64ada53..000000000
--- a/quickstart/image_recognition/tensorflow/inceptionv4/inference/cpu/.docs/baremetal.md
+++ /dev/null
@@ -1,98 +0,0 @@
-<!--- 50. AI Kit -->
-## Run the model
-
-Setup your environment using the instructions below, depending on if you are
-using [AI Kit](/docs/general/tensorflow/AIKit.md):
-
-<table>
-  <tr>
-    <th>Setup using AI Kit on Linux</th>
-    <th>Setup without AI Kit on Linux</th>
-    <th>Setup without AI Kit on Windows</th>
-  </tr>
-  <tr>
-    <td>
-      <p>To run using AI Kit on Linux you will need:</p>
-      <ul>
-        <li>numactl
-        <li>wget
-        <li>Activate the tensorflow conda environment
-        <pre>conda activate tensorflow</pre>
-      </ul>
-    </td>
-    <td>
-      <p>To run without AI Kit on Linux you will need:</p>
-      <ul>
-        <li>Python 3
-        <li><a href="https://pypi.org/project/intel-tensorflow/">intel-tensorflow>=2.5.0</a>
-        <li>git
-        <li>numactl
-        <li>wget
-        <li>A clone of the Model Zoo repo<br />
-        <pre>git clone https://github.com/IntelAI/models.git</pre>
-      </ul>
-    </td>
-    <td>
-      <p>To run without AI Kit on Windows you will need:</p>
-      <ul>
-        <li><a href="/docs/general/Windows.md">Intel Model Zoo on Windows Systems prerequisites</a>
-        <li>A clone of the Model Zoo repo<br />
-        <pre>git clone https://github.com/IntelAI/models.git</pre>
-      </ul>
-    </td>
-  </tr>
-</table>
-
-After finishing the setup above, download the pretrained model based on `PRECISION` and set the
-`PRETRAINED_MODEL` environment var to the path to the frozen graph.
-If you run on Windows, please use a browser to download the pretrained model using the link below.
-For Linux, run:
-```
-# Int8 Pretrained model
-wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/inceptionv4_int8_pretrained_model.pb
-export PRETRAINED_MODEL=$(pwd)/inceptionv4_int8_pretrained_model.pb
-
-# FP32 Pretrained model
-wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/inceptionv4_fp32_pretrained_model.pb
-export PRETRAINED_MODEL=$(pwd)/inceptionv4_fp32_pretrained_model.pb
-```
-
-Set the environment variables and run quickstart script on either Linux or Windows systems. See the [list of quickstart scripts](#quick-start-scripts) for details on the different options.
-
-### Run on Linux
-```
-# cd to your model zoo directory
-cd models
-
-export PRETRAINED_MODEL=<path to the frozen graph downloaded above>
-export DATASET_DIR=<path to the ImageNet TF records>
-export PRECISION=<set the precision to "int8" or "fp32">
-export OUTPUT_DIR=<path to the directory where log files will be written>
-# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
-export BATCH_SIZE=<customized batch size value>
-
-./quickstart/image_recognition/tensorflow/inceptionv4/inference/cpu/<script name>.sh
-```
-
-### Run on Windows
-Using `cmd.exe` run:
-```
-# cd to your model zoo directory
-cd models
-
-set PRETRAINED_MODEL=<path to the frozen graph downloaded above>
-set DATASET_DIR=<path to the ImageNet TF records>
-set PRECISION=<set the precision to "int8" or "fp32">
-set OUTPUT_DIR=<directory where log files will be written>
-# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
-set BATCH_SIZE=<customized batch size value>
-
-bash quickstart\image_recognition\tensorflow\inceptionv4\inference\cpu\<script name>.sh
-```
-> Note: You may use `cygpath` to convert the Windows paths to Unix paths before setting the environment variables. 
-As an example, if the dataset location on Windows is `D:\user\ImageNet`, convert the Windows path to Unix as shown:
-> ```
-> cygpath D:\user\ImageNet
-> /d/user/ImageNet
->```
->Then, set the `DATASET_DIR` environment variable `set DATASET_DIR=/d/user/ImageNet`.
diff --git a/quickstart/image_recognition/tensorflow/inceptionv4/inference/cpu/.docs/datasets.md b/quickstart/image_recognition/tensorflow/inceptionv4/inference/cpu/.docs/datasets.md
deleted file mode 100644
index df92f3539..000000000
--- a/quickstart/image_recognition/tensorflow/inceptionv4/inference/cpu/.docs/datasets.md
+++ /dev/null
@@ -1,8 +0,0 @@
-<!--- 30. Datasets -->
-## Datasets
-
-Download and preprocess the ImageNet dataset using the [instructions here](/datasets/imagenet/README.md).
-After running the conversion script you should have a directory with the
-ImageNet dataset in the TF records format.
-
-Set the `DATASET_DIR` to point to this directory when running <model name>.
diff --git a/quickstart/image_recognition/tensorflow/inceptionv4/inference/cpu/.docs/description.md b/quickstart/image_recognition/tensorflow/inceptionv4/inference/cpu/.docs/description.md
deleted file mode 100644
index 106b45dbe..000000000
--- a/quickstart/image_recognition/tensorflow/inceptionv4/inference/cpu/.docs/description.md
+++ /dev/null
@@ -1,5 +0,0 @@
-<!-- 10. Description -->
-## Description
-
-This document has instructions for running <model name> <mode> using
-Intel-optimized TensorFlow.
diff --git a/quickstart/image_recognition/tensorflow/inceptionv4/inference/cpu/.docs/docker.md b/quickstart/image_recognition/tensorflow/inceptionv4/inference/cpu/.docs/docker.md
deleted file mode 100644
index b20adb462..000000000
--- a/quickstart/image_recognition/tensorflow/inceptionv4/inference/cpu/.docs/docker.md
+++ /dev/null
@@ -1,32 +0,0 @@
-<!--- 60. Docker -->
-## Docker
-
-The model container includes the scripts and libraries needed to run 
-<model name> <mode>. To run one of the quickstart scripts 
-using this container, you'll need to provide volume mounts for the dataset 
-and an output directory.
-
-```
-DATASET_DIR=<path to the dataset>
-PRECISION=<set the precision to "int8" or "fp32">
-OUTPUT_DIR=<directory where log files will be written>
-# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
-export BATCH_SIZE=<customized batch size value>
-
-docker run \
-  --env DATASET_DIR=${DATASET_DIR} \
-  --env PRECISION=${PRECISION} \
-  --env OUTPUT_DIR=${OUTPUT_DIR} \
-  --env BATCH_SIZE=${BATCH_SIZE} \
-  --env http_proxy=${http_proxy} \
-  --env https_proxy=${https_proxy} \
-  --volume ${DATASET_DIR}:${DATASET_DIR} \
-  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
-  --privileged --init -t \
-  <docker image> \
-  /bin/bash quickstart/<script name>.sh
-```
-
-If you are new to docker and are running into issues with the container,
-see [this document](https://github.com/IntelAI/models/tree/master/docs/general/docker.md)
-for troubleshooting tips.
diff --git a/quickstart/image_recognition/tensorflow/inceptionv4/inference/cpu/.docs/license.md b/quickstart/image_recognition/tensorflow/inceptionv4/inference/cpu/.docs/license.md
deleted file mode 100644
index a60f169f4..000000000
--- a/quickstart/image_recognition/tensorflow/inceptionv4/inference/cpu/.docs/license.md
+++ /dev/null
@@ -1,4 +0,0 @@
-<!--- 80. License -->
-## License
-
-[LICENSE](/LICENSE)
diff --git a/quickstart/image_recognition/tensorflow/inceptionv4/inference/cpu/.docs/quickstart.md b/quickstart/image_recognition/tensorflow/inceptionv4/inference/cpu/.docs/quickstart.md
deleted file mode 100644
index 5a80d9fb1..000000000
--- a/quickstart/image_recognition/tensorflow/inceptionv4/inference/cpu/.docs/quickstart.md
+++ /dev/null
@@ -1,8 +0,0 @@
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| [`online_inference.sh`](/quickstart/image_recognition/tensorflow/inceptionv4/inference/cpu/online_inference.sh) | Runs online inference (batch_size=1). |
-| [`batch_inference.sh`](/quickstart/image_recognition/tensorflow/inceptionv4/inference/cpu/batch_inference.sh) | Runs batch inference (batch_size=240). |
-| [`accuracy.sh`](/quickstart/image_recognition/tensorflow/inceptionv4/inference/cpu/accuracy.sh) | Measures the model accuracy (batch_size=100). |
diff --git a/quickstart/image_recognition/tensorflow/inceptionv4/inference/cpu/.docs/title.md b/quickstart/image_recognition/tensorflow/inceptionv4/inference/cpu/.docs/title.md
deleted file mode 100644
index 1d6c60187..000000000
--- a/quickstart/image_recognition/tensorflow/inceptionv4/inference/cpu/.docs/title.md
+++ /dev/null
@@ -1,2 +0,0 @@
-<!--- 0. Title -->
-# <model name> <mode>
diff --git a/quickstart/image_recognition/tensorflow/inceptionv4/inference/cpu/README.md b/quickstart/image_recognition/tensorflow/inceptionv4/inference/cpu/README.md
deleted file mode 100644
index f3ff561b1..000000000
--- a/quickstart/image_recognition/tensorflow/inceptionv4/inference/cpu/README.md
+++ /dev/null
@@ -1,65 +0,0 @@
-<!--- 0. Title -->
-# Inception V4 inference
-
-<!-- 10. Description -->
-## Description
-
-This document has instructions for running Inception V4 inference using
-Intel-optimized TensorFlow.
-
-<!--- 30. Datasets -->
-## Datasets
-
-Download and preprocess the ImageNet dataset using the [instructions here](/datasets/imagenet/README.md).
-After running the conversion script you should have a directory with the
-ImageNet dataset in the TF records format.
-
-Set the `DATASET_DIR` to point to this directory when running Inception V4.
-
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| [`online_inference.sh`](/quickstart/image_recognition/tensorflow/inceptionv4/inference/cpu/online_inference.sh) | Runs online inference (batch_size=1). |
-| [`batch_inference.sh`](/quickstart/image_recognition/tensorflow/inceptionv4/inference/cpu/batch_inference.sh) | Runs batch inference (batch_size=240). |
-| [`accuracy.sh`](/quickstart/image_recognition/tensorflow/inceptionv4/inference/cpu/accuracy.sh) | Measures the model accuracy (batch_size=100). |
-
-<!--- 60. Docker -->
-## Docker
-
-The model container includes the scripts and libraries needed to run 
-Inception V4 inference. To run one of the quickstart scripts 
-using this container, you'll need to provide volume mounts for the dataset 
-and an output directory.
-
-```
-DATASET_DIR=<path to the dataset>
-PRECISION=<set the precision to "int8" or "fp32">
-OUTPUT_DIR=<directory where log files will be written>
-# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
-export BATCH_SIZE=<customized batch size value>
-
-docker run \
-  --env DATASET_DIR=${DATASET_DIR} \
-  --env PRECISION=${PRECISION}
-  --env OUTPUT_DIR=${OUTPUT_DIR} \
-  --env BATCH_SIZE=${BATCH_SIZE} \
-  --env http_proxy=${http_proxy} \
-  --env https_proxy=${https_proxy} \
-  --volume ${DATASET_DIR}:${DATASET_DIR} \
-  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
-  --privileged --init -t \
-  intel/image-recognition:tf-latest-inceptionv4-inference \
-  /bin/bash quickstart/<script name>.sh
-```
-
-If you are new to docker and are running into issues with the container,
-see [this document](https://github.com/IntelAI/models/tree/master/docs/general/docker.md)
-for troubleshooting tips.
-
-<!--- 80. License -->
-## License
-
-[LICENSE](/LICENSE)
-
diff --git a/quickstart/image_recognition/tensorflow/inceptionv4/inference/cpu/accuracy.sh b/quickstart/image_recognition/tensorflow/inceptionv4/inference/cpu/accuracy.sh
deleted file mode 100755
index 9f2e0cc55..000000000
--- a/quickstart/image_recognition/tensorflow/inceptionv4/inference/cpu/accuracy.sh
+++ /dev/null
@@ -1,88 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2020 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-MODEL_DIR=${MODEL_DIR-$PWD}
-
-if [ -z "${OUTPUT_DIR}" ]; then
-  echo "The required environment variable OUTPUT_DIR has not been set"
-  exit 1
-fi
-
-# Create the output directory in case it doesn't already exist
-mkdir -p ${OUTPUT_DIR}
-
-if [ -z "${DATASET_DIR}" ]; then
-  echo "The required environment variable DATASET_DIR has not been set"
-  exit 1
-fi
-
-if [ ! -d "${DATASET_DIR}" ]; then
-  echo "The DATASET_DIR '${DATASET_DIR}' does not exist"
-  exit 1
-fi
-
-if [ -z "${PRECISION}" ]; then
-  echo "The required environment variable PRECISION has not been set"
-  echo "Please set PRECISION to fp32 or int8"
-  exit 1
-fi
-
-if [[ $PRECISION != "fp32" ]] && [[ $PRECISION != "int8" ]]; then
-  echo "The specified precision '${PRECISION}' is unsupported."
-  echo "Supported precisions are: fp32 and int8"
-  exit 1
-fi
-
-if [ -z "${PRETRAINED_MODEL}" ]; then
-  if [[ $PRECISION == "int8" ]]; then
-      PRETRAINED_MODEL="${MODEL_DIR}/inceptionv4_int8_pretrained_model.pb"
-  elif [[ $PRECISION == "fp32" ]]; then
-      PRETRAINED_MODEL="${MODEL_DIR}/inceptionv4_fp32_pretrained_model.pb"
-  else
-      echo "The specified precision '${PRECISION}' is unsupported."
-      echo "Supported precisions are: fp32 and int8"
-      exit 1
-  fi
-
-  if [[ ! -f "${PRETRAINED_MODEL}" ]]; then
-    echo "The pretrained model could not be found. Please set the PRETRAINED_MODEL env var to point to the frozen graph file."
-    exit 1
-  fi
-elif [[ ! -f "${PRETRAINED_MODEL}" ]]; then
-  echo "The file specified by the PRETRAINED_MODEL environment variable (${PRETRAINED_MODEL}) does not exist."
-  exit 1
-fi
-
-# If batch size env is not mentioned, then the workload will run with the default batch size.
-if [ -z "${BATCH_SIZE}"]; then
-  BATCH_SIZE="100"
-  echo "Running with default batch size of ${BATCH_SIZE}"
-fi
-
-source "${MODEL_DIR}/quickstart/common/utils.sh"
-_command python benchmarks/launch_benchmark.py \
-         --model-name=inceptionv4 \
-         --precision ${PRECISION} \
-         --mode=inference \
-         --framework tensorflow \
-         --in-graph ${PRETRAINED_MODEL} \
-         --data-location=${DATASET_DIR} \
-         --output-dir ${OUTPUT_DIR} \
-         --accuracy-only \
-         --batch-size=${BATCH_SIZE} \
-         --socket-id 0 \
-         $@
diff --git a/quickstart/image_recognition/tensorflow/inceptionv4/inference/cpu/batch_inference.sh b/quickstart/image_recognition/tensorflow/inceptionv4/inference/cpu/batch_inference.sh
deleted file mode 100755
index 132de03f5..000000000
--- a/quickstart/image_recognition/tensorflow/inceptionv4/inference/cpu/batch_inference.sh
+++ /dev/null
@@ -1,87 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2020 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-MODEL_DIR=${MODEL_DIR-$PWD}
-
-if [ -z "${OUTPUT_DIR}" ]; then
-  echo "The required environment variable OUTPUT_DIR has not been set"
-  exit 1
-fi
-
-# Create the output directory in case it doesn't already exist
-mkdir -p ${OUTPUT_DIR}
-
-if [ -z "${DATASET_DIR}" ]; then
-  echo "The required environment variable DATASET_DIR has not been set"
-  exit 1
-fi
-
-if [ ! -d "${DATASET_DIR}" ]; then
-  echo "The DATASET_DIR '${DATASET_DIR}' does not exist"
-  exit 1
-fi
-
-if [ -z "${PRECISION}" ]; then
-  echo "The required environment variable PRECISION has not been set"
-  echo "Please set PRECISION to fp32 or int8"
-  exit 1
-fi
-
-if [[ $PRECISION != "fp32" ]] && [[ $PRECISION != "int8" ]]; then
-  echo "The specified precision '${PRECISION}' is unsupported."
-  echo "Supported precisions are: fp32 and int8"
-  exit 1
-fi
-
-if [ -z "${PRETRAINED_MODEL}" ]; then
-    if [[ $PRECISION == "int8" ]]; then
-      PRETRAINED_MODEL="${MODEL_DIR}/inceptionv4_int8_pretrained_model.pb"
-  elif [[ $PRECISION == "fp32" ]]; then
-      PRETRAINED_MODEL="${MODEL_DIR}/inceptionv4_fp32_pretrained_model.pb"
-  else
-      echo "The specified precision '${PRECISION}' is unsupported."
-      echo "Supported precisions are: fp32 and int8"
-      exit 1
-  fi
-
-  if [[ ! -f "${PRETRAINED_MODEL}" ]]; then
-    echo "The pretrained model could not be found. Please set the PRETRAINED_MODEL env var to point to the frozen graph file."
-    exit 1
-  fi
-elif [[ ! -f "${PRETRAINED_MODEL}" ]]; then
-  echo "The file specified by the PRETRAINED_MODEL environment variable (${PRETRAINED_MODEL}) does not exist."
-  exit 1
-fi
-
-# If batch size env is not mentioned, then the workload will run with the default batch size.
-if [ -z "${BATCH_SIZE}"]; then
-  BATCH_SIZE="240"
-  echo "Running with default batch size of ${BATCH_SIZE}"
-fi
-
-source "${MODEL_DIR}/quickstart/common/utils.sh"
-_command python benchmarks/launch_benchmark.py \
-         --model-name=inceptionv4 \
-         --precision=fp32 \
-         --mode=inference \
-         --framework tensorflow \
-         --in-graph ${PRETRAINED_MODEL} \
-         --data-location=${DATASET_DIR} \
-         --output-dir ${OUTPUT_DIR} \
-         --batch-size=${BATCH_SIZE} \
-         --socket-id 0 \
-         $@
diff --git a/quickstart/image_recognition/tensorflow/inceptionv4/inference/cpu/online_inference.sh b/quickstart/image_recognition/tensorflow/inceptionv4/inference/cpu/online_inference.sh
deleted file mode 100755
index bdbb05747..000000000
--- a/quickstart/image_recognition/tensorflow/inceptionv4/inference/cpu/online_inference.sh
+++ /dev/null
@@ -1,87 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2020 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-MODEL_DIR=${MODEL_DIR-$PWD}
-
-if [ -z "${OUTPUT_DIR}" ]; then
-  echo "The required environment variable OUTPUT_DIR has not been set"
-  exit 1
-fi
-
-# Create the output directory in case it doesn't already exist
-mkdir -p ${OUTPUT_DIR}
-
-if [ -z "${DATASET_DIR}" ]; then
-  echo "The required environment variable DATASET_DIR has not been set"
-  exit 1
-fi
-
-if [ ! -d "${DATASET_DIR}" ]; then
-  echo "The DATASET_DIR '${DATASET_DIR}' does not exist"
-  exit 1
-fi
-
-if [ -z "${PRECISION}" ]; then
-  echo "The required environment variable PRECISION has not been set"
-  echo "Please set PRECISION to fp32 or int8"
-  exit 1
-fi
-
-if [[ $PRECISION != "fp32" ]] && [[ $PRECISION != "int8" ]]; then
-  echo "The specified precision '${PRECISION}' is unsupported."
-  echo "Supported precisions are: fp32 and int8"
-  exit 1
-fi
-
-if [ -z "${PRETRAINED_MODEL}" ]; then
-  if [[ $PRECISION == "int8" ]]; then
-      PRETRAINED_MODEL="${MODEL_DIR}/inceptionv4_int8_pretrained_model.pb"      
-  elif [[ $PRECISION == "fp32" ]]; then
-      PRETRAINED_MODEL="${MODEL_DIR}/inceptionv4_fp32_pretrained_model.pb"
-  else
-      echo "The specified precision '${PRECISION}' is unsupported."
-      echo "Supported precisions are: fp32 and int8"
-      exit 1
-  fi
-
-  if [[ ! -f "${PRETRAINED_MODEL}" ]]; then
-    echo "The pretrained model could not be found. Please set the PRETRAINED_MODEL env var to point to the frozen graph file."
-    exit 1
-  fi
-elif [[ ! -f "${PRETRAINED_MODEL}" ]]; then
-  echo "The file specified by the PRETRAINED_MODEL environment variable (${PRETRAINED_MODEL}) does not exist."
-  exit 1
-fi
-
-# If batch size env is not mentioned, then the workload will run with the default batch size.
-if [ -z "${BATCH_SIZE}"]; then
-  BATCH_SIZE="1"
-  echo "Running with default batch size of ${BATCH_SIZE}"
-fi
-
-source "${MODEL_DIR}/quickstart/common/utils.sh"
-_command python benchmarks/launch_benchmark.py \
-         --model-name=inceptionv4 \
-         --precision ${PRECISION} \
-         --mode=inference \
-         --framework tensorflow \
-         --in-graph ${PRETRAINED_MODEL} \
-         --data-location=${DATASET_DIR} \
-         --output-dir ${OUTPUT_DIR} \
-         --batch-size=${BATCH_SIZE} \
-         --socket-id 0 \
-         $@
diff --git a/quickstart/image_recognition/tensorflow/mobilenet_v1/inference/cpu/README_DEV_CAT.md b/quickstart/image_recognition/tensorflow/mobilenet_v1/inference/cpu/README_DEV_CAT.md
new file mode 100644
index 000000000..2f9b98ba9
--- /dev/null
+++ b/quickstart/image_recognition/tensorflow/mobilenet_v1/inference/cpu/README_DEV_CAT.md
@@ -0,0 +1,96 @@
+# TensorFlow MobileNet V1 inference
+
+## Description
+This document has instructions for running MobileNet V1 inference using Intel-optimized TensorFlow.
+
+## Pull Command
+```
+docker pull intel/image-recognition:centos-tf-cpu-mobilenet-v1-inference
+```
+
+<table>
+   <thead>
+      <tr>
+         <th>Script name</th>
+         <th>Description</th>
+      </tr>
+   </thead>
+   <tbody>
+      <tr>
+         <td>inference_realtime_multi_instance.sh</td>
+         <td>A multi-instance run that uses 4 cores per instance with `batch_size=1` for the specified precision (fp32, int8, bfloat16, bfloat32). Uses synthetic data if no DATASET_DIR is set</td>
+      </tr>
+      <tr>
+         <td>inference_throughput_multi_instance.sh</td>
+         <td>A multi-instance run that uses 4 cores per instance with `batch_size=448` for the specified precision (fp32, int8, bfloat16, bfloat32). Uses synthetic data if no DATASET_DIR is set</td>
+      </tr>
+      <tr>
+         <td>accuracy.sh</td>
+         <td>Measures the model accuracy (batch_size=100) for the specified precision (fp32, int8, bfloat16, bfloat32).</td>
+      </tr>
+   </tbody>
+</table>
+
+## Datasets
+Download and preprocess the ImageNet dataset using the [instructions here](https://github.com/IntelAI/models/blob/master/datasets/imagenet/README.md).
+After running the conversion script you should have a directory with the
+ImageNet dataset in the TF records format.
+
+Set the `DATASET_DIR` to point to the TF records directory when running MobileNet V1.
+
+## Docker Run
+(Optional) Export related proxy into docker environment.
+```bash
+export DOCKER_RUN_ENVS="-e ftp_proxy=${ftp_proxy} \
+  -e FTP_PROXY=${FTP_PROXY} -e http_proxy=${http_proxy} \
+  -e HTTP_PROXY=${HTTP_PROXY} -e https_proxy=${https_proxy} \
+  -e HTTPS_PROXY=${HTTPS_PROXY} -e no_proxy=${no_proxy} \
+  -e NO_PROXY=${NO_PROXY} -e socks_proxy=${socks_proxy} \
+  -e SOCKS_PROXY=${SOCKS_PROXY}"
+```
+
+To run MobileNet V1 inference, set environment variables to specify the dataset directory, precision and mode to run, and an output directory. 
+```bash
+# Set the required environment vars
+export DATASET_DIR=<path to the dataset>
+export OUTPUT_DIR=<directory where log files will be written>
+export SCRIPT=<specify the script to run>
+export PRECISION=<specify the precision to run>
+
+# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
+export BATCH_SIZE=<customized batch size value>
+
+docker run --rm \
+  --env BATCH_SIZE=${BATCH_SIZE} \
+  --env DATASET_DIR=${DATASET_DIR} \
+  --env OUTPUT_DIR=${OUTPUT_DIR} \
+  --env PRECISION=${PRECISION} \
+  --volume ${DATASET_DIR}:${DATASET_DIR} \
+  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
+  ${DOCKER_RUN_ENVS} \
+  --privileged --init -it \
+  --shm-size 8G \
+  -w /workspace/tf-mobilenet-v1-inference \
+  intel/image-recognition:centos-tf-cpu-mobilenet-v1-inference \
+  /bin/bash quickstart/${SCRIPT}
+```
+
+## Documentation and Sources
+#### Get Started​
+[Docker* Repository](https://hub.docker.com/r/intel/image-recognition)
+
+[Main GitHub*](https://github.com/IntelAI/models)
+
+[Release Notes](https://github.com/IntelAI/models/releases)
+
+[Get Started Guide](https://github.com/IntelAI/models/blob/master/quickstart/image_recognition/tensorflow/mobilenet_v1/inference/cpu/README_DEV_CAT.md)
+
+#### Code Sources
+[Dockerfile](https://github.com/IntelAI/models/tree/master/dockerfiles/tensorflow)
+
+[Report Issue](https://community.intel.com/t5/Intel-Optimized-AI-Frameworks/bd-p/optimized-ai-frameworks)
+
+## License Agreement
+LEGAL NOTICE: By accessing, downloading or using this software and any required dependent software (the “Software Package”), you agree to the terms and conditions of the software license agreements for the Software Package, which may also include notices, disclaimers, or license terms for third party software included with the Software Package. Please refer to the [license](https://github.com/IntelAI/models/tree/master/third_party) file for additional details.
+
+[View All Containers and Solutions 🡢](https://www.intel.com/content/www/us/en/developer/tools/software-catalog/containers.html?s=Newest)
diff --git a/quickstart/image_recognition/tensorflow/mobilenet_v1/inference/cpu/accuracy.sh b/quickstart/image_recognition/tensorflow/mobilenet_v1/inference/cpu/accuracy.sh
index bd4b7c6f7..5f4938c44 100755
--- a/quickstart/image_recognition/tensorflow/mobilenet_v1/inference/cpu/accuracy.sh
+++ b/quickstart/image_recognition/tensorflow/mobilenet_v1/inference/cpu/accuracy.sh
@@ -84,7 +84,6 @@ fi
 
 MODE="inference"
 source "${MODEL_DIR}/quickstart/common/utils.sh"
-_ht_status_spr
 _command python ${MODEL_DIR}/benchmarks/launch_benchmark.py \
   --model-name=mobilenet_v1 \
   --precision ${PRECISION} \
diff --git a/quickstart/image_recognition/tensorflow/resnet101/inference/cpu/.docs/docker.md b/quickstart/image_recognition/tensorflow/resnet101/inference/cpu/.docs/docker.md
index 22f7da57a..aeba1bbfa 100644
--- a/quickstart/image_recognition/tensorflow/resnet101/inference/cpu/.docs/docker.md
+++ b/quickstart/image_recognition/tensorflow/resnet101/inference/cpu/.docs/docker.md
@@ -8,7 +8,11 @@ and an output directory.
 
 ```
 DATASET_DIR=<path to the dataset>
+<<<<<<<< HEAD:quickstart/language_translation/tensorflow/mlperf_gnmt/inference/cpu/.docs/docker.md
+PRECISION=fp32
+========
 PRECISION=<set the precision "int8" or "fp32">
+>>>>>>>> r3.1:quickstart/image_recognition/tensorflow/resnet101/inference/cpu/.docs/docker.md
 OUTPUT_DIR=<directory where log files will be written>
 # For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
 export BATCH_SIZE=<customized batch size value>
@@ -17,7 +21,11 @@ docker run \
   --env DATASET_DIR=${DATASET_DIR} \
   --env PRECISION=${PRECISION} \
   --env OUTPUT_DIR=${OUTPUT_DIR} \
+<<<<<<<< HEAD:quickstart/language_translation/tensorflow/mlperf_gnmt/inference/cpu/.docs/docker.md
+  --env BATCH_SIZE=${batch_size} \
+========
   --env BATCH_SIZE=${BATCH_SIZE} \
+>>>>>>>> r3.1:quickstart/image_recognition/tensorflow/resnet101/inference/cpu/.docs/docker.md
   --env http_proxy=${http_proxy} \
   --env https_proxy=${https_proxy} \
   --volume ${DATASET_DIR}:${DATASET_DIR} \
diff --git a/quickstart/image_recognition/tensorflow/resnet50/inference/cpu/.docs/advancedoptions.md b/quickstart/image_recognition/tensorflow/resnet50/inference/cpu/.docs/advancedoptions.md
deleted file mode 100644
index aa7aa64e6..000000000
--- a/quickstart/image_recognition/tensorflow/resnet50/inference/cpu/.docs/advancedoptions.md
+++ /dev/null
@@ -1,4 +0,0 @@
-<!-- 61. Advanced Options -->
-
-See the [Advanced Options for Model Packages and Containers](/quickstart/common/tensorflow/ModelPackagesAdvancedOptions.md)
-document for more advanced use cases.
diff --git a/quickstart/image_recognition/tensorflow/resnet50/inference/cpu/.docs/baremetal.md b/quickstart/image_recognition/tensorflow/resnet50/inference/cpu/.docs/baremetal.md
deleted file mode 100644
index ee1247974..000000000
--- a/quickstart/image_recognition/tensorflow/resnet50/inference/cpu/.docs/baremetal.md
+++ /dev/null
@@ -1,105 +0,0 @@
-<!--- 50. AI Kit -->
-## Run the model
-
-Setup your environment using the instructions below, depending on if you are
-using [AI Kit](/docs/general/tensorflow/AIKit.md):
-
-<table>
-  <tr>
-    <th>Setup using AI Kit on Linux</th>
-    <th>Setup without AI Kit on Linux</th>
-    <th>Setup without AI Kit on Windows</th>
-  </tr>
-  <tr>
-    <td>
-      <p>To run using AI Kit on Linux you will need:</p>
-      <ul>
-        <li>numactl
-        <li>wget
-        <li>Activate the tensorflow conda environment
-        <pre>conda activate tensorflow</pre>
-      </ul>
-    </td>
-    <td>
-      <p>To run without AI Kit on Linux you will need:</p>
-      <ul>
-        <li>Python 3
-        <li><a href="https://pypi.org/project/intel-tensorflow/">intel-tensorflow>=2.5.0</a>
-        <li>git
-        <li>numactl
-        <li>wget
-        <li>A clone of the Model Zoo repo<br />
-        <pre>git clone https://github.com/IntelAI/models.git</pre>
-      </ul>
-    </td>
-    <td>
-      <p>To run without AI Kit on Windows you will need:</p>
-      <ul>
-        <li><a href="/docs/general/Windows.md">Intel Model Zoo on Windows Systems prerequisites</a>
-        <li>A clone of the Model Zoo repo<br />
-        <pre>git clone https://github.com/IntelAI/models.git</pre>
-      </ul>
-    </td>
-  </tr>
-</table>
-
-After finishing the setup above, download the pretrained model based on `PRECISION` and set the
-`PRETRAINED_MODEL` environment var to the path to the frozen graph.
-If you run on Windows, please use a browser to download the pretrained model using the link below.
-For Linux, run:
-```
-# FP32 Pretrained Model
-wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/resnet50_fp32_pretrained_model.pb
-export PRETRAINED_MODEL=$(pwd)/resnet50_fp32_pretrained_model.pb
-```
-```
-#Int8 Pretrained Model
-wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/resnet50_int8_pretrained_model.pb
-export PRETRAINED_MODEL=$(pwd)/resnet50_int8_pretrained_model.pb
-```
-Intel® Neural Compressor int8 quantized ResNet50 pre-trained model is available as another option to download and try.
-```
-wget https://storage.googleapis.com/intel-optimized-tensorflow/intel-neural-compressor/v1_13/resnet-50-inc-int8-inference.pb
-export PRETRAINED_MODEL=$(pwd)/resnet-50-inc-int8-inference.pb
-```
-Check the [instructions](/quickstart/image_recognition/tensorflow/generate_int8/README.md) for more details on how to quantize FP32 model using Intel® Neural Compressor.
-
-Set the environment variables and run quickstart script on either Linux or Windows systems. See the [list of quickstart scripts](#quick-start-scripts) for details on the different options.
-
-### Run on Linux
-```
-# cd to your model zoo directory
-cd models
-
-export PRETRAINED_MODEL=<path to the frozen graph downloaded above>
-export DATASET_DIR=<path to the ImageNet TF records>
-export PRECISION=<set the precision to "int8" or "fp32">
-export OUTPUT_DIR=<path to the directory where log files will be written>
-# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
-export BATCH_SIZE=<customized batch size value>
-
-./quickstart/image_recognition/tensorflow/resnet50/inference/cpu/<script name>.sh
-```
-
-### Run on Windows
-Using `cmd.exe`, run:
-```
-# cd to your model zoo directory
-cd models
-
-set PRETRAINED_MODEL=<path to the frozen graph downloaded above>
-set DATASET_DIR=<path to the ImageNet TF records>
-set PRECISION=<set the precision to "int8" or "fp32">
-set OUTPUT_DIR=<directory where log files will be written>
-# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
-set BATCH_SIZE=<customized batch size value>
-
-bash quickstart\image_recognition\tensorflow\resnet50\inference\cpu\<script name>.sh
-```
-> Note: You may use `cygpath` to convert the Windows paths to Unix paths before setting the environment variables. 
-As an example, if the dataset location on Windows is `D:\user\ImageNet`, convert the Windows path to Unix as shown:
-> ```
-> cygpath D:\user\ImageNet
-> /d/user/ImageNet
->```
->Then, set the `DATASET_DIR` environment variable `set DATASET_DIR=/d/user/ImageNet`.
diff --git a/quickstart/image_recognition/tensorflow/resnet50/inference/cpu/.docs/datasets.md b/quickstart/image_recognition/tensorflow/resnet50/inference/cpu/.docs/datasets.md
deleted file mode 100644
index c7dff167b..000000000
--- a/quickstart/image_recognition/tensorflow/resnet50/inference/cpu/.docs/datasets.md
+++ /dev/null
@@ -1,7 +0,0 @@
-<!--- 30. Datasets -->
-## Datasets
-
-Download and preprocess the ImageNet dataset using the [instructions here](/datasets/imagenet/README.md).
-After running the conversion script you should have a directory with the
-ImageNet dataset in the TF records format.
-Set the `DATASET_DIR` to point to this directory when running <model name>.
diff --git a/quickstart/image_recognition/tensorflow/resnet50/inference/cpu/.docs/description.md b/quickstart/image_recognition/tensorflow/resnet50/inference/cpu/.docs/description.md
deleted file mode 100644
index 21cf18b5b..000000000
--- a/quickstart/image_recognition/tensorflow/resnet50/inference/cpu/.docs/description.md
+++ /dev/null
@@ -1,4 +0,0 @@
-<!-- 10. Description -->
-
-This document has instructions for running <model name> <mode> using
-Intel-optimized TensorFlow.
diff --git a/quickstart/image_recognition/tensorflow/resnet50/inference/cpu/.docs/docker.md b/quickstart/image_recognition/tensorflow/resnet50/inference/cpu/.docs/docker.md
deleted file mode 100644
index 96ab17eb2..000000000
--- a/quickstart/image_recognition/tensorflow/resnet50/inference/cpu/.docs/docker.md
+++ /dev/null
@@ -1,32 +0,0 @@
-<!-- 60. Docker -->
-## Docker
-
-The model container `<docker image>` includes the scripts
-and libraries needed to run <model name> <mode>. To run one of the model
-inference quickstart scripts using this container, you'll need to provide volume mounts for
-the ImageNet dataset and an output directory where checkpoint files will be written.
-
-```
-DATASET_DIR=<path to the preprocessed imagenet dataset>
-PRECISION=<set the precision to "int8" or "fp32">
-OUTPUT_DIR=<directory where log files will be written>
-# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
-export BATCH_SIZE=<customized batch size value>
-
-docker run \
-  --env DATASET_DIR=${DATASET_DIR} \
-  --env PRECISION=${PRECISION} \
-  --env OUTPUT_DIR=${OUTPUT_DIR} \
-  --env BATCH_SIZE=${BATCH_SIZE} \
-  --env http_proxy=${http_proxy} \
-  --env https_proxy=${https_proxy} \
-  --volume ${DATASET_DIR}:${DATASET_DIR} \
-  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
-  --privileged --init -t \
-  <docker image> \
-  /bin/bash quickstart/<script name>.sh
-```
-
-If you are new to docker and are running into issues with the container,
-see [this document](https://github.com/IntelAI/models/tree/master/docs/general/docker.md)
-for troubleshooting tips.
diff --git a/quickstart/image_recognition/tensorflow/resnet50/inference/cpu/.docs/license.md b/quickstart/image_recognition/tensorflow/resnet50/inference/cpu/.docs/license.md
deleted file mode 100644
index a60f169f4..000000000
--- a/quickstart/image_recognition/tensorflow/resnet50/inference/cpu/.docs/license.md
+++ /dev/null
@@ -1,4 +0,0 @@
-<!--- 80. License -->
-## License
-
-[LICENSE](/LICENSE)
diff --git a/quickstart/image_recognition/tensorflow/resnet50/inference/cpu/.docs/quickstart.md b/quickstart/image_recognition/tensorflow/resnet50/inference/cpu/.docs/quickstart.md
deleted file mode 100644
index 7dcb6ae8f..000000000
--- a/quickstart/image_recognition/tensorflow/resnet50/inference/cpu/.docs/quickstart.md
+++ /dev/null
@@ -1,10 +0,0 @@
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| [`online_inference.sh`](/quickstart/image_recognition/tensorflow/resnet50/inference/cpu/online_inference.sh) | Runs online inference (batch_size=1). |
-| [`batch_inference.sh`](/quickstart/image_recognition/tensorflow/resnet50/inference/cpu/batch_inference.sh) | Runs batch inference (batch_size=128). |
-| [`accuracy.sh`](/quickstart/image_recognition/tensorflow/resnet50/inference/cpu/accuracy.sh) | Measures the model accuracy (batch_size=100). |
-
-
diff --git a/quickstart/image_recognition/tensorflow/resnet50/inference/cpu/.docs/title.md b/quickstart/image_recognition/tensorflow/resnet50/inference/cpu/.docs/title.md
deleted file mode 100644
index 1d6c60187..000000000
--- a/quickstart/image_recognition/tensorflow/resnet50/inference/cpu/.docs/title.md
+++ /dev/null
@@ -1,2 +0,0 @@
-<!--- 0. Title -->
-# <model name> <mode>
diff --git a/quickstart/image_recognition/tensorflow/resnet50/inference/cpu/README.md b/quickstart/image_recognition/tensorflow/resnet50/inference/cpu/README.md
deleted file mode 100644
index 54141d531..000000000
--- a/quickstart/image_recognition/tensorflow/resnet50/inference/cpu/README.md
+++ /dev/null
@@ -1,65 +0,0 @@
-<!--- 0. Title -->
-# ResNet50 inference
-
-<!-- 10. Description -->
-
-This document has instructions for running ResNet50 inference using
-Intel-optimized TensorFlow.
-
-<!--- 30. Datasets -->
-## Datasets
-
-Download and preprocess the ImageNet dataset using the [instructions here](/datasets/imagenet/README.md).
-After running the conversion script you should have a directory with the
-ImageNet dataset in the TF records format.
-Set the `DATASET_DIR` to point to this directory when running ResNet50.
-
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| [`online_inference.sh`](/quickstart/image_recognition/tensorflow/resnet50/inference/cpu/online_inference.sh) | Runs online inference (batch_size=1). |
-| [`batch_inference.sh`](/quickstart/image_recognition/tensorflow/resnet50/inference/cpu/batch_inference.sh) | Runs batch inference (batch_size=128). |
-| [`accuracy.sh`](/quickstart/image_recognition/tensorflow/resnet50/inference/cpu/accuracy.sh) | Measures the model accuracy (batch_size=100). |
-
-
-
-<!-- 60. Docker -->
-## Docker
-
-The model container `intel/image-recognition:tf-latest-resnet50-inference` includes the scripts
-and libraries needed to run ResNet50 inference. To run one of the model
-inference quickstart scripts using this container, you'll need to provide volume mounts for
-the ImageNet dataset and an output directory where checkpoint files will be written.
-
-```
-DATASET_DIR=<path to the preprocessed imagenet dataset>
-PRECISION=<set the precision to "int8" or "fp32">
-OUTPUT_DIR=<directory where log files will be written>
-# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
-export BATCH_SIZE=<customized batch size value>
-
-docker run \
-  --env DATASET_DIR=${DATASET_DIR} \
-  --env PRECISION=${PRECISION}
-  --env OUTPUT_DIR=${OUTPUT_DIR} \
-  --env BATCH_SIZE=${BATCH_SIZE} \
-  --env http_proxy=${http_proxy} \
-  --env https_proxy=${https_proxy} \
-  --volume ${DATASET_DIR}:${DATASET_DIR} \
-  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
-  --privileged --init -t \
-  intel/image-recognition:tf-latest-resnet50-inference \
-  /bin/bash quickstart/<script name>.sh
-```
-
-If you are new to docker and are running into issues with the container,
-see [this document](https://github.com/IntelAI/models/tree/master/docs/general/docker.md)
-for troubleshooting tips.
-
-<!--- 80. License -->
-## License
-
-[LICENSE](/LICENSE)
-
diff --git a/quickstart/image_recognition/tensorflow/resnet50/inference/cpu/accuracy.sh b/quickstart/image_recognition/tensorflow/resnet50/inference/cpu/accuracy.sh
deleted file mode 100755
index e3e2b5454..000000000
--- a/quickstart/image_recognition/tensorflow/resnet50/inference/cpu/accuracy.sh
+++ /dev/null
@@ -1,88 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2020 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-MODEL_DIR=${MODEL_DIR-$PWD}
-
-if [ -z "${OUTPUT_DIR}" ]; then
-  echo "The required environment variable OUTPUT_DIR has not been set"
-  exit 1
-fi
-
-# Create the output directory in case it doesn't already exist
-mkdir -p ${OUTPUT_DIR}
-
-if [ -z "${DATASET_DIR}" ]; then
-  echo "The required environment variable DATASET_DIR has not been set"
-  exit 1
-fi
-
-if [ ! -d "${DATASET_DIR}" ]; then
-  echo "The DATASET_DIR '${DATASET_DIR}' does not exist"
-  exit 1
-fi
-
-if [ -z "${PRECISION}" ]; then
-  echo "The required environment variable PRECISION has not been set"
-  echo "Please set PRECISION to fp32 or int8"
-  exit 1
-fi
-
-if [[ $PRECISION != "fp32" ]] && [[ $PRECISION != "int8" ]]; then
-  echo "The specified precision '${PRECISION}' is unsupported."
-  echo "Supported precisions are: fp32 and int8"
-  exit 1
-fi
-
-if [ -z "${PRETRAINED_MODEL}" ]; then
-  if [[ $PRECISION == "int8" ]]; then
-      PRETRAINED_MODEL="${MODEL_DIR}/resnet50_int8_pretrained_model.pb"
-  elif [[ $PRECISION == "fp32" ]]; then
-      PRETRAINED_MODEL="${MODEL_DIR}/resnet50_fp32_pretrained_model.pb"
-  else
-      echo "The specified precision '${PRECISION}' is unsupported."
-      echo "Supported precisions are: fp32 and int8"
-      exit 1
-  fi
-
-  if [[ ! -f "${PRETRAINED_MODEL}" ]]; then
-    echo "The pretrained model could not be found. Please set the PRETRAINED_MODEL env var to point to the frozen graph file."
-    exit 1
-  fi
-elif [[ ! -f "${PRETRAINED_MODEL}" ]]; then
-  echo "The file specified by the PRETRAINED_MODEL environment variable (${PRETRAINED_MODEL}) does not exist."
-  exit 1
-fi
-
-# If batch size env is not mentioned, then the workload will run with the default batch size.
-if [ -z "${BATCH_SIZE}"]; then
-  BATCH_SIZE="100"
-  echo "Running with default batch size of ${BATCH_SIZE}"
-fi
-
-source "${MODEL_DIR}/quickstart/common/utils.sh"
-_command python benchmarks/launch_benchmark.py \
-         --model-name=resnet50 \
-         --precision ${PRECISION} \
-         --mode=inference \
-         --framework tensorflow \
-         --in-graph ${PRETRAINED_MODEL} \
-         --data-location=${DATASET_DIR} \
-         --output-dir ${OUTPUT_DIR} \
-         --accuracy-only \
-         --batch-size=${BATCH_SIZE} \
-         --socket-id 0 \
-         $@
diff --git a/quickstart/image_recognition/tensorflow/resnet50/inference/cpu/batch_inference.sh b/quickstart/image_recognition/tensorflow/resnet50/inference/cpu/batch_inference.sh
deleted file mode 100755
index e841879d9..000000000
--- a/quickstart/image_recognition/tensorflow/resnet50/inference/cpu/batch_inference.sh
+++ /dev/null
@@ -1,89 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2020 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-MODEL_DIR=${MODEL_DIR-$PWD}
-ARG=""
-
-if [ -z "${OUTPUT_DIR}" ]; then
-  echo "The required environment variable OUTPUT_DIR has not been set"
-  exit 1
-fi
-
-# Create the output directory in case it doesn't already exist
-mkdir -p ${OUTPUT_DIR}
-
-if [ -z "${DATASET_DIR}" ]; then
-  echo "The required environment variable DATASET_DIR has not been set"
-  exit 1
-fi
-
-if [ ! -d "${DATASET_DIR}" ]; then
-  echo "The DATASET_DIR '${DATASET_DIR}' does not exist"
-  exit 1
-fi
-
-if [ -z "${PRECISION}" ]; then
-  echo "The required environment variable PRECISION has not been set"
-  echo "Please set PRECISION to fp32 or int8"
-  exit 1
-fi
-
-if [[ $PRECISION != "fp32" ]] && [[ $PRECISION != "int8" ]]; then
-  echo "The specified precision '${PRECISION}' is unsupported."
-  echo "Supported precisions are: fp32 and int8"
-  exit 1
-fi
-
-if [ -z "${PRETRAINED_MODEL}" ]; then
-  if [[ $PRECISION == "int8" ]]; then
-      PRETRAINED_MODEL="${MODEL_DIR}/resnet50_int8_pretrained_model.pb"
-      ARGS="$ARGS -- warmup_steps=50 steps=500"
-  elif [[ $PRECISION == "fp32" ]]; then
-      PRETRAINED_MODEL="${MODEL_DIR}/resnet50_fp32_pretrained_model.pb"
-  else
-      echo "The specified precision '${PRECISION}' is unsupported."
-      echo "Supported precisions are: fp32 and int8"
-      exit 1
-  fi
-  if [[ ! -f "${PRETRAINED_MODEL}" ]]; then
-    echo "The pretrained model could not be found. Please set the PRETRAINED_MODEL env var to point to the frozen graph file."
-    exit 1
-  fi
-elif [[ ! -f "${PRETRAINED_MODEL}" ]]; then
-  echo "The file specified by the PRETRAINED_MODEL environment variable (${PRETRAINED_MODEL}) does not exist."
-  exit 1
-fi
-
-# If batch size env is not mentioned, then the workload will run with the default batch size.
-if [ -z "${BATCH_SIZE}"]; then
-  BATCH_SIZE="128"
-  echo "Running with default batch size of ${BATCH_SIZE}"
-fi
-
-source "${MODEL_DIR}/quickstart/common/utils.sh"
-_command python benchmarks/launch_benchmark.py \
-         --model-name=resnet50 \
-         --precision ${PRECISION} \
-         --mode=inference \
-         --framework tensorflow \
-         --in-graph ${PRETRAINED_MODEL} \
-         --data-location=${DATASET_DIR} \
-         --output-dir ${OUTPUT_DIR} \
-         --batch-size=${BATCH_SIZE} \
-         --socket-id 0 \
-         $@ \
-         $ARGS
diff --git a/quickstart/image_recognition/tensorflow/resnet50/inference/cpu/online_inference.sh b/quickstart/image_recognition/tensorflow/resnet50/inference/cpu/online_inference.sh
deleted file mode 100755
index 422e64eed..000000000
--- a/quickstart/image_recognition/tensorflow/resnet50/inference/cpu/online_inference.sh
+++ /dev/null
@@ -1,89 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2020 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-MODEL_DIR=${MODEL_DIR-$PWD}
-ARS=""
-
-if [ -z "${OUTPUT_DIR}" ]; then
-  echo "The required environment variable OUTPUT_DIR has not been set"
-  exit 1
-fi
-
-# Create the output directory in case it doesn't already exist
-mkdir -p ${OUTPUT_DIR}
-
-if [ -z "${DATASET_DIR}" ]; then
-  echo "The required environment variable DATASET_DIR has not been set"
-  exit 1
-fi
-
-if [ ! -d "${DATASET_DIR}" ]; then
-  echo "The DATASET_DIR '${DATASET_DIR}' does not exist"
-  exit 1
-fi
-
-if [ -z "${PRECISION}" ]; then
-  echo "The required environment variable PRECISION has not been set"
-  echo "Please set PRECISION to fp32 or int8"
-  exit 1
-fi
-
-if [[ $PRECISION != "fp32" ]] && [[ $PRECISION != "int8" ]]; then
-  echo "The specified precision '${PRECISION}' is unsupported."
-  echo "Supported precisions are: fp32 and int8"
-  exit 1
-fi
-
-if [ -z "${PRETRAINED_MODEL}" ]; then
-  if [[ $PRECISION == "int8" ]]; then
-      PRETRAINED_MODEL="${MODEL_DIR}/resnet50_int8_pretrained_model.pb"
-      ARGS="$ARGS -- warmup_steps=50 steps=500"
-  elif [[ $PRECISION == "fp32" ]]; then
-      PRETRAINED_MODEL="${MODEL_DIR}/resnet50_fp32_pretrained_model.pb"
-  else
-      echo "The specified precision '${PRECISION}' is unsupported."
-      echo "Supported precisions are: fp32 and int8"
-      exit 1
-  fi
-  if [[ ! -f "${PRETRAINED_MODEL}" ]]; then
-    echo "The pretrained model could not be found. Please set the PRETRAINED_MODEL env var to point to the frozen graph file."
-    exit 1
-  fi
-elif [[ ! -f "${PRETRAINED_MODEL}" ]]; then
-  echo "The file specified by the PRETRAINED_MODEL environment variable (${PRETRAINED_MODEL}) does not exist."
-  exit 1
-fi
-
-# If batch size env is not mentioned, then the workload will run with the default batch size.
-if [ -z "${BATCH_SIZE}"]; then
-  BATCH_SIZE="1"
-  echo "Running with default batch size of ${BATCH_SIZE}"
-fi
-
-source "${MODEL_DIR}/quickstart/common/utils.sh"
-_command python benchmarks/launch_benchmark.py \
-         --model-name=resnet50 \
-         --precision ${PRECISION} \
-         --mode=inference \
-         --framework tensorflow \
-         --in-graph ${PRETRAINED_MODEL} \
-         --data-location=${DATASET_DIR} \
-         --output-dir ${OUTPUT_DIR} \
-         --batch-size=${BATCH_SIZE} \
-         --socket-id 0 \
-         $@ \
-         $ARGS
diff --git a/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/.docs/docker.md b/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/.docs/docker.md
index 9619d5190..7637ab888 100644
--- a/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/.docs/docker.md
+++ b/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/.docs/docker.md
@@ -8,7 +8,11 @@ the ImageNet dataset (if a real dataset is being used) and an output directory w
 
 ```
 DATASET_DIR=<path to the preprocessed imagenet dataset>
+<<<<<<<< HEAD:quickstart/image_recognition/tensorflow/resnet50/inference/cpu/.docs/docker.md
+PRECISION=<set the precision to "int8" or "fp32">
+========
 PRECISION=<set the precision to "int8" or "fp32" or "bfloat16" or "bfloat32">
+>>>>>>>> r3.1:quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/.docs/docker.md
 OUTPUT_DIR=<directory where log files will be written>
 # For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
 export BATCH_SIZE=<customized batch size value>
diff --git a/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/README_DEV_CAT.md b/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/README_DEV_CAT.md
new file mode 100644
index 000000000..8bfdc26d0
--- /dev/null
+++ b/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/README_DEV_CAT.md
@@ -0,0 +1,104 @@
+# TensorFlow ResNet50 v1.5 inference
+
+## Description
+This document has instructions for running ResNet50 v1.5 inference using Intel-optimized TensorFlow.
+
+## Pull Command
+```
+docker pull intel/image-recognition:centos-tf-cpu-resnet50v1-5-inference
+```
+
+<table>
+   <thead>
+      <tr>
+         <th>Script name</th>
+         <th>Description</th>
+      </tr>
+   </thead>
+   <tbody>
+      <tr>
+         <td>inference_realtime.sh</td>
+         <td>Runs multi instance realtime inference using 4 cores per instance for the specified precision (fp32, int8, bfloat16, fp16, bfloat32) with 1500 steps and 50 warmup steps. If no `DATASET_DIR` is set, synthetic data is used. Waits for all instances to complete, then prints a summarized throughput value.</td>
+      </tr>
+      <tr>
+         <td>inference_throughput.sh</td>
+         <td>Runs multi instance batch inference using 1 instance per socket for the specified precision (fp32, int8, bfloat16, fp16, bfloat32) with 1500 steps and 50 warmup steps. If no `DATASET_DIR` is set, synthetic data is used. Waits for all instances to complete, then prints a summarized throughput value.</td>
+      </tr>
+      <tr>
+         <td>inference_realtime_weightsharing.sh</td>
+         <td>Runs multi instance realtime inference with weight sharing for the specified precision (int8 or bfloat16) with 1500 steps and 100 warmup steps. If no `DATASET_DIR` is set, synthetic data is used. Waits for all instances to complete, then prints a summarized throughput value.</td>
+      </tr>
+      <tr>
+         <td>accuracy.sh</td>
+         <td>Measures the inference accuracy (providing a `DATASET_DIR` environment variable is required) for the specified precision (fp32, int8, bfloat16, fp16, bfloat32).</td>
+      </tr>
+   </tbody>
+</table>
+
+## Datasets
+Download and preprocess the ImageNet dataset using the [instructions here](https://github.com/IntelAI/models/blob/master/datasets/imagenet/README.md).
+After running the conversion script you should have a directory with the
+ImageNet dataset in the TF records format.
+
+Set the `DATASET_DIR` to point to the TF records directory when running ResNet50 v1.5.
+
+## Docker Run
+(Optional) Export related proxy into docker environment.
+```bash
+export DOCKER_RUN_ENVS="-e ftp_proxy=${ftp_proxy} \
+  -e FTP_PROXY=${FTP_PROXY} -e http_proxy=${http_proxy} \
+  -e HTTP_PROXY=${HTTP_PROXY} -e https_proxy=${https_proxy} \
+  -e HTTPS_PROXY=${HTTPS_PROXY} -e no_proxy=${no_proxy} \
+  -e NO_PROXY=${NO_PROXY} -e socks_proxy=${socks_proxy} \
+  -e SOCKS_PROXY=${SOCKS_PROXY}"
+```
+
+To run ResNet50 v1.5 inference, set environment variables to specify the dataset directory, precision, and an output directory. 
+```bash
+# Set the required environment vars
+export DATASET_DIR=<path to the dataset>
+export OUTPUT_DIR=<directory where log files will be written>
+export SCRIPT=<specify the script to run>
+export PRECISION=<specify the precision to run>
+export CORES_PER_INSTANCE=<specify the cores per instance. Default value is 4>
+export OMP_NUM_THREADS=<specify the number for OMP_NUM_THREADS. Default value is 4>
+
+# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
+export BATCH_SIZE=<customized batch size value>
+
+docker run --rm \
+  --env BATCH_SIZE=${BATCH_SIZE} \
+  --env DATASET_DIR=${DATASET_DIR} \
+  --env OUTPUT_DIR=${OUTPUT_DIR} \
+  --env PRECISION=${PRECISION} \
+  --env CORES_PER_INSTANCE=${CORES_PER_INSTANCE} \
+  --env OMP_NUM_THREADS=${OMP_NUM_THREADS} \
+  --volume ${DATASET_DIR}:${DATASET_DIR} \
+  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
+  ${DOCKER_RUN_ENVS} \
+  --privileged --init -it \
+  --shm-size 8G \
+  -w /workspace/tf-resnet50v1-5-inference \
+  intel/image-recognition:centos-tf-cpu-resnet50v1-5-inference \
+  /bin/bash quickstart/${SCRIPT}
+```
+
+## Documentation and Sources
+#### Get Started​
+[Docker* Repository](https://hub.docker.com/r/intel/image-recognition)
+
+[Main GitHub*](https://github.com/IntelAI/models)
+
+[Release Notes](https://github.com/IntelAI/models/releases)
+
+[Get Started Guide](https://github.com/IntelAI/models/blob/master/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/README_DEV_CAT.md)
+
+#### Code Sources
+[Dockerfile](https://github.com/IntelAI/models/tree/master/dockerfiles/tensorflow)
+
+[Report Issue](https://community.intel.com/t5/Intel-Optimized-AI-Frameworks/bd-p/optimized-ai-frameworks)
+
+## License Agreement
+LEGAL NOTICE: By accessing, downloading or using this software and any required dependent software (the “Software Package”), you agree to the terms and conditions of the software license agreements for the Software Package, which may also include notices, disclaimers, or license terms for third party software included with the Software Package. Please refer to the [license](https://github.com/IntelAI/models/tree/master/third_party) file for additional details.
+
+[View All Containers and Solutions 🡢](https://www.intel.com/content/www/us/en/developer/tools/software-catalog/containers.html?s=Newest)
diff --git a/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/accuracy.sh b/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/accuracy.sh
index 431e298c6..e29756ec2 100755
--- a/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/accuracy.sh
+++ b/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/accuracy.sh
@@ -86,7 +86,6 @@ BATCH_SIZE="${BATCH_SIZE:-"100"}"
 rm -rf  ${OUTPUT_DIR}/resnet50v1_5_${PRECISION}_${MODE}_bs${BATCH_SIZE}_accuracy.log
 
 source "${MODEL_DIR}/quickstart/common/utils.sh"
-_ht_status_spr
 _command python ${MODEL_DIR}/benchmarks/launch_benchmark.py \
   --model-name=resnet50v1_5 \
   --precision ${PRECISION} \
diff --git a/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/inference_realtime_multi_instance.sh b/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/inference_realtime_multi_instance.sh
index b1f56af2b..5be1ec743 100755
--- a/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/inference_realtime_multi_instance.sh
+++ b/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/inference_realtime_multi_instance.sh
@@ -109,6 +109,12 @@ echo "WARMUP_STEPS: $WARMUP_STEPS"
 export TF_ENABLE_MKL_NATIVE_FORMAT=1 
 export TF_ONEDNN_ENABLE_FAST_CONV=1 
 
+if [ -z "${TF_THREAD_PINNING_MODE}" ]; then
+  echo "TF_THREAD_PINNING_MODE is not set. Default configuration of thread pinning and spinning settings"
+  export TF_THREAD_PINNING_MODE=none,$(($CORES_PER_INSTANCE-1)),400
+  echo "TF_THREAD_PINNING_MODE: $TF_THREAD_PINNING_MODE"
+fi
+
 # Remove old log file
 rm -rf  ${OUTPUT_DIR}/resnet50v1_5_${PRECISION}_${MODE}_bs${BATCH_SIZE}_cores*_all_instances.log
 
@@ -124,6 +130,8 @@ _command python ${MODEL_DIR}/benchmarks/launch_benchmark.py \
   --output-dir ${OUTPUT_DIR} \
   --batch-size ${BATCH_SIZE} \
   --numa-cores-per-instance ${CORES_PER_INSTANCE} \
+  --num-cores=${CORES_PER_INSTANCE} \
+  --num-intra-threads ${CORES_PER_INSTANCE} --num-inter-threads 1 \
   --data-num-intra-threads ${CORES_PER_INSTANCE} --data-num-inter-threads 1 \
   $@ \
   -- \
diff --git a/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/inference_throughput_multi_instance.sh b/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/inference_throughput_multi_instance.sh
index 02a821aad..d860d8b67 100755
--- a/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/inference_throughput_multi_instance.sh
+++ b/quickstart/image_recognition/tensorflow/resnet50v1_5/inference/cpu/inference_throughput_multi_instance.sh
@@ -74,8 +74,17 @@ MODE="inference"
   
 # If cores per instance env is not mentioned, then the workload will run with the default value.
 if [ -z "${CORES_PER_INSTANCE}" ]; then
-  CORES_PER_INSTANCE="socket"
-  echo "Runs an instance per ${CORES_PER_INSTANCE}"
+  # Get number of cores per instance
+  CORES_PER_SOCKET=`lscpu | grep 'Core(s) per socket' | awk '{print $4}'`
+  SOCKETS=`lscpu | grep Socket | awk '{print $2}'`
+  NUMAS=`lscpu | grep 'NUMA node(s)' | awk '{print $3}'`
+  CORES_PER_INSTANCE=`expr $CORES_PER_SOCKET \* $SOCKETS / $NUMAS`
+  NUM_INSTANCES=`expr $cores_per_socket / $CORES_PER_NUMA`
+
+  echo "CORES_PER_SOCKET: $CORES_PER_SOCKET"
+  echo "SOCKETS: $SOCKETS"
+  echo "NUMAS: $NUMAS"
+  echo "CORES_PER_INSTANCE: $CORES_PER_INSTANCE"
 fi
 
 cores_per_socket=$(lscpu |grep 'Core(s) per socket:' |sed 's/[^0-9]//g')
@@ -83,11 +92,13 @@ cores_per_socket="${cores_per_socket//[[:blank:]]/}"
 
 # If OMP_NUM_THREADS env is not mentioned, then run with the default value
 if [ -z "${OMP_NUM_THREADS}" ]; then 
-  export OMP_NUM_THREADS=4
+  omp_num_threads=4
 else
-  export OMP_NUM_THREADS=${OMP_NUM_THREADS}
+  omp_num_threads=${OMP_NUM_THREADS}
 fi
 
+export OMP_NUM_THREADS=${omp_num_threads}
+
 #Set up env variable for bfloat32
 if [[ $PRECISION == "bfloat32" ]]; then
   export ONEDNN_DEFAULT_FPMATH_MODE=BF16
@@ -117,6 +128,12 @@ else
 fi
 echo "WARMUP_STEPS: $WARMUP_STEPS"
 
+if [ -z "${TF_THREAD_PINNING_MODE}" ]; then
+  echo "TF_THREAD_PINNING_MODE is not set. Default configuration of thread pinning and spinning settings"
+  export TF_THREAD_PINNING_MODE=none,$(($CORES_PER_INSTANCE-1)),400
+  echo "TF_THREAD_PINNING_MODE: $TF_THREAD_PINNING_MODE"
+fi
+
 # Remove old log file
 rm -rf  ${OUTPUT_DIR}/resnet50v1_5_${PRECISION}_${MODE}_bs${BATCH_SIZE}_cores*_all_instances.log
 
@@ -132,7 +149,9 @@ _command python ${MODEL_DIR}/benchmarks/launch_benchmark.py \
   --output-dir ${OUTPUT_DIR} \
   --batch-size ${BATCH_SIZE} \
   --numa-cores-per-instance ${CORES_PER_INSTANCE} \
-  --data-num-intra-threads ${cores_per_socket} --data-num-inter-threads 1 \
+  --num-cores=${CORES_PER_INSTANCE} \
+  --num-intra-threads ${CORES_PER_INSTANCE} --num-inter-threads 1 \
+  --data-num-intra-threads ${CORES_PER_INSTANCE} --data-num-inter-threads 1 \
   $@ \
   -- \
   $WARMUP_STEPS \
diff --git a/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/.docs/advancedoptions.md b/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/.docs/advancedoptions.md
deleted file mode 100644
index aa7aa64e6..000000000
--- a/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/.docs/advancedoptions.md
+++ /dev/null
@@ -1,4 +0,0 @@
-<!-- 61. Advanced Options -->
-
-See the [Advanced Options for Model Packages and Containers](/quickstart/common/tensorflow/ModelPackagesAdvancedOptions.md)
-document for more advanced use cases.
diff --git a/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/.docs/baremetal.md b/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/.docs/baremetal.md
deleted file mode 100644
index e448d67eb..000000000
--- a/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/.docs/baremetal.md
+++ /dev/null
@@ -1,59 +0,0 @@
-<!--- 50. AI Kit -->
-## Run the model
-
-Setup your environment using the instructions below, depending on if you are
-using [AI Kit](/docs/general/tensorflow/AIKit.md):
-
-<table>
-  <tr>
-    <th>Setup using AI Kit</th>
-    <th>Setup without AI Kit</th>
-  </tr>
-  <tr>
-    <td>
-      <p>To run using AI Kit you will need:</p>
-      <ul>
-        <li>numactl
-        <li>openmpi-bin (only required for multi-instance)
-        <li>openmpi-common (only required for multi-instance)
-        <li>openssh-client (only required for multi-instance)
-        <li>openssh-server (only required for multi-instance)
-        <li>libopenmpi-dev (only required for multi-instance)
-        <li>horovod==0.21.0 (only required for multi-instance)
-        <li>Activate the tensorflow conda environment
-        <pre>conda activate tensorflow</pre>
-      </ul>
-    </td>
-    <td>
-      <p>To run without AI Kit you will need:</p>
-      <ul>
-        <li>Python 3
-        <li>[intel-tensorflow>=2.5.0](https://pypi.org/project/intel-tensorflow/)
-        <li>git
-        <li>numactl
-        <li>openmpi-bin (only required for multi-instance)
-        <li>openmpi-common (only required for multi-instance)
-        <li>openssh-client (only required for multi-instance)
-        <li>openssh-server (only required for multi-instance)
-        <li>libopenmpi-dev (only required for multi-instance)
-        <li>horovod==0.21.0 (only required for multi-instance)
-        <li>A clone of the Model Zoo repo<br />
-        <pre>git clone https://github.com/IntelAI/models.git</pre>
-      </ul>
-    </td>
-  </tr>
-</table>
-
-Set the environment variables, navigate to your model zoo directory and run quickstart scripts. See the [list of quickstart scripts](#quick-start-scripts) for details on the different options.
-```
-# cd to your model zoo directory
-cd models
-
-export DATASET_DIR=<path to the ImageNet TF records>
-export OUTPUT_DIR=<path to the directory where log files and checkpoints will be written>
-export PRECISION=<set the precision to "fp32" or "bfloat16" or "bfloat32">
-# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
-export BATCH_SIZE=<customized batch size value>
-
-./quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/<script name>.sh
-```
diff --git a/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/.docs/container_build.md b/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/.docs/container_build.md
deleted file mode 100644
index 07716ac7f..000000000
--- a/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/.docs/container_build.md
+++ /dev/null
@@ -1,26 +0,0 @@
-## Build the container
-
-The <model name> <mode> package has scripts and a Dockerfile that are
-used to build a workload container that runs the model. This container
-uses the TensorFlow SPR container as it's base, so ensure that you have built
-the `tensorflow-spr.tar.gz` container prior to building this model container.
-
-Use `docker images` to verify that you have the base container built. For example:
-```
-$ docker images | grep tensorflow-spr
-model-zoo     tensorflow-spr    a5f08b0abf25     23 seconds ago   1.58GB
-```
-
-To build the <model name> <mode> container, extract the package and
-run the `build.sh` script.
-```
-# Extract the package
-tar -xzf <package name>
-cd <package dir>
-
-# Build the container
-./build.sh
-```
-
-After the build completes, you should have a container called
-`<docker image>` that will be used to run the model.
diff --git a/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/.docs/datasets.md b/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/.docs/datasets.md
deleted file mode 100644
index 52cf661c1..000000000
--- a/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/.docs/datasets.md
+++ /dev/null
@@ -1,9 +0,0 @@
-<!--- 30. Datasets -->
-## Datasets
-
-Note that the ImageNet dataset is used in these <model name> examples.
-Download and preprocess the ImageNet dataset using the [instructions here](/datasets/imagenet/README.md).
-After running the conversion script you should have a directory with the
-ImageNet dataset in the TF records format.
-
-Set the `DATASET_DIR` to point to this directory when running <model name>.
diff --git a/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/.docs/description.md b/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/.docs/description.md
deleted file mode 100644
index e1183ed3e..000000000
--- a/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/.docs/description.md
+++ /dev/null
@@ -1,5 +0,0 @@
-<!-- 10. Description -->
-
-This document has instructions for running <model name> <mode>
-using Intel-optimized TensorFlow.
-
diff --git a/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/.docs/docker.md b/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/.docs/docker.md
deleted file mode 100644
index b457799d4..000000000
--- a/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/.docs/docker.md
+++ /dev/null
@@ -1,33 +0,0 @@
-<!-- 60. Docker -->
-## Docker
-
-The model container `<docker image>` includes the scripts
-and libraries needed to run <model name> <mode>. To run one of the model
-inference quickstart scripts using this container, you'll need to provide volume mounts for
-the ImageNet dataset (if a real dataset is being used) and an output directory where
-checkpoint files will be written.
-
-```
-DATASET_DIR=<path to the preprocessed imagenet dataset>
-PRECISION=<set the precision to "fp32" or "bfloat16">
-OUTPUT_DIR=<directory where log files will be written>
-# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
-export BATCH_SIZE=<customized batch size value>
-
-docker run \
-  --env DATASET_DIR=${DATASET_DIR} \
-  --env PRECISION=${PRECISION} \
-  --env OUTPUT_DIR=${OUTPUT_DIR} \
-  --env BATCH_SIZE=${BATCH_SIZE} \
-  --env http_proxy=${http_proxy} \
-  --env https_proxy=${https_proxy} \
-  --volume ${DATASET_DIR}:${DATASET_DIR} \
-  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
-  --privileged --init -t \
-  <docker image> \
-  /bin/bash quickstart/<script name>.sh
-```
-
-If you are new to docker and are running into issues with the container,
-see [this document](https://github.com/IntelAI/models/tree/master/docs/general/docker.md)
-for troubleshooting tips.
diff --git a/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/.docs/docker_spr.md b/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/.docs/docker_spr.md
deleted file mode 100644
index e325bab3c..000000000
--- a/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/.docs/docker_spr.md
+++ /dev/null
@@ -1,21 +0,0 @@
-## Run the model
-
-After you've followed the instructions to [build the container](#build-the-container)
-and [prepare the dataset](#datasets), use the `run.sh` script from the container
-package to run <model name> <mode> in docker. Set environment variables to
-specify the dataset directory, precision to run, and
-an output directory. By default, the `run.sh` script will run the
-`multi_instance_training.sh` quickstart script.
-```
-# Navigate to the container package directory
-cd tf-spr-resnet50v1-5-training
-
-# Set the required environment vars
-export PRECISION=<specify the precision to run>
-export OUTPUT_DIR=<directory where log files will be written>
-# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
-export BATCH_SIZE=<customized batch size value>
-
-# Run the container with training.sh quickstart script
-./run.sh
-```
diff --git a/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/.docs/license.md b/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/.docs/license.md
deleted file mode 100644
index e547f148d..000000000
--- a/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/.docs/license.md
+++ /dev/null
@@ -1,4 +0,0 @@
-<!--- 80. License -->
-## License
-
-Licenses can be found in the model package, in the `licenses` directory.
diff --git a/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/.docs/quickstart.md b/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/.docs/quickstart.md
deleted file mode 100644
index b36a1e0b0..000000000
--- a/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/.docs/quickstart.md
+++ /dev/null
@@ -1,10 +0,0 @@
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| [`training_demo.sh`](/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/training_demo.sh) | Executes a short run using small batch sizes and a limited number of steps to demonstrate the training flow |
-| [`training_1_epoch.sh`](/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/training_1_epoch.sh) | Executes a test run that trains the model for 1 epoch and saves checkpoint files to an output directory. |
-| [`training_full.sh`](/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/training_full.sh) | Trains the model using the full dataset and runs until convergence (90 epochs) and saves checkpoint files to an output directory. Note that this will take a considerable amount of time. |
-| [`multi_instance_training_demo.sh`](/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/multi_instance_training_demo.sh) | Uses mpirun to execute 2 processes with 1 process per socket with a batch size of 256 for 50 steps. |
-| [`multi_instance_training.sh`](/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/multi_instance_training.sh) | Uses mpirun to execute 1 processes with 1 process per socket with a batch size of 1024 for the specified precision (fp32 or bfloat16 or bfloat32). Checkpoint files and logs for each instance are saved to the output directory.|
diff --git a/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/.docs/quickstart_spr.md b/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/.docs/quickstart_spr.md
deleted file mode 100644
index 488f3f757..000000000
--- a/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/.docs/quickstart_spr.md
+++ /dev/null
@@ -1,6 +0,0 @@
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| [`multi_instance_training.sh`](/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/multi_instance_training.sh) |	Uses mpirun to execute 1 processes with 1 process per socket with a batch size of 1024 for the specified precision (fp32 or bfloat16 or bfloat32). Checkpoint files and logs for each instance are saved to the output directory.|
diff --git a/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/.docs/title.md b/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/.docs/title.md
deleted file mode 100644
index 1d6c60187..000000000
--- a/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/.docs/title.md
+++ /dev/null
@@ -1,2 +0,0 @@
-<!--- 0. Title -->
-# <model name> <mode>
diff --git a/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/.docs/wrapper_package.md b/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/.docs/wrapper_package.md
deleted file mode 100644
index 5b543719d..000000000
--- a/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/.docs/wrapper_package.md
+++ /dev/null
@@ -1,16 +0,0 @@
-## Model Package
-
-The model package includes the Dockerfile and scripts needed to build and
-run <model name> <mode> in a container.
-```
-<package dir>
-├── README.md
-├── build.sh
-├── licenses
-│   ├── LICENSE
-│   └── third_party
-├── model_packages
-│   └── <package name>
-├──<package dir>.Dockerfile
-└── run.sh
-```
diff --git a/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/README_DEV_CAT.md b/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/README_DEV_CAT.md
new file mode 100644
index 000000000..4b1210034
--- /dev/null
+++ b/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/README_DEV_CAT.md
@@ -0,0 +1,90 @@
+# TensorFlow ResNet50 v1.5 training
+
+## Description
+This document has instructions for running ResNet50 v1.5 training using Intel-optimized TensorFlow.
+
+## Pull Command
+```
+docker pull intel/image-recognition:centos-tf-cpu-resnet50v1-5-training
+```
+
+<table>
+   <thead>
+      <tr>
+         <th>Script name</th>
+         <th>Description</th>
+      </tr>
+   </thead>
+   <tbody>
+      <tr>
+         <td>training.sh</td>
+         <td>Uses mpirun to execute 1 processes with 1 process per socket with a batch size of 1024 for the specified precision (fp32, bfloat16 or fp16). Checkpoint files and logs for each instance are saved to the output directory.</td>
+      </tr>
+   </tbody>
+</table>
+
+Note that the ImageNet dataset is used in these ResNet50 v1.5 examples.
+
+## Datasets
+
+Download and preprocess the ImageNet dataset using the [instructions here](https://github.com/IntelAI/models/blob/master/datasets/imagenet/README.md).
+After running the conversion script you should have a directory with the
+ImageNet dataset in the TF records format.
+
+Set the `DATASET_DIR` to point to this directory when running ResNet50 v1.5.
+
+## Docker Run
+(Optional) Export related proxy into docker environment.
+```bash
+export DOCKER_RUN_ENVS="-e ftp_proxy=${ftp_proxy} \
+  -e FTP_PROXY=${FTP_PROXY} -e http_proxy=${http_proxy} \
+  -e HTTP_PROXY=${HTTP_PROXY} -e https_proxy=${https_proxy} \
+  -e HTTPS_PROXY=${HTTPS_PROXY} -e no_proxy=${no_proxy} \
+  -e NO_PROXY=${NO_PROXY} -e socks_proxy=${socks_proxy} \
+  -e SOCKS_PROXY=${SOCKS_PROXY}"
+```
+
+To run ResNet50 v1.5 training, set environment variables to specify the dataset directory, precision to run, and an output directory. 
+```bash
+# Set the required environment vars
+export DATASET_DIR=<path to the dataset>
+export OUTPUT_DIR=<directory where log files will be written>
+export PRECISION=<specify the precision to run (fp32 or bfloat16 or bfloat32 or fp16)>
+
+# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
+export BATCH_SIZE=<customized batch size value>
+
+docker run --rm \
+  --env BATCH_SIZE=${BATCH_SIZE} \
+  --env DATASET_DIR=${DATASET_DIR} \
+  --env OUTPUT_DIR=${OUTPUT_DIR} \
+  --env PRECISION=${PRECISION} \
+  --volume ${DATASET_DIR}:${DATASET_DIR} \
+  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
+  ${DOCKER_RUN_ENVS} \
+  --privileged --init -it \
+  --shm-size 8G \
+  -w /workspace/tf-resnet50v1-5-training \
+  intel/image-recognition:centos-tf-cpu-resnet50v1-5-training \
+  /bin/bash quickstart/training.sh
+```
+
+## Documentation and Sources
+#### Get Started​
+[Docker* Repository](https://hub.docker.com/r/intel/image-recognition)
+
+[Main GitHub*](https://github.com/IntelAI/models)
+
+[Release Notes](https://github.com/IntelAI/models/releases)
+
+[Get Started Guide](https://github.com/IntelAI/models/blob/master/quickstart/image_recognition/tensorflow/resnet50v1_5/training/cpu/README_DEV_CAT.md)
+
+#### Code Sources
+[Dockerfile](https://github.com/IntelAI/models/tree/master/dockerfiles/tensorflow)
+
+[Report Issue](https://community.intel.com/t5/Intel-Optimized-AI-Frameworks/bd-p/optimized-ai-frameworks)
+
+## License Agreement
+LEGAL NOTICE: By accessing, downloading or using this software and any required dependent software (the “Software Package”), you agree to the terms and conditions of the software license agreements for the Software Package, which may also include notices, disclaimers, or license terms for third party software included with the Software Package. Please refer to the [license](https://github.com/IntelAI/models/tree/master/third_party) file for additional details.
+
+[View All Containers and Solutions 🡢](https://www.intel.com/content/www/us/en/developer/tools/software-catalog/containers.html?s=Newest)
diff --git a/quickstart/image_segmentation/tensorflow/3d_unet_mlperf/inference/cpu/README_DEV_CAT.md b/quickstart/image_segmentation/tensorflow/3d_unet_mlperf/inference/cpu/README_DEV_CAT.md
new file mode 100644
index 000000000..8d1aed0dc
--- /dev/null
+++ b/quickstart/image_segmentation/tensorflow/3d_unet_mlperf/inference/cpu/README_DEV_CAT.md
@@ -0,0 +1,95 @@
+# TensorFlow MLPerf 3D U-Net inference
+
+## Description
+This document has instructions for running MLPerf 3D U-Net inference using
+Intel-optimized TensorFlow.
+
+## Pull Command
+```
+docker pull intel/image-segmentation:centos-tf-cpu-3d-unet-mlperf-inference
+```
+
+<table>
+   <thead>
+      <tr>
+         <th>Script name</th>
+         <th>Description</th>
+      </tr>
+   </thead>
+   <tbody>
+      <tr>
+         <td>inference_realtime.sh</td>
+         <td>Runs multi instance realtime inference using 4 cores per instance for the specified precision (int8, fp32 or bfloat16) with 100 steps and 50 warmup steps. Dummy data is used for performance evaluation. Waits for all instances to complete, then prints a summarized throughput value.</td>
+      </tr>
+      <tr>
+         <td>inference_throughput.sh</td>
+         <td>Runs multi instance batch inference using 1 instance per socket for the specified precision (int8, fp32 or bfloat16) with 100 steps and 50 warmup steps. Dummy data is used for performance evaluation. Waits for all instances to complete, then prints a summarized throughput value.</td>
+      </tr>
+      <tr>
+         <td>accuracy.sh</td>
+         <td>Measures the inference accuracy (providing a `DATASET_DIR` environment variable is required) for the specified precision (int8, fp32 or bfloat16).</td>
+      </tr>
+   </tbody>
+</table>
+
+## Datasets
+Download [Brats 2019](https://www.med.upenn.edu/cbica/brats2019/data.html) separately and unzip the dataset.
+
+Set the `DATASET_DIR` to point to the directory that contains the dataset files when running MLPerf 3D U-Net accuracy script.
+
+## Docker Run
+(Optional) Export related proxy into docker environment.
+```bash
+export DOCKER_RUN_ENVS="-e ftp_proxy=${ftp_proxy} \
+  -e FTP_PROXY=${FTP_PROXY} -e http_proxy=${http_proxy} \
+  -e HTTP_PROXY=${HTTP_PROXY} -e https_proxy=${https_proxy} \
+  -e HTTPS_PROXY=${HTTPS_PROXY} -e no_proxy=${no_proxy} \
+  -e NO_PROXY=${NO_PROXY} -e socks_proxy=${socks_proxy} \
+  -e SOCKS_PROXY=${SOCKS_PROXY}"
+```
+
+To run MLPerf 3D U-Net inference, set environment variables to specify the dataset directory, precision and mode to run, and an output directory. 
+```bash
+# Set the required environment vars
+export DATASET_DIR=<path to the dataset>
+export OUTPUT_DIR=<directory where log files will be written>
+export SCRIPT=<specify the script to run>
+export PRECISION=<specify the precision to run>
+
+# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
+export BATCH_SIZE=<customized batch size value>
+
+docker run --rm \
+  --env BATCH_SIZE=${BATCH_SIZE} \
+  --env DATASET_DIR=${DATASET_DIR} \
+  --env OUTPUT_DIR=${OUTPUT_DIR} \
+  --env PRECISION=${PRECISION} \
+  --volume ${DATASET_DIR}:${DATASET_DIR} \
+  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
+  ${DOCKER_RUN_ENVS} \
+  --privileged --init -it \
+  --shm-size 8G \
+  -w /workspace/tf-3d-unet-mlperf-inference \
+  intel/image-segmentation:centos-tf-cpu-3d-unet-mlperf-inference \
+  /bin/bash quickstart/${SCRIPT}
+```
+
+## Documentation and Sources
+#### Get Started​
+[Docker* Repository](https://hub.docker.com/r/intel/image-segmentation)
+
+[Main GitHub*](https://github.com/IntelAI/models)
+
+[Release Notes](https://github.com/IntelAI/models/releases)
+
+[Get Started Guide](https://github.com/IntelAI/models/blob/master/quickstart/image-segmentation/tensorflow/3d_unet_mlperf/inference/cpu/README_DEV_CAT.md)
+
+#### Code Sources
+[Dockerfile](https://github.com/IntelAI/models/tree/master/dockerfiles/tensorflow)
+
+[Report Issue](https://community.intel.com/t5/Intel-Optimized-AI-Frameworks/bd-p/optimized-ai-frameworks)
+
+## License Agreement
+LEGAL NOTICE: By accessing, downloading or using this software and any required dependent software (the “Software Package”), you agree to the terms and conditions of the software license agreements for the Software Package, which may also include notices, disclaimers, or license terms for third party software included with the Software Package. Please refer to the [license](https://github.com/IntelAI/models/tree/master/third_party) file for additional details.
+
+[View All Containers and Solutions 🡢](https://www.intel.com/content/www/us/en/developer/tools/software-catalog/containers.html?s=Newest)
diff --git a/quickstart/image_segmentation/tensorflow/3d_unet_mlperf/inference/cpu/accuracy.sh b/quickstart/image_segmentation/tensorflow/3d_unet_mlperf/inference/cpu/accuracy.sh
index 64d341582..6a5862d79 100755
--- a/quickstart/image_segmentation/tensorflow/3d_unet_mlperf/inference/cpu/accuracy.sh
+++ b/quickstart/image_segmentation/tensorflow/3d_unet_mlperf/inference/cpu/accuracy.sh
@@ -79,7 +79,6 @@ if [[ $PRECISION == "bfloat32" ]]; then
 fi
 
 source "${MODEL_DIR}/quickstart/common/utils.sh"
-_ht_status_spr
 _command python ${MODEL_DIR}/benchmarks/launch_benchmark.py \
   --model-name=3d_unet_mlperf \
   --precision ${PRECISION} \
diff --git a/quickstart/language_modeling/pytorch/bert_large/inference/cpu/.docs/container_build.md b/quickstart/language_modeling/pytorch/bert_large/inference/cpu/.docs/container_build.md
deleted file mode 100644
index 9b4f9b303..000000000
--- a/quickstart/language_modeling/pytorch/bert_large/inference/cpu/.docs/container_build.md
+++ /dev/null
@@ -1,26 +0,0 @@
-## Build the container
-
-The <model name> <mode> package has scripts and a Dockerfile that are
-used to build a workload container that runs the model. This container
-uses the PyTorch/IPEX container as it's base, so ensure that you have built
-the `pytorch-ipex-spr.tar.gz` container prior to building this model container.
-
-Use `docker images` to verify that you have the base container built. For example:
-```
-$ docker images | grep pytorch-ipex-spr
-model-zoo         pytorch-ipex-spr         fecc7096a11e        40 minutes ago      8.31GB
-```
-
-To build the <model name> <mode> container, extract the package and
-run the `build.sh` script.
-```
-# Extract the package
-tar -xzf <package name>
-cd <package dir>
-
-# Build the container
-./build.sh
-```
-
-After the build completes, you should have a container called
-`<docker image>` that will be used to run the model.
diff --git a/quickstart/language_modeling/pytorch/bert_large/inference/cpu/.docs/dataset.md b/quickstart/language_modeling/pytorch/bert_large/inference/cpu/.docs/dataset.md
deleted file mode 100644
index a86fa90e1..000000000
--- a/quickstart/language_modeling/pytorch/bert_large/inference/cpu/.docs/dataset.md
+++ /dev/null
@@ -1,9 +0,0 @@
-## Datasets
-
-Please following this [link](https://github.com/huggingface/transformers/tree/v3.0.2/examples/question-answering)
-to get `dev-v1.1.json` and set the `EVAL_DATA_FILE` environment variable to point
-to the file:
-```
-wget https://rajpurkar.github.io/SQuAD-explorer/dataset/dev-v1.1.json
-export EVAL_DATA_FILE=$(pwd)/dev-v1.1.json
-```
diff --git a/quickstart/language_modeling/pytorch/bert_large/inference/cpu/.docs/description.md b/quickstart/language_modeling/pytorch/bert_large/inference/cpu/.docs/description.md
deleted file mode 100644
index c554ffab2..000000000
--- a/quickstart/language_modeling/pytorch/bert_large/inference/cpu/.docs/description.md
+++ /dev/null
@@ -1,5 +0,0 @@
-<!-- 10. Description -->
-## Description
-
-This document has instructions for running <model name> SQuAD1.1 <mode> using
-Intel-optimized PyTorch.
diff --git a/quickstart/language_modeling/pytorch/bert_large/inference/cpu/.docs/docker_spr.md b/quickstart/language_modeling/pytorch/bert_large/inference/cpu/.docs/docker_spr.md
deleted file mode 100644
index 1557543d2..000000000
--- a/quickstart/language_modeling/pytorch/bert_large/inference/cpu/.docs/docker_spr.md
+++ /dev/null
@@ -1,33 +0,0 @@
-## Run the model
-
-Download the `config.json` and fine tuned model from huggingface and set the `PRETRAINED_MODEL`
-environment variable to point to the directory that has both files:
-```
-mkdir bert_squad_model
-wget https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-whole-word-masking-finetuned-squad-config.json -O bert_squad_model/config.json
-wget https://cdn.huggingface.co/bert-large-uncased-whole-word-masking-finetuned-squad-pytorch_model.bin  -O bert_squad_model/pytorch_model.bin
-PRETRAINED_MODEL=$(pwd)/bert_squad_model
-```
-
-Once you have the pretarined model and have [built the container](#build-the-container),
-use the `run.sh` script from the container package to run <model name> <mode> in docker.
-Set environment variables to specify the precision to run, and an output directory.
-By default, the `run.sh` script will run the `run_multi_instance_realtime.sh` quickstart script.
-To run a different script, specify the name of the script using the `SCRIPT` environment
-variable.
-```
-# Navigate to the container package directory
-cd <package dir>
-
-# Set the required environment vars
-export PRETRAINED_MODEL=<path to the downloaded model files>
-export EVAL_DATA_FILE=<path to the dev-v1.1.json file>
-export PRECISION=<specify the precision to run>
-export OUTPUT_DIR=<directory where log files will be written>
-
-# Run the container with inference_realtime.sh quickstart script
-./run.sh
-
-# Use the SCRIPT env var to run a different quickstart script
-SCRIPT=run_accuracy.sh ./run.sh
-```
diff --git a/quickstart/language_modeling/pytorch/bert_large/inference/cpu/.docs/license.md b/quickstart/language_modeling/pytorch/bert_large/inference/cpu/.docs/license.md
deleted file mode 100644
index e547f148d..000000000
--- a/quickstart/language_modeling/pytorch/bert_large/inference/cpu/.docs/license.md
+++ /dev/null
@@ -1,4 +0,0 @@
-<!--- 80. License -->
-## License
-
-Licenses can be found in the model package, in the `licenses` directory.
diff --git a/quickstart/language_modeling/pytorch/bert_large/inference/cpu/.docs/quickstart.md b/quickstart/language_modeling/pytorch/bert_large/inference/cpu/.docs/quickstart.md
deleted file mode 100644
index 01cc6d564..000000000
--- a/quickstart/language_modeling/pytorch/bert_large/inference/cpu/.docs/quickstart.md
+++ /dev/null
@@ -1,12 +0,0 @@
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| `run_multi_instance_realtime.sh` | Runs multi instance realtime inference using 4 cores per instance for the specified precision (fp32, avx-fp32, int8, avx-int8, or bf16) using the [huggingface fine tuned model](https://cdn.huggingface.co/bert-large-uncased-whole-word-masking-finetuned-squad-pytorch_model.bin). |
-| `run_multi_instance_throughput.sh` | Runs multi instance batch inference using 1 instance per socket for the specified precision (fp32, avx-fp32, int8, avx-int8, or bf16) using the [huggingface fine tuned model](https://cdn.huggingface.co/bert-large-uncased-whole-word-masking-finetuned-squad-pytorch_model.bin). |
-| `run_accuracy.sh` | Measures the inference accuracy for the specified precision (fp32, avx-fp32, int8, avx-int8, or bf16) using the [huggingface fine tuned model](https://cdn.huggingface.co/bert-large-uncased-whole-word-masking-finetuned-squad-pytorch_model.bin). |
-
-> Note: The `avx-int8` and `avx-fp32` precisions run the same scripts as `int8` and `fp32`, except that the
-> `DNNL_MAX_CPU_ISA` environment variable is unset. The environment variable is
-> otherwise set to `DNNL_MAX_CPU_ISA=AVX512_CORE_AMX`.
diff --git a/quickstart/language_modeling/pytorch/bert_large/inference/cpu/.docs/title.md b/quickstart/language_modeling/pytorch/bert_large/inference/cpu/.docs/title.md
deleted file mode 100644
index a4ccbf863..000000000
--- a/quickstart/language_modeling/pytorch/bert_large/inference/cpu/.docs/title.md
+++ /dev/null
@@ -1,2 +0,0 @@
-<!--- 0. Title -->
-# PyTorch <model name> <mode>
diff --git a/quickstart/language_modeling/pytorch/bert_large/inference/cpu/.docs/wrapper_package.md b/quickstart/language_modeling/pytorch/bert_large/inference/cpu/.docs/wrapper_package.md
deleted file mode 100644
index 0d77199f3..000000000
--- a/quickstart/language_modeling/pytorch/bert_large/inference/cpu/.docs/wrapper_package.md
+++ /dev/null
@@ -1,16 +0,0 @@
-## Model Package
-
-The model package includes the Dockerfile and scripts needed to build and
-run <model name> <mode> in a container.
-```
-<package dir>
-├── README.md
-├── build.sh
-├── licenses
-│   ├── LICENSE
-│   └── third_party
-├── model_packages
-│   └── <package name>
-├── <package dir>.Dockerfile
-└── run.sh
-```
diff --git a/quickstart/language_modeling/pytorch/bert_large/inference/cpu/DEVCATALOG.md b/quickstart/language_modeling/pytorch/bert_large/inference/cpu/DEVCATALOG.md
new file mode 100644
index 000000000..aa77c6938
--- /dev/null
+++ b/quickstart/language_modeling/pytorch/bert_large/inference/cpu/DEVCATALOG.md
@@ -0,0 +1,104 @@
+# PyTorch BERT Large inference
+
+## Description 
+This document has instructions for running BERT Large inference using Intel Extension for PyTorch. 
+
+## Pull Command
+
+Docker image based on CentOS Stream8
+```
+docker pull intel/language-modeling:centos-pytorch-cpu-bert-large-inference
+```
+
+Docker image based on Ubuntu 22.04
+```
+docker pull intel/language-modeling:ubuntu-pytorch-cpu-bert-large-inference
+```
+## Quick Start Scripts
+| Script name | Description |
+|-------------|-------------|
+| `run_multi_instance_realtime.sh` | Runs multi instance realtime inference using 4 cores per instance for the specified precision (fp32, avx-fp32, int8, avx-int8, bf32 or bf16) using the [huggingface fine tuned model](https://cdn.huggingface.co/bert-large-uncased-whole-word-masking-finetuned-squad-pytorch_model.bin). |
+| `run_multi_instance_throughput.sh` | Runs multi instance batch inference using 1 instance per socket for the specified precision (fp32, avx-fp32, int8, avx-int8,bf32 or bf16) using the [huggingface fine tuned model](https://cdn.huggingface.co/bert-large-uncased-whole-word-masking-finetuned-squad-pytorch_model.bin). |
+| `run_accuracy.sh` | Measures the inference accuracy for the specified precision (fp32, avx-fp32, int8, avx-int8, or bf16) using the [huggingface fine tuned model](https://cdn.huggingface.co/bert-large-uncased-whole-word-masking-finetuned-squad-pytorch_model.bin). |
+
+**Note:** The `avx-int8` and `avx-fp32` precisions run the same scripts as `int8` and `fp32`, except that the `DNNL_MAX_CPU_ISA` environment variable is unset. The environment variable is otherwise set to `DNNL_MAX_CPU_ISA=AVX512_CORE_AMX`.
+
+## Datasets
+Please following this [link](https://github.com/huggingface/transformers/tree/v3.0.2/examples/question-answering) to get `dev-v1.1.json` and set the `EVAL_DATA_FILE` environment variable to point to the file:
+```
+wget https://rajpurkar.github.io/SQuAD-explorer/dataset/dev-v1.1.json
+export EVAL_DATA_FILE=$(pwd)/dev-v1.1.json
+```
+## Pre-Trained Model
+Download the `config.json` and fine tuned model from huggingface and set the `PRETRAINED_MODEL` environment variable to point to the directory that has both files:
+```
+mkdir bert_squad_model
+wget https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-whole-word-masking-finetuned-squad-config.json -O bert_squad_model/config.json
+wget https://cdn.huggingface.co/bert-large-uncased-whole-word-masking-finetuned-squad-pytorch_model.bin  -O bert_squad_model/pytorch_model.bin
+PRETRAINED_MODEL=$(pwd)/bert_squad_model
+```
+
+## Docker Run
+(Optional) Export related proxy into docker environment.
+```bash
+export DOCKER_RUN_ENVS="-e ftp_proxy=${ftp_proxy} \
+  -e FTP_PROXY=${FTP_PROXY} -e http_proxy=${http_proxy} \
+  -e HTTP_PROXY=${HTTP_PROXY} -e https_proxy=${https_proxy} \
+  -e HTTPS_PROXY=${HTTPS_PROXY} -e no_proxy=${no_proxy} \
+  -e NO_PROXY=${NO_PROXY} -e socks_proxy=${socks_proxy} \
+  -e SOCKS_PROXY=${SOCKS_PROXY}"
+```
+To run the BERT Large inference scripts, set environment variables to specify the dataset directory, precision and an output directory. 
+
+```bash
+export OS=<provide either centos or ubuntu>
+export EVAL_DATA_FILE=<path to the eval data>
+export OUTPUT_DIR=<directory where log files will be written>
+export PRECISION=<specify the precision>
+export SCRIPT=quickstart/<specify the quickstart script>
+export PRETRAINED_MODEL=<path to pre-trained model>
+
+IMAGE_NAME=intel/language-modeling:${OS}-pytorch-cpu-bert-large-inference
+DOCKER_ARGS="--privileged --init -it"
+WORKDIR=/workspace/pytorch-bert-large-inference
+EVAL_SCRIPT="${WORKDIR}/quickstart/transformers/examples/legacy/question-answering/run_squad.py"
+
+docker run --rm \
+  --env PRECISION=${PRECISION} \
+  --env EVAL_DATA_FILE=${EVAL_DATA_FILE} \
+  --env EVAL_SCRIPT=${EVAL_SCRIPT} \
+  --env FINETUNED_MODEL=${PRETRAINED_MODEL} \
+  --env INT8_CONFIG=${WORKDIR}/quickstart/configure.json \
+  --env OUTPUT_DIR=${OUTPUT_DIR} \
+  --env http_proxy=${http_proxy} \
+  --env https_proxy=${https_proxy} \
+  --env no_proxy=${no_proxy} \
+  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
+  --volume ${EVAL_DATA_FILE}:${EVAL_DATA_FILE} \
+  --volume ${PRETRAINED_MODEL}:${PRETRAINED_MODEL} \
+  ${DOCKER_RUN_ENVS} \
+  --shm-size 8G \
+  -w ${WORKDIR} \
+  ${DOCKER_ARGS} \
+  $IMAGE_NAME \
+  /bin/bash $SCRIPT ${PRECISION}
+  ```
+## Documentation and Sources
+#### Get Started​
+[Docker* Repository](https://hub.docker.com/r/intel/language-modeling)
+
+[Main GitHub*](https://github.com/IntelAI/models)
+
+[Release Notes](https://github.com/IntelAI/models/releases)
+
+[Get Started Guide](https://github.com/IntelAI/models/blob/master/quickstart/quickstart/language_modeling/pytorch/bert_large/inference/cpu/DEVCATALOG.md)
+
+#### Code Sources
+[Dockerfile](https://github.com/IntelAI/models/tree/master/docker/pyt-cpu)
+
+[Report Issue](https://community.intel.com/t5/Intel-Optimized-AI-Frameworks/bd-p/optimized-ai-frameworks)
+
+## License Agreement
+LEGAL NOTICE: By accessing, downloading or using this software and any required dependent software (the “Software Package”), you agree to the terms and conditions of the software license agreements for the Software Package, which may also include notices, disclaimers, or license terms for third party software included with the Software Package. Please refer to the [license](https://github.com/IntelAI/models/tree/master/third_party) file for additional details.
+
+[View All Containers and Solutions 🡢](https://www.intel.com/content/www/us/en/developer/tools/software-catalog/containers.html?s=Newest)
diff --git a/quickstart/language_modeling/pytorch/bert_large/inference/cpu/README.md b/quickstart/language_modeling/pytorch/bert_large/inference/cpu/README.md
index 42a1c930b..7e3b084c1 100644
--- a/quickstart/language_modeling/pytorch/bert_large/inference/cpu/README.md
+++ b/quickstart/language_modeling/pytorch/bert_large/inference/cpu/README.md
@@ -12,33 +12,19 @@ Intel-optimized PyTorch.
 
 Follow [link](/docs/general/pytorch/BareMetalSetup.md) to install Miniconda and build Pytorch, IPEX, TorchVison Jemalloc and TCMalloc.
 
-### Prepare model
-```
-  cd <clone of the model zoo>/quickstart/language_modeling/pytorch/bert_large/inference/cpu
-  git clone https://github.com/huggingface/transformers.git
-  cd transformers
-  git checkout v4.28.1
-  git apply ../../../../../../../models/language_modeling/pytorch/common/enable_ipex_for_transformers.diff
-  pip install -e ./
-  cd ../
-
-```
 ### Model Specific Setup
-* Install dependency
-```
-  conda install intel-openmp
-```
-
-* Download dataset
 
-  Please following this [link](https://github.com/huggingface/transformers/tree/v3.0.2/examples/question-answering) to get dev-v1.1.json
+* Set Jemalloc and tcmalloc Preload for better performance
 
-* Download fine-tuned model
+  The jemalloc should be built from the [General setup](#general-setup) section.
+  ```
+  export LD_PRELOAD="<path to the jemalloc directory>/lib/libjemalloc.so":"path_to/tcmalloc/lib/libtcmalloc.so":$LD_PRELOAD
+  export MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+  ```
+* Set IOMP preload for better performance
 ```
-  mkdir bert_squad_model
-  wget https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-whole-word-masking-finetuned-squad-config.json -O bert_squad_model/config.json
-  wget https://cdn.huggingface.co/bert-large-uncased-whole-word-masking-finetuned-squad-pytorch_model.bin  -O bert_squad_model/pytorch_model.bin
-  wget https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-whole-word-masking-finetuned-squad-vocab.txt -O bert_squad_model/vocab.txt
+  pip install packaging intel-openmp
+  export LD_PRELOAD=path/lib/libiomp5.so:$LD_PRELOAD
 ```
 
 * Set ENV to use fp16 AMX if you are using a supported platform
@@ -64,15 +50,23 @@ Follow [link](/docs/general/pytorch/BareMetalSetup.md) to install Miniconda and
   run_calibration.sh
 ```
 
-## Quick Start Scripts
+* [optional] Compile model with PyTorch Inductor backend
+```shell
+  export TORCH_INDUCTOR=1
+```
 
-|  DataType   | Throughput  |  Latency    |   Accuracy  |
-| ----------- | ----------- | ----------- | ----------- |
-| FP32        | bash run_multi_instance_throughput.sh fp32 | bash run_multi_instance_realtime.sh fp32 | bash run_accuracy.sh fp32 |
-| BF32        | bash run_multi_instance_throughput.sh bf32 | bash run_multi_instance_realtime.sh bf32 | bash run_accuracy.sh bf32 |
-| BF16        | bash run_multi_instance_throughput.sh bf16 | bash run_multi_instance_realtime.sh bf16 | bash run_accuracy.sh bf16 |
-| FP16        | bash run_multi_instance_throughput.sh fp16 | bash run_multi_instance_realtime.sh fp16 | bash run_accuracy.sh fp16 |
-| INT8        | bash run_multi_instance_throughput.sh int8 | bash run_multi_instance_realtime.sh int8 | bash run_accuracy.sh int8 |
+## Quick Start Scripts
+| Script name | Description |
+|-------------|-------------|
+| `run_multi_instance_realtime.sh` | Runs multi instance realtime inference using 4 cores per instance for the specified precision (fp32, avx-fp32, int8, avx-int8, bf32 or bf16) using the [huggingface fine tuned model](https://cdn.huggingface.co/bert-large-uncased-whole-word-masking-finetuned-squad-pytorch_model.bin). |
+| `run_multi_instance_throughput.sh` | Runs multi instance batch inference using 1 instance per socket for the specified precision (fp32, avx-fp32, int8, avx-int8,bf32 or bf16) using the [huggingface fine tuned model](https://cdn.huggingface.co/bert-large-uncased-whole-word-masking-finetuned-squad-pytorch_model.bin). |
+| `run_accuracy.sh` | Measures the inference accuracy for the specified precision (fp32, avx-fp32, int8, avx-int8, bf32 or bf16) using the [huggingface fine tuned model](https://cdn.huggingface.co/bert-large-uncased-whole-word-masking-finetuned-squad-pytorch_model.bin). |
+
+**Note**: The `avx-fp32` precision runs the same scripts as `fp32`, except that the `DNNL_MAX_CPU_ISA` environment variable is unset. The environment variable is otherwise set to `DNNL_MAX_CPU_ISA=AVX512_CORE_AMX`.
+* Set ENV to use AMX:
+  ```
+  export DNNL_MAX_CPU_ISA=AVX512_CORE_AMX
+  ```
 
 ## Quick Start Scripts for fast_bert with TPP optimization 
 
@@ -80,36 +74,49 @@ Follow [link](/docs/general/pytorch/BareMetalSetup.md) to install Miniconda and
 | ----------- |  ----------- |
 | BF16        | bash fast_bert_squad_infer.sh --use_tpp --unpad --tpp_bf16|
 
+## Datasets
+Please following this [link](https://github.com/huggingface/transformers/tree/v3.0.2/examples/question-answering) to get `dev-v1.1.json` and set the `EVAL_DATA_FILE` environment variable to point to the file:
+```
+wget https://rajpurkar.github.io/SQuAD-explorer/dataset/dev-v1.1.json
+export EVAL_DATA_FILE=$(pwd)/dev-v1.1.json
+```
+## Pre-Trained Model
+Download the `config.json` and fine tuned model from huggingface and set the `FINETUNED_MODEL` environment variable to point to the directory that has both files:
+```
+mkdir bert_squad_model
+wget https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-whole-word-masking-finetuned-squad-config.json -O bert_squad_model/config.json
+wget https://cdn.huggingface.co/bert-large-uncased-whole-word-masking-finetuned-squad-pytorch_model.bin  -O bert_squad_model/pytorch_model.bin
+export FINETUNED_MODEL=$(pwd)/bert_squad_model
+```
 
 ## Run the model
 
 Follow the instructions above to setup your bare metal environment, download and
 preprocess the dataset, and do the model specific setup. Once all the setup is done,
-the Model Zoo can be used to run a [quickstart script](#quick-start-scripts).
+the Intel® AI Reference Models can be used to run a [quickstart script](#quick-start-scripts).
 Ensure that you have enviornment variables set to point to the dataset directory
 and an output directory.
 
 ```
-# Clone the model zoo repo and set the MODEL_DIR
+# Clone the Intel® AI Reference Models repo and set the MODEL_DIR
 git clone https://github.com/IntelAI/models.git
 cd models
 export MODEL_DIR=$(pwd)
 
 # Clone the Transformers repo in the BERT large inference directory
-cd quickstart/language_modeling/pytorch/bert_large/inference/cpu
-git clone https://github.com/huggingface/transformers.git
-cd transformers
-git checkout v4.28.1
-git apply ../../../../../../../models/language_modeling/pytorch/common/enable_ipex_for_transformers.diff
-pip install -e ./
+./quickstart/language_modeling/pytorch/bert_large/inference/cpu/setup.sh
 
 # Env vars
 export FINETUNED_MODEL=<path to the fine tuned model>
 export EVAL_DATA_FILE=<path to dev-v1.1.json file>
 export OUTPUT_DIR=<path to an output directory>
+export PRECISION=< select from :- fp32, bf32, bf16, int8, avx-int8, avx-fp32>
+
+# Optional environemnt variables:
+export BATCH_SIZE=<set a value for batch size, else it will run with default batch size>
 
-# Run a quickstart script (for example, FP32 multi-instance realtime inference)
-bash run_multi_instance_realtime.sh fp32
+# Run a quickstart script:
+./quickstart/language_modeling/pytorch/bert_large/inference/cpu/<script.sh>
 ```
 
 <!--- 80. License -->
diff --git a/quickstart/language_modeling/pytorch/bert_large/inference/cpu/run_accuracy.sh b/quickstart/language_modeling/pytorch/bert_large/inference/cpu/run_accuracy.sh
index 192a40876..2534c8664 100755
--- a/quickstart/language_modeling/pytorch/bert_large/inference/cpu/run_accuracy.sh
+++ b/quickstart/language_modeling/pytorch/bert_large/inference/cpu/run_accuracy.sh
@@ -20,40 +20,42 @@
 ARGS=""
 precision=fp32
 
-if [[ "$1" == *"avx"* ]]; then
+if [[ "$PRECISION" == *"avx"* ]]; then
     unset DNNL_MAX_CPU_ISA
 fi
 
-if [[ "$1" == "bf16" ]]
+if [[ "$PRECISION" == "bf16" ]]
 then
     precision=bf16
     ARGS="$ARGS --bf16"
     echo "### running bf16 mode"
-elif [[ "$1" == "fp16" ]]
+elif [[ "$PRECISION" == "fp16" ]]
 then
     precision=fp16
     ARGS="$ARGS --fp16_cpu"
     echo "### running fp16 mode"
 
-elif [[ "$1" == "int8" || "$1" == "avx-int8" ]]
+elif [[ "$PRECISION" == "int8" || "$PRECISION" == "avx-int8" ]]
 then
     precision=int8
-    ARGS="$ARGS --int8"
+    ARGS="$ARGS --int8 --int8_bf16"
     echo "### running int8 mode"
-elif [[ "$1" == "bf32" ]]
+elif [[ "$PRECISION" == "bf32" ]]
 then
     precision=bf32
     ARGS="$ARGS --bf32"
     echo "### running bf32 mode"
-elif [[ "$1" == "fp32" || "$1" == "avx-fp32" ]]
+elif [[ "$PRECISION" == "fp32" || "$PRECISION" == "avx-fp32" ]]
 then
     precision=fp32
     echo "### running fp32 mode"
-elif [[ "$1" == "int8-fp32" ]]
+elif [[ "$PRECISION" == "int8-fp32" ]]
 then
     precision=int8
     ARGS="$ARGS --int8_fp32"
     echo "### running int8-fp32 mode"
+else
+    echo "Please set PRECISION to : fp32, int8, bf32, bf26, avx-int8 or avx-fp32"
 fi
 
 rm -f ${OUTPUT_DIR}/accuracy_log*
@@ -65,6 +67,7 @@ OUTPUT_DIR=${OUTPUT_DIR:-"${PWD}"}
 EVAL_SCRIPT=${EVAL_SCRIPT:-"./transformers/examples/legacy/question-answering/run_squad.py"}
 work_space=${work_space:-"${OUTPUT_DIR}"}
 
+TORCH_INDUCTOR=${TORCH_INDUCTOR:-"0"}
 if [ ${WEIGHT_SHAREING} ]; then
   CORES=`lscpu | grep Core | awk '{print $4}'`
   SOCKETS=`lscpu | grep Socket | awk '{print $2}'`
@@ -88,10 +91,12 @@ if [ ${WEIGHT_SHAREING} ]; then
   ARGS="$ARGS --num_streams $STREAM_PER_INSTANCE"
   ARGS="$ARGS --instance_number $numa_node_i"
 
-  numactl -C $start_core_i-$end_core_i --membind=$numa_node_i python $EVAL_SCRIPT $ARGS --model_type bert --model_name_or_path ${FINETUNED_MODEL}  --do_eval --do_lower_case --predict_file $EVAL_DATA_FILE  --per_gpu_eval_batch_size $BATCH_SIZE --learning_rate 3e-5 --num_train_epochs 2.0 --max_seq_length 384 --doc_stride 128 --output_dir ./tmp --tokenizer_name bert-large-uncased-whole-word-masking-finetuned-squad --use_jit --int8_config ${INT8_CONFIG} \
+  numactl -C $start_core_i-$end_core_i --membind=$numa_node_i python $EVAL_SCRIPT $ARGS --model_type bert --model_name_or_path ${FINETUNED_MODEL}  --do_eval --do_lower_case --predict_file $EVAL_DATA_FILE  --per_gpu_eval_batch_size $BATCH_SIZE --learning_rate 3e-5 --num_train_epochs 2.0 --max_seq_length 384 --doc_stride 128 --output_dir ./tmp --tokenizer_name bert-large-uncased-whole-word-masking-finetuned-squad --use_jit --ipex --int8_config ${INT8_CONFIG} \
   2>&1 | tee $LOG_0
+elif [[ "0" == ${TORCH_INDUCTOR} ]];then
+  python -m intel_extension_for_pytorch.cpu.launch --log_path=${OUTPUT_DIR} --log_file_prefix="accuracy_log" $EVAL_SCRIPT $ARGS --model_type bert --model_name_or_path ${FINETUNED_MODEL}  --do_eval --do_lower_case --predict_file $EVAL_DATA_FILE  --per_gpu_eval_batch_size $BATCH_SIZE --learning_rate 3e-5 --num_train_epochs 2.0 --max_seq_length 384 --doc_stride 128 --output_dir ./tmp --tokenizer_name bert-large-uncased-whole-word-masking-finetuned-squad --use_jit --ipex --int8_config ${INT8_CONFIG} 2>&1 | tee $LOG_0
 else
-  python -m intel_extension_for_pytorch.cpu.launch --log_path=${OUTPUT_DIR} --log_file_prefix="accuracy_log" $EVAL_SCRIPT $ARGS --model_type bert --model_name_or_path ${FINETUNED_MODEL}  --do_eval --do_lower_case --predict_file $EVAL_DATA_FILE  --per_gpu_eval_batch_size $BATCH_SIZE --learning_rate 3e-5 --num_train_epochs 2.0 --max_seq_length 384 --doc_stride 128 --output_dir ./tmp --tokenizer_name bert-large-uncased-whole-word-masking-finetuned-squad --use_jit --int8_config ${INT8_CONFIG} 2>&1 | tee $LOG_0
+  python -m intel_extension_for_pytorch.cpu.launch --log_path=${OUTPUT_DIR} --log_file_prefix="accuracy_log" $EVAL_SCRIPT $ARGS --model_type bert --model_name_or_path ${FINETUNED_MODEL}  --do_eval --do_lower_case --predict_file $EVAL_DATA_FILE  --per_gpu_eval_batch_size $BATCH_SIZE --learning_rate 3e-5 --num_train_epochs 2.0 --max_seq_length 384 --doc_stride 128 --output_dir ./tmp --tokenizer_name bert-large-uncased-whole-word-masking-finetuned-squad --inductor --int8_config ${INT8_CONFIG} 2>&1 | tee $LOG_0
 fi
 
 
diff --git a/quickstart/language_modeling/pytorch/bert_large/inference/cpu/run_multi_instance_realtime.sh b/quickstart/language_modeling/pytorch/bert_large/inference/cpu/run_multi_instance_realtime.sh
index 84cdb8032..d7bab0d5b 100755
--- a/quickstart/language_modeling/pytorch/bert_large/inference/cpu/run_multi_instance_realtime.sh
+++ b/quickstart/language_modeling/pytorch/bert_large/inference/cpu/run_multi_instance_realtime.sh
@@ -21,34 +21,36 @@
 ARGS="--benchmark"
 precision=fp32
 
-if [[ "$1" == *"avx"* ]]; then
+if [[ "$PRECISION" == *"avx"* ]]; then
     unset DNNL_MAX_CPU_ISA
 fi
 
-if [[ "$1" == "bf16" ]]
+if [[ "$PRECISION" == "bf16" ]]
 then
     precision=bf16
     ARGS="$ARGS --bf16"
     echo "### running bf16 mode"
-elif [[ "$1" == "fp16" ]]
+elif [[ "$PRECISION" == "fp16" ]]
 then
     precision=fp16
     ARGS="$ARGS --fp16_cpu"
     echo "### running fp16 mode"
-elif [[ "$1" == "bf32" ]]
+elif [[ "$PRECISION" == "bf32" ]]
 then
     precision=bf32
     ARGS="$ARGS --bf32"
     echo "### running bf32 mode"
-elif [[ "$1" == "int8" || "$1" == "avx-int8" ]]
+elif [[ "$PRECISION" == "int8" || "$PRECISION" == "avx-int8" ]]
 then
     precision=int8
-    ARGS="$ARGS --int8"
+    ARGS="$ARGS --int8 --int8_bf16"
     echo "### running int8 mode"
-elif [[ "$1" == "fp32" || "$1" == "avx-fp32" ]]
+elif [[ "$PRECISION" == "fp32" || "$PRECISION" == "avx-fp32" ]]
 then
     precision=fp32
     echo "### running fp32 mode"
+else
+    echo "Please set PRECISION to : fp32, int8, bf32, bf26, avx-int8 or avx-fp32"
 fi
 
 rm -rf ${OUTPUT_DIR}/latency_log*
@@ -65,7 +67,12 @@ OUTPUT_DIR=${OUTPUT_DIR:-${PWD}}
 EVAL_SCRIPT=${EVAL_SCRIPT:-"./transformers/examples/legacy/question-answering/run_squad.py"}
 work_space=${work_space:-${OUTPUT_DIR}}
 
-python -m intel_extension_for_pytorch.cpu.launch --ninstance ${NUMAS} --log_path=${OUTPUT_DIR} --log_file_prefix="./latency_log_${precision}" ${EVAL_SCRIPT} $ARGS --model_type bert --model_name_or_path ${FINETUNED_MODEL} --tokenizer_name bert-large-uncased-whole-word-masking-finetuned-squad  --do_eval --do_lower_case --predict_file $EVAL_DATA_FILE  --per_gpu_eval_batch_size $BATCH_SIZE --learning_rate 3e-5 --num_train_epochs 2.0 --max_seq_length 384 --doc_stride 128 --output_dir ./tmp --perf_begin_iter 20 --perf_run_iters 100 --use_jit --int8_config ${INT8_CONFIG} --use_share_weight --total_cores ${CORES_PER_NUMA}
+TORCH_INDUCTOR=${TORCH_INDUCTOR:-"0"}
+if [[ "0" == ${TORCH_INDUCTOR} ]];then
+    python -m intel_extension_for_pytorch.cpu.launch --ninstance ${NUMAS} --log_path=${OUTPUT_DIR} --log_file_prefix="./latency_log_${precision}" ${EVAL_SCRIPT} $ARGS --model_type bert --model_name_or_path ${FINETUNED_MODEL} --tokenizer_name bert-large-uncased-whole-word-masking-finetuned-squad  --do_eval --do_lower_case --predict_file $EVAL_DATA_FILE  --per_gpu_eval_batch_size $BATCH_SIZE --learning_rate 3e-5 --num_train_epochs 2.0 --max_seq_length 384 --doc_stride 128 --output_dir ./tmp --perf_begin_iter 20 --perf_run_iters 100 --use_jit --ipex --int8_config ${INT8_CONFIG} --use_share_weight --total_cores ${CORES_PER_NUMA}
+else
+    python -m intel_extension_for_pytorch.cpu.launch --ninstance ${NUMAS} --log_path=${OUTPUT_DIR} --log_file_prefix="./latency_log_${precision}" ${EVAL_SCRIPT} $ARGS --model_type bert --model_name_or_path ${FINETUNED_MODEL} --tokenizer_name bert-large-uncased-whole-word-masking-finetuned-squad  --do_eval --do_lower_case --predict_file $EVAL_DATA_FILE  --per_gpu_eval_batch_size $BATCH_SIZE --learning_rate 3e-5 --num_train_epochs 2.0 --max_seq_length 384 --doc_stride 128 --output_dir ./tmp --perf_begin_iter 20 --perf_run_iters 100 --inductor --int8_config ${INT8_CONFIG} --use_share_weight --total_cores ${CORES_PER_NUMA}
+fi
 CORES_PER_INSTANCE=4
 TOTAL_CORES=`expr $CORES \* $SOCKETS`
 INSTANCES=`expr $TOTAL_CORES / $CORES_PER_INSTANCE`
diff --git a/quickstart/language_modeling/pytorch/bert_large/inference/cpu/run_multi_instance_throughput.sh b/quickstart/language_modeling/pytorch/bert_large/inference/cpu/run_multi_instance_throughput.sh
index 667014c56..fd897479f 100755
--- a/quickstart/language_modeling/pytorch/bert_large/inference/cpu/run_multi_instance_throughput.sh
+++ b/quickstart/language_modeling/pytorch/bert_large/inference/cpu/run_multi_instance_throughput.sh
@@ -19,35 +19,37 @@
 ARGS="--benchmark"
 precision=fp32
 
-if [[ "$1" == *"avx"* ]]; then
+if [[ "$PRECISION" == *"avx"* ]]; then
     unset DNNL_MAX_CPU_ISA
 fi
 
-if [[ "$1" == "bf16" ]]
+if [[ "$PRECISION" == "bf16" ]]
 then
     precision=bf16
     ARGS="$ARGS --bf16"
     echo "### running bf16 mode"
-elif [[ "$1" == "fp16" ]]
+elif [[ "$PRECISION" == "fp16" ]]
 then
     precision=fp16
     ARGS="$ARGS --fp16_cpu"
     echo "### running fp16 mode"
 
-elif [[ "$1" == "bf32" ]]
+elif [[ "$PRECISION" == "bf32" ]]
 then
     precision=bf32
     ARGS="$ARGS --bf32"
     echo "### running bf32 mode"
-elif [[ "$1" == "int8" || "$1" == "avx-int8" ]]
+elif [[ "$PRECISION" == "int8" || "$PRECISION" == "avx-int8" ]]
 then
     precision=int8
-    ARGS="$ARGS --int8"
+    ARGS="$ARGS --int8 --int8_bf16"
     echo "### running int8 mode"
-elif [[ "$1" == "fp32" || "$1" == "avx-fp32" ]]
+elif [[ "$PRECISION" == "fp32" || "$PRECISION" == "avx-fp32" ]]
 then
     precision=fp32
     echo "### running fp32 mode"
+else
+    echo "Please set PRECISION to : fp32, int8, bf32, bf26, avx-int8 or avx-fp32"
 fi
 
 export MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000";
@@ -61,6 +63,7 @@ work_space=${work_space:-${OUTPUT_DIR}}
 
 rm -rf ${OUTPUT_DIR}/throughput_log*
 
+TORCH_INDUCTOR=${TORCH_INDUCTOR:-"0"}
 if [ ${WEIGHT_SHAREING} ]; then
   CORES=`lscpu | grep Core | awk '{print $4}'`
   SOCKETS=`lscpu | grep Socket | awk '{print $2}'`
@@ -84,12 +87,14 @@ if [ ${WEIGHT_SHAREING} ]; then
     ARGS="$ARGS --num_streams $STREAM_PER_INSTANCE"
     ARGS="$ARGS --instance_number $numa_node_i"
 
-    numactl -C $start_core_i-$end_core_i --membind=$numa_node_i python ${EVAL_SCRIPT} $ARGS --model_type bert --model_name_or_path ${FINETUNED_MODEL} --tokenizer_name bert-large-uncased-whole-word-masking-finetuned-squad  --do_eval --do_lower_case --predict_file $EVAL_DATA_FILE --per_gpu_eval_batch_size $BATCH_SIZE --learning_rate 3e-5 --num_train_epochs 2.0 --max_seq_length 384 --doc_stride 128 --output_dir ./tmp --perf_begin_iter 15 --use_jit --perf_run_iters 40 --int8_config ${INT8_CONFIG} \
+    numactl -C $start_core_i-$end_core_i --membind=$numa_node_i python ${EVAL_SCRIPT} $ARGS --model_type bert --model_name_or_path ${FINETUNED_MODEL} --tokenizer_name bert-large-uncased-whole-word-masking-finetuned-squad  --do_eval --do_lower_case --predict_file $EVAL_DATA_FILE --per_gpu_eval_batch_size $BATCH_SIZE --learning_rate 3e-5 --num_train_epochs 2.0 --max_seq_length 384 --doc_stride 128 --output_dir ./tmp --perf_begin_iter 15 --use_jit --ipex --perf_run_iters 40 --int8_config ${INT8_CONFIG} \
     2>&1 | tee ${LOG_i} &
   done
   wait
+elif [[ "0" == ${TORCH_INDUCTOR} ]];then
+  python -m intel_extension_for_pytorch.cpu.launch --throughput_mode --enable_jemalloc --log_path=${OUTPUT_DIR} --log_file_prefix="./throughput_log_${precision}" ${EVAL_SCRIPT} $ARGS --model_type bert --model_name_or_path ${FINETUNED_MODEL} --tokenizer_name bert-large-uncased-whole-word-masking-finetuned-squad  --do_eval --do_lower_case --predict_file $EVAL_DATA_FILE --per_gpu_eval_batch_size $BATCH_SIZE --learning_rate 3e-5 --num_train_epochs 2.0 --max_seq_length 384 --doc_stride 128 --output_dir ./tmp --perf_begin_iter 15 --use_jit --ipex --perf_run_iters 40 --int8_config ${INT8_CONFIG}
 else
-  python -m intel_extension_for_pytorch.cpu.launch --throughput_mode --enable_jemalloc --log_path=${OUTPUT_DIR} --log_file_prefix="./throughput_log_${precision}" ${EVAL_SCRIPT} $ARGS --model_type bert --model_name_or_path ${FINETUNED_MODEL} --tokenizer_name bert-large-uncased-whole-word-masking-finetuned-squad  --do_eval --do_lower_case --predict_file $EVAL_DATA_FILE --per_gpu_eval_batch_size $BATCH_SIZE --learning_rate 3e-5 --num_train_epochs 2.0 --max_seq_length 384 --doc_stride 128 --output_dir ./tmp --perf_begin_iter 15 --use_jit --perf_run_iters 40 --int8_config ${INT8_CONFIG}
+  python -m intel_extension_for_pytorch.cpu.launch --throughput_mode --enable_jemalloc --log_path=${OUTPUT_DIR} --log_file_prefix="./throughput_log_${precision}" ${EVAL_SCRIPT} $ARGS --model_type bert --model_name_or_path ${FINETUNED_MODEL} --tokenizer_name bert-large-uncased-whole-word-masking-finetuned-squad  --do_eval --do_lower_case --predict_file $EVAL_DATA_FILE --per_gpu_eval_batch_size $BATCH_SIZE --learning_rate 3e-5 --num_train_epochs 2.0 --max_seq_length 384 --doc_stride 128 --output_dir ./tmp --perf_begin_iter 15 --inductor --perf_run_iters 40 --int8_config ${INT8_CONFIG}
 fi
 
 throughput=$(grep 'Throughput:' ${OUTPUT_DIR}/throughput_log* |sed -e 's/.*Throughput//;s/[^0-9.]//g' |awk '
diff --git a/quickstart/language_modeling/pytorch/bert_large/inference/cpu/setup.sh b/quickstart/language_modeling/pytorch/bert_large/inference/cpu/setup.sh
new file mode 100755
index 000000000..6603718a0
--- /dev/null
+++ b/quickstart/language_modeling/pytorch/bert_large/inference/cpu/setup.sh
@@ -0,0 +1,37 @@
+#!/usr/bin/env bash
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+pip install tensorboard
+
+# Check the operating system type
+os_type=$(awk -F= '/^NAME/{print $2}' /etc/os-release)
+
+# Install model specific dependencies:
+if [[ "$os_name" == *"CentOS"* ]]; then
+    yum install -y git-lfs
+elif [[ "$os_name" == *"Ubuntu"* ]]; then
+    apt install -y git-lfs
+fi
+
+rm -rf transformers
+git clone https://github.com/huggingface/transformers.git
+cd transformers
+git checkout v4.28.1
+git lfs pull
+git apply ../models/language_modeling/pytorch/common/enable_ipex_for_transformers.diff
+pip install -e ./
+cd ..
diff --git a/quickstart/language_modeling/pytorch/bert_large/training/cpu/.docs/container_build.md b/quickstart/language_modeling/pytorch/bert_large/training/cpu/.docs/container_build.md
deleted file mode 100644
index 9b4f9b303..000000000
--- a/quickstart/language_modeling/pytorch/bert_large/training/cpu/.docs/container_build.md
+++ /dev/null
@@ -1,26 +0,0 @@
-## Build the container
-
-The <model name> <mode> package has scripts and a Dockerfile that are
-used to build a workload container that runs the model. This container
-uses the PyTorch/IPEX container as it's base, so ensure that you have built
-the `pytorch-ipex-spr.tar.gz` container prior to building this model container.
-
-Use `docker images` to verify that you have the base container built. For example:
-```
-$ docker images | grep pytorch-ipex-spr
-model-zoo         pytorch-ipex-spr         fecc7096a11e        40 minutes ago      8.31GB
-```
-
-To build the <model name> <mode> container, extract the package and
-run the `build.sh` script.
-```
-# Extract the package
-tar -xzf <package name>
-cd <package dir>
-
-# Build the container
-./build.sh
-```
-
-After the build completes, you should have a container called
-`<docker image>` that will be used to run the model.
diff --git a/quickstart/language_modeling/pytorch/bert_large/training/cpu/.docs/dataset.md b/quickstart/language_modeling/pytorch/bert_large/training/cpu/.docs/dataset.md
deleted file mode 100644
index 76993929e..000000000
--- a/quickstart/language_modeling/pytorch/bert_large/training/cpu/.docs/dataset.md
+++ /dev/null
@@ -1,92 +0,0 @@
-## Datasets
-
-### Input files
-
-Download the following files from the [MLCommons members Google Drive](https://drive.google.com/drive/u/0/folders/1oQF4diVHNPCclykwdvQJw8n_VIWwV0PT):
-* TensorFlow checkpoint (bert_model.ckpt) containing the pre-trained weights (which is actually 3 files).
-* Vocab file (vocab.txt) to map WordPiece to word id.
-* Config file (bert_config.json) which specifies the hyperparameters of the model.
-
-### Download and extract the dataset
-
-From the [MLCommons BERT Processed dataset
-directory](https://drive.google.com/drive/folders/1cywmDnAsrP5-2vsr8GDc6QUc7VWe-M3v?usp=sharing)
-download the `results_text.tar.gz` and `bert_reference_results_text_md5.txt` files. Perform the
-following steps to extract the tar file and check the MD5 checksums.
-```shell
-tar xf results_text.tar.gz
-cd results4
-md5sum --check ../bert_reference_results_text_md5.txt
-cd ..
-```
-After completing this step you should have a directory called `results4/` that
-contains 502 files for a total of about 13Gbytes.
-
-### Generate the BERT input dataset
-
-There are input preprocessing scripts in the `<package dir>/input_preprocessing` folder.
-Note that the pretraining dataset is about `539GB`, so ensure that you have enough disk
-space. First, navigate to the folder with the preprocessing scripts.
-```
-cd <package dir>/input_preprocessing
-```
-> Note that the `results4` folder and the `vocab.txt` file will need to be in the
-> `input_preprocessing` before running the preprocessing scripts.
-
-The `create_pretraining_data.py` script duplicates the input plain text, replaces
-different sets of words with masks for each duplication, and serializes the
-output into the HDF5 file format.
-
-The following snippet shows how `create_pretraining_data.py` is called by a parallelized
-script that can be called as shown below.  The script reads the text data from
-the `results4/` subdirectory and outputs the resulting 500 hdf5 files to a
-subdirectory named `hdf5/`.
-
-```shell
-pip install tensorflow-cpu
-```
-
-For phase1 the seq_len=128:
-```shell
-export SEQ_LEN=128
-./parallel_create_hdf5.sh
-```
-For phase2 the seq_len=512:
-```shell
-export SEQ_LEN=512
-./parallel_create_hdf5.sh
-```
-
-The resulting `hdf5` subdirs will have 500 files named
-`part-00???-of-0500.hdf5` and have a size of about 539 Gigabytes.
-
-Next we need to shard the data into 2048 chunks.  This is done by calling the
-chop_hdf5_files.py script.  This script reads the 500 hdf5 files from
-subdirectory `hdf5` and creates 2048 hdf5 files in subdirectory
-`2048_shards_uncompressed`.
-
-For phase1:
-
-```shell
-export SEQ_LEN=128
-python3 ./chop_hdf5_files.py
-```
-
-For phase2:
-
-```shell
-export SEQ_LEN=512
-python3 ./chop_hdf5_files.py
-```
-
-The above will produce a subdirectory named `2048_shards_uncompressed/`
-containing 2048 files named `part_*_of_2048.hdf5` and have a size of about 539 Gigabytes.
-you can use "SHARD_NUM" to control the shard files number. the default "SHARD_NUM" if 2048.
-
-```
-<DATASET_DIR>
-├── 2048_shards_uncompressed_512
-│   └── part-00000-of-00xxx
-└── 2048_shards_uncompressed_128
-    └── part-00000-of-00xxx
-```
diff --git a/quickstart/language_modeling/pytorch/bert_large/training/cpu/.docs/description.md b/quickstart/language_modeling/pytorch/bert_large/training/cpu/.docs/description.md
deleted file mode 100644
index 0421f1e0b..000000000
--- a/quickstart/language_modeling/pytorch/bert_large/training/cpu/.docs/description.md
+++ /dev/null
@@ -1,5 +0,0 @@
-<!-- 10. Description -->
-## Description
-
-This document has instructions for running <model name> pre-<mode> using
-Intel-optimized PyTorch.
diff --git a/quickstart/language_modeling/pytorch/bert_large/training/cpu/.docs/docker_spr.md b/quickstart/language_modeling/pytorch/bert_large/training/cpu/.docs/docker_spr.md
deleted file mode 100644
index 929a0bcf6..000000000
--- a/quickstart/language_modeling/pytorch/bert_large/training/cpu/.docs/docker_spr.md
+++ /dev/null
@@ -1,32 +0,0 @@
-## Run the model
-
-After you've followed the instructions to [build the container](#build-the-container),
-download the [dataset and config file](#datasets), use the `run.sh` script from the
-container package to run BERT Large training in docker. Set environment variables to
-specify the precision to run, dataset directory, config file directory, and an
-output directory. Use an empty `OUTPUT_DIR` to start with to prevent any previously
-generated checkpoints from getting picked up. By default, the `run.sh` script will
-run the `run_bert_pretrain_phase1.sh` quickstart script.
-```
-# Navigate to the container package directory
-cd <package dir>
-
-# Set the required environment vars to run phase 1
-export PRECISION=<specify the precision to run>
-export OUTPUT_DIR=<directory where log and model files will be written for phase 1>
-export DATASET_DIR=<path to the preprocessed dataset>
-export CONFIG_FILE=<path to the bert_config.json>
-
-# Run the container with the default run_bert_pretrain_phase1.sh quickstart script
-./run.sh
-```
-To run phase 2, to the directory where checkpoints were generated during phase 1
-pretraining. This `CHECKPOINT_DIR` should also have the `bert_config.json` file.
-```
-# To run phase 2, set the CHECKPOINT_DIR to the folder with checkpoints generated during phase 1
-export CHECKPOINT_DIR=<directory with checkpoints and the bert_config.json file>
-
-# Define a new directory for phase 2 output, and set the SCRIPT var to run phase 2
-export OUTPUT_DIR=<directory where log and model files will be written for phase 2>
-SCRIPT=run_bert_pretrain_phase2.sh ./run.sh
-```
diff --git a/quickstart/language_modeling/pytorch/bert_large/training/cpu/.docs/license.md b/quickstart/language_modeling/pytorch/bert_large/training/cpu/.docs/license.md
deleted file mode 100644
index e547f148d..000000000
--- a/quickstart/language_modeling/pytorch/bert_large/training/cpu/.docs/license.md
+++ /dev/null
@@ -1,4 +0,0 @@
-<!--- 80. License -->
-## License
-
-Licenses can be found in the model package, in the `licenses` directory.
diff --git a/quickstart/language_modeling/pytorch/bert_large/training/cpu/.docs/quickstart.md b/quickstart/language_modeling/pytorch/bert_large/training/cpu/.docs/quickstart.md
deleted file mode 100644
index a3a1f12d3..000000000
--- a/quickstart/language_modeling/pytorch/bert_large/training/cpu/.docs/quickstart.md
+++ /dev/null
@@ -1,11 +0,0 @@
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| `run_bert_pretrain_phase1.sh` | Runs BERT large pretraining phase 1 using max_seq_len=128 for the first 90% dataset for the specified precision (fp32, avx-fp32, or bf16). The script saves the model to the `OUTPUT_DIR` in a directory called `model_save`. |
-| `run_bert_pretrain_phase2.sh` | Runs BERT large pretraining phase 2 using max_seq_len=512 with the remaining 10% of the dataset for the specified precision (fp32, avx-fp32, or bf16). Use path to the `model_save` directory from phase one as the `CHECKPOINT_DIR` for phase 2. |
-
-> Note: The `avx-fp32` precision runs the same scripts as `fp32`, except that the
-> `DNNL_MAX_CPU_ISA` environment variable is unset. The environment variable is
-> otherwise set to `DNNL_MAX_CPU_ISA=AVX512_CORE_AMX`.
diff --git a/quickstart/language_modeling/pytorch/bert_large/training/cpu/.docs/title.md b/quickstart/language_modeling/pytorch/bert_large/training/cpu/.docs/title.md
deleted file mode 100644
index a4ccbf863..000000000
--- a/quickstart/language_modeling/pytorch/bert_large/training/cpu/.docs/title.md
+++ /dev/null
@@ -1,2 +0,0 @@
-<!--- 0. Title -->
-# PyTorch <model name> <mode>
diff --git a/quickstart/language_modeling/pytorch/bert_large/training/cpu/.docs/wrapper_package.md b/quickstart/language_modeling/pytorch/bert_large/training/cpu/.docs/wrapper_package.md
deleted file mode 100644
index bf3632aba..000000000
--- a/quickstart/language_modeling/pytorch/bert_large/training/cpu/.docs/wrapper_package.md
+++ /dev/null
@@ -1,22 +0,0 @@
-## Model Package
-
-The model package includes the Dockerfile and scripts needed to build and
-run <model name> <mode> in a container.
-```
-<package dir>
-├── README.md
-├── build.sh
-├── input_preprocessing
-│   ├── chop_hdf5_files.py
-│   ├── create_pretraining_data.py
-│   ├── create_pretraining_data_wrapper.sh
-│   ├── prallel_create_hdf5.sh
-│   └── tokenization.py
-├── licenses
-│   ├── LICENSE
-│   └── third_party
-├── model_packages
-│   └── <package name>
-├── <package dir>.Dockerfile
-└── run.sh
-```
diff --git a/quickstart/language_modeling/pytorch/bert_large/training/cpu/DEVCATALOG.md b/quickstart/language_modeling/pytorch/bert_large/training/cpu/DEVCATALOG.md
new file mode 100644
index 000000000..0f6b77d85
--- /dev/null
+++ b/quickstart/language_modeling/pytorch/bert_large/training/cpu/DEVCATALOG.md
@@ -0,0 +1,100 @@
+# PyTorch BERT Large training
+
+## Description 
+This document has instructions for running BERT-Large training using Intel Extension for PyTorch. 
+
+## Pull Command
+
+Docker image based on CentOS Stream8
+```
+docker pull intel/language-modeling:centos-pytorch-cpu-bert-large-training
+```
+
+Docker image based on Ubuntu 22.04
+```
+docker pull intel/language-modeling:ubuntu-pytorch-cpu-bert-large-training
+```
+
+## Quick Start Scripts
+| Script name | Description |
+|-------------|-------------|
+| `run_bert_pretrain_phase1.sh` | Runs BERT large pretraining phase 1 using max_seq_len=128 for the first 90% dataset for the specified precision (fp32, avx-fp32, bf32 or bf16). The script saves the model to the `OUTPUT_DIR` in a directory called `model_save`. |
+| `run_bert_pretrain_phase2.sh` | Runs BERT large pretraining phase 2 using max_seq_len=512 with the remaining 10% of the dataset for the specified precision (fp32, avx-fp32, bf32 or bf16). Use path to the `model_save` directory from phase one as the `CHECKPOINT_DIR` for phase 2. |
+
+**Note:** The `avx-fp32` precision runs the same scripts as `fp32`, except that the `DNNL_MAX_CPU_ISA` environment variable is unset. The environment variable is otherwise set to `DNNL_MAX_CPU_ISA=AVX512_CORE_AMX`.
+
+## Datasets
+Follow instructions to [download and preprocess](https://github.com/IntelAI/models/blob/v2.9.0/quickstart/language_modeling/pytorch/bert_large/training/cpu/README.md#datasets)  the text dataset and set the `DATASET_DIR` to point to the pre-processed dataset.
+
+# BERT Config File
+BERT Training happens in two stages. For stage 1, download the BERT Config file from [here](https://drive.google.com/drive/folders/1oQF4diVHNPCclykwdvQJw8n_VIWwV0PT) and export `CONFIG_FILE` variable to point to this file path. 
+
+# Checkpoint Directory
+The checkpoint directory is created as a result of Phase 1 Training. Please set the `CHECKPOINT_DIR` to point to the pre-trained model path and volume mount it for Phase 2 training. 
+
+## Docker Run
+(Optional) Export related proxy into docker environment.
+```bash
+export DOCKER_RUN_ENVS="-e ftp_proxy=${ftp_proxy} \
+  -e FTP_PROXY=${FTP_PROXY} -e http_proxy=${http_proxy} \
+  -e HTTP_PROXY=${HTTP_PROXY} -e https_proxy=${https_proxy} \
+  -e HTTPS_PROXY=${HTTPS_PROXY} -e no_proxy=${no_proxy} \
+  -e NO_PROXY=${NO_PROXY} -e socks_proxy=${socks_proxy} \
+  -e SOCKS_PROXY=${SOCKS_PROXY}"
+```
+To run the BERT-Large training scripts, set environment variables to specify the dataset directory, precision and an output directory. 
+
+```bash
+export OS=<provide either centos or ubuntu>
+export DATASET_DIR=<path to the dataset>
+export OUTPUT_DIR=<directory where log files will be written>
+export SCRIPT=quickstart/<specify the training script>
+export PRECISION=<specify the precision to run>
+export BERT_MODEL_CONFIG=<path to bert configuration file>
+export CHECKPOINT_DIR=<path to checkpoint to Directory>
+
+DOCKER_ARGS="--privileged --init -it"
+IMAGE_NAME=intel/language-modeling:${OS}-pytorch-cpu-bert-large-training
+WORKDIR=/workspace/pytorch-bert-large-training
+TRAIN_SCRIPT=${WORKDIR}/models/language_modeling/pytorch/bert_large/training/run_pretrain_mlperf.py  
+
+docker run --rm \
+  --env PRECISION=${PRECISION} \
+  --env OUTPUT_DIR=${OUTPUT_DIR} \
+  --env TRAIN_SCRIPT=${TRAIN_SCRIPT} \
+  --env DATASET_DIR=${DATASET_DIR} \
+  --env BERT_MODEL_CONFIG=${BERT_MODEL_CONFIG} \
+  --env PRETRAINED_MODEL=${CHECKPOINT_DIR} \
+  --env http_proxy=${http_proxy} \
+  --env https_proxy=${https_proxy} \
+  --env no_proxy=${no_proxy} \
+  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
+  --volume ${DATASET_DIR}:${DATASET_DIR} \
+  --volume ${BERT_MODEL_CONFIG}:${BERT_MODEL_CONFIG} \
+  --volume ${CHECKPOINT_DIR}:${CHECKPOINT_DIR} \
+  ${DOCKER_RUN_ENVS} \
+  --shm-size 8G \
+  -w ${WORKDIR} \
+  ${DOCKER_ARGS} \
+  $IMAGE_NAME \
+  /bin/bash $SCRIPT $PRECISION
+```
+## Documentation and Sources
+#### Get Started​
+[Docker* Repository](https://hub.docker.com/r/intel/language-modeling)
+
+[Main GitHub*](https://github.com/IntelAI/models)
+
+[Release Notes](https://github.com/IntelAI/models/releases)
+
+[Get Started Guide](https://github.com/IntelAI/models/blob/master/quickstart/quickstart/language_modeling/pytorch/bert_large/training/cpu/DEVCATALOG.md)
+
+#### Code Sources
+[Dockerfile](https://github.com/IntelAI/models/tree/master/docker/pyt-cpu)
+
+[Report Issue](https://community.intel.com/t5/Intel-Optimized-AI-Frameworks/bd-p/optimized-ai-frameworks)
+
+## License Agreement
+LEGAL NOTICE: By accessing, downloading or using this software and any required dependent software (the “Software Package”), you agree to the terms and conditions of the software license agreements for the Software Package, which may also include notices, disclaimers, or license terms for third party software included with the Software Package. Please refer to the [license](https://github.com/IntelAI/models/tree/master/third_party) file for additional details.
+
+[View All Containers and Solutions 🡢](https://www.intel.com/content/www/us/en/developer/tools/software-catalog/containers.html?s=Newest)
diff --git a/quickstart/language_modeling/pytorch/bert_large/training/cpu/README.md b/quickstart/language_modeling/pytorch/bert_large/training/cpu/README.md
index c0647f221..f0e5c98cc 100644
--- a/quickstart/language_modeling/pytorch/bert_large/training/cpu/README.md
+++ b/quickstart/language_modeling/pytorch/bert_large/training/cpu/README.md
@@ -36,7 +36,7 @@ contains 502 files for a total of about 13Gbytes.
 
 # Generate the BERT input dataset
 
-The create_pretraining_data.py script duplicates the input plain text, replaces
+The [create_pretraining_data.py](/models/language_modeling/pytorch/bert_large/training/input_preprocessing/create_pretraining_data.py) script duplicates the input plain text, replaces
 different sets of words with masks for each duplication, and serializes the
 output into the HDF5 file format.
 
@@ -54,12 +54,12 @@ pip install tensorflow-cpu
 For phase1 the seq_len=128:
 ```shell
 export SEQ_LEN=128
-./parallel_create_hdf5.sh
+./models/language_modeling/pytorch/bert_large/training/input_preprocessing/parallel_create_hdf5.sh
 ```
 For phase2 the seq_len=512:
 ```shell
 export SEQ_LEN=512
-./parallel_create_hdf5.sh
+./models/language_modeling/pytorch/bert_large/training/input_preprocessing/parallel_create_hdf5.sh
 ```
 
 The resulting `hdf5/` subdir will have 500 files named
@@ -74,14 +74,14 @@ For phase1:
 
 ```shell
 export SEQ_LEN=128
-python3 ./chop_hdf5_files.py
+python3 ./models/language_modeling/pytorch/bert_large/training/input_preprocessing/chop_hdf5_files.py
 ```
 
 For phase2:
 
 ```shell
 export SEQ_LEN=512
-python3 ./chop_hdf5_files.py
+python3 ./models/language_modeling/pytorch/bert_large/training/input_preprocessing/chop_hdf5_files.py
 ```
 
 The above will produce a subdirectory named `2048_shards_uncompressed/`
@@ -97,24 +97,30 @@ you can use "SHARD_NUM" to control the shard files number. the default "SHARD_NU
 ```
 
 ## Bare Metal
-
 ### General setup
 
-Follow [link](/docs/general/pytorch/BareMetalSetup.md) to install Miniconda and build Pytorch, IPEX, TorchVison, Torch-CCL, Jemalloc and TCMalloc.
+Follow [link](/docs/general/pytorch/BareMetalSetup.md) to install Miniconda and build Pytorch, IPEX, TorchVison Jemalloc and TCMalloc.
 
 ### Model Specific Setup
-* Install dependence
+
+* Set Jemalloc and tcmalloc Preload for better performance
+
+  The jemalloc should be built from the [General setup](#general-setup) section.
   ```
-  pip install datasets accelerate tfrecord
-  conda install openblas
-  conda install faiss-cpu -c pytorch
-  conda install intel-openmp
+  export LD_PRELOAD="<path to the jemalloc directory>/lib/libjemalloc.so":"path_to/tcmalloc/lib/libtcmalloc.so":$LD_PRELOAD
+  export MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
   ```
+* Set IOMP preload for better performance
+```
+  pip install packaging intel-openmp
+  export LD_PRELOAD=path/lib/libiomp5.so:$LD_PRELOAD
+```
 
 * Set ENV to use fp16 AMX if you are using a supported platform
 ```
   export DNNL_MAX_CPU_ISA=AVX512_CORE_AMX_FP16
 ```
+
 * Set ENV to use multi-nodes distributed training (no need for single-node multi-sockets)
 
   In this case, we use data-parallel distributed training and every rank will hold same model replica. The NNODES is the number of ip in the HOSTFILE. To use multi-nodes distributed training you should firstly setup the passwordless login (you can refer to [link](https://linuxize.com/post/how-to-setup-passwordless-ssh-login/)) between these nodes. 
@@ -122,14 +128,17 @@ Follow [link](/docs/general/pytorch/BareMetalSetup.md) to install Miniconda and
   export NNODES=#your_node_number
   export HOSTFILE=your_ip_list_file #one ip per line 
   ```
-## Quick Start Scripts
 
-|  DataType   | Phase 1  |  Phase 2 |
-| ----------- | ----------- | ----------- |
-| FP32        | bash run_bert_pretrain_phase1.sh fp32 | bash run_bert_pretrain_phase2.sh fp32 |
-| BF32        | bash run_bert_pretrain_phase1.sh bf32 | bash run_bert_pretrain_phase2.sh bf32 |
-| BF16        | bash run_bert_pretrain_phase1.sh bf16 | bash run_bert_pretrain_phase2.sh bf16 |
-| FP16        | bash run_bert_pretrain_phase1.sh fp16 | bash run_bert_pretrain_phase2.sh fp16 |
+* [optional] Compile model with PyTorch Inductor backend
+```shell
+  export TORCH_INDUCTOR=1
+```
+
+## Quick Start Scripts
+| Script name | Description |
+|-------------|-------------|
+| `run_bert_pretrain_phase1.sh` | Runs BERT large pretraining phase 1 using max_seq_len=128 for the first 90% dataset for the specified precision (fp32, avx-fp32, bf32 or bf16). The script saves the model to the `OUTPUT_DIR` in a directory called `model_save`. |
+| `run_bert_pretrain_phase2.sh` | Runs BERT large pretraining phase 2 using max_seq_len=512 with the remaining 10% of the dataset for the specified precision (fp32, avx-fp32, bf32 or bf16). Use path to the `model_save` directory from phase one as the `CHECKPOINT_DIR` for phase 2. |
 
 |  DataType   | Distributed Training Phase 1  |  Distributed Training Phase 2 |
 | ----------- | ----------- | ----------- |
@@ -153,45 +162,53 @@ export PROCESS_PER_NODE=2
 export GBS=1024
 ```
 will run 16 ranks on 8 nodes with `global batch size=1024`.
+
+**Note**: The `avx-fp32` precision runs the same scripts as `fp32`, except that the `DNNL_MAX_CPU_ISA` environment variable is unset. The environment variable is otherwise set to `DNNL_MAX_CPU_ISA=AVX512_CORE_AMX`.
+* Set ENV to use AMX:
+  ```
+  export DNNL_MAX_CPU_ISA=AVX512_CORE_AMX
+  ```
+
 ## Run the model
 
 Follow the instructions above to setup your bare metal environment, download and
 preprocess the dataset, and do the model specific setup. Once all the setup is done,
-the Model Zoo can be used to run a [quickstart script](#quick-start-scripts).
+the Intel® AI Reference Models can be used to run a [quickstart script](#quick-start-scripts).
 Ensure that you have enviornment variables set to point to the dataset directory
 and an output directory.
 
 ```
-# Clone the model zoo repo and set the MODEL_DIR
+# Clone the Intel® AI Reference Models repo and set the MODEL_DIR
 git clone https://github.com/IntelAI/models.git
 cd models
 export MODEL_DIR=$(pwd)
 
-# Clone the Transformers repo in the BERT large training directory
-cd quickstart/language_modeling/pytorch/bert_large/training/cpu
-git clone https://github.com/huggingface/transformers.git
-cd transformers
-git checkout v4.28.1
-git apply ../../../../../../../models/language_modeling/pytorch/common/enable_ipex_for_transformers.diff
-pip install -e ./
-cd ..
+# Install dependencies:
+./quickstart/language_modeling/pytorch/bert_large/training/cpu/setup.sh
 
 # Env vars
 export OUTPUT_DIR=<path to an output directory>
-export DATASET_DIR=</path/to/dataset/tfrecord_dir>
+export DATASET_DIR=<path to the dataset>
 export TRAIN_SCRIPT=${MODEL_DIR}/models/language_modeling/pytorch/bert_large/training/run_pretrain_mlperf.py
+export PRECISION=<specify the precision to run: fp32, avx-fp32, bf16 or bf32>
+
+# Optional environemnt variables:
+export BATCH_SIZE=<set a value for batch size, else it will run with default batch size>
 
-# For phase 1 get the bert config from https://drive.google.com/drive/folders/1oQF4diVHNPCclykwdvQJw8n_VIWwV0PT
-export BERT_MODEL_CONFIG=/path/to/bert_config.json
+# For phase 1 get the bert_config.json from [here](https://drive.google.com/drive/folders/1oQF4diVHNPCclykwdvQJw8n_VIWwV0PT)
+wget https://drive.google.com/drive/folders/1oQF4diVHNPCclykwdvQJw8n_VIWwV0PT
+export BERT_MODEL_CONFIG=$(pwd)/bert_config.json
 
-# Run the phase 1 quickstart script for fp32 (or bf16)
-bash run_bert_pretrain_phase1.sh fp32
+# Run the phase 1 quickstart script:
+# This downloads the checkpoints in 'CHECKPOINT_DIR'
+export CHECKPOINT_DIR=$(pwd)/checkpoint_phase1_dir
+./quickstart/language_modeling/pytorch/bert_large/training/cpu/run_bert_pretrain_phase1.sh
 
 # For phase 2 set the pretrained model path to the checkpoints generated during phase 1
-export PRETRAINED_MODEL=/path/to/bert_large_mlperf_checkpoint/checkpoint/
+export PRETRAINED_MODEL=$(pwd)/checkpoint_phase1_dir
 
-# Run the phase 2 quickstart script for fp32 (or bf16)
-bash run_bert_pretrain_phase2.sh fp32
+# Run the phase 2 quickstart script:
+./quickstart/language_modeling/pytorch/bert_large/training/cpu/run_bert_pretrain_phase2.sh
 ```
 
 ## CheckPoint in Your Training Phase 1 or Phase 2
diff --git a/quickstart/language_modeling/pytorch/bert_large/training/cpu/run_bert_pretrain_phase1.sh b/quickstart/language_modeling/pytorch/bert_large/training/cpu/run_bert_pretrain_phase1.sh
index 8e3ce128e..e26c232c5 100755
--- a/quickstart/language_modeling/pytorch/bert_large/training/cpu/run_bert_pretrain_phase1.sh
+++ b/quickstart/language_modeling/pytorch/bert_large/training/cpu/run_bert_pretrain_phase1.sh
@@ -15,36 +15,36 @@
 # limitations under the License.
 #
 
-MODEL_DIR=${MODEL_DIR-../../../../../../}
+MODEL_DIR=${MODEL_DIR-$PWD}
 
 #export DNNL_MAX_CPU_ISA=AVX512_CORE_AMX
 ARGS="--benchmark"
 precision=fp32
-batch_size=224
 
-if [[ "$1" == *"avx"* ]]; then
+batch_size=${batch_size:-224}
+if [[ "$PRECISION" == *"avx"* ]]; then
     unset DNNL_MAX_CPU_ISA
 fi
 
-if [[ "$1" == "bf16" ]]
+if [[ "$PRECISION" == "bf16" ]]
 then
     ARGS="$ARGS --bf16"
     precision=bf16
-    batch_size=448
+    batch_size=${batch_size:-448}
     echo "### running bf16 mode"
-elif [[ $1 == "bf32" ]]; then
+elif [[ $PRECISION == "bf32" ]]; then
     echo "### running BF32 mode"
     ARGS="$ARGS --bf32"
     precision=bf32
-elif [[ $1 == "fp16" ]]; then
+elif [[ $PRECISION == "fp16" ]]; then
     echo "### running FP16 mode"
     ARGS="$ARGS --fp16"
     precision=fp16
-elif [[ $1 == "fp32" || $1 == "avx-fp32" ]]; then
+elif [[ $PRECISION == "fp32" || $PRECISION == "avx-fp32" ]]; then
     echo "### running FP32 mode"
 
 else
-    echo "The specified precision '$1' is unsupported."
+    echo "The specified precision '$PRECISION' is unsupported."
     echo "Supported precisions are: fp32, bf32, avx-fp32, bf16"
     exit 1
 fi
@@ -61,16 +61,32 @@ NUM_RANKS=1
 LBS=$(( batch_size / NUM_RANKS ))
 params="--train_batch_size=$LBS     --learning_rate=3.5e-4     --opt_lamb_beta_1=0.9     --opt_lamb_beta_2=0.999     --warmup_proportion=0.0     --warmup_steps=0.0     --start_warmup_step=0     --max_steps=13700    --max_predictions_per_seq=76      --do_train   --train_mlm_accuracy_window_size=0     --target_mlm_accuracy=0.720     --weight_decay_rate=0.01     --max_samples_termination=4500000     --eval_iter_start_samples=150000 --eval_iter_samples=150000     --eval_batch_size=16  --gradient_accumulation_steps=1 --num_samples_per_checkpoint 1 --min_samples_to_start_checkpoints 1 --log_freq 1 "
 
-python -m intel_extension_for_pytorch.cpu.launch --node_id 0 --enable_jemalloc --log_path=${OUTPUT_DIR} --log_file_prefix="./throughput_log_phase1_${precision}" ${TRAIN_SCRIPT} \
-    --input_dir ${DATASET_DIR}/2048_shards_uncompressed_128/ \
-    --eval_dir ${DATASET_DIR}/eval_set_uncompressed/ \
-    --model_type 'bert' \
-    --benchmark \
-    --output_dir $OUTPUT_DIR/model_save \
-    --dense_seq_output \
-    --config_name ${BERT_MODEL_CONFIG} \
-    $ARGS \
-    $params \
+TORCH_INDUCTOR=${TORCH_INDUCTOR:-"0"}
+if [[ "0" == ${TORCH_INDUCTOR} ]];then
+    python -m intel_extension_for_pytorch.cpu.launch --node_id 0 --enable_jemalloc --log_path=${OUTPUT_DIR} --log_file_prefix="./throughput_log_phase1_${precision}" ${TRAIN_SCRIPT} \
+        --input_dir ${DATASET_DIR}/2048_shards_uncompressed_128/ \
+        --eval_dir ${DATASET_DIR}/eval_set_uncompressed/ \
+        --model_type 'bert' \
+        --benchmark \
+        --ipex \
+        --output_dir $OUTPUT_DIR/model_save \
+        --dense_seq_output \
+        --config_name ${BERT_MODEL_CONFIG} \
+        $ARGS \
+        $params
+else
+    python -m intel_extension_for_pytorch.cpu.launch --node_id 0 --enable_jemalloc --log_path=${OUTPUT_DIR} --log_file_prefix="./throughput_log_phase1_${precision}" ${TRAIN_SCRIPT} \
+        --input_dir ${DATASET_DIR}/2048_shards_uncompressed_128/ \
+        --eval_dir ${DATASET_DIR}/eval_set_uncompressed/ \
+        --model_type 'bert' \
+        --benchmark \
+        --inductor \
+        --output_dir $OUTPUT_DIR/model_save \
+        --dense_seq_output \
+        --config_name ${BERT_MODEL_CONFIG} \
+        $ARGS \
+        $params
+fi
     
 
 
diff --git a/quickstart/language_modeling/pytorch/bert_large/training/cpu/run_bert_pretrain_phase2.sh b/quickstart/language_modeling/pytorch/bert_large/training/cpu/run_bert_pretrain_phase2.sh
index abd594b2d..b56ac6264 100755
--- a/quickstart/language_modeling/pytorch/bert_large/training/cpu/run_bert_pretrain_phase2.sh
+++ b/quickstart/language_modeling/pytorch/bert_large/training/cpu/run_bert_pretrain_phase2.sh
@@ -15,36 +15,36 @@
 # limitations under the License.
 #
 
-MODEL_DIR=${MODEL_DIR-../../../../../..}
+MODEL_DIR=${MODEL_DIR-$PWD}
 
 #export DNNL_MAX_CPU_ISA=AVX512_CORE_AMX
 ARGS="--benchmark"
 precision=fp32
-batch_size=28
+batch_size=${batch_size:-28}
 
-if [[ "$1" == *"avx"* ]]; then
+if [[ "$PRECISION" == *"avx"* ]]; then
     unset DNNL_MAX_CPU_ISA
 fi
 
-if [[ "$1" == "bf16" ]]
+if [[ "$PRECISION" == "bf16" ]]
 then
     ARGS="$ARGS --bf16"
     precision=bf16
-    batch_size=128
+    batch_size=${batch_size:-128}
     echo "### running bf16 mode"
-elif [[ $1 == "bf32" ]]; then
+elif [[ $PRECISION == "bf32" ]]; then
     echo "### running BF32 mode"
     ARGS="$ARGS --bf32"
     precision=bf32
-elif [[ $1 == "fp16" ]]; then
+elif [[ $PRECISION == "fp16" ]]; then
     echo "### running FP16 mode"
     ARGS="$ARGS --fp16"
     precision=fp16
-elif [[ $1 == "fp32" || $1 == "avx-fp32" ]]; then
+elif [[ $PRECISION == "fp32" || $PRECISION == "avx-fp32" ]]; then
     echo "### running FP32 mode"
 
 else
-    echo "The specified precision '$1' is unsupported."
+    echo "The specified precision '$PRECISION' is unsupported."
     echo "Supported precisions are: fp32, bf32 avx-fp32, bf16"
     exit 1
 fi
@@ -61,16 +61,32 @@ NUM_RANKS=1
 LBS=$(( batch_size / NUM_RANKS ))
 params="--train_batch_size=$LBS     --learning_rate=3.5e-4     --opt_lamb_beta_1=0.9     --opt_lamb_beta_2=0.999     --warmup_proportion=0.0     --warmup_steps=0.0     --start_warmup_step=0     --max_steps=13700     --phase2    --max_predictions_per_seq=76      --do_train     --skip_checkpoint     --train_mlm_accuracy_window_size=0     --target_mlm_accuracy=0.720     --weight_decay_rate=0.01     --max_samples_termination=4500000     --eval_iter_start_samples=150000 --eval_iter_samples=150000     --eval_batch_size=16  --gradient_accumulation_steps=1     --log_freq=0 "
 
-python -m intel_extension_for_pytorch.cpu.launch --node_id 0 --enable_jemalloc --log_path=${OUTPUT_DIR} --log_file_prefix="./throughput_log_phase2_${precision}" ${TRAIN_SCRIPT} \
-    --input_dir ${DATASET_DIR}/2048_shards_uncompressed_512/ \
-    --eval_dir ${DATASET_DIR}/eval_set_uncompressed/ \
-    --model_type 'bert' \
-    --model_name_or_path ${PRETRAINED_MODEL} \
-    --benchmark \
-    --dense_seq_output \
-    --output_dir $OUTPUT_DIR/model_save \
-    $ARGS \
-    $params \
+TORCH_INDUCTOR=${TORCH_INDUCTOR:-"0"}
+if [[ "0" == ${TORCH_INDUCTOR} ]];then
+    python -m intel_extension_for_pytorch.cpu.launch --node_id 0 --enable_jemalloc --log_path=${OUTPUT_DIR} --log_file_prefix="./throughput_log_phase2_${precision}" ${TRAIN_SCRIPT} \
+        --input_dir ${DATASET_DIR}/2048_shards_uncompressed_512/ \
+        --eval_dir ${DATASET_DIR}/eval_set_uncompressed/ \
+        --model_type 'bert' \
+        --model_name_or_path ${PRETRAINED_MODEL} \
+        --benchmark \
+        --ipex \
+        --dense_seq_output \
+        --output_dir $OUTPUT_DIR/model_save \
+        $ARGS \
+        $params
+else
+    python -m intel_extension_for_pytorch.cpu.launch --node_id 0 --enable_jemalloc --log_path=${OUTPUT_DIR} --log_file_prefix="./throughput_log_phase2_${precision}" ${TRAIN_SCRIPT} \
+        --input_dir ${DATASET_DIR}/2048_shards_uncompressed_512/ \
+        --eval_dir ${DATASET_DIR}/eval_set_uncompressed/ \
+        --model_type 'bert' \
+        --model_name_or_path ${PRETRAINED_MODEL} \
+        --benchmark \
+        --inductor \
+        --dense_seq_output \
+        --output_dir $OUTPUT_DIR/model_save \
+        $ARGS \
+        $params
+fi
     
 
 
diff --git a/quickstart/language_modeling/pytorch/bert_large/training/cpu/run_ddp_bert_pretrain_phase1.sh b/quickstart/language_modeling/pytorch/bert_large/training/cpu/run_ddp_bert_pretrain_phase1.sh
index af806583d..8c280f6fb 100755
--- a/quickstart/language_modeling/pytorch/bert_large/training/cpu/run_ddp_bert_pretrain_phase1.sh
+++ b/quickstart/language_modeling/pytorch/bert_large/training/cpu/run_ddp_bert_pretrain_phase1.sh
@@ -22,12 +22,12 @@ MODEL_DIR=${MODEL_DIR-../../../../../../}
 #export DNNL_MAX_CPU_ISA=AVX512_CORE_AMX
 ARGS="--benchmark"
 precision=fp32
-batch_size=224
+batch_size=${batch_size:-224}
 if [[ "$1" == "bf16" ]]
 then
     ARGS="$ARGS --bf16"
     precision=bf16
-    batch_size=448
+    batch_size=${batch_size:-448}
     echo "### running bf16 mode"
 elif [[ $1 == "bf32" ]]; then
     echo "### running BF32 mode"
@@ -55,18 +55,32 @@ rm -rf ${OUTPUT_DIR}/throughput_log_phase1_*
 NUM_RANKS=$(( NNODES * SOCKETS ))
 LBS=$(( batch_size / NUM_RANKS ))
 params="--train_batch_size=$LBS     --learning_rate=3.5e-4     --opt_lamb_beta_1=0.9     --opt_lamb_beta_2=0.999     --warmup_proportion=0.0     --warmup_steps=0.0     --start_warmup_step=0     --max_steps=13700   --max_predictions_per_seq=76      --do_train     --skip_checkpoint     --train_mlm_accuracy_window_size=0     --target_mlm_accuracy=0.720     --weight_decay_rate=0.01     --max_samples_termination=4500000     --eval_iter_start_samples=150000 --eval_iter_samples=150000     --eval_batch_size=16  --gradient_accumulation_steps=1     --log_freq=0 "
-
-python -m intel_extension_for_pytorch.cpu.launch --distributed  --nnodes ${NNODES} --hostfile ${HOSTFILE} --nproc_per_node $SOCKETS --log_path=${OUTPUT_DIR} --log_file_prefix="./throughput_log_phase1_${precision}" ${TRAIN_SCRIPT} \
-    --input_dir ${DATASET_DIR}/2048_shards_uncompressed_128/ \
-    --eval_dir ${DATASET_DIR}/eval_set_uncompressed/ \
-    --model_type 'bert' \
-    --output_dir model_save \
-    --dense_seq_output \
-    --config_name ${BERT_MODEL_CONFIG} \
-    $ARGS \
-    $params \
+TORCH_INDUCTOR=${TORCH_INDUCTOR:-"0"}
+if [[ "0" == ${TORCH_INDUCTOR} ]];then
+    python -m intel_extension_for_pytorch.cpu.launch --distributed  --nnodes ${NNODES} --hostfile ${HOSTFILE} --nproc_per_node $SOCKETS --log_path=${OUTPUT_DIR} --log_file_prefix="./throughput_log_phase1_${precision}" ${TRAIN_SCRIPT} \
+        --input_dir ${DATASET_DIR}/2048_shards_uncompressed_128/ \
+        --eval_dir ${DATASET_DIR}/eval_set_uncompressed/ \
+        --model_type 'bert' \
+        --ipex \
+        --output_dir model_save \
+        --dense_seq_output \
+        --config_name ${BERT_MODEL_CONFIG} \
+        $ARGS \
+        $params \
+    2>&1 | tee ${OUTPUT_DIR}/throughput_log_phase1_${precision}.log
+else
+    export TORCHINDUCTOR_FREEZING=1
+    python -m intel_extension_for_pytorch.cpu.launch --distributed  --nnodes ${NNODES} --hostfile ${HOSTFILE} --nproc_per_node $SOCKETS --log_path=${OUTPUT_DIR} --log_file_prefix="./throughput_log_phase1_${precision}" ${TRAIN_SCRIPT} \
+        --input_dir ${DATASET_DIR}/2048_shards_uncompressed_128/ \
+        --eval_dir ${DATASET_DIR}/eval_set_uncompressed/ \
+        --model_type 'bert' \
+        --inductor \
+        --output_dir model_save \
+        --dense_seq_output \
+        --config_name ${BERT_MODEL_CONFIG} \
+        $ARGS \
+        $params \
     2>&1 | tee ${OUTPUT_DIR}/throughput_log_phase1_${precision}.log
-
 # For the summary of results
 wait
 throughput=$(grep 'Throughput:' ${OUTPUT_DIR}/throughput_log_phase1_${precision}* |sed -e 's/.*Throughput//;s/[^0-9.]//g' |awk '
diff --git a/quickstart/language_modeling/pytorch/bert_large/training/cpu/run_ddp_bert_pretrain_phase2.sh b/quickstart/language_modeling/pytorch/bert_large/training/cpu/run_ddp_bert_pretrain_phase2.sh
index cdef7ba3e..e9fd6d8eb 100755
--- a/quickstart/language_modeling/pytorch/bert_large/training/cpu/run_ddp_bert_pretrain_phase2.sh
+++ b/quickstart/language_modeling/pytorch/bert_large/training/cpu/run_ddp_bert_pretrain_phase2.sh
@@ -22,12 +22,12 @@ MODEL_DIR=${MODEL_DIR-../../../../../..}
 #export DNNL_MAX_CPU_ISA=AVX512_CORE_AMX
 ARGS="--benchmark"
 precision=fp32
-batch_size=28
+batch_size=${batch_size:-28}
 if [[ "$1" == "bf16" ]]
 then
     ARGS="$ARGS --bf16"
     precision=bf16
-    batch_size=56
+    batch_size=${batch_size:-56}
     echo "### running bf16 mode"
 elif [[ $1 == "bf32" ]]; then
     echo "### running BF32 mode"
@@ -56,16 +56,33 @@ NUM_RANKS=$(( NNODES * SOCKETS ))
 LBS=$(( batch_size / NUM_RANKS ))
 params="--train_batch_size=$LBS     --learning_rate=3.5e-4     --opt_lamb_beta_1=0.9     --opt_lamb_beta_2=0.999     --warmup_proportion=0.0     --warmup_steps=0.0     --start_warmup_step=0     --max_steps=13700     --phase2    --max_predictions_per_seq=76      --do_train     --skip_checkpoint     --train_mlm_accuracy_window_size=0     --target_mlm_accuracy=0.720     --weight_decay_rate=0.01     --max_samples_termination=4500000     --eval_iter_start_samples=150000 --eval_iter_samples=150000     --eval_batch_size=16  --gradient_accumulation_steps=1     --log_freq=0 "
 
-python -m intel_extension_for_pytorch.cpu.launch --distributed --nnodes ${NNODES} --hostfile ${HOSTFILE}  --log_path=${OUTPUT_DIR} --log_file_prefix="./throughput_log_phase2_${precision}" ${TRAIN_SCRIPT} \
-    --input_dir ${DATASET_DIR}/2048_shards_uncompressed_512/ \
-    --eval_dir ${DATASET_DIR}/eval_set_uncompressed/ \
-    --model_type 'bert' \
-    --model_name_or_path ${PRETRAINED_MODEL} \
-    --output_dir model_save \
-    --dense_seq_output \
-    $ARGS \
-    $params \
-    2>&1 | tee ${OUTPUT_DIR}/throughput_log_phase2_${precision}.log
+TORCH_INDUCTOR=${TORCH_INDUCTOR:-"0"}
+if [[ "0" == ${TORCH_INDUCTOR} ]];then
+    python -m intel_extension_for_pytorch.cpu.launch --distributed --nnodes ${NNODES} --hostfile ${HOSTFILE}  --log_path=${OUTPUT_DIR} --log_file_prefix="./throughput_log_phase2_${precision}" ${TRAIN_SCRIPT} \
+        --input_dir ${DATASET_DIR}/2048_shards_uncompressed_512/ \
+        --eval_dir ${DATASET_DIR}/eval_set_uncompressed/ \
+        --model_type 'bert' \
+        --ipex \
+        --model_name_or_path ${PRETRAINED_MODEL} \
+        --output_dir model_save \
+        --dense_seq_output \
+        $ARGS \
+        $params \
+        2>&1 | tee ${OUTPUT_DIR}/throughput_log_phase2_${precision}.log
+else
+    export TORCHINDUCTOR_FREEZING=1
+    python -m intel_extension_for_pytorch.cpu.launch --distributed --nnodes ${NNODES} --hostfile ${HOSTFILE}  --log_path=${OUTPUT_DIR} --log_file_prefix="./throughput_log_phase2_${precision}" ${TRAIN_SCRIPT} \
+        --input_dir ${DATASET_DIR}/2048_shards_uncompressed_512/ \
+        --eval_dir ${DATASET_DIR}/eval_set_uncompressed/ \
+        --model_type 'bert' \
+        --inductor \
+        --model_name_or_path ${PRETRAINED_MODEL} \
+        --output_dir model_save \
+        --dense_seq_output \
+        $ARGS \
+        $params \
+        2>&1 | tee ${OUTPUT_DIR}/throughput_log_phase2_${precision}.log
+fi
 
 # For the summary of results
 wait
diff --git a/quickstart/language_modeling/pytorch/bert_large/training/cpu/setup.sh b/quickstart/language_modeling/pytorch/bert_large/training/cpu/setup.sh
new file mode 100755
index 000000000..b5ddaab32
--- /dev/null
+++ b/quickstart/language_modeling/pytorch/bert_large/training/cpu/setup.sh
@@ -0,0 +1,41 @@
+#!/usr/bin/env bash
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# Install dependency
+pip install tensorboard
+pip install datasets==1.11.0 accelerate tfrecord intel-openmp faiss-cpu tfrecord
+pip install --upgrade huggingface_hub
+pip install h5py
+
+# Check the operating system type
+os_type=$(awk -F= '/^NAME/{print $2}' /etc/os-release)
+
+# Install model specific dependencies:
+if [[ "$os_name" == *"CentOS"* ]]; then
+    yum install -y git-lfs
+elif [[ "$os_name" == *"Ubuntu"* ]]; then
+    apt install -y git-lfs
+fi
+
+rm -rf transformers
+git clone https://github.com/huggingface/transformers.git
+cd transformers
+git checkout v4.28.1
+git lfs pull
+git apply ../models/language_modeling/pytorch/common/enable_ipex_for_transformers.diff
+pip install -e ./
+cd ..
diff --git a/quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/DEVCATALOG.md b/quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/DEVCATALOG.md
new file mode 100644
index 000000000..8a863f05b
--- /dev/null
+++ b/quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/DEVCATALOG.md
@@ -0,0 +1,116 @@
+# PyTorch DistilBERT Base inference
+
+## Description 
+This document has instructions for running DistilBERT Base inference using Intel Extension for PyTorch. 
+
+## Pull Command
+
+Docker image based on CentOS Stream8
+```
+docker pull intel/language-modeling:centos-pytorch-cpu-distilbert-inference
+```
+
+Docker image based on Ubuntu 22.04
+```
+docker pull intel/language-modeling:ubuntu-pytorch-cpu-distilbert-inference
+```
+
+# Quick Start Scripts
+|  DataType   | Throughput  |  Latency    |   Accuracy  |
+| ----------- | ----------- | ----------- | ----------- |
+| FP32        | run_multi_instance_throughput.sh fp32 | run_multi_instance_realtime.sh fp32 | run_accuracy.sh fp32 |
+| BF32        | run_multi_instance_throughput.sh bf32 | run_multi_instance_realtime.sh bf32 | run_accuracy.sh bf32 |
+| BF16        | run_multi_instance_throughput.sh bf16 | run_multi_instance_realtime.sh bf16 | run_accuracy.sh bf16 |
+| INT8-FP32        | run_multi_instance_throughput.sh int8-fp32 | run_multi_instance_realtime.sh int8-fp32 | run_accuracy.sh int8-fp32 |
+| INT8-BF16       | run_multi_instance_throughput.sh int8-bf16 | run_multi_instance_realtime.sh int8-bf16 | run_accuracy.sh int8-bf16 |
+
+# Datasets
+Use the following instructions to download the SST-2 dataset.
+Also, clone the AI Reference Models GitHub Repository and set the `MODEL_DIR` directory.
+```
+git clone https://github.com/IntelAI/models.git
+cd models
+git checkout v2.9.0
+export MODEL_DIR=$(pwd)
+cd -
+export DATASET_DIR=$(pwd)
+wget https://dl.fbaipublicfiles.com/glue/data/SST-2.zip -O $DATASET_DIR/SST-2.zip
+unzip $DATASET_DIR/SST-2.zip -d $DATASET_DIR/
+python $MODEL_DIR/quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/convert.py $DATASET_DIR
+```
+
+# Pre-Trained Model
+Follow the instructions below to download the pre-trained model. 
+
+```
+git clone https://huggingface.co/distilbert-base-uncased-finetuned-sst-2-english
+export PRETRAINED_MODEL=$(pwd)/distilbert-base-uncased-finetuned-sst-2-english
+```
+## Docker Run
+(Optional) Export related proxy into docker environment.
+```bash
+export DOCKER_RUN_ENVS="-e ftp_proxy=${ftp_proxy} \
+  -e FTP_PROXY=${FTP_PROXY} -e http_proxy=${http_proxy} \
+  -e HTTP_PROXY=${HTTP_PROXY} -e https_proxy=${https_proxy} \
+  -e HTTPS_PROXY=${HTTPS_PROXY} -e no_proxy=${no_proxy} \
+  -e NO_PROXY=${NO_PROXY} -e socks_proxy=${socks_proxy} \
+  -e SOCKS_PROXY=${SOCKS_PROXY}"
+```
+To run DistilBERT inference, set environment variables to specify the dataset directory, precision to run, model directory and an output directory. 
+```bash
+export OS=<provide either centos or ubuntu>
+export HF_DATASETS_OFFLINE=<provide 0 or 1. providing 1 downloads the datasets>
+export SEQUENCE_LENGTH=128
+export CORE_PER_INSTANCE=<recommended core per instance are 4 for realtime inference and 32 for throughput inference and accuracy>
+export PRECISION=<provide the precision>
+export PRETRAINED_MODEL=<path to pre-trained model>
+export DATASET_DIR=<path to dataset directory>
+export OUTPUT_DIR=<path to output directory>
+
+IMAGE_NAME=intel/language-modeling:${OS}-pytorch-cpu-distilbert-inference
+SCRIPT=quickstart/<name of script>
+DOCKER_ARGS="--privileged --init -it"
+WORKDIR=/workspace/pytorch-distilbert-inference
+EVAL_SCRIPT="${WORKDIR}/quickstart/transformers/examples/pytorch/text-classification/run_glue.py"
+
+docker run --rm \
+  --env PRECISION=${PRECISION} \
+  --env EVAL_SCRIPT=${EVAL_SCRIPT} \
+  --env HF_DATASETS_OFFLINE=${HF_DATASETS_OFFLINE} \
+  --env FINETUNED_MODEL=${PRETRAINED_MODEL} \
+  --env OUTPUT_DIR=${OUTPUT_DIR} \
+  --env DATASET_DIR=${DATASET_DIR} \
+  --env SEQUENCE_LENGTH=${SEQUENCE_LENGTH} \
+  --env CORE_PER_INSTANCE=${CORE_PER_INSTANCE} \
+  --env http_proxy=${http_proxy} \
+  --env https_proxy=${https_proxy} \
+  --env no_proxy=${no_proxy} \
+  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
+  --volume ${DATASET_DIR}:${DATASET_DIR} \
+  --volume ${PRETRAINED_MODEL}:${PRETRAINED_MODEL} \
+  ${DOCKER_RUN_ENVS} \
+  --shm-size 8G \
+  -w ${WORKDIR} \
+  ${DOCKER_ARGS} \
+  $IMAGE_NAME \
+  /bin/bash $SCRIPT ${PRECISION}
+  ```
+## Documentation and Sources
+#### Get Started​
+[Docker* Repository](https://hub.docker.com/r/intel/language-modeling)
+
+[Main GitHub*](https://github.com/IntelAI/models)
+
+[Release Notes](https://github.com/IntelAI/models/releases)
+
+[Get Started Guide](https://github.com/IntelAI/models/blob/master/quickstart/quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/DEVCATALOG.md)
+
+#### Code Sources
+[Dockerfile](https://github.com/IntelAI/models/tree/master/docker/pyt-cpu)
+
+[Report Issue](https://community.intel.com/t5/Intel-Optimized-AI-Frameworks/bd-p/optimized-ai-frameworks)
+
+## License Agreement
+LEGAL NOTICE: By accessing, downloading or using this software and any required dependent software (the “Software Package”), you agree to the terms and conditions of the software license agreements for the Software Package, which may also include notices, disclaimers, or license terms for third party software included with the Software Package. Please refer to the [license](https://github.com/IntelAI/models/tree/master/third_party) file for additional details.
+
+[View All Containers and Solutions 🡢](https://www.intel.com/content/www/us/en/developer/tools/software-catalog/containers.html?s=Newest)
diff --git a/quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/README.md b/quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/README.md
index 569ae62e2..88727589f 100644
--- a/quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/README.md
+++ b/quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/README.md
@@ -11,28 +11,20 @@ This document has instructions for running [DistilBERT base uncased finetuned SS
 
 Follow [link](/docs/general/pytorch/BareMetalSetup.md) to install Miniconda and build Pytorch, IPEX, TorchVison Jemalloc and TCMalloc.
 
-### Prepare model
-```
-  cd <clone of the model zoo>/quickstart/language_modeling/pytorch/distilbert_base/inference/cpu
-  git clone https://github.com/huggingface/transformers.git
-  cd transformers
-  git checkout v4.28.1
-  pip install -r ./examples/pytorch/text-classification/requirements.txt
-  git apply ../../../../../../../models/language_modeling/pytorch/common/enable_ipex_for_transformers.diff
-  pip install -e ./
-  cd ..
- ```
 ### Model Specific Setup
 
-* Install Intel OpenMP
-  ```
-  conda install intel-openmp
-  ```
+* Set Jemalloc and tcmalloc Preload for better performance
 
-* Install datasets
+  The jemalloc should be built from the [General setup](#general-setup) section.
   ```
-  pip install datasets
+  export LD_PRELOAD="<path to the jemalloc directory>/lib/libjemalloc.so":"path_to/tcmalloc/lib/libtcmalloc.so":$LD_PRELOAD
+  export MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
   ```
+* Set IOMP preload for better performance
+```
+  pip install packaging intel-openmp
+  export LD_PRELOAD=path/lib/libiomp5.so:$LD_PRELOAD
+```
 
 * Set SEQUENCE_LENGTH before running the model
   ```
@@ -82,45 +74,76 @@ Follow [link](/docs/general/pytorch/BareMetalSetup.md) to install Miniconda and
   export DNNL_MAX_CPU_ISA=AVX512_CORE_AMX_FP16
   ```
 
-# Quick Start Scripts
+* [optional] Compile model with PyTorch Inductor backend
+  ```shell
+  export TORCH_INDUCTOR=1
+  ```
 
+# Quick Start Scripts
 |  DataType   | Throughput  |  Latency    |   Accuracy  |
 | ----------- | ----------- | ----------- | ----------- |
-| FP32        | bash run_multi_instance_throughput.sh fp32 | bash run_multi_instance_realtime.sh fp32 | bash run_accuracy.sh fp32 |
-| BF32        | bash run_multi_instance_throughput.sh bf32 | bash run_multi_instance_realtime.sh bf32 | bash run_accuracy.sh bf32 |
-| BF16        | bash run_multi_instance_throughput.sh bf16 | bash run_multi_instance_realtime.sh bf16 | bash run_accuracy.sh bf16 |
-| FP16        | bash run_multi_instance_throughput.sh fp16 | bash run_multi_instance_realtime.sh fp16 | bash run_accuracy.sh fp16 |
-| INT8-FP32        | bash run_multi_instance_throughput.sh int8-fp32 | bash run_multi_instance_realtime.sh int8-fp32 | bash run_accuracy.sh int8-fp32 |
-| INT8-BF16       | bash run_multi_instance_throughput.sh int8-bf16 | bash run_multi_instance_realtime.sh int8-bf16 | bash run_accuracy.sh int8-bf16 |
+| FP32        | run_multi_instance_throughput.sh fp32 | run_multi_instance_realtime.sh fp32 | run_accuracy.sh fp32 |
+| BF32        | run_multi_instance_throughput.sh bf32 | run_multi_instance_realtime.sh bf32 | run_accuracy.sh bf32 |
+| BF16        | run_multi_instance_throughput.sh bf16 | run_multi_instance_realtime.sh bf16 | run_accuracy.sh bf16 |
+| INT8-FP32        | run_multi_instance_throughput.sh int8-fp32 | run_multi_instance_realtime.sh int8-fp32 | run_accuracy.sh int8-fp32 |
+| INT8-BF16       | run_multi_instance_throughput.sh int8-bf16 | run_multi_instance_realtime.sh int8-bf16 | run_accuracy.sh int8-bf16 |
+
+**Note**: The `avx-fp32` precision runs the same scripts as `fp32`, except that the `DNNL_MAX_CPU_ISA` environment variable is unset. The environment variable is otherwise set to `DNNL_MAX_CPU_ISA=AVX512_CORE_AMX`.
+* Set ENV to use AMX:
+  ```
+  export DNNL_MAX_CPU_ISA=AVX512_CORE_AMX
+  ```
+
+# Datasets
+Use the following instructions to download the SST-2 dataset.
+Also, clone the Intel® AI Reference Models GitHub Repository and set the `MODEL_DIR` directory.
+```
+git clone https://github.com/IntelAI/models.git
+cd models
+export MODEL_DIR=$(pwd)
+cd -
+export DATASET_DIR=$(pwd)
+wget https://dl.fbaipublicfiles.com/glue/data/SST-2.zip -O $DATASET_DIR/SST-2.zip
+unzip $DATASET_DIR/SST-2.zip -d $DATASET_DIR/
+python $MODEL_DIR/quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/convert.py $DATASET_DIR
+```
+
+# Pre-Trained Model
+Follow the instructions below to download the pre-trained model. 
+
+```
+git clone https://huggingface.co/distilbert-base-uncased-finetuned-sst-2-english
+export FINETUNED_MODEL=$(pwd)/distilbert-base-uncased-finetuned-sst-2-english
+```
 
 ## Run the model
 
 Follow the instructions above to setup your bare metal environment, download and
 preprocess the dataset, and do the model specific setup. Once all the setup is done,
-the Model Zoo can be used to run a [quickstart script](#quick-start-scripts).
+the Intel® AI Reference Models can be used to run a [quickstart script](#quick-start-scripts).
 Ensure that you have an enviornment variable set to point to an output directory.
 
 ```
-# Clone the model zoo repo and set the MODEL_DIR
-git clone https://github.com/IntelAI/models.git
+# Navigate to the Intel® AI Reference Models repo and set the MODEL_DIR
 cd models
-export MODEL_DIR=$(pwd)
 
-# Clone the Transformers repo in the DistilBERT Base inference directory
-cd quickstart/language_modeling/pytorch/distilbert_base/inference/cpu
-git clone https://github.com/huggingface/transformers.git
-cd transformers
-git checkout v4.28.1
-pip install -r ./examples/pytorch/text-classification/requirements.txt
-git apply ../../../../../../../models/language_modeling/pytorch/common/enable_ipex_for_transformers.diff
-pip install -e ./
-cd ..
+# Prepare model and install dependencies:
+./quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/setup.sh
 
 # Env vars
 export OUTPUT_DIR=<path to an output directory>
-
-# Run a quickstart script (for example, FP32 multi-instance realtime inference)
-bash run_multi_instance_realtime.sh fp32
+export PRECISION=< select from :- fp32, bf32, bf16, int8-fp32, int8-bf16>
+export HF_DATASETS_OFFLINE=0
+export SEQUENCE_LENGTH=128 
+export CORE_PER_INSTANCE=4
+export FINETUNED_MODEL=<path to pre-trained model>
+export DATASET_DIR=<path to dataset directory>
+
+# Optional environemnt variables:
+export BATCH_SIZE=<set a value for batch size, else it will run with default batch size>
+
+# Run a quickstart script
+./quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/<script.sh>
 ```
 
 <!--- 80. License -->
diff --git a/quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/configure.json b/quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/configure.json
index c8bc61fe1..91809d1f2 100644
--- a/quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/configure.json
+++ b/quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/configure.json
@@ -12,10 +12,10 @@
                         "inf_dtype": "torch.quint8",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.025485554710030556
+                            0.027204349637031555
                         ],
                         "zero_point": [
-                            146
+                            151
                         ]
                     }
                 ],
@@ -1571,10 +1571,10 @@
                         "orig_dtype": "torch.float32",
                         "inf_dtype": "torch.float32",
                         "scale": [
-                            0.014870311133563519
+                            0.015292411670088768
                         ],
                         "zero_point": [
-                            103
+                            104
                         ]
                     }
                 ],
@@ -1584,7 +1584,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -1593,7 +1594,8 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             },
             "1": {
@@ -1607,10 +1609,10 @@
                         "inf_dtype": "torch.float32",
                         "force_dtype": "torch.float32",
                         "scale": [
-                            0.014870311133563519
+                            0.015292411670088768
                         ],
                         "zero_point": [
-                            103
+                            104
                         ]
                     }
                 ],
@@ -1621,7 +1623,7 @@
                         "orig_dtype": "torch.float32",
                         "inf_dtype": "torch.float32",
                         "scale": [
-                            0.008879014290869236
+                            0.009052076376974583
                         ],
                         "zero_point": [
                             0
@@ -1634,7 +1636,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -1643,7 +1646,8 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             },
             "2": {
@@ -1657,7 +1661,7 @@
                         "inf_dtype": "torch.quint8",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.008879014290869236
+                            0.009052076376974583
                         ],
                         "zero_point": [
                             0
@@ -1691,7 +1695,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -1700,13 +1705,14 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             }
         },
         "nonq_op_infos": {
             "0": {
-                "op_type": "<slot wrapper '__getitem__' of 'torch._C._TensorBase' objects>",
+                "op_type": "<slot wrapper '__getitem__' of 'torch._C.TensorBase' objects>",
                 "fqn": "",
                 "input_tensor_infos": [
                     {
@@ -1744,7 +1750,7 @@
                 ]
             },
             "2": {
-                "op_type": "<method 'view' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'view' of 'torch._C.TensorBase' objects>",
                 "fqn": "",
                 "input_tensor_infos": [
                     {
@@ -1764,7 +1770,7 @@
                 ]
             },
             "3": {
-                "op_type": "<method 'view' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'view' of 'torch._C.TensorBase' objects>",
                 "fqn": "",
                 "input_tensor_infos": [
                     {
@@ -1823,7 +1829,7 @@
         "q_op_infos": {},
         "nonq_op_infos": {
             "0": {
-                "op_type": "<method 'size' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'size' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert",
                 "input_tensor_infos": [
                     {
@@ -1846,7 +1852,7 @@
     "distilbert:embeddings": {
         "q_op_infos": {
             "0": {
-                "op_type": "<method 'add' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'add' of 'torch._C.TensorBase' objects>",
                 "op_type_is_module": false,
                 "fqn": "distilbert.embeddings",
                 "input_tensor_infos": [
@@ -1889,7 +1895,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -1898,7 +1905,8 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             }
         },
@@ -1922,7 +1930,7 @@
                 ]
             },
             "1": {
-                "op_type": "<method 'size' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'size' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.embeddings",
                 "input_tensor_infos": [
                     {
@@ -1984,7 +1992,7 @@
                     {
                         "id": 6,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8"
+                        "inf_dtype": "torch.float32"
                     }
                 ]
             }
@@ -1993,7 +2001,7 @@
             {
                 "id": 6,
                 "orig_dtype": "torch.float32",
-                "inf_dtype": "torch.quint8"
+                "inf_dtype": "torch.float32"
             }
         ]
     },
@@ -2016,7 +2024,7 @@
     "distilbert:transformer:layer:0": {
         "q_op_infos": {
             "0": {
-                "op_type": "<method 'add' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'add' of 'torch._C.TensorBase' objects>",
                 "op_type_is_module": false,
                 "fqn": "distilbert.transformer.layer.0",
                 "input_tensor_infos": [
@@ -2026,22 +2034,22 @@
                         "inf_dtype": "torch.float32",
                         "force_dtype": "torch.float32",
                         "scale": [
-                            0.10660196840763092
+                            0.10801716148853302
                         ],
                         "zero_point": [
-                            198
+                            196
                         ]
                     },
                     {
                         "id": 6,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8",
-                        "force_dtype": "torch.quint8",
+                        "inf_dtype": "torch.float32",
+                        "force_dtype": "torch.float32",
                         "scale": [
-                            0.046401847153902054
+                            0.04691799730062485
                         ],
                         "zero_point": [
-                            184
+                            182
                         ]
                     }
                 ],
@@ -2059,7 +2067,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -2068,11 +2077,12 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             },
             "1": {
-                "op_type": "<method 'add' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'add' of 'torch._C.TensorBase' objects>",
                 "op_type_is_module": false,
                 "fqn": "distilbert.transformer.layer.0",
                 "input_tensor_infos": [
@@ -2082,7 +2092,7 @@
                         "inf_dtype": "torch.float32",
                         "force_dtype": "torch.float32",
                         "scale": [
-                            0.31826239824295044
+                            0.31827905774116516
                         ],
                         "zero_point": [
                             214
@@ -2091,10 +2101,10 @@
                     {
                         "id": 33,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8",
-                        "force_dtype": "torch.quint8",
+                        "inf_dtype": "torch.float32",
+                        "force_dtype": "torch.float32",
                         "scale": [
-                            0.3026157021522522
+                            0.3027787208557129
                         ],
                         "zero_point": [
                             239
@@ -2115,7 +2125,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -2124,7 +2135,8 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             }
         },
@@ -2143,7 +2155,7 @@
                     {
                         "id": 33,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8"
+                        "inf_dtype": "torch.float32"
                     }
                 ]
             },
@@ -2161,7 +2173,7 @@
                     {
                         "id": 39,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8"
+                        "inf_dtype": "torch.float32"
                     }
                 ]
             }
@@ -2170,7 +2182,7 @@
             {
                 "id": 39,
                 "orig_dtype": "torch.float32",
-                "inf_dtype": "torch.quint8"
+                "inf_dtype": "torch.float32"
             }
         ]
     },
@@ -2184,13 +2196,13 @@
                     {
                         "id": 6,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8",
+                        "inf_dtype": "torch.float32",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.046401847153902054
+                            0.04691799730062485
                         ],
                         "zero_point": [
-                            184
+                            182
                         ]
                     }
                 ],
@@ -3753,7 +3765,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -3762,7 +3775,8 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             },
             "1": {
@@ -3773,13 +3787,13 @@
                     {
                         "id": 6,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8",
+                        "inf_dtype": "torch.float32",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.046401847153902054
+                            0.04691799730062485
                         ],
                         "zero_point": [
-                            184
+                            182
                         ]
                     }
                 ],
@@ -5342,7 +5356,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -5351,7 +5366,8 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             },
             "2": {
@@ -5362,13 +5378,13 @@
                     {
                         "id": 6,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8",
+                        "inf_dtype": "torch.float32",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.046401847153902054
+                            0.04691799730062485
                         ],
                         "zero_point": [
-                            184
+                            182
                         ]
                     }
                 ],
@@ -6931,7 +6947,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -6940,43 +6957,44 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             },
             "3": {
-                "op_type": "<built-in method matmul of type object at 0x7fb59440e700>",
+                "op_type": "<built-in method matmul of type object at 0x7fd649399960>",
                 "op_type_is_module": false,
                 "fqn": "distilbert.transformer.layer.0.attention",
                 "input_tensor_infos": [
                     {
-                        "id": 16,
+                        "id": 9,
                         "orig_dtype": "torch.float32",
                         "inf_dtype": "torch.quint8",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.00648243073374033
+                            0.05323055759072304
                         ],
                         "zero_point": [
-                            133
+                            136
                         ]
                     },
                     {
-                        "id": 17,
+                        "id": 16,
                         "orig_dtype": "torch.float32",
                         "inf_dtype": "torch.quint8",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.05702095851302147
+                            0.06305121630430222
                         ],
                         "zero_point": [
-                            122
+                            134
                         ]
                     }
                 ],
                 "weight_tensor_infos": [],
                 "output_tensor_infos": [
                     {
-                        "id": 18,
+                        "id": 17,
                         "orig_dtype": "torch.float32",
                         "inf_dtype": "torch.float32"
                     }
@@ -6987,7 +7005,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -6996,11 +7015,12 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             },
             "4": {
-                "op_type": "<built-in method matmul of type object at 0x7fb59440e700>",
+                "op_type": "<built-in method matmul of type object at 0x7fd649399960>",
                 "op_type_is_module": false,
                 "fqn": "distilbert.transformer.layer.0.attention",
                 "input_tensor_infos": [
@@ -7010,7 +7030,7 @@
                         "inf_dtype": "torch.quint8",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.00392026687040925
+                            0.003921240568161011
                         ],
                         "zero_point": [
                             0
@@ -7022,10 +7042,10 @@
                         "inf_dtype": "torch.quint8",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.0453609935939312
+                            0.0456799641251564
                         ],
                         "zero_point": [
-                            97
+                            98
                         ]
                     }
                 ],
@@ -7043,7 +7063,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -7052,7 +7073,8 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             },
             "5": {
@@ -7066,10 +7088,10 @@
                         "inf_dtype": "torch.quint8",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.039291027933359146
+                            0.04080498591065407
                         ],
                         "zero_point": [
-                            93
+                            97
                         ]
                     }
                 ],
@@ -8632,7 +8654,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -8641,38 +8664,39 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             }
         },
         "nonq_op_infos": {
             "0": {
-                "op_type": "<method 'size' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'size' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.0.attention",
                 "input_tensor_infos": [
                     {
                         "id": 6,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8"
+                        "inf_dtype": "torch.float32"
                     }
                 ],
                 "output_tensor_infos": []
             },
             "1": {
-                "op_type": "<method 'size' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'size' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.0.attention",
                 "input_tensor_infos": [
                     {
                         "id": 6,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8"
+                        "inf_dtype": "torch.float32"
                     },
                     {}
                 ],
                 "output_tensor_infos": []
             },
             "2": {
-                "op_type": "<method 'view' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'view' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.0.attention",
                 "input_tensor_infos": [
                     {
@@ -8694,7 +8718,7 @@
                 ]
             },
             "3": {
-                "op_type": "<method 'transpose' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'transpose' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.0.attention",
                 "input_tensor_infos": [
                     {
@@ -8709,12 +8733,12 @@
                     {
                         "id": 9,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.float32"
+                        "inf_dtype": "torch.quint8"
                     }
                 ]
             },
             "4": {
-                "op_type": "<method 'view' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'view' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.0.attention",
                 "input_tensor_infos": [
                     {
@@ -8736,7 +8760,7 @@
                 ]
             },
             "5": {
-                "op_type": "<method 'transpose' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'transpose' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.0.attention",
                 "input_tensor_infos": [
                     {
@@ -8756,7 +8780,7 @@
                 ]
             },
             "6": {
-                "op_type": "<method 'view' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'view' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.0.attention",
                 "input_tensor_infos": [
                     {
@@ -8778,7 +8802,7 @@
                 ]
             },
             "7": {
-                "op_type": "<method 'transpose' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'transpose' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.0.attention",
                 "input_tensor_infos": [
                     {
@@ -8798,14 +8822,15 @@
                 ]
             },
             "8": {
-                "op_type": "<method 'div' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'transpose' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.0.attention",
                 "input_tensor_infos": [
                     {
-                        "id": 9,
+                        "id": 12,
                         "orig_dtype": "torch.float32",
                         "inf_dtype": "torch.float32"
                     },
+                    {},
                     {}
                 ],
                 "output_tensor_infos": [
@@ -8817,27 +8842,26 @@
                 ]
             },
             "9": {
-                "op_type": "<method 'transpose' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'div' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.0.attention",
                 "input_tensor_infos": [
                     {
-                        "id": 12,
+                        "id": 17,
                         "orig_dtype": "torch.float32",
                         "inf_dtype": "torch.float32"
                     },
-                    {},
                     {}
                 ],
                 "output_tensor_infos": [
                     {
-                        "id": 17,
+                        "id": 18,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8"
+                        "inf_dtype": "torch.float32"
                     }
                 ]
             },
             "10": {
-                "op_type": "<method 'eq' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'eq' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.0.attention",
                 "input_tensor_infos": [
                     {
@@ -8856,7 +8880,7 @@
                 ]
             },
             "11": {
-                "op_type": "<method 'view' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'view' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.0.attention",
                 "input_tensor_infos": [
                     {
@@ -8878,7 +8902,7 @@
                 ]
             },
             "12": {
-                "op_type": "<method 'expand_as' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'expand_as' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.0.attention",
                 "input_tensor_infos": [
                     {
@@ -8901,7 +8925,7 @@
                 ]
             },
             "13": {
-                "op_type": "<method 'masked_fill' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'masked_fill' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.0.attention",
                 "input_tensor_infos": [
                     {
@@ -8929,7 +8953,7 @@
                 ]
             },
             "14": {
-                "op_type": "<function softmax at 0x7fb5dc102790>",
+                "op_type": "<function softmax at 0x7fd6d16608b0>",
                 "fqn": "distilbert.transformer.layer.0.attention",
                 "input_tensor_infos": [
                     {
@@ -8965,7 +8989,7 @@
                 ]
             },
             "16": {
-                "op_type": "<method 'transpose' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'transpose' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.0.attention",
                 "input_tensor_infos": [
                     {
@@ -8985,7 +9009,7 @@
                 ]
             },
             "17": {
-                "op_type": "<method 'contiguous' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'contiguous' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.0.attention",
                 "input_tensor_infos": [
                     {
@@ -9003,7 +9027,7 @@
                 ]
             },
             "18": {
-                "op_type": "<method 'view' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'view' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.0.attention",
                 "input_tensor_infos": [
                     {
@@ -9042,10 +9066,10 @@
                     {
                         "id": 33,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8",
+                        "inf_dtype": "torch.float32",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.3026157021522522
+                            0.3027787208557129
                         ],
                         "zero_point": [
                             239
@@ -15219,7 +15243,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -15228,7 +15253,8 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             },
             "1": {
@@ -15242,7 +15268,7 @@
                         "inf_dtype": "torch.quint8",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.07182127982378006
+                            0.07252021878957748
                         ],
                         "zero_point": [
                             2
@@ -16808,7 +16834,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -16817,7 +16844,8 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             }
         },
@@ -16882,7 +16910,7 @@
     "distilbert:transformer:layer:1": {
         "q_op_infos": {
             "0": {
-                "op_type": "<method 'add' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'add' of 'torch._C.TensorBase' objects>",
                 "op_type_is_module": false,
                 "fqn": "distilbert.transformer.layer.1",
                 "input_tensor_infos": [
@@ -16892,7 +16920,7 @@
                         "inf_dtype": "torch.float32",
                         "force_dtype": "torch.float32",
                         "scale": [
-                            0.10847434401512146
+                            0.11154649406671524
                         ],
                         "zero_point": [
                             195
@@ -16901,13 +16929,13 @@
                     {
                         "id": 39,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8",
-                        "force_dtype": "torch.quint8",
+                        "inf_dtype": "torch.float32",
+                        "force_dtype": "torch.float32",
                         "scale": [
-                            0.0508233904838562
+                            0.051652878522872925
                         ],
                         "zero_point": [
-                            191
+                            188
                         ]
                     }
                 ],
@@ -16925,7 +16953,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -16934,11 +16963,12 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             },
             "1": {
-                "op_type": "<method 'add' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'add' of 'torch._C.TensorBase' objects>",
                 "op_type_is_module": false,
                 "fqn": "distilbert.transformer.layer.1",
                 "input_tensor_infos": [
@@ -16948,7 +16978,7 @@
                         "inf_dtype": "torch.float32",
                         "force_dtype": "torch.float32",
                         "scale": [
-                            0.7653557062149048
+                            0.7658050656318665
                         ],
                         "zero_point": [
                             249
@@ -16957,10 +16987,10 @@
                     {
                         "id": 65,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8",
-                        "force_dtype": "torch.quint8",
+                        "inf_dtype": "torch.float32",
+                        "force_dtype": "torch.float32",
                         "scale": [
-                            0.20951531827449799
+                            0.21209503710269928
                         ],
                         "zero_point": [
                             230
@@ -16981,7 +17011,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -16990,7 +17021,8 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             }
         },
@@ -17009,7 +17041,7 @@
                     {
                         "id": 65,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8"
+                        "inf_dtype": "torch.float32"
                     }
                 ]
             },
@@ -17027,7 +17059,7 @@
                     {
                         "id": 71,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8"
+                        "inf_dtype": "torch.float32"
                     }
                 ]
             }
@@ -17036,7 +17068,7 @@
             {
                 "id": 71,
                 "orig_dtype": "torch.float32",
-                "inf_dtype": "torch.quint8"
+                "inf_dtype": "torch.float32"
             }
         ]
     },
@@ -17050,13 +17082,13 @@
                     {
                         "id": 39,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8",
+                        "inf_dtype": "torch.float32",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.0508233904838562
+                            0.051652878522872925
                         ],
                         "zero_point": [
-                            191
+                            188
                         ]
                     }
                 ],
@@ -18619,7 +18651,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -18628,7 +18661,8 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             },
             "1": {
@@ -18639,13 +18673,13 @@
                     {
                         "id": 39,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8",
+                        "inf_dtype": "torch.float32",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.0508233904838562
+                            0.051652878522872925
                         ],
                         "zero_point": [
-                            191
+                            188
                         ]
                     }
                 ],
@@ -20208,7 +20242,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -20217,7 +20252,8 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             },
             "2": {
@@ -20228,13 +20264,13 @@
                     {
                         "id": 39,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8",
+                        "inf_dtype": "torch.float32",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.0508233904838562
+                            0.051652878522872925
                         ],
                         "zero_point": [
-                            191
+                            188
                         ]
                     }
                 ],
@@ -21797,7 +21833,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -21806,43 +21843,44 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             },
             "3": {
-                "op_type": "<built-in method matmul of type object at 0x7fb59440e700>",
+                "op_type": "<built-in method matmul of type object at 0x7fd649399960>",
                 "op_type_is_module": false,
                 "fqn": "distilbert.transformer.layer.1.attention",
                 "input_tensor_infos": [
                     {
-                        "id": 49,
+                        "id": 42,
                         "orig_dtype": "torch.float32",
                         "inf_dtype": "torch.quint8",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.009408144280314445
+                            0.07946352660655975
                         ],
                         "zero_point": [
-                            131
+                            129
                         ]
                     },
                     {
-                        "id": 50,
+                        "id": 49,
                         "orig_dtype": "torch.float32",
                         "inf_dtype": "torch.quint8",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.07137680053710938
+                            0.07269971072673798
                         ],
                         "zero_point": [
-                            137
+                            134
                         ]
                     }
                 ],
                 "weight_tensor_infos": [],
                 "output_tensor_infos": [
                     {
-                        "id": 51,
+                        "id": 50,
                         "orig_dtype": "torch.float32",
                         "inf_dtype": "torch.float32"
                     }
@@ -21853,7 +21891,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -21862,11 +21901,12 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             },
             "4": {
-                "op_type": "<built-in method matmul of type object at 0x7fb59440e700>",
+                "op_type": "<built-in method matmul of type object at 0x7fd649399960>",
                 "op_type_is_module": false,
                 "fqn": "distilbert.transformer.layer.1.attention",
                 "input_tensor_infos": [
@@ -21888,10 +21928,10 @@
                         "inf_dtype": "torch.quint8",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.03892306238412857
+                            0.04066462814807892
                         ],
                         "zero_point": [
-                            111
+                            114
                         ]
                     }
                 ],
@@ -21909,7 +21949,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -21918,7 +21959,8 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             },
             "5": {
@@ -21932,10 +21974,10 @@
                         "inf_dtype": "torch.quint8",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.030531667172908783
+                            0.03188679367303848
                         ],
                         "zero_point": [
-                            116
+                            122
                         ]
                     }
                 ],
@@ -23498,7 +23540,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -23507,38 +23550,39 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             }
         },
         "nonq_op_infos": {
             "0": {
-                "op_type": "<method 'size' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'size' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.1.attention",
                 "input_tensor_infos": [
                     {
                         "id": 39,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8"
+                        "inf_dtype": "torch.float32"
                     }
                 ],
                 "output_tensor_infos": []
             },
             "1": {
-                "op_type": "<method 'size' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'size' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.1.attention",
                 "input_tensor_infos": [
                     {
                         "id": 39,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8"
+                        "inf_dtype": "torch.float32"
                     },
                     {}
                 ],
                 "output_tensor_infos": []
             },
             "2": {
-                "op_type": "<method 'view' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'view' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.1.attention",
                 "input_tensor_infos": [
                     {
@@ -23560,7 +23604,7 @@
                 ]
             },
             "3": {
-                "op_type": "<method 'transpose' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'transpose' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.1.attention",
                 "input_tensor_infos": [
                     {
@@ -23575,12 +23619,12 @@
                     {
                         "id": 42,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.float32"
+                        "inf_dtype": "torch.quint8"
                     }
                 ]
             },
             "4": {
-                "op_type": "<method 'view' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'view' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.1.attention",
                 "input_tensor_infos": [
                     {
@@ -23602,7 +23646,7 @@
                 ]
             },
             "5": {
-                "op_type": "<method 'transpose' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'transpose' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.1.attention",
                 "input_tensor_infos": [
                     {
@@ -23622,7 +23666,7 @@
                 ]
             },
             "6": {
-                "op_type": "<method 'view' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'view' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.1.attention",
                 "input_tensor_infos": [
                     {
@@ -23644,7 +23688,7 @@
                 ]
             },
             "7": {
-                "op_type": "<method 'transpose' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'transpose' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.1.attention",
                 "input_tensor_infos": [
                     {
@@ -23664,14 +23708,15 @@
                 ]
             },
             "8": {
-                "op_type": "<method 'div' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'transpose' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.1.attention",
                 "input_tensor_infos": [
                     {
-                        "id": 42,
+                        "id": 45,
                         "orig_dtype": "torch.float32",
                         "inf_dtype": "torch.float32"
                     },
+                    {},
                     {}
                 ],
                 "output_tensor_infos": [
@@ -23683,27 +23728,26 @@
                 ]
             },
             "9": {
-                "op_type": "<method 'transpose' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'div' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.1.attention",
                 "input_tensor_infos": [
                     {
-                        "id": 45,
+                        "id": 50,
                         "orig_dtype": "torch.float32",
                         "inf_dtype": "torch.float32"
                     },
-                    {},
                     {}
                 ],
                 "output_tensor_infos": [
                     {
-                        "id": 50,
+                        "id": 51,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8"
+                        "inf_dtype": "torch.float32"
                     }
                 ]
             },
             "10": {
-                "op_type": "<method 'eq' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'eq' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.1.attention",
                 "input_tensor_infos": [
                     {
@@ -23722,7 +23766,7 @@
                 ]
             },
             "11": {
-                "op_type": "<method 'view' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'view' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.1.attention",
                 "input_tensor_infos": [
                     {
@@ -23744,7 +23788,7 @@
                 ]
             },
             "12": {
-                "op_type": "<method 'expand_as' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'expand_as' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.1.attention",
                 "input_tensor_infos": [
                     {
@@ -23767,7 +23811,7 @@
                 ]
             },
             "13": {
-                "op_type": "<method 'masked_fill' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'masked_fill' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.1.attention",
                 "input_tensor_infos": [
                     {
@@ -23795,7 +23839,7 @@
                 ]
             },
             "14": {
-                "op_type": "<function softmax at 0x7fb5dc102790>",
+                "op_type": "<function softmax at 0x7fd6d16608b0>",
                 "fqn": "distilbert.transformer.layer.1.attention",
                 "input_tensor_infos": [
                     {
@@ -23831,7 +23875,7 @@
                 ]
             },
             "16": {
-                "op_type": "<method 'transpose' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'transpose' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.1.attention",
                 "input_tensor_infos": [
                     {
@@ -23851,7 +23895,7 @@
                 ]
             },
             "17": {
-                "op_type": "<method 'contiguous' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'contiguous' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.1.attention",
                 "input_tensor_infos": [
                     {
@@ -23869,7 +23913,7 @@
                 ]
             },
             "18": {
-                "op_type": "<method 'view' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'view' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.1.attention",
                 "input_tensor_infos": [
                     {
@@ -23908,10 +23952,10 @@
                     {
                         "id": 65,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8",
+                        "inf_dtype": "torch.float32",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.20951531827449799
+                            0.21209503710269928
                         ],
                         "zero_point": [
                             230
@@ -30085,7 +30129,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -30094,7 +30139,8 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             },
             "1": {
@@ -31674,7 +31720,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -31683,7 +31730,8 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             }
         },
@@ -31748,7 +31796,7 @@
     "distilbert:transformer:layer:2": {
         "q_op_infos": {
             "0": {
-                "op_type": "<method 'add' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'add' of 'torch._C.TensorBase' objects>",
                 "op_type_is_module": false,
                 "fqn": "distilbert.transformer.layer.2",
                 "input_tensor_infos": [
@@ -31758,7 +31806,7 @@
                         "inf_dtype": "torch.float32",
                         "force_dtype": "torch.float32",
                         "scale": [
-                            0.0439857542514801
+                            0.04897928610444069
                         ],
                         "zero_point": [
                             139
@@ -31767,10 +31815,10 @@
                     {
                         "id": 71,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8",
-                        "force_dtype": "torch.quint8",
+                        "inf_dtype": "torch.float32",
+                        "force_dtype": "torch.float32",
                         "scale": [
-                            0.0711798146367073
+                            0.07145479321479797
                         ],
                         "zero_point": [
                             201
@@ -31791,7 +31839,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -31800,11 +31849,12 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             },
             "1": {
-                "op_type": "<method 'add' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'add' of 'torch._C.TensorBase' objects>",
                 "op_type_is_module": false,
                 "fqn": "distilbert.transformer.layer.2",
                 "input_tensor_infos": [
@@ -31814,22 +31864,22 @@
                         "inf_dtype": "torch.float32",
                         "force_dtype": "torch.float32",
                         "scale": [
-                            0.9983832240104675
+                            1.005752444267273
                         ],
                         "zero_point": [
-                            251
+                            250
                         ]
                     },
                     {
                         "id": 97,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8",
-                        "force_dtype": "torch.quint8",
+                        "inf_dtype": "torch.float32",
+                        "force_dtype": "torch.float32",
                         "scale": [
-                            0.434603750705719
+                            0.43636801838874817
                         ],
                         "zero_point": [
-                            244
+                            243
                         ]
                     }
                 ],
@@ -31847,7 +31897,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -31856,7 +31907,8 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             }
         },
@@ -31875,7 +31927,7 @@
                     {
                         "id": 97,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8"
+                        "inf_dtype": "torch.float32"
                     }
                 ]
             },
@@ -31893,7 +31945,7 @@
                     {
                         "id": 103,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8"
+                        "inf_dtype": "torch.float32"
                     }
                 ]
             }
@@ -31902,7 +31954,7 @@
             {
                 "id": 103,
                 "orig_dtype": "torch.float32",
-                "inf_dtype": "torch.quint8"
+                "inf_dtype": "torch.float32"
             }
         ]
     },
@@ -31916,10 +31968,10 @@
                     {
                         "id": 71,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8",
+                        "inf_dtype": "torch.float32",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.0711798146367073
+                            0.07145479321479797
                         ],
                         "zero_point": [
                             201
@@ -33485,7 +33537,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -33494,7 +33547,8 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             },
             "1": {
@@ -33505,10 +33559,10 @@
                     {
                         "id": 71,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8",
+                        "inf_dtype": "torch.float32",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.0711798146367073
+                            0.07145479321479797
                         ],
                         "zero_point": [
                             201
@@ -35074,7 +35128,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -35083,7 +35138,8 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             },
             "2": {
@@ -35094,10 +35150,10 @@
                     {
                         "id": 71,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8",
+                        "inf_dtype": "torch.float32",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.0711798146367073
+                            0.07145479321479797
                         ],
                         "zero_point": [
                             201
@@ -36663,7 +36719,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -36672,33 +36729,34 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             },
             "3": {
-                "op_type": "<built-in method matmul of type object at 0x7fb59440e700>",
+                "op_type": "<built-in method matmul of type object at 0x7fd649399960>",
                 "op_type_is_module": false,
                 "fqn": "distilbert.transformer.layer.2.attention",
                 "input_tensor_infos": [
                     {
-                        "id": 81,
+                        "id": 74,
                         "orig_dtype": "torch.float32",
                         "inf_dtype": "torch.quint8",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.007862313650548458
+                            0.06289852410554886
                         ],
                         "zero_point": [
                             124
                         ]
                     },
                     {
-                        "id": 82,
+                        "id": 81,
                         "orig_dtype": "torch.float32",
                         "inf_dtype": "torch.quint8",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.05403390899300575
+                            0.05672023072838783
                         ],
                         "zero_point": [
                             139
@@ -36708,7 +36766,7 @@
                 "weight_tensor_infos": [],
                 "output_tensor_infos": [
                     {
-                        "id": 83,
+                        "id": 82,
                         "orig_dtype": "torch.float32",
                         "inf_dtype": "torch.float32"
                     }
@@ -36719,7 +36777,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -36728,11 +36787,12 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             },
             "4": {
-                "op_type": "<built-in method matmul of type object at 0x7fb59440e700>",
+                "op_type": "<built-in method matmul of type object at 0x7fd649399960>",
                 "op_type_is_module": false,
                 "fqn": "distilbert.transformer.layer.2.attention",
                 "input_tensor_infos": [
@@ -36742,7 +36802,7 @@
                         "inf_dtype": "torch.quint8",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.003921302501112223
+                            0.003921313676983118
                         ],
                         "zero_point": [
                             0
@@ -36754,7 +36814,7 @@
                         "inf_dtype": "torch.quint8",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.04678581655025482
+                            0.04692273586988449
                         ],
                         "zero_point": [
                             134
@@ -36775,7 +36835,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -36784,7 +36845,8 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             },
             "5": {
@@ -36798,10 +36860,10 @@
                         "inf_dtype": "torch.quint8",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.0362231619656086
+                            0.03697829321026802
                         ],
                         "zero_point": [
-                            139
+                            136
                         ]
                     }
                 ],
@@ -38364,7 +38426,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -38373,38 +38436,39 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             }
         },
         "nonq_op_infos": {
             "0": {
-                "op_type": "<method 'size' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'size' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.2.attention",
                 "input_tensor_infos": [
                     {
                         "id": 71,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8"
+                        "inf_dtype": "torch.float32"
                     }
                 ],
                 "output_tensor_infos": []
             },
             "1": {
-                "op_type": "<method 'size' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'size' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.2.attention",
                 "input_tensor_infos": [
                     {
                         "id": 71,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8"
+                        "inf_dtype": "torch.float32"
                     },
                     {}
                 ],
                 "output_tensor_infos": []
             },
             "2": {
-                "op_type": "<method 'view' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'view' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.2.attention",
                 "input_tensor_infos": [
                     {
@@ -38426,7 +38490,7 @@
                 ]
             },
             "3": {
-                "op_type": "<method 'transpose' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'transpose' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.2.attention",
                 "input_tensor_infos": [
                     {
@@ -38441,12 +38505,12 @@
                     {
                         "id": 74,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.float32"
+                        "inf_dtype": "torch.quint8"
                     }
                 ]
             },
             "4": {
-                "op_type": "<method 'view' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'view' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.2.attention",
                 "input_tensor_infos": [
                     {
@@ -38468,7 +38532,7 @@
                 ]
             },
             "5": {
-                "op_type": "<method 'transpose' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'transpose' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.2.attention",
                 "input_tensor_infos": [
                     {
@@ -38488,7 +38552,7 @@
                 ]
             },
             "6": {
-                "op_type": "<method 'view' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'view' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.2.attention",
                 "input_tensor_infos": [
                     {
@@ -38510,7 +38574,7 @@
                 ]
             },
             "7": {
-                "op_type": "<method 'transpose' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'transpose' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.2.attention",
                 "input_tensor_infos": [
                     {
@@ -38530,14 +38594,15 @@
                 ]
             },
             "8": {
-                "op_type": "<method 'div' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'transpose' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.2.attention",
                 "input_tensor_infos": [
                     {
-                        "id": 74,
+                        "id": 77,
                         "orig_dtype": "torch.float32",
                         "inf_dtype": "torch.float32"
                     },
+                    {},
                     {}
                 ],
                 "output_tensor_infos": [
@@ -38549,27 +38614,26 @@
                 ]
             },
             "9": {
-                "op_type": "<method 'transpose' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'div' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.2.attention",
                 "input_tensor_infos": [
                     {
-                        "id": 77,
+                        "id": 82,
                         "orig_dtype": "torch.float32",
                         "inf_dtype": "torch.float32"
                     },
-                    {},
                     {}
                 ],
                 "output_tensor_infos": [
                     {
-                        "id": 82,
+                        "id": 83,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8"
+                        "inf_dtype": "torch.float32"
                     }
                 ]
             },
             "10": {
-                "op_type": "<method 'eq' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'eq' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.2.attention",
                 "input_tensor_infos": [
                     {
@@ -38588,7 +38652,7 @@
                 ]
             },
             "11": {
-                "op_type": "<method 'view' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'view' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.2.attention",
                 "input_tensor_infos": [
                     {
@@ -38610,7 +38674,7 @@
                 ]
             },
             "12": {
-                "op_type": "<method 'expand_as' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'expand_as' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.2.attention",
                 "input_tensor_infos": [
                     {
@@ -38633,7 +38697,7 @@
                 ]
             },
             "13": {
-                "op_type": "<method 'masked_fill' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'masked_fill' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.2.attention",
                 "input_tensor_infos": [
                     {
@@ -38661,7 +38725,7 @@
                 ]
             },
             "14": {
-                "op_type": "<function softmax at 0x7fb5dc102790>",
+                "op_type": "<function softmax at 0x7fd6d16608b0>",
                 "fqn": "distilbert.transformer.layer.2.attention",
                 "input_tensor_infos": [
                     {
@@ -38697,7 +38761,7 @@
                 ]
             },
             "16": {
-                "op_type": "<method 'transpose' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'transpose' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.2.attention",
                 "input_tensor_infos": [
                     {
@@ -38717,7 +38781,7 @@
                 ]
             },
             "17": {
-                "op_type": "<method 'contiguous' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'contiguous' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.2.attention",
                 "input_tensor_infos": [
                     {
@@ -38735,7 +38799,7 @@
                 ]
             },
             "18": {
-                "op_type": "<method 'view' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'view' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.2.attention",
                 "input_tensor_infos": [
                     {
@@ -38774,13 +38838,13 @@
                     {
                         "id": 97,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8",
+                        "inf_dtype": "torch.float32",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.434603750705719
+                            0.43636801838874817
                         ],
                         "zero_point": [
-                            244
+                            243
                         ]
                     }
                 ],
@@ -44951,7 +45015,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -44960,7 +45025,8 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             },
             "1": {
@@ -44974,7 +45040,7 @@
                         "inf_dtype": "torch.quint8",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.09822209924459457
+                            0.09847944974899292
                         ],
                         "zero_point": [
                             2
@@ -46540,7 +46606,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -46549,7 +46616,8 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             }
         },
@@ -46614,7 +46682,7 @@
     "distilbert:transformer:layer:3": {
         "q_op_infos": {
             "0": {
-                "op_type": "<method 'add' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'add' of 'torch._C.TensorBase' objects>",
                 "op_type_is_module": false,
                 "fqn": "distilbert.transformer.layer.3",
                 "input_tensor_infos": [
@@ -46624,19 +46692,19 @@
                         "inf_dtype": "torch.float32",
                         "force_dtype": "torch.float32",
                         "scale": [
-                            0.05528954043984413
+                            0.059873275458812714
                         ],
                         "zero_point": [
-                            155
+                            149
                         ]
                     },
                     {
                         "id": 103,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8",
-                        "force_dtype": "torch.quint8",
+                        "inf_dtype": "torch.float32",
+                        "force_dtype": "torch.float32",
                         "scale": [
-                            0.07015957683324814
+                            0.07034638524055481
                         ],
                         "zero_point": [
                             195
@@ -46657,7 +46725,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -46666,11 +46735,12 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             },
             "1": {
-                "op_type": "<method 'add' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'add' of 'torch._C.TensorBase' objects>",
                 "op_type_is_module": false,
                 "fqn": "distilbert.transformer.layer.3",
                 "input_tensor_infos": [
@@ -46680,7 +46750,7 @@
                         "inf_dtype": "torch.float32",
                         "force_dtype": "torch.float32",
                         "scale": [
-                            0.08726980537176132
+                            0.08726981282234192
                         ],
                         "zero_point": [
                             148
@@ -46689,13 +46759,13 @@
                     {
                         "id": 129,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8",
-                        "force_dtype": "torch.quint8",
+                        "inf_dtype": "torch.float32",
+                        "force_dtype": "torch.float32",
                         "scale": [
-                            0.387818843126297
+                            0.3892805576324463
                         ],
                         "zero_point": [
-                            244
+                            243
                         ]
                     }
                 ],
@@ -46713,7 +46783,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -46722,7 +46793,8 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             }
         },
@@ -46741,7 +46813,7 @@
                     {
                         "id": 129,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8"
+                        "inf_dtype": "torch.float32"
                     }
                 ]
             },
@@ -46759,7 +46831,7 @@
                     {
                         "id": 135,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8"
+                        "inf_dtype": "torch.float32"
                     }
                 ]
             }
@@ -46768,7 +46840,7 @@
             {
                 "id": 135,
                 "orig_dtype": "torch.float32",
-                "inf_dtype": "torch.quint8"
+                "inf_dtype": "torch.float32"
             }
         ]
     },
@@ -46782,10 +46854,10 @@
                     {
                         "id": 103,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8",
+                        "inf_dtype": "torch.float32",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.07015957683324814
+                            0.07034638524055481
                         ],
                         "zero_point": [
                             195
@@ -48351,7 +48423,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -48360,7 +48433,8 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             },
             "1": {
@@ -48371,10 +48445,10 @@
                     {
                         "id": 103,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8",
+                        "inf_dtype": "torch.float32",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.07015957683324814
+                            0.07034638524055481
                         ],
                         "zero_point": [
                             195
@@ -49940,7 +50014,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -49949,7 +50024,8 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             },
             "2": {
@@ -49960,10 +50036,10 @@
                     {
                         "id": 103,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8",
+                        "inf_dtype": "torch.float32",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.07015957683324814
+                            0.07034638524055481
                         ],
                         "zero_point": [
                             195
@@ -51529,7 +51605,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -51538,43 +51615,44 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             },
             "3": {
-                "op_type": "<built-in method matmul of type object at 0x7fb59440e700>",
+                "op_type": "<built-in method matmul of type object at 0x7fd649399960>",
                 "op_type_is_module": false,
                 "fqn": "distilbert.transformer.layer.3.attention",
                 "input_tensor_infos": [
                     {
-                        "id": 113,
+                        "id": 106,
                         "orig_dtype": "torch.float32",
                         "inf_dtype": "torch.quint8",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.007643361110240221
+                            0.06411602348089218
                         ],
                         "zero_point": [
-                            138
+                            141
                         ]
                     },
                     {
-                        "id": 114,
+                        "id": 113,
                         "orig_dtype": "torch.float32",
                         "inf_dtype": "torch.quint8",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.06173744425177574
+                            0.06230035796761513
                         ],
                         "zero_point": [
-                            127
+                            126
                         ]
                     }
                 ],
                 "weight_tensor_infos": [],
                 "output_tensor_infos": [
                     {
-                        "id": 115,
+                        "id": 114,
                         "orig_dtype": "torch.float32",
                         "inf_dtype": "torch.float32"
                     }
@@ -51585,7 +51663,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -51594,11 +51673,12 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             },
             "4": {
-                "op_type": "<built-in method matmul of type object at 0x7fb59440e700>",
+                "op_type": "<built-in method matmul of type object at 0x7fd649399960>",
                 "op_type_is_module": false,
                 "fqn": "distilbert.transformer.layer.3.attention",
                 "input_tensor_infos": [
@@ -51620,10 +51700,10 @@
                         "inf_dtype": "torch.quint8",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.04956842213869095
+                            0.05577118322253227
                         ],
                         "zero_point": [
-                            132
+                            139
                         ]
                     }
                 ],
@@ -51641,7 +51721,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -51650,7 +51731,8 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             },
             "5": {
@@ -51664,7 +51746,7 @@
                         "inf_dtype": "torch.quint8",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.04299405217170715
+                            0.04299403354525566
                         ],
                         "zero_point": [
                             125
@@ -53230,7 +53312,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -53239,38 +53322,39 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             }
         },
         "nonq_op_infos": {
             "0": {
-                "op_type": "<method 'size' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'size' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.3.attention",
                 "input_tensor_infos": [
                     {
                         "id": 103,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8"
+                        "inf_dtype": "torch.float32"
                     }
                 ],
                 "output_tensor_infos": []
             },
             "1": {
-                "op_type": "<method 'size' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'size' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.3.attention",
                 "input_tensor_infos": [
                     {
                         "id": 103,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8"
+                        "inf_dtype": "torch.float32"
                     },
                     {}
                 ],
                 "output_tensor_infos": []
             },
             "2": {
-                "op_type": "<method 'view' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'view' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.3.attention",
                 "input_tensor_infos": [
                     {
@@ -53292,7 +53376,7 @@
                 ]
             },
             "3": {
-                "op_type": "<method 'transpose' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'transpose' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.3.attention",
                 "input_tensor_infos": [
                     {
@@ -53307,12 +53391,12 @@
                     {
                         "id": 106,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.float32"
+                        "inf_dtype": "torch.quint8"
                     }
                 ]
             },
             "4": {
-                "op_type": "<method 'view' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'view' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.3.attention",
                 "input_tensor_infos": [
                     {
@@ -53334,7 +53418,7 @@
                 ]
             },
             "5": {
-                "op_type": "<method 'transpose' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'transpose' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.3.attention",
                 "input_tensor_infos": [
                     {
@@ -53354,7 +53438,7 @@
                 ]
             },
             "6": {
-                "op_type": "<method 'view' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'view' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.3.attention",
                 "input_tensor_infos": [
                     {
@@ -53376,7 +53460,7 @@
                 ]
             },
             "7": {
-                "op_type": "<method 'transpose' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'transpose' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.3.attention",
                 "input_tensor_infos": [
                     {
@@ -53396,14 +53480,15 @@
                 ]
             },
             "8": {
-                "op_type": "<method 'div' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'transpose' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.3.attention",
                 "input_tensor_infos": [
                     {
-                        "id": 106,
+                        "id": 109,
                         "orig_dtype": "torch.float32",
                         "inf_dtype": "torch.float32"
                     },
+                    {},
                     {}
                 ],
                 "output_tensor_infos": [
@@ -53415,27 +53500,26 @@
                 ]
             },
             "9": {
-                "op_type": "<method 'transpose' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'div' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.3.attention",
                 "input_tensor_infos": [
                     {
-                        "id": 109,
+                        "id": 114,
                         "orig_dtype": "torch.float32",
                         "inf_dtype": "torch.float32"
                     },
-                    {},
                     {}
                 ],
                 "output_tensor_infos": [
                     {
-                        "id": 114,
+                        "id": 115,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8"
+                        "inf_dtype": "torch.float32"
                     }
                 ]
             },
             "10": {
-                "op_type": "<method 'eq' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'eq' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.3.attention",
                 "input_tensor_infos": [
                     {
@@ -53454,7 +53538,7 @@
                 ]
             },
             "11": {
-                "op_type": "<method 'view' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'view' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.3.attention",
                 "input_tensor_infos": [
                     {
@@ -53476,7 +53560,7 @@
                 ]
             },
             "12": {
-                "op_type": "<method 'expand_as' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'expand_as' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.3.attention",
                 "input_tensor_infos": [
                     {
@@ -53499,7 +53583,7 @@
                 ]
             },
             "13": {
-                "op_type": "<method 'masked_fill' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'masked_fill' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.3.attention",
                 "input_tensor_infos": [
                     {
@@ -53527,7 +53611,7 @@
                 ]
             },
             "14": {
-                "op_type": "<function softmax at 0x7fb5dc102790>",
+                "op_type": "<function softmax at 0x7fd6d16608b0>",
                 "fqn": "distilbert.transformer.layer.3.attention",
                 "input_tensor_infos": [
                     {
@@ -53563,7 +53647,7 @@
                 ]
             },
             "16": {
-                "op_type": "<method 'transpose' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'transpose' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.3.attention",
                 "input_tensor_infos": [
                     {
@@ -53583,7 +53667,7 @@
                 ]
             },
             "17": {
-                "op_type": "<method 'contiguous' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'contiguous' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.3.attention",
                 "input_tensor_infos": [
                     {
@@ -53601,7 +53685,7 @@
                 ]
             },
             "18": {
-                "op_type": "<method 'view' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'view' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.3.attention",
                 "input_tensor_infos": [
                     {
@@ -53640,13 +53724,13 @@
                     {
                         "id": 129,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8",
+                        "inf_dtype": "torch.float32",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.387818843126297
+                            0.3892805576324463
                         ],
                         "zero_point": [
-                            244
+                            243
                         ]
                     }
                 ],
@@ -59817,7 +59901,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -59826,7 +59911,8 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             },
             "1": {
@@ -59840,10 +59926,10 @@
                         "inf_dtype": "torch.quint8",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.027507111430168152
+                            0.03402227535843849
                         ],
                         "zero_point": [
-                            6
+                            5
                         ]
                     }
                 ],
@@ -61406,7 +61492,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -61415,7 +61502,8 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             }
         },
@@ -61480,7 +61568,7 @@
     "distilbert:transformer:layer:4": {
         "q_op_infos": {
             "0": {
-                "op_type": "<method 'add' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'add' of 'torch._C.TensorBase' objects>",
                 "op_type_is_module": false,
                 "fqn": "distilbert.transformer.layer.4",
                 "input_tensor_infos": [
@@ -61490,22 +61578,22 @@
                         "inf_dtype": "torch.float32",
                         "force_dtype": "torch.float32",
                         "scale": [
-                            0.06198941171169281
+                            0.06525913625955582
                         ],
                         "zero_point": [
-                            150
+                            155
                         ]
                     },
                     {
                         "id": 135,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8",
-                        "force_dtype": "torch.quint8",
+                        "inf_dtype": "torch.float32",
+                        "force_dtype": "torch.float32",
                         "scale": [
-                            0.08957237005233765
+                            0.09207678586244583
                         ],
                         "zero_point": [
-                            214
+                            213
                         ]
                     }
                 ],
@@ -61523,7 +61611,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -61532,11 +61621,12 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             },
             "1": {
-                "op_type": "<method 'add' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'add' of 'torch._C.TensorBase' objects>",
                 "op_type_is_module": false,
                 "fqn": "distilbert.transformer.layer.4",
                 "input_tensor_infos": [
@@ -61546,7 +61636,7 @@
                         "inf_dtype": "torch.float32",
                         "force_dtype": "torch.float32",
                         "scale": [
-                            5.07191801071167
+                            5.462576389312744
                         ],
                         "zero_point": [
                             168
@@ -61555,13 +61645,13 @@
                     {
                         "id": 161,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8",
-                        "force_dtype": "torch.quint8",
+                        "inf_dtype": "torch.float32",
+                        "force_dtype": "torch.float32",
                         "scale": [
-                            0.24626225233078003
+                            0.25417351722717285
                         ],
                         "zero_point": [
-                            218
+                            216
                         ]
                     }
                 ],
@@ -61579,7 +61669,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -61588,7 +61679,8 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             }
         },
@@ -61607,7 +61699,7 @@
                     {
                         "id": 161,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8"
+                        "inf_dtype": "torch.float32"
                     }
                 ]
             },
@@ -61625,7 +61717,7 @@
                     {
                         "id": 167,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8"
+                        "inf_dtype": "torch.float32"
                     }
                 ]
             }
@@ -61634,7 +61726,7 @@
             {
                 "id": 167,
                 "orig_dtype": "torch.float32",
-                "inf_dtype": "torch.quint8"
+                "inf_dtype": "torch.float32"
             }
         ]
     },
@@ -61648,13 +61740,13 @@
                     {
                         "id": 135,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8",
+                        "inf_dtype": "torch.float32",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.08957237005233765
+                            0.09207678586244583
                         ],
                         "zero_point": [
-                            214
+                            213
                         ]
                     }
                 ],
@@ -63217,7 +63309,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -63226,7 +63319,8 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             },
             "1": {
@@ -63237,13 +63331,13 @@
                     {
                         "id": 135,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8",
+                        "inf_dtype": "torch.float32",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.08957237005233765
+                            0.09207678586244583
                         ],
                         "zero_point": [
-                            214
+                            213
                         ]
                     }
                 ],
@@ -64806,7 +64900,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -64815,7 +64910,8 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             },
             "2": {
@@ -64826,13 +64922,13 @@
                     {
                         "id": 135,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8",
+                        "inf_dtype": "torch.float32",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.08957237005233765
+                            0.09207678586244583
                         ],
                         "zero_point": [
-                            214
+                            213
                         ]
                     }
                 ],
@@ -66395,7 +66491,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -66404,43 +66501,44 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             },
             "3": {
-                "op_type": "<built-in method matmul of type object at 0x7fb59440e700>",
+                "op_type": "<built-in method matmul of type object at 0x7fd649399960>",
                 "op_type_is_module": false,
                 "fqn": "distilbert.transformer.layer.4.attention",
                 "input_tensor_infos": [
                     {
-                        "id": 145,
+                        "id": 138,
                         "orig_dtype": "torch.float32",
                         "inf_dtype": "torch.quint8",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.0077769155614078045
+                            0.06345818936824799
                         ],
                         "zero_point": [
-                            132
+                            130
                         ]
                     },
                     {
-                        "id": 146,
+                        "id": 145,
                         "orig_dtype": "torch.float32",
                         "inf_dtype": "torch.quint8",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.0497259683907032
+                            0.052396662533283234
                         ],
                         "zero_point": [
-                            122
+                            127
                         ]
                     }
                 ],
                 "weight_tensor_infos": [],
                 "output_tensor_infos": [
                     {
-                        "id": 147,
+                        "id": 146,
                         "orig_dtype": "torch.float32",
                         "inf_dtype": "torch.float32"
                     }
@@ -66451,7 +66549,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -66460,11 +66559,12 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             },
             "4": {
-                "op_type": "<built-in method matmul of type object at 0x7fb59440e700>",
+                "op_type": "<built-in method matmul of type object at 0x7fd649399960>",
                 "op_type_is_module": false,
                 "fqn": "distilbert.transformer.layer.4.attention",
                 "input_tensor_infos": [
@@ -66486,7 +66586,7 @@
                         "inf_dtype": "torch.quint8",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.047192152589559555
+                            0.048682961612939835
                         ],
                         "zero_point": [
                             126
@@ -66507,7 +66607,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -66516,7 +66617,8 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             },
             "5": {
@@ -66530,10 +66632,10 @@
                         "inf_dtype": "torch.quint8",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.03943760693073273
+                            0.04368660971522331
                         ],
                         "zero_point": [
-                            132
+                            121
                         ]
                     }
                 ],
@@ -68096,7 +68198,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -68105,38 +68208,39 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             }
         },
         "nonq_op_infos": {
             "0": {
-                "op_type": "<method 'size' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'size' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.4.attention",
                 "input_tensor_infos": [
                     {
                         "id": 135,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8"
+                        "inf_dtype": "torch.float32"
                     }
                 ],
                 "output_tensor_infos": []
             },
             "1": {
-                "op_type": "<method 'size' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'size' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.4.attention",
                 "input_tensor_infos": [
                     {
                         "id": 135,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8"
+                        "inf_dtype": "torch.float32"
                     },
                     {}
                 ],
                 "output_tensor_infos": []
             },
             "2": {
-                "op_type": "<method 'view' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'view' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.4.attention",
                 "input_tensor_infos": [
                     {
@@ -68158,7 +68262,7 @@
                 ]
             },
             "3": {
-                "op_type": "<method 'transpose' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'transpose' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.4.attention",
                 "input_tensor_infos": [
                     {
@@ -68173,12 +68277,12 @@
                     {
                         "id": 138,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.float32"
+                        "inf_dtype": "torch.quint8"
                     }
                 ]
             },
             "4": {
-                "op_type": "<method 'view' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'view' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.4.attention",
                 "input_tensor_infos": [
                     {
@@ -68200,7 +68304,7 @@
                 ]
             },
             "5": {
-                "op_type": "<method 'transpose' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'transpose' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.4.attention",
                 "input_tensor_infos": [
                     {
@@ -68220,7 +68324,7 @@
                 ]
             },
             "6": {
-                "op_type": "<method 'view' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'view' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.4.attention",
                 "input_tensor_infos": [
                     {
@@ -68242,7 +68346,7 @@
                 ]
             },
             "7": {
-                "op_type": "<method 'transpose' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'transpose' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.4.attention",
                 "input_tensor_infos": [
                     {
@@ -68262,14 +68366,15 @@
                 ]
             },
             "8": {
-                "op_type": "<method 'div' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'transpose' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.4.attention",
                 "input_tensor_infos": [
                     {
-                        "id": 138,
+                        "id": 141,
                         "orig_dtype": "torch.float32",
                         "inf_dtype": "torch.float32"
                     },
+                    {},
                     {}
                 ],
                 "output_tensor_infos": [
@@ -68281,27 +68386,26 @@
                 ]
             },
             "9": {
-                "op_type": "<method 'transpose' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'div' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.4.attention",
                 "input_tensor_infos": [
                     {
-                        "id": 141,
+                        "id": 146,
                         "orig_dtype": "torch.float32",
                         "inf_dtype": "torch.float32"
                     },
-                    {},
                     {}
                 ],
                 "output_tensor_infos": [
                     {
-                        "id": 146,
+                        "id": 147,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8"
+                        "inf_dtype": "torch.float32"
                     }
                 ]
             },
             "10": {
-                "op_type": "<method 'eq' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'eq' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.4.attention",
                 "input_tensor_infos": [
                     {
@@ -68320,7 +68424,7 @@
                 ]
             },
             "11": {
-                "op_type": "<method 'view' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'view' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.4.attention",
                 "input_tensor_infos": [
                     {
@@ -68342,7 +68446,7 @@
                 ]
             },
             "12": {
-                "op_type": "<method 'expand_as' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'expand_as' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.4.attention",
                 "input_tensor_infos": [
                     {
@@ -68365,7 +68469,7 @@
                 ]
             },
             "13": {
-                "op_type": "<method 'masked_fill' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'masked_fill' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.4.attention",
                 "input_tensor_infos": [
                     {
@@ -68393,7 +68497,7 @@
                 ]
             },
             "14": {
-                "op_type": "<function softmax at 0x7fb5dc102790>",
+                "op_type": "<function softmax at 0x7fd6d16608b0>",
                 "fqn": "distilbert.transformer.layer.4.attention",
                 "input_tensor_infos": [
                     {
@@ -68429,7 +68533,7 @@
                 ]
             },
             "16": {
-                "op_type": "<method 'transpose' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'transpose' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.4.attention",
                 "input_tensor_infos": [
                     {
@@ -68449,7 +68553,7 @@
                 ]
             },
             "17": {
-                "op_type": "<method 'contiguous' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'contiguous' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.4.attention",
                 "input_tensor_infos": [
                     {
@@ -68467,7 +68571,7 @@
                 ]
             },
             "18": {
-                "op_type": "<method 'view' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'view' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.4.attention",
                 "input_tensor_infos": [
                     {
@@ -68506,13 +68610,13 @@
                     {
                         "id": 161,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8",
+                        "inf_dtype": "torch.float32",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.24626225233078003
+                            0.25417351722717285
                         ],
                         "zero_point": [
-                            218
+                            216
                         ]
                     }
                 ],
@@ -74683,7 +74787,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -74692,7 +74797,8 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             },
             "1": {
@@ -74706,7 +74812,7 @@
                         "inf_dtype": "torch.quint8",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.31002089381217957
+                            0.3301960527896881
                         ],
                         "zero_point": [
                             1
@@ -76272,7 +76378,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -76281,7 +76388,8 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             }
         },
@@ -76346,7 +76454,7 @@
     "distilbert:transformer:layer:5": {
         "q_op_infos": {
             "0": {
-                "op_type": "<method 'add' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'add' of 'torch._C.TensorBase' objects>",
                 "op_type_is_module": false,
                 "fqn": "distilbert.transformer.layer.5",
                 "input_tensor_infos": [
@@ -76356,22 +76464,22 @@
                         "inf_dtype": "torch.float32",
                         "force_dtype": "torch.float32",
                         "scale": [
-                            0.09456482529640198
+                            0.13014645874500275
                         ],
                         "zero_point": [
-                            137
+                            147
                         ]
                     },
                     {
                         "id": 167,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8",
-                        "force_dtype": "torch.quint8",
+                        "inf_dtype": "torch.float32",
+                        "force_dtype": "torch.float32",
                         "scale": [
-                            0.10347627103328705
+                            0.10631092637777328
                         ],
                         "zero_point": [
-                            220
+                            216
                         ]
                     }
                 ],
@@ -76389,7 +76497,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -76398,11 +76507,12 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             },
             "1": {
-                "op_type": "<method 'add' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'add' of 'torch._C.TensorBase' objects>",
                 "op_type_is_module": false,
                 "fqn": "distilbert.transformer.layer.5",
                 "input_tensor_infos": [
@@ -76412,22 +76522,22 @@
                         "inf_dtype": "torch.float32",
                         "force_dtype": "torch.float32",
                         "scale": [
-                            0.04283904656767845
+                            0.045247096568346024
                         ],
                         "zero_point": [
-                            154
+                            153
                         ]
                     },
                     {
                         "id": 193,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8",
-                        "force_dtype": "torch.quint8",
+                        "inf_dtype": "torch.float32",
+                        "force_dtype": "torch.float32",
                         "scale": [
-                            0.18003985285758972
+                            0.18901820480823517
                         ],
                         "zero_point": [
-                            192
+                            183
                         ]
                     }
                 ],
@@ -76445,7 +76555,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -76454,7 +76565,8 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             }
         },
@@ -76473,7 +76585,7 @@
                     {
                         "id": 193,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8"
+                        "inf_dtype": "torch.float32"
                     }
                 ]
             },
@@ -76514,13 +76626,13 @@
                     {
                         "id": 167,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8",
+                        "inf_dtype": "torch.float32",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.10347627103328705
+                            0.10631092637777328
                         ],
                         "zero_point": [
-                            220
+                            216
                         ]
                     }
                 ],
@@ -78083,7 +78195,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -78092,7 +78205,8 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             },
             "1": {
@@ -78103,13 +78217,13 @@
                     {
                         "id": 167,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8",
+                        "inf_dtype": "torch.float32",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.10347627103328705
+                            0.10631092637777328
                         ],
                         "zero_point": [
-                            220
+                            216
                         ]
                     }
                 ],
@@ -79672,7 +79786,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -79681,7 +79796,8 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             },
             "2": {
@@ -79692,13 +79808,13 @@
                     {
                         "id": 167,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8",
+                        "inf_dtype": "torch.float32",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.10347627103328705
+                            0.10631092637777328
                         ],
                         "zero_point": [
-                            220
+                            216
                         ]
                     }
                 ],
@@ -81261,7 +81377,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -81270,43 +81387,44 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             },
             "3": {
-                "op_type": "<built-in method matmul of type object at 0x7fb59440e700>",
+                "op_type": "<built-in method matmul of type object at 0x7fd649399960>",
                 "op_type_is_module": false,
                 "fqn": "distilbert.transformer.layer.5.attention",
                 "input_tensor_infos": [
                     {
-                        "id": 177,
+                        "id": 170,
                         "orig_dtype": "torch.float32",
                         "inf_dtype": "torch.quint8",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.006569200195372105
+                            0.053713031113147736
                         ],
                         "zero_point": [
-                            130
+                            127
                         ]
                     },
                     {
-                        "id": 178,
+                        "id": 177,
                         "orig_dtype": "torch.float32",
                         "inf_dtype": "torch.quint8",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.05540752410888672
+                            0.05568793788552284
                         ],
                         "zero_point": [
-                            132
+                            131
                         ]
                     }
                 ],
                 "weight_tensor_infos": [],
                 "output_tensor_infos": [
                     {
-                        "id": 179,
+                        "id": 178,
                         "orig_dtype": "torch.float32",
                         "inf_dtype": "torch.float32"
                     }
@@ -81317,7 +81435,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -81326,11 +81445,12 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             },
             "4": {
-                "op_type": "<built-in method matmul of type object at 0x7fb59440e700>",
+                "op_type": "<built-in method matmul of type object at 0x7fd649399960>",
                 "op_type_is_module": false,
                 "fqn": "distilbert.transformer.layer.5.attention",
                 "input_tensor_infos": [
@@ -81340,7 +81460,7 @@
                         "inf_dtype": "torch.quint8",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.003876183880493045
+                            0.003903247881680727
                         ],
                         "zero_point": [
                             0
@@ -81352,10 +81472,10 @@
                         "inf_dtype": "torch.quint8",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.057793986052274704
+                            0.06308483332395554
                         ],
                         "zero_point": [
-                            124
+                            126
                         ]
                     }
                 ],
@@ -81373,7 +81493,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -81382,7 +81503,8 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             },
             "5": {
@@ -81396,10 +81518,10 @@
                         "inf_dtype": "torch.quint8",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.03462733328342438
+                            0.03768916055560112
                         ],
                         "zero_point": [
-                            111
+                            115
                         ]
                     }
                 ],
@@ -82962,7 +83084,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -82971,38 +83094,39 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             }
         },
         "nonq_op_infos": {
             "0": {
-                "op_type": "<method 'size' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'size' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.5.attention",
                 "input_tensor_infos": [
                     {
                         "id": 167,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8"
+                        "inf_dtype": "torch.float32"
                     }
                 ],
                 "output_tensor_infos": []
             },
             "1": {
-                "op_type": "<method 'size' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'size' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.5.attention",
                 "input_tensor_infos": [
                     {
                         "id": 167,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8"
+                        "inf_dtype": "torch.float32"
                     },
                     {}
                 ],
                 "output_tensor_infos": []
             },
             "2": {
-                "op_type": "<method 'view' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'view' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.5.attention",
                 "input_tensor_infos": [
                     {
@@ -83024,7 +83148,7 @@
                 ]
             },
             "3": {
-                "op_type": "<method 'transpose' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'transpose' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.5.attention",
                 "input_tensor_infos": [
                     {
@@ -83039,12 +83163,12 @@
                     {
                         "id": 170,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.float32"
+                        "inf_dtype": "torch.quint8"
                     }
                 ]
             },
             "4": {
-                "op_type": "<method 'view' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'view' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.5.attention",
                 "input_tensor_infos": [
                     {
@@ -83066,7 +83190,7 @@
                 ]
             },
             "5": {
-                "op_type": "<method 'transpose' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'transpose' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.5.attention",
                 "input_tensor_infos": [
                     {
@@ -83086,7 +83210,7 @@
                 ]
             },
             "6": {
-                "op_type": "<method 'view' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'view' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.5.attention",
                 "input_tensor_infos": [
                     {
@@ -83108,7 +83232,7 @@
                 ]
             },
             "7": {
-                "op_type": "<method 'transpose' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'transpose' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.5.attention",
                 "input_tensor_infos": [
                     {
@@ -83128,14 +83252,15 @@
                 ]
             },
             "8": {
-                "op_type": "<method 'div' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'transpose' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.5.attention",
                 "input_tensor_infos": [
                     {
-                        "id": 170,
+                        "id": 173,
                         "orig_dtype": "torch.float32",
                         "inf_dtype": "torch.float32"
                     },
+                    {},
                     {}
                 ],
                 "output_tensor_infos": [
@@ -83147,27 +83272,26 @@
                 ]
             },
             "9": {
-                "op_type": "<method 'transpose' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'div' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.5.attention",
                 "input_tensor_infos": [
                     {
-                        "id": 173,
+                        "id": 178,
                         "orig_dtype": "torch.float32",
                         "inf_dtype": "torch.float32"
                     },
-                    {},
                     {}
                 ],
                 "output_tensor_infos": [
                     {
-                        "id": 178,
+                        "id": 179,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8"
+                        "inf_dtype": "torch.float32"
                     }
                 ]
             },
             "10": {
-                "op_type": "<method 'eq' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'eq' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.5.attention",
                 "input_tensor_infos": [
                     {
@@ -83186,7 +83310,7 @@
                 ]
             },
             "11": {
-                "op_type": "<method 'view' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'view' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.5.attention",
                 "input_tensor_infos": [
                     {
@@ -83208,7 +83332,7 @@
                 ]
             },
             "12": {
-                "op_type": "<method 'expand_as' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'expand_as' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.5.attention",
                 "input_tensor_infos": [
                     {
@@ -83231,7 +83355,7 @@
                 ]
             },
             "13": {
-                "op_type": "<method 'masked_fill' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'masked_fill' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.5.attention",
                 "input_tensor_infos": [
                     {
@@ -83259,7 +83383,7 @@
                 ]
             },
             "14": {
-                "op_type": "<function softmax at 0x7fb5dc102790>",
+                "op_type": "<function softmax at 0x7fd6d16608b0>",
                 "fqn": "distilbert.transformer.layer.5.attention",
                 "input_tensor_infos": [
                     {
@@ -83295,7 +83419,7 @@
                 ]
             },
             "16": {
-                "op_type": "<method 'transpose' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'transpose' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.5.attention",
                 "input_tensor_infos": [
                     {
@@ -83315,7 +83439,7 @@
                 ]
             },
             "17": {
-                "op_type": "<method 'contiguous' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'contiguous' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.5.attention",
                 "input_tensor_infos": [
                     {
@@ -83333,7 +83457,7 @@
                 ]
             },
             "18": {
-                "op_type": "<method 'view' of 'torch._C._TensorBase' objects>",
+                "op_type": "<method 'view' of 'torch._C.TensorBase' objects>",
                 "fqn": "distilbert.transformer.layer.5.attention",
                 "input_tensor_infos": [
                     {
@@ -83372,13 +83496,13 @@
                     {
                         "id": 193,
                         "orig_dtype": "torch.float32",
-                        "inf_dtype": "torch.quint8",
+                        "inf_dtype": "torch.float32",
                         "force_dtype": "torch.quint8",
                         "scale": [
-                            0.18003985285758972
+                            0.18901820480823517
                         ],
                         "zero_point": [
-                            192
+                            183
                         ]
                     }
                 ],
@@ -89549,7 +89673,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -89558,7 +89683,8 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             },
             "1": {
@@ -91138,7 +91264,8 @@
                     "qscheme": "torch.per_tensor_affine",
                     "reduce_range": false,
                     "quant_min": 0,
-                    "quant_max": 255
+                    "quant_max": 255,
+                    "is_dynamic": false
                 },
                 "weight_observer": {
                     "name": "PerChannelMinMaxObserver",
@@ -91147,7 +91274,8 @@
                     "qscheme": "torch.per_channel_symmetric",
                     "reduce_range": false,
                     "quant_min": -128,
-                    "quant_max": 127
+                    "quant_max": 127,
+                    "is_dynamic": false
                 }
             }
         },
diff --git a/quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/do_calibration.sh b/quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/do_calibration.sh
index 1196ced15..7aecf1bc4 100644
--- a/quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/do_calibration.sh
+++ b/quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/do_calibration.sh
@@ -47,5 +47,5 @@ python -m intel_extension_for_pytorch.cpu.launch --ninstance 1 --node_id 0 --ena
   --output_dir ./tmp \
   --per_device_eval_batch_size $BATCH_SIZE \
   --do_calibration \
-  --calibration_iters 100 \
+  --calibration_iters 400 \
   --int8_config configure.json
diff --git a/quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/run_accuracy.sh b/quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/run_accuracy.sh
index cfa819c89..f63eaefdc 100755
--- a/quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/run_accuracy.sh
+++ b/quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/run_accuracy.sh
@@ -21,49 +21,41 @@ ARGS=""
 export DNNL_PRIMITIVE_CACHE_CAPACITY=1024
 export MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
 
-path="ipex"
-ARGS="$ARGS --use_ipex"
-echo "### running with intel extension for pytorch"
-
 precision="fp32"
-if [[ "$1" == "bf16" ]]
+if [[ "$PRECISION" == "bf16" ]]
 then
     precision="bf16"
     ARGS="$ARGS --bf16"
     echo "### running bf16 mode"
-elif [[ "$1" == "fp16" ]]
+elif [[ "$PRECISION" == "fp16" ]]
 then
     precision=fp16
     ARGS="$ARGS --fp16_cpu"
     echo "### running fp16 mode"
-elif [[ "$1" == "fp32" ]]
+elif [[ "$PRECISION" == "fp32" ]]
 then
     echo "### running fp32 mode"
-elif [[ "$1" == "bf32" ]]
+elif [[ "$PRECISION" == "bf32" ]]
 then
     precision="bf32"
     ARGS="$ARGS --bf32 --auto_kernel_selection"
     echo "### running bf32 mode"
-elif [[ "$1" == "int8-fp32" ]]
+elif [[ "$PRECISION" == "int8-fp32" ]]
 then
     precision="int8-fp32"
     ARGS="$ARGS --int8 --int8_config configure.json"
     echo "### running int8-fp32 mode"
-elif [[ "$1" == "int8-bf16" ]]
+elif [[ "$PRECISION" == "int8-bf16" ]]
 then
     precision="int8-bf16"
     ARGS="$ARGS --bf16 --int8 --int8_config configure.json"
     echo "### running int8-bf16 mode"
 else
-    echo "The specified precision '$1' is unsupported."
+    echo "The specified precision '$PRECISION' is unsupported."
     echo "Supported precisions are: fp32, bf32, bf16, int8-fp32, int8-bf16"
     exit 1
 fi
 
-mode="jit"
-ARGS="$ARGS --jit_mode_eval"
-echo "### running with jit mode"
-
 if [ -z "${OUTPUT_DIR}" ]; then
   echo "The required environment variable OUTPUT_DIR has not been set, please create the output path and set it to OUTPUT_DIR"
   exit 1
@@ -80,14 +72,35 @@ FINETUNED_MODEL=${FINETUNED_MODEL:-"distilbert-base-uncased-finetuned-sst-2-engl
 EVAL_SCRIPT=${EVAL_SCRIPT:-"./transformers/examples/pytorch/text-classification/run_glue.py"}
 WORK_SPACE=${WORK_SPACE:-${OUTPUT_DIR}}
 rm -rf ${OUTPUT_DIR}/accuracy_log*
-python -m intel_extension_for_pytorch.cpu.launch --ninstance 1 --node_id 0  --enable_jemalloc --log_path=${OUTPUT_DIR} --log_file_prefix="accuracy_log_${precision}_${mode}" \
-  ${EVAL_SCRIPT} $ARGS \
-  --model_name_or_path   ${FINETUNED_MODEL} \
-  --task_name sst2 \
-  --do_eval \
-  --max_seq_length ${SEQUENCE_LENGTH} \
-  --output_dir ./tmp \
-  --per_device_eval_batch_size $BATCH_SIZE \
+
+TORCH_INDUCTOR=${TORCH_INDUCTOR:-"0"}
+if [[ "0" == ${TORCH_INDUCTOR} ]];then
+    path="ipex"
+    echo "### running with intel extension for pytorch"
+    mode="jit"
+    ARGS="$ARGS --jit_mode_eval"
+    echo "### running with jit mode"
+    python -m intel_extension_for_pytorch.cpu.launch --ninstance 1 --node_id 0  --enable_jemalloc --log_path=${OUTPUT_DIR} --log_file_prefix="accuracy_log_${precision}_${mode}" \
+        ${EVAL_SCRIPT} $ARGS \
+        --use_ipex \
+        --model_name_or_path   ${FINETUNED_MODEL} \
+        --task_name sst2 \
+        --do_eval \
+        --max_seq_length ${SEQUENCE_LENGTH} \
+        --output_dir ./tmp \
+        --per_device_eval_batch_size $BATCH_SIZE
+else
+    echo "Running inference with torch.compile inductor backend."
+    ARGS="$ARGS --inductor"
+    python -m intel_extension_for_pytorch.cpu.launch --ninstance 1 --node_id 0  --enable_jemalloc --log_path=${OUTPUT_DIR} --log_file_prefix="accuracy_log_${precision}_${mode}" \
+        ${EVAL_SCRIPT} $ARGS \
+        --model_name_or_path   ${FINETUNED_MODEL} \
+        --task_name sst2 \
+        --do_eval \
+        --max_seq_length ${SEQUENCE_LENGTH} \
+        --output_dir ./tmp \
+        --per_device_eval_batch_size $BATCH_SIZE
+fi
 
 accuracy=$(cat ${OUTPUT_DIR}/accuracy_log* | grep "eval_accuracy" |sed -e 's/.*= //;s/[^0-9.]//g')
 f1=$(cat ${OUTPUT_DIR}/accuracy_log* | grep "eval_f1" |sed -e 's/.*= //;s/[^0-9.]//g')
diff --git a/quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/run_multi_instance_realtime.sh b/quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/run_multi_instance_realtime.sh
index 7c579e1e3..91da4ef85 100755
--- a/quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/run_multi_instance_realtime.sh
+++ b/quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/run_multi_instance_realtime.sh
@@ -22,41 +22,40 @@ ARGS=""
 export DNNL_PRIMITIVE_CACHE_CAPACITY=1024
 export MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
 
-path="ipex"
-ARGS="$ARGS --use_ipex --benchmark --perf_begin_iter 500 --perf_run_iters 2000 "
+ARGS="$ARGS --benchmark --perf_begin_iter 500 --perf_run_iters 2000 "
 echo "### running with intel extension for pytorch"
 
 precision="fp32"
-if [[ "$1" == "bf16" ]]
+if [[ "$PRECISION" == "bf16" ]]
 then
     precision="bf16"
     ARGS="$ARGS --bf16"
     echo "### running bf16 mode"
-elif [[ "$1" == "fp32" ]]
+elif [[ "$PRECISION" == "fp32" ]]
 then
     echo "### running fp32 mode"
-elif [[ "$1" == "fp16" ]]
+elif [[ "$PRECISION" == "fp16" ]]
 then
     precision=fp16
     ARGS="$ARGS --fp16_cpu"
     echo "### running fp16 mode"
-elif [[ "$1" == "bf32" ]]
+elif [[ "$PRECISION" == "bf32" ]]
 then
     precision="bf32"
     ARGS="$ARGS --bf32 --auto_kernel_selection"
     echo "### running bf32 mode"
-elif [[ "$1" == "int8-fp32" ]]
+elif [[ "$PRECISION" == "int8-fp32" ]]
 then
     precision="int8-fp32"
     ARGS="$ARGS --int8 --int8_config configure.json"
     echo "### running int8-fp32 mode"
-elif [[ "$1" == "int8-bf16" ]]
+elif [[ "$PRECISION" == "int8-bf16" ]]
 then
     precision="int8-bf16"
     ARGS="$ARGS --bf16 --int8 --int8_config configure.json"
     echo "### running int8-bf16 mode"
 else
-    echo "The specified precision '$1' is unsupported."
+    echo "The specified precision '$PRECISION' is unsupported."
     echo "Supported precisions are: fp32, bf32, bf16, int8-fp32, int8-bf16"
     exit 1
 fi
@@ -74,11 +73,6 @@ if [ -z "${OUTPUT_DIR}" ]; then
   exit 1
 fi
 
-mode="jit"
-ARGS="$ARGS --jit_mode_eval"
-echo "### running with jit mode"
-
-
 export OMP_NUM_THREADS=${CORE_PER_INSTANCE}
 CORES=`lscpu | grep Core | awk '{print $4}'`
 SOCKETS=`lscpu | grep Socket | awk '{print $2}'`
@@ -91,14 +85,34 @@ ARGS="$ARGS --use_share_weight --total_cores ${CORES_PER_NUMA} --cores_per_insta
 EVAL_SCRIPT=${EVAL_SCRIPT:-"./transformers/examples/pytorch/text-classification/run_glue.py"}
 WORK_SPACE=${WORK_SPACE:-${OUTPUT_DIR}}
 rm -rf ${OUTPUT_DIR}/latency_log*
-python -m intel_extension_for_pytorch.cpu.launch --ninstances $NUMAS --enable_jemalloc --log_path=${OUTPUT_DIR} --log_file_prefix="./latency_log_${precision}_${mode}" \
-  ${EVAL_SCRIPT} $ARGS \
-  --model_name_or_path   ${FINETUNED_MODEL} \
-  --task_name sst2 \
-  --do_eval \
-  --max_seq_length ${SEQUENCE_LENGTH} \
-  --output_dir ./tmp \
-  --per_device_eval_batch_size $BATCH_SIZE \
+
+TORCH_INDUCTOR=${TORCH_INDUCTOR:-"0"}
+if [[ "0" == ${TORCH_INDUCTOR} ]];then
+    path="ipex"
+    mode="jit"
+    ARGS="$ARGS --jit_mode_eval"
+    echo "### running with jit mode"
+    python -m intel_extension_for_pytorch.cpu.launch --ninstances $NUMAS --enable_jemalloc --log_path=${OUTPUT_DIR} --log_file_prefix="./latency_log_${precision}_${mode}" \
+        ${EVAL_SCRIPT} $ARGS \
+        --use_ipex \
+        --model_name_or_path   ${FINETUNED_MODEL} \
+        --task_name sst2 \
+        --do_eval \
+        --max_seq_length ${SEQUENCE_LENGTH} \
+        --output_dir ./tmp \
+        --per_device_eval_batch_size $BATCH_SIZE
+else
+    echo "Running inference with torch.compile inductor backend."
+    ARGS="$ARGS --inductor"
+    python -m intel_extension_for_pytorch.cpu.launch --ninstances $NUMAS --enable_jemalloc --log_path=${OUTPUT_DIR} --log_file_prefix="./latency_log_${precision}_${mode}" \
+        ${EVAL_SCRIPT} $ARGS \
+        --model_name_or_path   ${FINETUNED_MODEL} \
+        --task_name sst2 \
+        --do_eval \
+        --max_seq_length ${SEQUENCE_LENGTH} \
+        --output_dir ./tmp \
+        --per_device_eval_batch_size $BATCH_SIZE
+fi
 
 CORES_PER_INSTANCE=${OMP_NUM_THREADS}
 TOTAL_CORES=`expr $CORES \* $SOCKETS`
diff --git a/quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/run_multi_instance_throughput.sh b/quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/run_multi_instance_throughput.sh
index b7b0cd479..f9014f562 100755
--- a/quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/run_multi_instance_throughput.sh
+++ b/quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/run_multi_instance_throughput.sh
@@ -22,49 +22,44 @@ ARGS=""
 export DNNL_PRIMITIVE_CACHE_CAPACITY=1024
 export MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
 
-path="ipex"
-ARGS="$ARGS --use_ipex --benchmark --perf_begin_iter 10 --perf_run_iters 100"
+ARGS="$ARGS --benchmark --perf_begin_iter 10 --perf_run_iters 100"
 echo "### running with intel extension for pytorch"
 
 precision="fp32"
-if [[ "$1" == "bf16" ]]
+if [[ "$PRECISION" == "bf16" ]]
 then
     precision="bf16"
     ARGS="$ARGS --bf16"
     echo "### running bf16 mode"
-elif [[ "$1" == "fp16" ]]
+elif [[ "$PRECISION" == "fp16" ]]
 then
     precision=fp16
     ARGS="$ARGS --fp16_cpu"
     echo "### running fp16 mode"
-elif [[ "$1" == "fp32" ]]
+elif [[ "$PRECISION" == "fp32" ]]
 then
     echo "### running fp32 mode"
-elif [[ "$1" == "bf32" ]]
+elif [[ "$PRECISION" == "bf32" ]]
 then
     precision="bf32"
     ARGS="$ARGS --bf32 --auto_kernel_selection"
     echo "### running bf32 mode"
-elif [[ "$1" == "int8-fp32" ]]
+elif [[ "$PRECISION" == "int8-fp32" ]]
 then
     precision="int8-fp32"
     ARGS="$ARGS --int8 --int8_config configure.json"
     echo "### running int8-fp32 mode"
-elif [[ "$1" == "int8-bf16" ]]
+elif [[ "$PRECISION" == "int8-bf16" ]]
 then
     precision="int8-bf16"
     ARGS="$ARGS --bf16 --int8 --int8_config configure.json"
     echo "### running int8-bf16 mode"
 else
-    echo "The specified precision '$1' is unsupported."
+    echo "The specified precision '$PRECISION' is unsupported."
     echo "Supported precisions are: fp32, bf32, bf16, int8-fp32, int8-bf16"
     exit 1
 fi
 
-mode="jit"
-ARGS="$ARGS --jit_mode_eval"
-echo "### running with jit mode"
-
 if [ -z "${OUTPUT_DIR}" ]; then
   echo "The required environment variable OUTPUT_DIR has not been set, please create the output path and set it to OUTPUT_DIR"
   exit 1
@@ -82,15 +77,36 @@ EVAL_SCRIPT=${EVAL_SCRIPT:-"./transformers/examples/pytorch/text-classification/
 WORK_SPACE=${WORK_SPACE:-${OUTPUT_DIR}}
 
 rm -rf ${OUTPUT_DIR}/throughput_log*
-python -m intel_extension_for_pytorch.cpu.launch --throughput_mode  --enable_jemalloc --log_path=${OUTPUT_DIR} --log_file_prefix="./throughput_log_${path}_${precision}_${mode}" \
-  ${EVAL_SCRIPT} $ARGS \
-  --model_name_or_path   ${FINETUNED_MODEL} \
-  --task_name sst2 \
-  --do_eval \
-  --max_seq_length ${SEQUENCE_LENGTH} \
-  --output_dir ./tmp \
-  --per_device_eval_batch_size $BATCH_SIZE \
-  --dataloader_drop_last \
+
+TORCH_INDUCTOR=${TORCH_INDUCTOR:-"0"}
+if [[ "0" == ${TORCH_INDUCTOR} ]];then
+    path="ipex"
+    mode="jit"
+    ARGS="$ARGS --jit_mode_eval"
+    echo "### running with jit mode"
+    python -m intel_extension_for_pytorch.cpu.launch --throughput_mode  --enable_jemalloc --log_path=${OUTPUT_DIR} --log_file_prefix="./throughput_log_${path}_${precision}_${mode}" \
+        ${EVAL_SCRIPT} $ARGS \
+        --use_ipex \
+        --model_name_or_path   ${FINETUNED_MODEL} \
+        --task_name sst2 \
+        --do_eval \
+        --max_seq_length ${SEQUENCE_LENGTH} \
+        --output_dir ./tmp \
+        --per_device_eval_batch_size $BATCH_SIZE \
+        --dataloader_drop_last
+else
+    echo "Running inference with torch.compile inductor backend."
+    ARGS="$ARGS --inductor"
+    python -m intel_extension_for_pytorch.cpu.launch --throughput_mode  --enable_jemalloc --log_path=${OUTPUT_DIR} --log_file_prefix="./throughput_log_${path}_${precision}_${mode}" \
+        ${EVAL_SCRIPT} $ARGS \
+        --model_name_or_path   ${FINETUNED_MODEL} \
+        --task_name sst2 \
+        --do_eval \
+        --max_seq_length ${SEQUENCE_LENGTH} \
+        --output_dir ./tmp \
+        --per_device_eval_batch_size $BATCH_SIZE \
+        --dataloader_drop_last
+fi
 
 throughput=$(grep 'Throughput:' ${OUTPUT_DIR}/throughput_log* |sed -e 's/.*Throughput//;s/[^0-9.]//g' |awk '
 BEGIN {
diff --git a/quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/setup.sh b/quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/setup.sh
new file mode 100755
index 000000000..7baf701e3
--- /dev/null
+++ b/quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/setup.sh
@@ -0,0 +1,39 @@
+#!/usr/bin/env bash
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+pip install datasets evaluate 
+pip install scikit-learn scipy
+
+# Check the operating system type
+os_type=$(awk -F= '/^NAME/{print $2}' /etc/os-release)
+
+# Install model specific dependencies:
+if [[ "$os_name" == *"CentOS"* ]]; then
+    yum install -y git-lfs
+elif [[ "$os_name" == *"Ubuntu"* ]]; then
+    apt install -y git-lfs
+fi
+
+rm -rf transformers
+git clone https://github.com/huggingface/transformers.git
+cd transformers
+git checkout v4.28.1
+git lfs pull
+pip install -r examples/pytorch/text-classification/requirements.txt
+git apply ../models/language_modeling/pytorch/common/enable_ipex_for_transformers.diff
+pip install -e ./
+cd ..
diff --git a/quickstart/language_modeling/pytorch/rnnt/inference/cpu/.docs/container_build.md b/quickstart/language_modeling/pytorch/rnnt/inference/cpu/.docs/container_build.md
deleted file mode 100644
index d57bf6489..000000000
--- a/quickstart/language_modeling/pytorch/rnnt/inference/cpu/.docs/container_build.md
+++ /dev/null
@@ -1,26 +0,0 @@
-## Build the container
-
-The <model name> <mode> package has scripts and a Dockerfile that are
-used to build a workload container that runs the model. This container
-uses the PyTorch/IPEX container as it's base, so ensure that you have built
-the `pytorch-ipex-spr.tar.gz` container prior to building this model container.
-
-Use `docker images` to verify that you have the base container built. For example:
-```
-$ docker images | grep pytorch-ipex-spr
-model-zoo         pytorch-ipex-spr         f5b473554295        2 hours ago         4.08GB
-```
-
-To build the <model name> <mode> container, extract the package and
-run the `build.sh` script.
-```
-# Extract the package
-tar -xzf <package name>
-cd <package dir>
-
-# Build the container
-./build.sh
-```
-
-After the build completes, you should have a container called
-`<docker image>` that will be used to run the model.
diff --git a/quickstart/language_modeling/pytorch/rnnt/inference/cpu/.docs/datasets.md b/quickstart/language_modeling/pytorch/rnnt/inference/cpu/.docs/datasets.md
deleted file mode 100644
index 2453c888c..000000000
--- a/quickstart/language_modeling/pytorch/rnnt/inference/cpu/.docs/datasets.md
+++ /dev/null
@@ -1,23 +0,0 @@
-## Datasets
-
-The LibriSpeech dataset is used by RNN-T. Use the <model name> <mode> container
-to download and prepare the inference dataset. Specify a directory for the dataset to be
-downloaded to when running the container:
-```
-export DATASET_DIR=<folder where the inference dataset will be downloaded>
-mkdir -p $DATASET_DIR
-
-docker run --rm \
-  --env DATASET_DIR=${DATASET_DIR} \
-  --env http_proxy=${http_proxy} \
-  --env https_proxy=${https_proxy} \
-  --env no_proxy=${no_proxy} \
-  --volume ${DATASET_DIR}:${DATASET_DIR} \
-  -w /workspace/<package dir> \
-  -it \
-  <docker image> \
-  /bin/bash quickstart/download_dataset.sh
-```
-
-This `DATASET_DIR` environment variable will be used again when
-[running the model](#run-the-model).
diff --git a/quickstart/language_modeling/pytorch/rnnt/inference/cpu/.docs/description.md b/quickstart/language_modeling/pytorch/rnnt/inference/cpu/.docs/description.md
deleted file mode 100644
index 49f899094..000000000
--- a/quickstart/language_modeling/pytorch/rnnt/inference/cpu/.docs/description.md
+++ /dev/null
@@ -1,5 +0,0 @@
-<!-- 10. Description -->
-## Description
-
-This document has instructions for running <model name> <mode> using
-Intel-optimized PyTorch.
diff --git a/quickstart/language_modeling/pytorch/rnnt/inference/cpu/.docs/docker_spr.md b/quickstart/language_modeling/pytorch/rnnt/inference/cpu/.docs/docker_spr.md
deleted file mode 100644
index 299e7f7ac..000000000
--- a/quickstart/language_modeling/pytorch/rnnt/inference/cpu/.docs/docker_spr.md
+++ /dev/null
@@ -1,36 +0,0 @@
-## Run the model
-
-Set the `CHECKPOINT_DIR` environment variable and run the script to download the
-pretrained model:
-```
-export CHECKPOINT_DIR=<directory to download the pretrained model>
-mkdir -p $CHECKPOINT_DIR
-cd <package dir>
-bash download_model.sh
-```
-
-After you've downloaded the pretrained model and followed the instructions to
-[build the container](#build-the-container) and [prepare the dataset](#datasets),
-use the `run.sh` script from the container package to run <model name> <mode> in
-docker. Set environment variables to specify the dataset directory, precision to run,
-and an output directory. By default, the `inference_realtime.sh` quickstart script will
-be run. To run a different quickstart script, set the `SCRIPT` environment variable
-to the script of your choice.
-
-The snippet below demonstrates how to run <model name> <mode>:
-```
-# Navigate to the container package directory
-cd <package dir>
-
-# Set the required environment vars
-export DATASET_DIR=<path to the dataset>
-export CHECKPOINT_DIR=<path to the downloaded model weights directory>
-export PRECISION=<specify the precision to run (fp32 or bf16)>
-export OUTPUT_DIR=<directory where log files will be written>
-
-# Run the container with inference_realtime.sh quickstart script
-./run.sh
-
-# Run a different quickstart script
-SCRIPT=accuracy.sh ./run.sh
-```
diff --git a/quickstart/language_modeling/pytorch/rnnt/inference/cpu/.docs/license.md b/quickstart/language_modeling/pytorch/rnnt/inference/cpu/.docs/license.md
deleted file mode 100644
index e547f148d..000000000
--- a/quickstart/language_modeling/pytorch/rnnt/inference/cpu/.docs/license.md
+++ /dev/null
@@ -1,4 +0,0 @@
-<!--- 80. License -->
-## License
-
-Licenses can be found in the model package, in the `licenses` directory.
diff --git a/quickstart/language_modeling/pytorch/rnnt/inference/cpu/.docs/quickstart.md b/quickstart/language_modeling/pytorch/rnnt/inference/cpu/.docs/quickstart.md
deleted file mode 100644
index 0d594d2f6..000000000
--- a/quickstart/language_modeling/pytorch/rnnt/inference/cpu/.docs/quickstart.md
+++ /dev/null
@@ -1,13 +0,0 @@
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| `download_dataset.sh` | Download and prepare the LibriSpeech inference dataset. See the [datasets section](#datasets) for instructions on it's usage. |
-| `inference_realtime.sh` | Runs multi-instance inference using 4 cores per instance for the specified precision (fp32, avx-fp32, or bf16). |
-| `inference_throughput.sh` | Runs multi-instance inference using 1 instance per socket for the specified precision (fp32, avx-fp32, or bf16). |
-| `accuracy.sh` | Runs an inference accuracy test for the specified precision (fp32, avx-fp32, or bf16). |
-
-> Note: The `avx-fp32` precision runs the same scripts as `fp32`, except that the
-> `DNNL_MAX_CPU_ISA` environment variable is unset. The environment variable is
-> otherwise set to `DNNL_MAX_CPU_ISA=AVX512_CORE_AMX`.
diff --git a/quickstart/language_modeling/pytorch/rnnt/inference/cpu/.docs/title.md b/quickstart/language_modeling/pytorch/rnnt/inference/cpu/.docs/title.md
deleted file mode 100644
index a4ccbf863..000000000
--- a/quickstart/language_modeling/pytorch/rnnt/inference/cpu/.docs/title.md
+++ /dev/null
@@ -1,2 +0,0 @@
-<!--- 0. Title -->
-# PyTorch <model name> <mode>
diff --git a/quickstart/language_modeling/pytorch/rnnt/inference/cpu/.docs/wrapper_package.md b/quickstart/language_modeling/pytorch/rnnt/inference/cpu/.docs/wrapper_package.md
deleted file mode 100644
index 0d77199f3..000000000
--- a/quickstart/language_modeling/pytorch/rnnt/inference/cpu/.docs/wrapper_package.md
+++ /dev/null
@@ -1,16 +0,0 @@
-## Model Package
-
-The model package includes the Dockerfile and scripts needed to build and
-run <model name> <mode> in a container.
-```
-<package dir>
-├── README.md
-├── build.sh
-├── licenses
-│   ├── LICENSE
-│   └── third_party
-├── model_packages
-│   └── <package name>
-├── <package dir>.Dockerfile
-└── run.sh
-```
diff --git a/quickstart/language_modeling/pytorch/rnnt/inference/cpu/DEVCATALOG.md b/quickstart/language_modeling/pytorch/rnnt/inference/cpu/DEVCATALOG.md
new file mode 100644
index 000000000..f30aca3e4
--- /dev/null
+++ b/quickstart/language_modeling/pytorch/rnnt/inference/cpu/DEVCATALOG.md
@@ -0,0 +1,92 @@
+# PyTorch RNN-T inference
+
+## Description
+This document has instructions for running RNN-T inference using Intel Extension for PyTorch.
+
+## Pull Command
+
+Docker image based on CentOS Stream8
+```
+docker pull intel/language-modeling:centos-pytorch-cpu-rnnt-inference
+```
+
+Docker image based on Ubuntu 22.04
+```
+docker pull intel/language-modeling:ubuntu-pytorch-cpu-rnnt-inference
+```
+
+## Quick Start Scripts
+| Script name | Description |
+|-------------|-------------|
+| `inference_realtime.sh` | Runs multi-instance inference using 4 cores per instance for the specified precision (fp32, avx-fp32, bf32 or bf16). |
+| `inference_throughput.sh` | Runs multi-instance inference using 1 instance per socket for the specified precision (fp32, avx-fp32, bf32 or bf16). |
+| `accuracy.sh` | Runs an inference accuracy test for the specified precision (fp32, avx-fp32, or bf16). |
+
+**Note:** The `avx-fp32` precision runs the same scripts as `fp32`, except that the `DNNL_MAX_CPU_ISA` environment variable is unset. The environment variable is otherwise set to `DNNL_MAX_CPU_ISA=AVX512_CORE_AMX`.
+
+# Datasets 
+Follow instructions to download and pre-process instructions from [here](https://github.com/IntelAI/models/blob/v2.9.0/quickstart/language_modeling/pytorch/rnnt/inference/cpu/download_dataset.sh) and set `DATASET_DIR` variable to point to the dataset. 
+
+# Pre-Trained Model
+Follow instructions [here](https://github.com/IntelAI/models/blob/v2.9.0/quickstart/language_modeling/pytorch/rnnt/inference/cpu/download_model.sh) to download the RNNT pre-trained model.
+
+## Docker Run
+(Optional) Export related proxy into docker environment.
+```bash
+export DOCKER_RUN_ENVS="-e ftp_proxy=${ftp_proxy} \
+  -e FTP_PROXY=${FTP_PROXY} -e http_proxy=${http_proxy} \
+  -e HTTP_PROXY=${HTTP_PROXY} -e https_proxy=${https_proxy} \
+  -e HTTPS_PROXY=${HTTPS_PROXY} -e no_proxy=${no_proxy} \
+  -e NO_PROXY=${NO_PROXY} -e socks_proxy=${socks_proxy} \
+  -e SOCKS_PROXY=${SOCKS_PROXY}"
+```
+
+To run RNNT inference, set environment variables to specify the dataset directory, precision to run, model directory and an output directory. 
+```
+export OS=<provide either centos or ubuntu>
+export OUTPUT_DIR=<specify path to output directory>
+export PRECISION=<specify the precision>
+export DATASET_DIR=<specify path to processed dataset>
+export CHECKPOINT_DIR=<specify path to pre-trained model>
+
+IMAGE_NAME=intel/language-modeling:${OS}-pytorch-cpu-rnnt-inference
+SCRIPT=quickstart/<specify script name>
+WORKDIR=/workspace/pytorch-rnnt-inference
+DOCKER_ARGS="--privileged --init -it"
+
+docker run --rm \
+  --env OUTPUT_DIR=${OUTPUT_DIR} \
+  --env DATASET_DIR=${DATASET_DIR} \
+  --env CHECKPOINT_DIR=${CHECKPOINT_DIR} \
+  --env http_proxy=${http_proxy} \
+  --env https_proxy=${https_proxy} \
+  --env no_proxy=${no_proxy} \
+  --volume ${DATASET_DIR}:${DATASET_DIR} \
+  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
+  --volume ${CHECKPOINT_DIR}:${CHECKPOINT_DIR} \
+  ${DOCKER_RUN_ENVS} \
+  --shm-size 8G \
+  -w ${WORKDIR} \
+  ${DOCKER_ARGS} \
+  $IMAGE_NAME \
+  /bin/bash $SCRIPT $PRECISION
+```
+## Documentation and Sources
+#### Get Started​
+[Docker* Repository](https://hub.docker.com/r/intel/language-modeling)
+
+[Main GitHub*](https://github.com/IntelAI/models)
+
+[Release Notes](https://github.com/IntelAI/models/releases)
+
+[Get Started Guide](https://github.com/IntelAI/models/blob/master/quickstart/quickstart/language_modeling/pytorch/rnnt/inference/cpu/DEVCATALOG.md)
+
+#### Code Sources
+[Dockerfile](https://github.com/IntelAI/models/tree/master/docker/pyt-cpu)
+
+[Report Issue](https://community.intel.com/t5/Intel-Optimized-AI-Frameworks/bd-p/optimized-ai-frameworks)
+
+## License Agreement
+LEGAL NOTICE: By accessing, downloading or using this software and any required dependent software (the “Software Package”), you agree to the terms and conditions of the software license agreements for the Software Package, which may also include notices, disclaimers, or license terms for third party software included with the Software Package. Please refer to the [license](https://github.com/IntelAI/models/tree/master/third_party) file for additional details.
+
+[View All Containers and Solutions 🡢](https://www.intel.com/content/www/us/en/developer/tools/software-catalog/containers.html?s=Newest)
diff --git a/quickstart/language_modeling/pytorch/rnnt/inference/cpu/README.md b/quickstart/language_modeling/pytorch/rnnt/inference/cpu/README.md
index b94fab9af..4346695a2 100644
--- a/quickstart/language_modeling/pytorch/rnnt/inference/cpu/README.md
+++ b/quickstart/language_modeling/pytorch/rnnt/inference/cpu/README.md
@@ -4,80 +4,74 @@
 
 This document has instructions for running RNN-T inference using Intel-optimized PyTorch.
 
+## Quick Start Scripts
+| Script name | Description |
+|-------------|-------------|
+| `inference_realtime.sh` | Runs multi-instance inference using 4 cores per instance for the specified precision (fp32, avx-fp32, bf32 or bf16). |
+| `inference_throughput.sh` | Runs multi-instance inference using 1 instance per socket for the specified precision (fp32, avx-fp32, bf32 or bf16). |
+| `accuracy.sh` | Runs an inference accuracy test for the specified precision (fp32, avx-fp32, bf32 or bf16). |
+
+**Note:** The `avx-fp32` precision runs the same scripts as `fp32`, except that the `DNNL_MAX_CPU_ISA` environment variable is unset. The environment variable is otherwise set to `DNNL_MAX_CPU_ISA=AVX512_CORE_AMX`. 
+* Set ENV to use AMX:
+  ```
+  export DNNL_MAX_CPU_ISA=AVX512_CORE_AMX
+  ```
+
 ## Bare Metal
 ### General setup
 
-Follow [link](/docs/general/pytorch/BareMetalSetup.md) to install Miniconda and build Pytorch, IPEX, TorchVison and Jemalloc.
+Follow [link](/docs/general/pytorch/BareMetalSetup.md) to install Miniconda and build Pytorch, IPEX, TorchVison and Jemalloc and TCmalloc
 
 ### Model Specific Setup
-* Install dependencies
-  ```bash
-  export MODEL_DIR=<path to your clone of the model zoo>
-  bash ${MODEL_DIR}/quickstart/language_modeling/pytorch/rnnt/inference/cpu/install_dependency_baremetal.sh
-  ```
-
-* Download and preprocess RNN-T dataset:
-  ```bash
-  export DATASET_DIR=#Where_to_save_Dataset
-  bash ${MODEL_DIR}/quickstart/language_modeling/pytorch/rnnt/inference/cpu/download_dataset.sh
-  ```
-
-* Download pretrained model
-  ```bash
-  export CHECKPOINT_DIR=#Where_to_save_pretrained_model
-  bash ${MODEL_DIR}/quickstart/language_modeling/pytorch/rnnt/inference/cpu/download_model.sh
-  ```
-
-* Set Jemalloc Preload for better performance
+* Set Jemalloc and tcmalloc Preload for better performance
 
   The jemalloc should be built from the [General setup](#general-setup) section.
-  ```bash
-  export LD_PRELOAD="<path to the jemalloc directory>/lib/libjemalloc.so":$LD_PRELOAD
+  ```
+  export LD_PRELOAD="<path to the jemalloc directory>/lib/libjemalloc.so":"path_to/tcmalloc/lib/libtcmalloc.so":$LD_PRELOAD
   export MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
   ```
 
 * Set IOMP preload for better performance
 
   IOMP should be installed in your conda env from the [General setup](#general-setup) section.
-  ```bash
-  export LD_PRELOAD=<path to the intel-openmp directory>/lib/libiomp5.so:$LD_PRELOAD
   ```
-
-* Set ENV to use AMX if you are using SPR
-  ```bash
-  export DNNL_MAX_CPU_ISA=AVX512_CORE_AMX
+  pip install packaging intel-openmp
+  export LD_PRELOAD=<path to the intel-openmp directory>/lib/libiomp5.so:$LD_PRELOAD
   ```
 
-# Quick Start Scripts
-
-|  DataType   | Throughput  |  Latency    |   Accuracy  |
-| ----------- | ----------- | ----------- | ----------- |
-| FP32        | bash inference_throughput.sh fp32 | bash inference_realtime.sh fp32 | bash accuracy.sh fp32 |
-| BF16        | bash inference_throughput.sh bf16 | bash inference_realtime.sh bf16 | bash accuracy.sh bf16 |
-| BF32        | bash inference_throughput.sh bf32 | bash inference_realtime.sh bf32 | bash accuracy.sh bf32 |
-
-## Run the model
-
-Follow the instructions above to setup your bare metal environment, download and
-preprocess the dataset, and do the model specific setup. Once all the setup is done,
-the Model Zoo can be used to run a [quickstart script](#quick-start-scripts).
+### Run the model
+Once all the setup is done, the Intel® AI Reference Models can be used to run a [quickstart script](#quick-start-scripts).
 Ensure that you have enviornment variables set to point to the dataset directory,
 an output directory and the checkpoint directory.
 
 ```
-# Clone the model zoo repo and set the MODEL_DIR
+# Clone the Intel® AI Reference Models repo and set the MODEL_DIR
 git clone https://github.com/IntelAI/models.git
 cd models
 export MODEL_DIR=$(pwd)
 
+# Install dependencies
+./quickstart/language_modeling/pytorch/rnnt/inference/cpu/install_dependency_baremetal.sh
+
+# If the dataset is not downloaded on the machine, then download and preprocess RNN-T dataset:
+export DATASET_DIR=<Where_to_save_Dataset>
+./quickstart/language_modeling/pytorch/rnnt/inference/cpu/download_dataset.sh
+
+# Download pretrained model
+export CHECKPOINT_DIR=#Where_to_save_pretrained_model
+./quickstart/language_modeling/pytorch/rnnt/inference/cpu/download_model.sh
+
 # Env vars
 export OUTPUT_DIR=<path to an output directory>
 export CHECKPOINT_DIR=<path to the pretrained model checkpoints>
 export DATASET_DIR=<path to the dataset>
+export PRECISION=< select from :- fp32, avx-fp32, bf16, or bf32>
+
+# Optional environemnt variables:
+export BATCH_SIZE=<set a value for batch size, else it will run with default batch size>
 
-# Run a quickstart script (for example, FP32 batch inference)
-cd ${MODEL_DIR}/quickstart/language_modeling/pytorch/rnnt/inference/cpu
-bash inference_throughput.sh fp32
+# Run a quickstart script:
+./quickstart/language_modeling/pytorch/rnnt/inference/cpu/<script.sh>
 ```
 
 <!--- 80. License -->
diff --git a/quickstart/language_modeling/pytorch/rnnt/inference/cpu/accuracy.sh b/quickstart/language_modeling/pytorch/rnnt/inference/cpu/accuracy.sh
index 17953cc83..13ebc36e4 100755
--- a/quickstart/language_modeling/pytorch/rnnt/inference/cpu/accuracy.sh
+++ b/quickstart/language_modeling/pytorch/rnnt/inference/cpu/accuracy.sh
@@ -38,15 +38,21 @@ if [ ! -d "${OUTPUT_DIR}" ]; then
   exit 1
 fi
 
-if [[ "$1" == *"avx"* ]]; then
+if [ -z "${PRECISION}" ]; then
+  echo "The required environment variable PRECISION has not been set"
+  echo "Please set PRECISION to fp32, avx-fp32, bf32 or bf16."
+  exit 1
+fi
+
+if [[ "$PRECISION" == *"avx"* ]]; then
     unset DNNL_MAX_CPU_ISA
 fi
 
 ARGS=""
-if [ "$1" == "bf16" ]; then
+if [ "$PRECISION" == "bf16" ]; then
     ARGS="$ARGS --mix-precision"
     echo "### running bf16 datatype"
-elif [ "$1" == "bf32" ]; then
+elif [ "$PRECISION" == "bf32" ]; then
     ARGS="$ARGS --bf32"
     echo "### running bf32 datatype"
 else
@@ -57,8 +63,7 @@ export DNNL_PRIMITIVE_CACHE_CAPACITY=1024
 export KMP_BLOCKTIME=1
 export KMP_AFFINITY=granularity=fine,compact,1,0
 
-BATCH_SIZE=64
-PRECISION=$1
+BATCH_SIZE=${BATCH_SIZE:-64}
 
 rm -rf ${OUTPUT_DIR}/rnnt_${PRECISION}_inference_accuracy*
 
@@ -79,5 +84,5 @@ wait
 
 accuracy=$(grep 'Accuracy:' ${OUTPUT_DIR}/rnnt_${PRECISION}_inference_accuracy* |sed -e 's/.*Accuracy//;s/[^0-9.]//g')
 WER=$(grep 'Evaluation WER:' ${OUTPUT_DIR}/rnnt_${PRECISION}_inference_accuracy* |sed -e 's/.*Evaluation WER//;s/[^0-9.]//g')
-echo ""RNN-T";"accuracy";$1; ${BATCH_SIZE};${accuracy}" | tee -a ${OUTPUT_DIR}/summary.log
-echo ""RNN-T";"WER";$1; ${BATCH_SIZE};${WER}" | tee -a ${work_space}/summary.log
+echo ""RNN-T";"accuracy";$PRECISION; ${BATCH_SIZE};${accuracy}" | tee -a ${OUTPUT_DIR}/summary.log
+echo ""RNN-T";"WER";$PRECISION; ${BATCH_SIZE};${WER}" | tee -a ${work_space}/summary.log
diff --git a/quickstart/language_modeling/pytorch/rnnt/inference/cpu/enable_warprnnt_c++17.diff b/quickstart/language_modeling/pytorch/rnnt/inference/cpu/enable_warprnnt_c++17.diff
new file mode 100644
index 000000000..6d0f78459
--- /dev/null
+++ b/quickstart/language_modeling/pytorch/rnnt/inference/cpu/enable_warprnnt_c++17.diff
@@ -0,0 +1,28 @@
+diff --git a/CMakeLists.txt b/CMakeLists.txt
+index 2c570e0..61942de 100755
+--- a/CMakeLists.txt
++++ b/CMakeLists.txt
+@@ -4,6 +4,7 @@ ELSE()
+     cmake_minimum_required(VERSION 2.8)
+ ENDIF()
+ 
++set(CMAKE_CXX_STANDARD 17)
+ project(rnnt_release)
+ 
+ IF (NOT APPLE)
+diff --git a/pytorch_binding/setup.py b/pytorch_binding/setup.py
+index ad960ea..0fc23f3 100644
+--- a/pytorch_binding/setup.py
++++ b/pytorch_binding/setup.py
+@@ -8,10 +8,7 @@
+ 
+ 
+ extra_compile_args = ['-fPIC']
+-if LooseVersion(torch.__version__) >= LooseVersion("1.5.0"):
+-    extra_compile_args += ['-std=c++14']
+-else:
+-    extra_compile_args += ['-std=c++11']
++extra_compile_args += ['-std=c++17']
+ warp_rnnt_path = "../build"
+ 
+ if torch.cuda.is_available() or "CUDA_HOME" in os.environ:
diff --git a/quickstart/language_modeling/pytorch/rnnt/inference/cpu/inference_realtime.sh b/quickstart/language_modeling/pytorch/rnnt/inference/cpu/inference_realtime.sh
index 7de4ea89f..252ab31f8 100755
--- a/quickstart/language_modeling/pytorch/rnnt/inference/cpu/inference_realtime.sh
+++ b/quickstart/language_modeling/pytorch/rnnt/inference/cpu/inference_realtime.sh
@@ -38,15 +38,21 @@ if [ ! -d "${OUTPUT_DIR}" ]; then
   exit 1
 fi
 
-if [[ "$1" == *"avx"* ]]; then
+if [ -z "${PRECISION}" ]; then
+  echo "The required environment variable PRECISION has not been set"
+  echo "Please set PRECISION to fp32, avx-fp32, bf32 or bf16."
+  exit 1
+fi
+
+if [[ "$PRECISION" == *"avx"* ]]; then
     unset DNNL_MAX_CPU_ISA
 fi
 
 ARGS=""
-if [ "$1" == "bf16" ]; then
+if [ "$PRECISION" == "bf16" ]; then
     ARGS="$ARGS --mix-precision"
     echo "### running bf16 datatype"
-elif [ "$1" == "bf32" ]; then
+elif [ "$PRECISION" == "bf32" ]; then
     ARGS="$ARGS --bf32"
     echo "### running bf32 datatype"
 else
@@ -57,8 +63,8 @@ export DNNL_PRIMITIVE_CACHE_CAPACITY=1024
 export KMP_BLOCKTIME=1
 export KMP_AFFINITY=granularity=fine,compact,1,0
 
-BATCH_SIZE=1
-PRECISION=$1
+# Runs with default value if BATCH_SIZE is not mentioned
+BATCH_SIZE=${BATCH_SIZE:-1}
 
 rm -rf ${OUTPUT_DIR}/rnnt_${PRECISION}_inference_realtime*
 
@@ -112,5 +118,5 @@ END   {
     sum = sum / i;
     printf("%.3f ms", sum);
 }')
-echo ""RNN-T";"latency";$1; ${BATCH_SIZE};${throughput}" | tee -a ${OUTPUT_DIR}/summary.log
-echo ""RNN-T";"p99_latency";$1; ${BATCH_SIZE};${p99_latency}" | tee -a ${OUTPUT_DIR}/summary.log
+echo ""RNN-T";"latency";$PRECISION; ${BATCH_SIZE};${throughput}" | tee -a ${OUTPUT_DIR}/summary.log
+echo ""RNN-T";"p99_latency";$PRECISION; ${BATCH_SIZE};${p99_latency}" | tee -a ${OUTPUT_DIR}/summary.log
diff --git a/quickstart/language_modeling/pytorch/rnnt/inference/cpu/inference_throughput.sh b/quickstart/language_modeling/pytorch/rnnt/inference/cpu/inference_throughput.sh
index 1604f00df..8c1a57203 100755
--- a/quickstart/language_modeling/pytorch/rnnt/inference/cpu/inference_throughput.sh
+++ b/quickstart/language_modeling/pytorch/rnnt/inference/cpu/inference_throughput.sh
@@ -38,15 +38,21 @@ if [ ! -d "${OUTPUT_DIR}" ]; then
   exit 1
 fi
 
-if [[ "$1" == *"avx"* ]]; then
+if [ -z "${PRECISION}" ]; then
+  echo "The required environment variable PRECISION has not been set"
+  echo "Please set PRECISION to fp32, avx-fp32, bf32 or bf16."
+  exit 1
+fi
+
+if [[ "$PRECISION" == *"avx"* ]]; then
     unset DNNL_MAX_CPU_ISA
 fi
 
 ARGS=""
-if [ "$1" == "bf16" ]; then
+if [ "$PRECISION" == "bf16" ]; then
     ARGS="$ARGS --mix-precision"
     echo "### running bf16 datatype"
-elif [ "$1" == "bf32" ]; then
+elif [ "$PRECISION" == "bf32" ]; then
     ARGS="$ARGS --bf32"
     echo "### running bf32 datatype"
 else
@@ -57,8 +63,8 @@ export DNNL_PRIMITIVE_CACHE_CAPACITY=1024
 export KMP_BLOCKTIME=1
 export KMP_AFFINITY=granularity=fine,compact,1,0
 
-BATCH_SIZE=448
-PRECISION=$1
+# Runs with default value when BATCH_SIZE is not set
+BATCH_SIZE=${BATCH_SIZE:-448}
 
 rm -rf ${OUTPUT_DIR}/rnnt_${PRECISION}_inference_throughput*
 
@@ -95,4 +101,4 @@ END   {
 sum = sum / i;
         printf("%.3f", sum);
 }')
-echo ""RNN-T";"throughput";$1; ${BATCH_SIZE};${throughput}" | tee -a ${OUTPUT_DIR}/summary.log
+echo ""RNN-T";"throughput";$PRECISION; ${BATCH_SIZE};${throughput}" | tee -a ${OUTPUT_DIR}/summary.log
diff --git a/quickstart/language_modeling/pytorch/rnnt/inference/cpu/install_dependency_baremetal.sh b/quickstart/language_modeling/pytorch/rnnt/inference/cpu/install_dependency_baremetal.sh
old mode 100644
new mode 100755
index 8857889fa..7cb27d2ef
--- a/quickstart/language_modeling/pytorch/rnnt/inference/cpu/install_dependency_baremetal.sh
+++ b/quickstart/language_modeling/pytorch/rnnt/inference/cpu/install_dependency_baremetal.sh
@@ -23,15 +23,18 @@ if [ ! -e "${MODEL_DIR}/models/language_modeling/pytorch/rnnt/inference/cpu/infe
   exit 1
 fi
 
-
 dir=$(pwd)
 cd ${MODEL_DIR}/models/language_modeling/pytorch/rnnt/inference/cpu
 pip install -r requirements.txt
 pip install unidecode inflect
+pip install librosa==0.9.1
 
 # warp-transducer:
 git clone https://github.com/HawkAaron/warp-transducer
 cd warp-transducer
+git checkout master
+git apply ${MODEL_DIR}/quickstart/language_modeling/pytorch/rnnt/inference/cpu/enable_warprnnt_c++17.diff
+rm -rf build
 mkdir build; cd build
 cmake .. 
 make 
diff --git a/quickstart/language_modeling/pytorch/rnnt/training/cpu/.docs/container_build.md b/quickstart/language_modeling/pytorch/rnnt/training/cpu/.docs/container_build.md
deleted file mode 100644
index d57bf6489..000000000
--- a/quickstart/language_modeling/pytorch/rnnt/training/cpu/.docs/container_build.md
+++ /dev/null
@@ -1,26 +0,0 @@
-## Build the container
-
-The <model name> <mode> package has scripts and a Dockerfile that are
-used to build a workload container that runs the model. This container
-uses the PyTorch/IPEX container as it's base, so ensure that you have built
-the `pytorch-ipex-spr.tar.gz` container prior to building this model container.
-
-Use `docker images` to verify that you have the base container built. For example:
-```
-$ docker images | grep pytorch-ipex-spr
-model-zoo         pytorch-ipex-spr         f5b473554295        2 hours ago         4.08GB
-```
-
-To build the <model name> <mode> container, extract the package and
-run the `build.sh` script.
-```
-# Extract the package
-tar -xzf <package name>
-cd <package dir>
-
-# Build the container
-./build.sh
-```
-
-After the build completes, you should have a container called
-`<docker image>` that will be used to run the model.
diff --git a/quickstart/language_modeling/pytorch/rnnt/training/cpu/.docs/datasets.md b/quickstart/language_modeling/pytorch/rnnt/training/cpu/.docs/datasets.md
deleted file mode 100644
index 0739072f6..000000000
--- a/quickstart/language_modeling/pytorch/rnnt/training/cpu/.docs/datasets.md
+++ /dev/null
@@ -1,24 +0,0 @@
-## Datasets
-
-The LibriSpeech dataset is used by RNN-T. Use the <model name> <mode> container
-to download and prepare the training dataset. Specify a directory the dataset will be
-downloaded to when running the container:
-
-```
-export DATASET_DIR=<folder where the training dataset will be downloaded>
-mkdir -p $DATASET_DIR
-
-docker run --rm \
-  --env DATASET_DIR=${DATASET_DIR} \
-  --env http_proxy=${http_proxy} \
-  --env https_proxy=${https_proxy} \
-  --env no_proxy=${no_proxy} \
-  --volume ${DATASET_DIR}:${DATASET_DIR} \
-  -w /workspace/<package dir> \
-  -it \
-  <docker image> \
-  /bin/bash quickstart/download_dataset.sh
-```
-
-This `DATASET_DIR` environment variable will be used again when
-[running the model](#run-the-model).
diff --git a/quickstart/language_modeling/pytorch/rnnt/training/cpu/.docs/description.md b/quickstart/language_modeling/pytorch/rnnt/training/cpu/.docs/description.md
deleted file mode 100644
index 49f899094..000000000
--- a/quickstart/language_modeling/pytorch/rnnt/training/cpu/.docs/description.md
+++ /dev/null
@@ -1,5 +0,0 @@
-<!-- 10. Description -->
-## Description
-
-This document has instructions for running <model name> <mode> using
-Intel-optimized PyTorch.
diff --git a/quickstart/language_modeling/pytorch/rnnt/training/cpu/.docs/docker_spr.md b/quickstart/language_modeling/pytorch/rnnt/training/cpu/.docs/docker_spr.md
deleted file mode 100644
index 1d113ef79..000000000
--- a/quickstart/language_modeling/pytorch/rnnt/training/cpu/.docs/docker_spr.md
+++ /dev/null
@@ -1,19 +0,0 @@
-## Run the model
-
-After you've followed the instructions to [build the container](#build-the-container)
-and [prepare the dataset](#datasets), use the `run.sh` script from the container package
-to run <model name> <mode> using docker. Set environment variables to specify the dataset
-directory, precision to run, and an output directory. The `run.sh` script will execute the
-`training.sh` script in the container that was built.
-```
-# Navigate to the container package directory
-cd <package dir>
-
-# Set the required environment vars
-export DATASET_DIR=<path to the dataset>
-export PRECISION=<specify the precision to run (fp32 or bf16)>
-export OUTPUT_DIR=<directory where log files will be written>
-
-# Run the container using the training.sh script
-./run.sh
-```
diff --git a/quickstart/language_modeling/pytorch/rnnt/training/cpu/.docs/license.md b/quickstart/language_modeling/pytorch/rnnt/training/cpu/.docs/license.md
deleted file mode 100644
index e547f148d..000000000
--- a/quickstart/language_modeling/pytorch/rnnt/training/cpu/.docs/license.md
+++ /dev/null
@@ -1,4 +0,0 @@
-<!--- 80. License -->
-## License
-
-Licenses can be found in the model package, in the `licenses` directory.
diff --git a/quickstart/language_modeling/pytorch/rnnt/training/cpu/.docs/quickstart.md b/quickstart/language_modeling/pytorch/rnnt/training/cpu/.docs/quickstart.md
deleted file mode 100644
index ad8aa6c7f..000000000
--- a/quickstart/language_modeling/pytorch/rnnt/training/cpu/.docs/quickstart.md
+++ /dev/null
@@ -1,11 +0,0 @@
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| `download_dataset.sh` | Download and prepare the LibriSpeech training dataset |
-| `training.sh` | Runs RNN-T training for the specified precision (fp32, avx-fp32, or bf16). |
-
-> Note: The `avx-fp32` precision runs the same scripts as `fp32`, except that the
-> `DNNL_MAX_CPU_ISA` environment variable is unset. The environment variable is
-> otherwise set to `DNNL_MAX_CPU_ISA=AVX512_CORE_AMX`.
diff --git a/quickstart/language_modeling/pytorch/rnnt/training/cpu/.docs/title.md b/quickstart/language_modeling/pytorch/rnnt/training/cpu/.docs/title.md
deleted file mode 100644
index a4ccbf863..000000000
--- a/quickstart/language_modeling/pytorch/rnnt/training/cpu/.docs/title.md
+++ /dev/null
@@ -1,2 +0,0 @@
-<!--- 0. Title -->
-# PyTorch <model name> <mode>
diff --git a/quickstart/language_modeling/pytorch/rnnt/training/cpu/.docs/wrapper_package.md b/quickstart/language_modeling/pytorch/rnnt/training/cpu/.docs/wrapper_package.md
deleted file mode 100644
index 5b543719d..000000000
--- a/quickstart/language_modeling/pytorch/rnnt/training/cpu/.docs/wrapper_package.md
+++ /dev/null
@@ -1,16 +0,0 @@
-## Model Package
-
-The model package includes the Dockerfile and scripts needed to build and
-run <model name> <mode> in a container.
-```
-<package dir>
-├── README.md
-├── build.sh
-├── licenses
-│   ├── LICENSE
-│   └── third_party
-├── model_packages
-│   └── <package name>
-├──<package dir>.Dockerfile
-└── run.sh
-```
diff --git a/quickstart/language_modeling/pytorch/rnnt/training/cpu/DEVCATALOG.md b/quickstart/language_modeling/pytorch/rnnt/training/cpu/DEVCATALOG.md
new file mode 100644
index 000000000..8571d5ae4
--- /dev/null
+++ b/quickstart/language_modeling/pytorch/rnnt/training/cpu/DEVCATALOG.md
@@ -0,0 +1,89 @@
+# PyTorch RNN-T Training
+
+## Description
+This document has instructions for running RNNT training using Intel Extension for PyTorch.
+
+## Pull Command
+
+Docker image based on CentOS Stream8
+```
+docker pull intel/language-modeling:centos-pytorch-cpu-rnnt-training
+```
+
+Docker image based on Ubuntu 22.04
+```
+docker pull intel/language-modeling:ubuntu-pytorch-cpu-rnnt-training
+```
+
+
+## Quick Start Scripts
+| Script name | Description |
+|-------------|-------------|
+| `training.sh` | Runs training for the specified precision (fp32,bf32 or bf16). |
+
+# Datasets 
+Follow instructions to download and pre-process instructions from [here](https://github.com/IntelAI/models/blob/v2.9.0/quickstart/language_modeling/pytorch/rnnt/training/cpu/download_dataset.sh) and set `DATASET_DIR` variable to point to the dataset. 
+
+# Pre-Trained Model
+Follow instructions [here](https://github.com/IntelAI/models/blob/v2.9.0/quickstart/language_modeling/pytorch/rnnt/training/cpu/download_dataset.sh) to download the RNNT pre-trained model.
+
+## Docker Run
+(Optional) Export related proxy into docker environment.
+```bash
+export DOCKER_RUN_ENVS="-e ftp_proxy=${ftp_proxy} \
+  -e FTP_PROXY=${FTP_PROXY} -e http_proxy=${http_proxy} \
+  -e HTTP_PROXY=${HTTP_PROXY} -e https_proxy=${https_proxy} \
+  -e HTTPS_PROXY=${HTTPS_PROXY} -e no_proxy=${no_proxy} \
+  -e NO_PROXY=${NO_PROXY} -e socks_proxy=${socks_proxy} \
+  -e SOCKS_PROXY=${SOCKS_PROXY}"
+```
+
+To run RNNT training, set environment variables to specify the dataset directory, precision to run, model directory and an output directory. 
+```
+export OS=<provide either centos or ubuntu>
+export OUTPUT_DIR=<specify path to output directory>
+export PRECISION=<specify the precision>
+export DATASET_DIR=<specify path to processed dataset>
+export CHECKPOINT_DIR=<specify path to pre-trained model>
+
+IMAGE_NAME=intel/language-modeling:${OS}-pytorch-cpu-rnnt-training
+SCRIPT=quickstart/<specify script name>
+WORKDIR=/workspace/pytorch-rnnt-training
+DOCKER_ARGS="--privileged --init -it"
+
+docker run --rm \
+  --env OUTPUT_DIR=${OUTPUT_DIR} \
+  --env DATASET_DIR=${DATASET_DIR} \
+  --env CHECKPOINT_DIR=${CHECKPOINT_DIR} \
+  --env http_proxy=${http_proxy} \
+  --env https_proxy=${https_proxy} \
+  --env no_proxy=${no_proxy} \
+  --volume ${DATASET_DIR}:${DATASET_DIR} \
+  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
+  --volume ${CHECKPOINT_DIR}:${CHECKPOINT_DIR} \
+  ${DOCKER_RUN_ENVS} \
+  --shm-size 8G \
+  -w ${WORKDIR} \
+  ${DOCKER_ARGS} \
+  $IMAGE_NAME \
+  /bin/bash $SCRIPT $PRECISION
+```
+## Documentation and Sources
+#### Get Started​
+[Docker* Repository](https://hub.docker.com/r/intel/language-modeling)
+
+[Main GitHub*](https://github.com/IntelAI/models)
+
+[Release Notes](https://github.com/IntelAI/models/releases)
+
+[Get Started Guide](https://github.com/IntelAI/models/blob/master/quickstart/quickstart/language_modeling/pytorch/rnnt/training/cpu/DEVCATALOG.md)
+
+#### Code Sources
+[Dockerfile](https://github.com/IntelAI/models/tree/master/docker/pyt-cpu)
+
+[Report Issue](https://community.intel.com/t5/Intel-Optimized-AI-Frameworks/bd-p/optimized-ai-frameworks)
+
+## License Agreement
+LEGAL NOTICE: By accessing, downloading or using this software and any required dependent software (the “Software Package”), you agree to the terms and conditions of the software license agreements for the Software Package, which may also include notices, disclaimers, or license terms for third party software included with the Software Package. Please refer to the [license](https://github.com/IntelAI/models/tree/master/third_party) file for additional details.
+
+[View All Containers and Solutions 🡢](https://www.intel.com/content/www/us/en/developer/tools/software-catalog/containers.html?s=Newest)
diff --git a/quickstart/language_modeling/pytorch/rnnt/training/cpu/README.md b/quickstart/language_modeling/pytorch/rnnt/training/cpu/README.md
index 9996a73e9..aca9ed83f 100644
--- a/quickstart/language_modeling/pytorch/rnnt/training/cpu/README.md
+++ b/quickstart/language_modeling/pytorch/rnnt/training/cpu/README.md
@@ -4,43 +4,46 @@
 
 This document has instructions for running RNN-T training using Intel-optimized PyTorch.
 
+## Quick Start Scripts
+| Script name | Description |
+|-------------|-------------|
+| `training.sh` | Runs training for the specified precision (fp32, avx-fp32, bf32 or bf16). |
+
+**Note:** The `avx-fp32` precision runs the same scripts as `fp32`, except that the `DNNL_MAX_CPU_ISA` environment variable is unset. The environment variable is otherwise set to `DNNL_MAX_CPU_ISA=AVX512_CORE_AMX`. 
+* Set ENV to use AMX:
+  ```
+  export DNNL_MAX_CPU_ISA=AVX512_CORE_AMX
+  ```
+
+|               Distributed Training                    |
+|  DataType   | Throughput  
+| ----------- | ----------- |
+| FP32 (run 100 steps)       | bash training_multinode.sh fp32 |
+| BF16        | bash training_multinode.sh bf16 |
+| BF16 (run 500 steps)        | NUM_STEPS=500 bash training_multinode.sh bf16 |
+| BF32        | bash training_multinode.sh bf32 |
+| BF32 (run 100 steps)        | NUM_STEPS=100 bash training_multinode.sh bf32 |
+
 ## Bare Metal
 ### General setup
 
-Follow [link](/docs/general/pytorch/BareMetalSetup.md) to install Miniconda and build Pytorch, IPEX, TorchVison, Torch-CCL and Jemalloc.
+Follow [link](/docs/general/pytorch/BareMetalSetup.md) to install Miniconda and build Pytorch, IPEX, TorchVison, Torch-CCL and Jemalloc and TCmalloc
 
 ### Model Specific Setup
-* Install dependencies
-  ```bash
-  export MODEL_DIR=<path to your clone of the model zoo>
-  bash ${MODEL_DIR}/quickstart/language_modeling/pytorch/rnnt/training/cpu/install_dependency.sh
-  ```
-
-* Download and preprocess RNN-T dataset:
-  Dataset takes up 60+ GB disk space. After they are decompressed, they will need 60GB more disk space. Next step is preprocessing dataset, it will generate 110+ GB WAV file. Please make sure the disk space is enough.
-  ```bash
-  export DATASET_DIR=#Where_to_save_Dataset
-  bash ${MODEL_DIR}/quickstart/language_modeling/pytorch/rnnt/training/cpu/download_dataset.sh
-  ```
-
-* Set Jemalloc Preload for better performance
+* Set Jemalloc and tcmalloc Preload for better performance
 
   The jemalloc should be built from the [General setup](#general-setup) section.
-  ```bash
-  export LD_PRELOAD="path/lib/libjemalloc.so":$LD_PRELOAD
+  ```
+  export LD_PRELOAD="<path to the jemalloc directory>/lib/libjemalloc.so":"path_to/tcmalloc/lib/libtcmalloc.so":$LD_PRELOAD
   export MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
   ```
 
 * Set IOMP preload for better performance
 
   IOMP should be installed in your conda env from the [General setup](#general-setup) section.
-  ```bash
-  export LD_PRELOAD=path/lib/libiomp5.so:$LD_PRELOAD
   ```
-
-* Set ENV to use AMX if you are using SPR
-  ```bash
-  export DNNL_MAX_CPU_ISA=AVX512_CORE_AMX
+  pip install packaging intel-openmp
+  export LD_PRELOAD=<path to the intel-openmp directory>/lib/libiomp5.so:$LD_PRELOAD
   ```
 * Set ENV to use multi-node distributed training (no need for single-node multi-sockets)
 
@@ -50,54 +53,45 @@ Follow [link](/docs/general/pytorch/BareMetalSetup.md) to install Miniconda and
   export HOSTFILE=your_ip_list_file #one ip per line
   ```
 
-## Quick Start Scripts
-
-|  DataType   | Throughput  |  Latency    |   Accuracy  |
-| ----------- | ----------- | ----------- | ----------- |
-| FP32 (run 100 steps)        | bash training.sh fp32 | --- | --- |
-| BF16        | bash training.sh bf16 | --- | --- |
-| BF16 (run 500 steps)       | NUM_STEPS=500 bash training.sh bf16 | --- | --- |
-| BF32        | bash training.sh bf32 | --- | --- |
-| BF32 (run 100 steps)       | NUM_STEPS=100 bash training.sh bf32 | --- | --- |
-
-|               Distributed Training                    |
-|  DataType   | Throughput  |  Latency    |   Accuracy  |
-| ----------- | ----------- | ----------- | ----------- |
-| FP32 (run 100 steps)       | bash training_multinode.sh fp32 | --- | --- |
-| BF16        | bash training_multinode.sh bf16 | --- | --- |
-| BF16 (run 500 steps)        | NUM_STEPS=500 bash training_multinode.sh bf16 | --- | --- |
-| BF32        | bash training_multinode.sh bf32 | --- | --- |
-| BF32 (run 100 steps)        | NUM_STEPS=100 bash training_multinode.sh bf32 | --- | --- |
-## Run the model
-
-Follow the instructions above to setup your bare metal environment, download and
-preprocess the dataset, and do the model specific setup. Once all the setup is done,
-the Model Zoo can be used to run a [quickstart script](#quick-start-scripts).
+### Run the model
+Once all the setup is done, the Intel® AI Reference Models repo can be used to run a [quickstart script](#quick-start-scripts).
 Ensure that you have enviornment variables set to point to the dataset directory and
 an output directory.
 
 ```
-# Clone the model zoo repo and set the MODEL_DIR
+# Clone the Intel® AI Reference Models repo and set the MODEL_DIR
 git clone https://github.com/IntelAI/models.git
 cd models
 export MODEL_DIR=$(pwd)
 
+# Install dependencies
+./quickstart/language_modeling/pytorch/rnnt/training/cpu/install_dependency.sh
+
+# If dataset is not dowloaded on the machine, then download and preprocess RNN-T dataset:
+#Dataset takes up 60+ GB disk space. After they are decompressed, they will need 60GB more disk space. Next step is preprocessing #dataset, it will generate 110+ GB WAV file. Please make sure the disk space is enough.
+export DATASET_DIR=<Where_to_save_Dataset>
+./quickstart/language_modeling/pytorch/rnnt/training/cpu/download_dataset.sh
+
 # Env vars
 export OUTPUT_DIR=<path to an output directory>
 export DATASET_DIR=<path to the dataset>
+export PRECISION=< select from :- fp32, avx-fp32, bf16, or bf32>
 
-# Run a quickstart script (for example, FP32 training)
-cd ${MODEL_DIR}/quickstart/language_modeling/pytorch/rnnt/training/cpu
-bash training.sh fp32
+# Optional environemnt variables:
+export BATCH_SIZE=<set a value for batch size, else it will run with default batch size>
 
-# Run distributed training script (for example, FP32 distributed training)
+# Run a quickstart script:
+./quickstart/language_modeling/pytorch/rnnt/training/cpu/training.sh
+```
+# Run distributed training script
 cd ${MODEL_DIR}/quickstart/language_modeling/pytorch/rnnt/training/cpu
-bash training_multinode.sh fp32
+bash training_multinode.sh 
 
-# Run a quickstart script and terminate in advance (for example, BF16 training)
+# Run a quickstart script and terminate in advance
 cd ${MODEL_DIR}/quickstart/language_modeling/pytorch/rnnt/training/cpu
-NUM_STEPS=500 bash training.sh bf16
+NUM_STEPS=500 bash training.sh
 ```
+
 ## Attention please:
 1. According to MLPerf RNN-T training README, three RNN-T datasets may need 500GB disk space to preprocess the dataset. After preprocessing done, datasets take up 110+ GB disk.
 
diff --git a/quickstart/language_modeling/pytorch/rnnt/training/cpu/install_dependency.sh b/quickstart/language_modeling/pytorch/rnnt/training/cpu/install_dependency.sh
index 1562694ad..29ca2ac3a 100644
--- a/quickstart/language_modeling/pytorch/rnnt/training/cpu/install_dependency.sh
+++ b/quickstart/language_modeling/pytorch/rnnt/training/cpu/install_dependency.sh
@@ -22,7 +22,6 @@ if [ ! -e "${MODEL_DIR}/models/language_modeling/pytorch/rnnt/training/cpu/train
   exit 1
 fi
 
-
 dir=$(pwd)
 cd ${MODEL_DIR}/models/language_modeling/pytorch/rnnt/training/cpu
 
@@ -30,15 +29,17 @@ pip install -r requirements.txt
 pip install unidecode inflect
 pip install --upgrade pip
 pip install librosa sox
-yum install -y libsndfile
+pip install librosa==0.9.1 protobuf==3.20.3 numpy==1.23.4
 
 # warp-transducer:
 cd ${MODEL_DIR}/quickstart/language_modeling/pytorch/rnnt/training/cpu
 git clone https://github.com/HawkAaron/warp-transducer
 cd warp-transducer
 git checkout master
-git apply ../enable_warprnnt_c++17.diff
-mkdir build; cd build
+git apply ${MODEL_DIR}/quickstart/language_modeling/pytorch/rnnt/training/cpu/enable_warprnnt_c++17.diff
+rm -rf build
+mkdir build 
+cd build
 cmake .. 
 make 
 cd ../pytorch_binding
diff --git a/quickstart/language_modeling/pytorch/rnnt/training/cpu/training.sh b/quickstart/language_modeling/pytorch/rnnt/training/cpu/training.sh
index 6b6179b00..d9fe18ad8 100755
--- a/quickstart/language_modeling/pytorch/rnnt/training/cpu/training.sh
+++ b/quickstart/language_modeling/pytorch/rnnt/training/cpu/training.sh
@@ -33,42 +33,48 @@ if [ ! -d "${OUTPUT_DIR}" ]; then
   exit 1
 fi
 
+if [ -z "${PRECISION}" ]; then
+  echo "The required environment variable PRECISION has not been set"
+  echo "Please set PRECISION to fp32, avx-fp32, bf32 or bf16."
+  exit 1
+fi
+
 MODEL_CONFIG=${5:-"${MODEL_DIR}/models/language_modeling/pytorch/rnnt/training/cpu/configs/rnnt.toml"}
 RESULT_DIR=${6:-"${MODEL_DIR}/models/language_modeling/pytorch/rnnt/training/cpu/results"}
 CHECKPOINT=${7:-"none"}
 CREATE_LOGFILE=${8:-"true"}
 CUDNN_BENCHMARK=${9:-"true"}
 NUM_GPUS=${10:-0}
-PRECISION=${11:-"fp32"}
+PRECISION=${PRECISION:-"fp32"}
 EPOCHS=${12:-1}
 SEED=${13:-2021}
-BATCH_SIZE=${14:-64}
+BATCH_SIZE=${BATCH_SIZE:-64}
 EVAL_BATCH_SIZE=${15:-2}
 LEARNING_RATE=${16:-"0.001"}
 LEARNING_RATE_WARMUP=${17:-"8000"}
 GRADIENT_ACCUMULATION_STEPS=${18:-1}
 LAUNCH_OPT=${LAUNCH_OPT:-"none"}
 
-if [[ $1 == "avx-fp32" ]]; then
+if [[ $PRECISION == "avx-fp32" ]]; then
     unset DNNL_MAX_CPU_ISA
 fi
 
 PREC=""
-if [ "$1" = "bf16" ]; then
+if [ "$PRECISION" = "bf16" ]; then
     PREC="--bf16"
     precision="bf16"
     echo "### running bf16 datatype"
-elif [ "$1" = "fp32" ] ; then
+elif [ "$PRECISION" = "fp32" ] ; then
     PREC="--fp32"
     precision="fp32"
     echo "### running fp32 datatype"
-elif [ "$1" = "bf32" ]; then
+elif [ "$PRECISION" = "bf32" ]; then
     PREC="--bf32"
     precision="bf32"
     echo "### running bf32 datatype"
 else
-    echo "The specified precision '${1}' is unsupported."
-    echo "Supported precisions now are: fp32, bf16 and bf32"
+    echo "The specified precision '$PRECISION' is unsupported."
+    echo "Supported precisions now are: fp32, avx-fp32, bf16 and bf32"
 fi
 
 IPEX="--ipex"
@@ -111,7 +117,7 @@ CMD+=" --warmup=$WARMUP"
 CMD+=" $PROFILE"
 # TODO: FP32 is still under development. For current validation,
 # in FP32, it only runs 100 iterations. NUM_STEPS is disabled in FP32.
-if [ "$1" = "fp32" ] ; then
+if [ "$PRECISION" = "fp32" ] ; then
     CMD+=" --num_steps=100"
 elif [[ ! -z "${NUM_STEPS}" ]]; then
     CMD+=" --num_steps=$NUM_STEPS"
diff --git a/quickstart/language_modeling/tensorflow/bert_large/inference/cpu/README_DEV_CAT.md b/quickstart/language_modeling/tensorflow/bert_large/inference/cpu/README_DEV_CAT.md
new file mode 100644
index 000000000..5d735743e
--- /dev/null
+++ b/quickstart/language_modeling/tensorflow/bert_large/inference/cpu/README_DEV_CAT.md
@@ -0,0 +1,111 @@
+# TensorFlow BERT Large inference
+
+## Description
+This document has instructions for running BERT Large inference using
+Intel-optimized TensorFlow.
+
+## Pull Command
+```
+docker pull intel/language-modeling:centos-tf-cpu-bert-large-inference
+```
+
+<table>
+   <thead>
+      <tr>
+         <th>Script name</th>
+         <th>Description</th>
+      </tr>
+   </thead>
+   <tbody>
+      <tr>
+         <td>inference_realtime.sh</td>
+         <td>Runs multi instance realtime inference for BERT large (SQuAD) using 4 cores per instance with batch size 1 ( for precisions: fp32, int8, bfloat16 and bfloat32) to compute latency. Waits for all instances to complete, then prints a summarized throughput value.</td>
+      </tr>
+      <tr>
+        <td>inference_realtime_weight_sharing.sh</td>
+        <td>Runs multi instance realtime inference with weight sharing for BERT large (SQuAD) using 4 cores per instance with batch size 1 ( for precisions: fp32, int8, bfloat16 and bfloat32) to compute latency for weight sharing. Waits for all instances to complete, then prints a summarized throughput value.</td>
+      </tr>
+      <tr>
+         <td>inference_throughput.sh</td>
+         <td>Runs multi instance batch inference for BERT large (SQuAD) using 1 instance per socket with batch size 128 (for precisions: fp32, int8 or bfloat16) to compute throughput. Waits for all instances to complete, then prints a summarized throughput value.</td>
+      </tr>
+      <tr>
+         <td>accuracy.sh</td>
+         <td>Measures BERT large (SQuAD) inference accuracy for the specified precision (fp32, int8 or bfloat16 and bfloat32).</td>
+      </tr>
+   </tbody>
+</table>
+
+## Datasets
+Download and unzip the BERT Large uncased (whole word masking) model from the
+[google bert repo](https://github.com/google-research/bert#pre-trained-models).
+Then, download the Stanford Question Answering Dataset (SQuAD) dataset file `dev-v1.1.json` into the `wwm_uncased_L-24_H-1024_A-16` directory that was just unzipped.
+
+```
+wget https://storage.googleapis.com/bert_models/2019_05_30/wwm_uncased_L-24_H-1024_A-16.zip
+unzip wwm_uncased_L-24_H-1024_A-16.zip
+
+wget https://rajpurkar.github.io/SQuAD-explorer/dataset/dev-v1.1.json -P wwm_uncased_L-24_H-1024_A-16
+```
+Set the `DATASET_DIR` to point to that directory when running BERT Large inference using the SQuAD data.
+
+## Docker Run
+(Optional) Export related proxy into docker environment.
+```bash
+export DOCKER_RUN_ENVS="-e ftp_proxy=${ftp_proxy} \
+  -e FTP_PROXY=${FTP_PROXY} -e http_proxy=${http_proxy} \
+  -e HTTP_PROXY=${HTTP_PROXY} -e https_proxy=${https_proxy} \
+  -e HTTPS_PROXY=${HTTPS_PROXY} -e no_proxy=${no_proxy} \
+  -e NO_PROXY=${NO_PROXY} -e socks_proxy=${socks_proxy} \
+  -e SOCKS_PROXY=${SOCKS_PROXY}"
+```
+
+To run BERT Large inference, set environment variables to specify the dataset directory, precision and mode to run, and an output directory. 
+```bash
+# Set the required environment vars
+export DATASET_DIR=<path to the dataset>
+export OUTPUT_DIR=<directory where log files will be written>
+export SCRIPT=<specify the script to run>
+export PRECISION=<specify the precision to run>
+export CORES_PER_INSTANCE=<specify the cores per instance. Default value is 4>
+export OMP_NUM_THREADS=<specify the number for OMP_NUM_THREADS. Default value is 4>
+
+# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
+export BATCH_SIZE=<customized batch size value>
+
+docker run --rm \
+  --env BATCH_SIZE=${BATCH_SIZE} \
+  --env DATASET_DIR=${DATASET_DIR} \
+  --env OUTPUT_DIR=${OUTPUT_DIR} \
+  --env PRECISION=${PRECISION} \
+  --env CORES_PER_INSTANCE=${CORES_PER_INSTANCE} \
+  --env OMP_NUM_THREADS=${OMP_NUM_THREADS} \
+  --volume ${DATASET_DIR}:${DATASET_DIR} \
+  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
+  --privileged --init -it \
+  --shm-size 8G \
+  -w /workspace/tf-bert-large-inference \
+  ${DOCKER_RUN_ENVS} \
+  intel/language-modeling:centos-tf-cpu-bert-large-inference \
+  /bin/bash quickstart/${SCRIPT}
+```
+
+## Documentation and Sources
+#### Get Started​
+[Docker* Repository](https://hub.docker.com/r/intel/language-modeling)
+
+[Main GitHub*](https://github.com/IntelAI/models)
+
+[Release Notes](https://github.com/IntelAI/models/releases)
+
+[Get Started Guide](https://github.com/IntelAI/models/blob/master/quickstart/language_modeling/tensorflow/bert_large/inference/cpu/README_DEV_CAT.md)
+
+#### Code Sources
+[Dockerfile](https://github.com/IntelAI/models/tree/master/dockerfiles/tensorflow)
+
+[Report Issue](https://community.intel.com/t5/Intel-Optimized-AI-Frameworks/bd-p/optimized-ai-frameworks)
+
+## License Agreement
+LEGAL NOTICE: By accessing, downloading or using this software and any required dependent software (the “Software Package”), you agree to the terms and conditions of the software license agreements for the Software Package, which may also include notices, disclaimers, or license terms for third party software included with the Software Package. Please refer to the [license](https://github.com/IntelAI/models/tree/master/third_party) file for additional details.
+
+[View All Containers and Solutions 🡢](https://www.intel.com/content/www/us/en/developer/tools/software-catalog/containers.html?s=Newest)
diff --git a/quickstart/language_modeling/tensorflow/bert_large/inference/cpu/accuracy.sh b/quickstart/language_modeling/tensorflow/bert_large/inference/cpu/accuracy.sh
index 6cfbcb2b7..df39daa2f 100755
--- a/quickstart/language_modeling/tensorflow/bert_large/inference/cpu/accuracy.sh
+++ b/quickstart/language_modeling/tensorflow/bert_large/inference/cpu/accuracy.sh
@@ -110,7 +110,6 @@ if [[ $PRECISION == "bfloat32" ]]; then
 fi
 
 source "${MODEL_DIR}/quickstart/common/utils.sh"
-_ht_status_spr
 _command python ${MODEL_DIR}/benchmarks/launch_benchmark.py \
   --model-name=bert_large \
   --precision ${PRECISION} \
diff --git a/quickstart/language_modeling/tensorflow/bert_large/training/cpu/README_DEV_CAT.md b/quickstart/language_modeling/tensorflow/bert_large/training/cpu/README_DEV_CAT.md
new file mode 100644
index 000000000..1674f7d56
--- /dev/null
+++ b/quickstart/language_modeling/tensorflow/bert_large/training/cpu/README_DEV_CAT.md
@@ -0,0 +1,96 @@
+# TensorFlow BERT Large Pretraining
+
+## Description
+This document has instructions for running BERT Large Pretraining using Intel-optimized TensorFlow.
+
+## Pull Command
+```
+docker pull intel/language-modeling:centos-tf-cpu-bert-large-pretraining
+```
+
+<table>
+   <thead>
+      <tr>
+         <th>Script name</th>
+         <th>Description</th>
+      </tr>
+   </thead>
+   <tbody>
+      <tr>
+         <td>pretraining.sh</td>
+         <td>Uses mpirun to execute 2 processes with 1 process per socket for BERT Large pretraining with the specified precision (fp32 or bfloat16). Logs for each instance are saved to the output directory.</td>
+      </tr>
+   </tbody>
+</table>
+
+## Datasets
+Download and unzip the BERT Large uncased (whole word masking) model from the
+[google bert repo](https://github.com/google-research/bert#pre-trained-models).
+Set the `DATASET_DIR` to point to this directory when running BERT Large.
+```
+mkdir -p $DATASET_DIR && cd $DATASET_DIR
+wget https://storage.googleapis.com/bert_models/2019_05_30/wwm_uncased_L-24_H-1024_A-16.zip
+unzip wwm_uncased_L-24_H-1024_A-16.zip
+
+wget https://rajpurkar.github.io/SQuAD-explorer/dataset/dev-v1.1.json -P wwm_uncased_L-24_H-1024_A-16
+```
+
+Follow [instructions to generate BERT pre-training dataset](https://github.com/IntelAI/models/blob/bert-lamb-pretraining-tf-2.2/quickstart/language_modeling/tensorflow/bert_large/training/bfloat16/HowToGenerateBERTPretrainingDataset.txt)
+in TensorFlow record file format. The output TensorFlow record files are expected to be located in the dataset directory `${DATASET_DIR}/tf_records`. An example for the TF record file path should be
+`${DATASET_DIR}/tf_records/part-00430-of-00500`.
+
+## Docker Run
+(Optional) Export related proxy into docker environment.
+```bash
+export DOCKER_RUN_ENVS="-e ftp_proxy=${ftp_proxy} \
+  -e FTP_PROXY=${FTP_PROXY} -e http_proxy=${http_proxy} \
+  -e HTTP_PROXY=${HTTP_PROXY} -e https_proxy=${https_proxy} \
+  -e HTTPS_PROXY=${HTTPS_PROXY} -e no_proxy=${no_proxy} \
+  -e NO_PROXY=${NO_PROXY} -e socks_proxy=${socks_proxy} \
+  -e SOCKS_PROXY=${SOCKS_PROXY}"
+```
+
+To run BERT Large Pretraining, set environment variables to specify the dataset directory, precision to run, and an output directory. 
+```bash
+# Set the required environment vars
+export DATASET_DIR=<path to the dataset>
+export OUTPUT_DIR=<directory where log files will be written>
+export PRECISION=<specify the precision to run (fp32 or bfloat16)>
+
+# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
+export BATCH_SIZE=<customized batch size value>
+
+docker run --rm \
+  --env BATCH_SIZE=${BATCH_SIZE} \
+  --env DATASET_DIR=${DATASET_DIR} \
+  --env OUTPUT_DIR=${OUTPUT_DIR} \
+  --env PRECISION=${PRECISION} \
+  --volume ${DATASET_DIR}:${DATASET_DIR} \
+  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
+  ${DOCKER_RUN_ENVS} \
+  --privileged --init -it \
+  --shm-size 8G \
+  -w /workspace/tf-spr-bert-large-pretraining \
+  intel/language-modeling:centos-tf-cpu-bert-large-pretraining \
+  /bin/bash quickstart/pretraining.sh
+```
+
+## Documentation and Sources
+#### Get Started​
+[Docker* Repository](https://hub.docker.com/r/intel/language-modeling)
+
+[Main GitHub*](https://github.com/IntelAI/models)
+
+[Release Notes](https://github.com/IntelAI/models/releases)
+
+[Get Started Guide](https://github.com/IntelAI/models/blob/master/quickstart/language_modeling/tensorflow/bert_large/training/cpu/README_DEV_CAT.md)
+
+#### Code Sources
+[Dockerfile](https://github.com/IntelAI/models/tree/master/dockerfiles/tensorflow)
+
+[Report Issue](https://community.intel.com/t5/Intel-Optimized-AI-Frameworks/bd-p/optimized-ai-frameworks)
+
+## License Agreement
+LEGAL NOTICE: By accessing, downloading or using this software and any required dependent software (the “Software Package”), you agree to the terms and conditions of the software license agreements for the Software Package, which may also include notices, disclaimers, or license terms for third party software included with the Software Package. Please refer to the [license](https://github.com/IntelAI/models/tree/master/third_party) file for additional details.
+
+[View All Containers and Solutions 🡢](https://www.intel.com/content/www/us/en/developer/tools/software-catalog/containers.html?s=Newest)
diff --git a/quickstart/language_modeling/tensorflow/bert_large/training/cpu/training_squad.sh b/quickstart/language_modeling/tensorflow/bert_large/training/cpu/training_squad.sh
index 0cfdbde48..b9a22547a 100755
--- a/quickstart/language_modeling/tensorflow/bert_large/training/cpu/training_squad.sh
+++ b/quickstart/language_modeling/tensorflow/bert_large/training/cpu/training_squad.sh
@@ -50,8 +50,15 @@ if [ -z "${BATCH_SIZE}" ]; then
 fi
 
 cores_per_socket=$(lscpu |grep 'Core(s) per socket:' |sed 's/[^0-9]//g')
+cores_per_socket="${cores_per_socket//[[:blank:]]/}"
+
+CORES=`lscpu | grep Core | awk '{print $4}'`
+SOCKETS=`lscpu | grep Socket | awk '{print $2}'`
+NUMAS=`lscpu | grep 'NUMA node(s)' | awk '{print $3}'`
+CORES_PER_NUMA=`expr $CORES \* $SOCKETS / $NUMAS`
+
+NUM_INSTANCES=`expr $cores_per_socket / $CORES_PER_NUMA`
 export OMP_NUM_THREADS=${cores_per_socket}
-NUM_INSTANCES="1"
 
 source "${MODEL_DIR}/quickstart/common/utils.sh"
 _ht_status_spr
@@ -62,10 +69,10 @@ _command python ${MODEL_DIR}/benchmarks/launch_benchmark.py \
   --framework tensorflow \
   --output-dir ${OUTPUT_DIR} \
   --mpi_num_processes=${NUM_INSTANCES} \
-  --mpi_num_processes_per_socket=1 \
+  --mpi_num_processes_per_socket=${NUM_INSTANCES} \
   --batch-size ${BATCH_SIZE} \
-  --num-intra-threads ${cores_per_socket} \
-  --num-inter-threads 1 \
+  --num-intra-threads $CORES_PER_NUMA \
+  --num-inter-threads 2 \
   $@ \
   -- DEBIAN_FRONTEND=noninteractive \
   train-option=SQuAD do-predict=False do-train=True profile=False \
diff --git a/quickstart/language_modeling/tensorflow/distilbert_base/inference/cpu/README_DEV_CAT.md b/quickstart/language_modeling/tensorflow/distilbert_base/inference/cpu/README_DEV_CAT.md
new file mode 100644
index 000000000..3714593cf
--- /dev/null
+++ b/quickstart/language_modeling/tensorflow/distilbert_base/inference/cpu/README_DEV_CAT.md
@@ -0,0 +1,104 @@
+# TensorFlow DistilBERT inference
+
+## Description
+This document has instructions for running DistilBERT inference using
+Intel-optimized TensorFlow.
+
+## Pull Command
+```
+docker pull intel/language-modeling:centos-tf-cpu-distilbert-inference
+```
+
+<table>
+   <thead>
+      <tr>
+         <th>Script name</th>
+         <th>Description</th>
+      </tr>
+   </thead>
+   <tbody>
+      <tr>
+         <td>inference_realtime.sh</td>
+         <td>Runs multi instance realtime inference for DistilBERT with batch size 1 ( for precisions: fp32, int8, bfloat16) to compute latency. Waits for all instances to complete, then prints a summarized throughput value.</td>
+      </tr>
+      <tr>
+        <td>inference_realtime_weight_sharing.sh</td>
+        <td>Runs multi instance realtime inference with weight sharing for DistilBERT with batch size 1 ( for precisions: fp32, int8, bfloat16 and bfloat32) to compute latency for weight sharing. Waits for all instances to complete, then prints a summarized throughput value.</td>
+      </tr>
+      <tr>
+         <td>inference_throughput.sh</td>
+         <td>Runs multi instance batch inference for DistilBERT with batch size 56 (for precisions: fp32, int8 or bfloat16) to compute throughput. Waits for all instances to complete, then prints a summarized throughput value.</td>
+      </tr>
+      <tr>
+         <td>accuracy.sh</td>
+         <td>Measures DistilBERT inference accuracy for the specified precision (fp32, int8 or bfloat16 and bfloat32).</td>
+      </tr>
+   </tbody>
+</table>
+
+## Datasets
+We use a part of Stanford Sentiment Treebank corpus for our task. Specifically, the validation split present in the SST2 dataset in the hugging face [repository](https://huggingface.co/datasets/sst2). It contains 872 labeled English sentences. Instructions to get the frozen graph and dataset are given below:
+
+```
+
+Downloading dataset:
+cd to directory:  <model_zoo_dir>/models/language_modeling/tensorflow/distilbert_base/inference/
+python download_dataset.py --path_to_save_dataset <enter path to save dataset>
+
+```
+
+## Docker Run
+(Optional) Export related proxy into docker environment.
+```bash
+export DOCKER_RUN_ENVS="-e ftp_proxy=${ftp_proxy} \
+  -e FTP_PROXY=${FTP_PROXY} -e http_proxy=${http_proxy} \
+  -e HTTP_PROXY=${HTTP_PROXY} -e https_proxy=${https_proxy} \
+  -e HTTPS_PROXY=${HTTPS_PROXY} -e no_proxy=${no_proxy} \
+  -e NO_PROXY=${NO_PROXY} -e socks_proxy=${socks_proxy} \
+  -e SOCKS_PROXY=${SOCKS_PROXY}"
+```
+
+To run DistilBERT inference, set environment variables to specify the dataset directory, precision and mode to run, and an output directory. 
+```bash
+# Set the required environment vars
+export DATASET_DIR=<path to the dataset>
+export OUTPUT_DIR=<directory where log files will be written>
+export SCRIPT=<specify the script to run>
+export PRECISION=<specify the precision to run>
+
+# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
+export BATCH_SIZE=<customized batch size value>
+
+docker run --rm \
+  --env BATCH_SIZE=${BATCH_SIZE} \
+  --env DATASET_DIR=${DATASET_DIR} \
+  --env OUTPUT_DIR=${OUTPUT_DIR} \
+  --env PRECISION=${PRECISION} \
+  --volume ${DATASET_DIR}:${DATASET_DIR} \
+  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
+  --privileged --init -it \
+  --shm-size 8G \
+  -w /workspace/tf-distilbert-inference \
+  intel/language-modeling:centos-tf-cpu-distilbert-inference \
+  /bin/bash quickstart/${SCRIPT}
+```
+
+## Documentation and Sources
+#### Get Started​
+[Docker* Repository](https://hub.docker.com/r/intel/language-modeling)
+
+[Main GitHub*](https://github.com/IntelAI/models)
+
+[Release Notes](https://github.com/IntelAI/models/releases)
+
+[Get Started Guide](https://github.com/IntelAI/models/blob/master/quickstart/language_modeling/tensorflow/distilbert_base/inference/cpu/README_DEV_CAT.md)
+
+#### Code Sources
+[Dockerfile](https://github.com/IntelAI/models/tree/master/dockerfiles/tensorflow)
+
+[Report Issue](https://community.intel.com/t5/Intel-Optimized-AI-Frameworks/bd-p/optimized-ai-frameworks)
+
+## License Agreement
+LEGAL NOTICE: By accessing, downloading or using this software and any required dependent software (the “Software Package”), you agree to the terms and conditions of the software license agreements for the Software Package, which may also include notices, disclaimers, or license terms for third party software included with the Software Package. Please refer to the [license](https://github.com/IntelAI/models/tree/master/third_party) file for additional details.
+
+[View All Containers and Solutions 🡢](https://www.intel.com/content/www/us/en/developer/tools/software-catalog/containers.html?s=Newest)
\ No newline at end of file
diff --git a/quickstart/language_modeling/tensorflow/distilbert_base/inference/cpu/inference_realtime_multi_instance.sh b/quickstart/language_modeling/tensorflow/distilbert_base/inference/cpu/inference_realtime_multi_instance.sh
index 6374284fa..05830ee6f 100755
--- a/quickstart/language_modeling/tensorflow/distilbert_base/inference/cpu/inference_realtime_multi_instance.sh
+++ b/quickstart/language_modeling/tensorflow/distilbert_base/inference/cpu/inference_realtime_multi_instance.sh
@@ -71,6 +71,35 @@ else
     exit 1
 fi
 
+echo "CORES_PER_INSTANCE: $CORES_PER_INSTANCE"
+
+# Setting environment variables
+echo "Advanced settings for improved performance: "
+echo "Setting TF_USE_ADVANCED_CPU_OPS to 1, to enhace the overall performance"
+export TF_USE_ADVANCED_CPU_OPS=1
+echo "TF_USE_ADVANCED_CPU_OPS = ${TF_USE_ADVANCED_CPU_OPS}"
+
+if [[ ${TF_USE_ADVANCED_CPU_OPS} == "1" ]]; then
+	if [[ $PRECISION == "bfloat16" ]]; then
+		echo "TF_USE_ADVANCED_CPU_OPS is on for bfloat16 precision"
+		export TF_AUTO_MIXED_PRECISION_GRAPH_REWRITE_INFERLIST_ADD=Mean
+    export TF_AUTO_MIXED_PRECISION_GRAPH_REWRITE_DENYLIST_REMOVE=Mean
+		echo "TF_AUTO_MIXED_PRECISION_GRAPH_REWRITE_INFERLIST_ADD = ${TF_AUTO_MIXED_PRECISION_GRAPH_REWRITE_INFERLIST_ADD}"
+	elif [[ $PRECISION == "fp16" ]]; then
+		echo "TF_USE_ADVANCED_CPU_OPS is on for fp16 precision"
+		export TF_AUTO_MIXED_PRECISION_GRAPH_REWRITE_INFERLIST_ADD=Mean
+		export TF_AUTO_MIXED_PRECISION_GRAPH_REWRITE_DENYLIST_REMOVE=Mean
+    export ONEDNN_MAX_CPU_ISA=AVX512_CORE_AMX_FP16
+		echo "TF_AUTO_MIXED_PRECISION_GRAPH_REWRITE_INFERLIST_ADD = ${TF_AUTO_MIXED_PRECISION_GRAPH_REWRITE_INFERLIST_ADD}"
+		echo "TF_AUTO_MIXED_PRECISION_GRAPH_REWRITE_DENYLIST_REMOVE = ${TF_AUTO_MIXED_PRECISION_GRAPH_REWRITE_DENYLIST_REMOVE}"
+    echo "ONEDNN_MAX_CPU_ISA=AVX512_CORE_AMX_FP16 = ${ONEDNN_MAX_CPU_ISA}"
+  fi
+fi
+
+echo "Configuring thread pinning and spinning settings"
+export TF_THREAD_PINNING_MODE=none,$(($CORES_PER_INSTANCE-1)),400
+echo "TF_THREAD_PINNING_MODE: $TF_THREAD_PINNING_MODE"
+
 source "${MODEL_DIR}/quickstart/common/utils.sh"
 _ht_status_spr
 _command python benchmarks/launch_benchmark.py \
diff --git a/quickstart/language_modeling/tensorflow/distilbert_base/inference/cpu/inference_throughput_multi_instance.sh b/quickstart/language_modeling/tensorflow/distilbert_base/inference/cpu/inference_throughput_multi_instance.sh
index e2375e5bb..60472620f 100755
--- a/quickstart/language_modeling/tensorflow/distilbert_base/inference/cpu/inference_throughput_multi_instance.sh
+++ b/quickstart/language_modeling/tensorflow/distilbert_base/inference/cpu/inference_throughput_multi_instance.sh
@@ -76,6 +76,36 @@ fi
 BATCH_SIZE="${BATCH_SIZE:-"56"}"
 
 source "${MODEL_DIR}/quickstart/common/utils.sh"
+_get_numa_cores_lists
+echo "Cores per node: ${cores_per_node}"
+
+# Setting environment variables
+echo "Advanced settings for improved performance: "
+echo "Setting TF_USE_ADVANCED_CPU_OPS to 1, to enhace the overall performance"
+export TF_USE_ADVANCED_CPU_OPS=1
+echo "TF_USE_ADVANCED_CPU_OPS = ${TF_USE_ADVANCED_CPU_OPS}"
+
+if [[ ${TF_USE_ADVANCED_CPU_OPS} == "1" ]]; then
+	if [[ $PRECISION == "bfloat16" ]]; then
+		echo "TF_USE_ADVANCED_CPU_OPS is on for bfloat16 precision"
+		export TF_AUTO_MIXED_PRECISION_GRAPH_REWRITE_INFERLIST_ADD=Mean
+    export TF_AUTO_MIXED_PRECISION_GRAPH_REWRITE_DENYLIST_REMOVE=Mean
+		echo "TF_AUTO_MIXED_PRECISION_GRAPH_REWRITE_INFERLIST_ADD = ${TF_AUTO_MIXED_PRECISION_GRAPH_REWRITE_INFERLIST_ADD}"
+	elif [[ $PRECISION == "fp16" ]]; then
+		echo "TF_USE_ADVANCED_CPU_OPS is on for fp16 precision"
+		export TF_AUTO_MIXED_PRECISION_GRAPH_REWRITE_INFERLIST_ADD=Mean
+		export TF_AUTO_MIXED_PRECISION_GRAPH_REWRITE_DENYLIST_REMOVE=Mean
+    export ONEDNN_MAX_CPU_ISA=AVX512_CORE_AMX_FP16
+		echo "TF_AUTO_MIXED_PRECISION_GRAPH_REWRITE_INFERLIST_ADD = ${TF_AUTO_MIXED_PRECISION_GRAPH_REWRITE_INFERLIST_ADD}"
+		echo "TF_AUTO_MIXED_PRECISION_GRAPH_REWRITE_DENYLIST_REMOVE = ${TF_AUTO_MIXED_PRECISION_GRAPH_REWRITE_DENYLIST_REMOVE}"
+    echo "ONEDNN_MAX_CPU_ISA=AVX512_CORE_AMX_FP16 = ${ONEDNN_MAX_CPU_ISA}"
+  fi
+fi
+
+echo "Configuring thread pinning and spinning settings"
+export TF_THREAD_PINNING_MODE=none,$((${cores_per_node} - 1)),400
+echo "TF_THREAD_PINNING_MODE: $TF_THREAD_PINNING_MODE"
+
 _ht_status_spr
 _command python benchmarks/launch_benchmark.py \
          --model-name=distilbert_base \
diff --git a/quickstart/language_translation/tensorflow/mlperf_gnmt/inference/cpu/.docs/baremetal.md b/quickstart/language_translation/tensorflow/mlperf_gnmt/inference/cpu/.docs/baremetal.md
deleted file mode 100644
index 571ef4ce4..000000000
--- a/quickstart/language_translation/tensorflow/mlperf_gnmt/inference/cpu/.docs/baremetal.md
+++ /dev/null
@@ -1,89 +0,0 @@
-<!--- 50. AI Kit -->
-## Run the model
-
-Setup your environment using the instructions below, depending on if you are
-using [AI Kit](/docs/general/tensorflow/AIKit.md):
-
-<table>
-  <tr>
-    <th>Setup using AI Kit</th>
-    <th>Setup without AI Kit</th>
-  </tr>
-  <tr>
-    <td>
-      <p>To run using AI Kit you will need:</p>
-      <ul>
-        <li>git
-        <li>numactl
-        <li>pip
-        <li>wget
-        <li><a href="https://bazel.build/">Bazel</a> to build tensorflow addons
-        <li>Activate the `tensorflow` conda environment
-        <pre>conda activate tensorflow</pre>
-      </ul>
-    </td>
-    <td>
-      <p>To run without AI Kit you will need:</p>
-      <ul>
-        <li>Python 3
-        <li><a href="https://pypi.org/project/intel-tensorflow/">intel-tensorflow>=2.5.0</a>
-        <li>git
-        <li>numactl
-        <li>pip
-        <li>wget
-        <li><a href="https://bazel.build/">Bazel</a> to build tensorflow addons
-        <li>A clone of the Model Zoo repo<br />
-        <pre>git clone https://github.com/IntelAI/models.git</pre>
-      </ul>
-    </td>
-  </tr>
-</table>
-
-After installing the prerequisites, download the pretrained model and set
-the `PRETRAINED_MODEL` environment variable to the .pb file path:
-```
-wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/mlperf_gnmt_fp32_pretrained_model.pb
-export PRETRAINED_MODEL=$(pwd)/mlperf_gnmt_fp32_pretrained_model.pb
-```
-
-<model name> requires TensorFlow addons to be built with a patch from the model
-zoo. The snippet below shows how to clone the addons repo, apply the patch, and
-then build and install the TensorFlow addons wheel.
-```
-# TensorFlow addons (r0.5) build and installation instructions:
-#   Clone TensorFlow addons (r0.5) and apply a patch: A patch file
-#   is attached in Intel Model Zoo MLpref GNMT model scripts,
-#   it fixes TensorFlow addons (r0.5) to work with TensorFlow
-#   version 2.11, and prevents TensorFlow 2.0.0 to be installed
-#   by default as a required dependency.
-git clone --single-branch --branch=r0.5 https://github.com/tensorflow/addons.git
-cd addons
-git apply ../models/language_translation/tensorflow/mlperf_gnmt/gnmt-fix.patch
-
-#   Build TensorFlow addons source code and create TensorFlow addons
-#   pip wheel. Use bazel 6.0.0 version :
-#   Answer yes to questions while running configure.sh
-bash configure.sh
-bazel build --enable_runfiles build_pip_pkg
-bazel-bin/build_pip_pkg artifacts
-pip install artifacts/tensorflow_addons-*.whl --no-deps
-```
-
-Once that has been completed, ensure you have the required environment variables
-set, and then run a quickstart script.
-
-```
-# cd to your model zoo directory
-cd models
-
-# Set env var paths
-export DATASET_DIR=<path to the dataset>
-export PRECISION=fp32
-export OUTPUT_DIR=<path to the directory where log files will be written>
-export PRETRAINED_MODEL=<path to the pretrained model frozen graph>
-# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
-export BATCH_SIZE=<customized batch size value>
-
-# Run a quickstart script
-./quickstart/language_translation/tensorflow/mlperf_gnmt/inference/cpu/<script name>.sh
-```
diff --git a/quickstart/language_translation/tensorflow/mlperf_gnmt/inference/cpu/.docs/datasets.md b/quickstart/language_translation/tensorflow/mlperf_gnmt/inference/cpu/.docs/datasets.md
deleted file mode 100644
index 3f7bfea8f..000000000
--- a/quickstart/language_translation/tensorflow/mlperf_gnmt/inference/cpu/.docs/datasets.md
+++ /dev/null
@@ -1,12 +0,0 @@
-<!--- 30. Datasets -->
-## Datasets
-
-Download and unzip the <model name> model benchmarking data.
-
-```
-wget https://zenodo.org/record/2531868/files/gnmt_inference_data.zip
-unzip gnmt_inference_data.zip
-export DATASET_DIR=$(pwd)/nmt/data
-```
-
-Set the `DATASET_DIR` to point as instructed above  when running <model name>.
diff --git a/quickstart/language_translation/tensorflow/mlperf_gnmt/inference/cpu/.docs/description.md b/quickstart/language_translation/tensorflow/mlperf_gnmt/inference/cpu/.docs/description.md
deleted file mode 100644
index 106b45dbe..000000000
--- a/quickstart/language_translation/tensorflow/mlperf_gnmt/inference/cpu/.docs/description.md
+++ /dev/null
@@ -1,5 +0,0 @@
-<!-- 10. Description -->
-## Description
-
-This document has instructions for running <model name> <mode> using
-Intel-optimized TensorFlow.
diff --git a/quickstart/language_translation/tensorflow/mlperf_gnmt/inference/cpu/.docs/docker.md b/quickstart/language_translation/tensorflow/mlperf_gnmt/inference/cpu/.docs/docker.md
deleted file mode 100644
index 82ac84f20..000000000
--- a/quickstart/language_translation/tensorflow/mlperf_gnmt/inference/cpu/.docs/docker.md
+++ /dev/null
@@ -1,32 +0,0 @@
-<!--- 60. Docker -->
-## Docker
-
-The model container includes the scripts and libraries needed to run 
-<model name> <mode>. To run one of the quickstart scripts 
-using this container, you'll need to provide volume mounts for the dataset 
-and an output directory.
-
-```
-DATASET_DIR=<path to the dataset>
-PRECISION=fp32
-OUTPUT_DIR=<directory where log files will be written>
-# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
-export BATCH_SIZE=<customized batch size value>
-
-docker run \
-  --env DATASET_DIR=${DATASET_DIR} \
-  --env PRECISION=${PRECISION} \
-  --env OUTPUT_DIR=${OUTPUT_DIR} \
-  --env BATCH_SIZE=${batch_size} \
-  --env http_proxy=${http_proxy} \
-  --env https_proxy=${https_proxy} \
-  --volume ${DATASET_DIR}:${DATASET_DIR} \
-  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
-  --privileged --init -t \
-  <docker image> \
-  /bin/bash quickstart/<script name>.sh
-```
-
-If you are new to docker and are running into issues with the container,
-see [this document](https://github.com/IntelAI/models/tree/master/docs/general/docker.md)
-for troubleshooting tips.
diff --git a/quickstart/language_translation/tensorflow/mlperf_gnmt/inference/cpu/.docs/license.md b/quickstart/language_translation/tensorflow/mlperf_gnmt/inference/cpu/.docs/license.md
deleted file mode 100644
index a60f169f4..000000000
--- a/quickstart/language_translation/tensorflow/mlperf_gnmt/inference/cpu/.docs/license.md
+++ /dev/null
@@ -1,4 +0,0 @@
-<!--- 80. License -->
-## License
-
-[LICENSE](/LICENSE)
diff --git a/quickstart/language_translation/tensorflow/mlperf_gnmt/inference/cpu/.docs/quickstart.md b/quickstart/language_translation/tensorflow/mlperf_gnmt/inference/cpu/.docs/quickstart.md
deleted file mode 100644
index 88db20dd5..000000000
--- a/quickstart/language_translation/tensorflow/mlperf_gnmt/inference/cpu/.docs/quickstart.md
+++ /dev/null
@@ -1,8 +0,0 @@
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| [`online_inference.sh`](/quickstart/language_translation/tensorflow/mlperf_gnmt/inference/cpu/online_inference.sh) | Runs online inference (batch_size=1). |
-| [`batch_inference.sh`](/quickstart/language_translation/tensorflow/mlperf_gnmt/inference/cpu/batch_inference.sh) | Runs batch inference (batch_size=32). |
-| [`accuracy.sh`](/quickstart/language_translation/tensorflow/mlperf_gnmt/inference/cpu/accuracy.sh) | Runs accuracy |
diff --git a/quickstart/language_translation/tensorflow/mlperf_gnmt/inference/cpu/.docs/title.md b/quickstart/language_translation/tensorflow/mlperf_gnmt/inference/cpu/.docs/title.md
deleted file mode 100644
index 1d6c60187..000000000
--- a/quickstart/language_translation/tensorflow/mlperf_gnmt/inference/cpu/.docs/title.md
+++ /dev/null
@@ -1,2 +0,0 @@
-<!--- 0. Title -->
-# <model name> <mode>
diff --git a/quickstart/language_translation/tensorflow/mlperf_gnmt/inference/cpu/README.md b/quickstart/language_translation/tensorflow/mlperf_gnmt/inference/cpu/README.md
deleted file mode 100644
index b632018d5..000000000
--- a/quickstart/language_translation/tensorflow/mlperf_gnmt/inference/cpu/README.md
+++ /dev/null
@@ -1,69 +0,0 @@
-<!--- 0. Title -->
-# MLPerf GNMT inference
-
-<!-- 10. Description -->
-## Description
-
-This document has instructions for running MLPerf GNMT inference using
-Intel-optimized TensorFlow.
-
-<!--- 30. Datasets -->
-## Datasets
-
-Download and unzip the MLPerf GNMT model benchmarking data.
-
-```
-wget https://zenodo.org/record/2531868/files/gnmt_inference_data.zip
-unzip gnmt_inference_data.zip
-export DATASET_DIR=$(pwd)/nmt/data
-```
-
-Set the `DATASET_DIR` to point as instructed above  when running MLPerf GNMT.
-
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| [`online_inference.sh`](/quickstart/language_translation/tensorflow/mlperf_gnmt/inference/cpu/online_inference.sh) | Runs online inference (batch_size=1). |
-| [`batch_inference.sh`](/quickstart/language_translation/tensorflow/mlperf_gnmt/inference/cpu/batch_inference.sh) | Runs batch inference (batch_size=32). |
-| [`accuracy.sh`](/quickstart/language_translation/tensorflow/mlperf_gnmt/inference/cpu/accuracy.sh) | Runs accuracy |
-
-<!--- 60. Docker -->
-## Docker
-
-The model container includes the scripts and libraries needed to run 
-MLPerf GNMT inference. To run one of the quickstart scripts 
-using this container, you'll need to provide volume mounts for the dataset 
-and an output directory.
-
-```
-DATASET_DIR=<path to the dataset>
-PRECISION=fp32
-OUTPUT_DIR=<directory where log files will be written>
-# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
-export BATCH_SIZE=<customized batch size value>
-
-docker run \
-  --env DATASET_DIR=${DATASET_DIR} \
-  --env PRECISION=${PRECISION} \
-  --env OUTPUT_DIR=${OUTPUT_DIR} \
-  --env BATCH_SIZE=${batch_size} \
-  --env http_proxy=${http_proxy} \
-  --env https_proxy=${https_proxy} \
-  --volume ${DATASET_DIR}:${DATASET_DIR} \
-  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
-  --privileged --init -t \
-  intel/language-translation:tf-latest-mlperf-gnmt-inference \
-  /bin/bash quickstart/<script name>.sh
-```
-
-If you are new to docker and are running into issues with the container,
-see [this document](https://github.com/IntelAI/models/tree/master/docs/general/docker.md)
-for troubleshooting tips.
-
-<!--- 80. License -->
-## License
-
-[LICENSE](/LICENSE)
-
diff --git a/quickstart/language_translation/tensorflow/mlperf_gnmt/inference/cpu/accuracy.sh b/quickstart/language_translation/tensorflow/mlperf_gnmt/inference/cpu/accuracy.sh
deleted file mode 100755
index 87696b520..000000000
--- a/quickstart/language_translation/tensorflow/mlperf_gnmt/inference/cpu/accuracy.sh
+++ /dev/null
@@ -1,67 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2020 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-MODEL_DIR=${MODEL_DIR-$PWD}
-
-if [ -z "${OUTPUT_DIR}" ]; then
-  echo "The required environment variable OUTPUT_DIR has not been set"
-  exit 1
-fi
-
-# Create the output directory in case it doesn't already exist
-mkdir -p ${OUTPUT_DIR}
-
-if [ -z "${DATASET_DIR}" ]; then
-  echo "The required environment variable DATASET_DIR has not been set"
-  exit 1
-fi
-
-if [ ! -d "${DATASET_DIR}" ]; then
-  echo "The DATASET_DIR '${DATASET_DIR}' does not exist"
-  exit 1
-fi
-
-if [ -z "${PRETRAINED_MODEL}" ]; then
-  PRETRAINED_MODEL="${MODEL_DIR}/mlperf_gnmt_fp32_pretrained_model.pb"
-
-  if [[ ! -f "${PRETRAINED_MODEL}" ]]; then
-    echo "The pretrained model could not be found. Please set the PRETRAINED_MODEL env var to point to the frozen graph file."
-    exit 1
-  fi
-elif [[ ! -f "${PRETRAINED_MODEL}" ]]; then
-  echo "The file specified by the PRETRAINED_MODEL environment variable (${PRETRAINED_MODEL}) does not exist."
-  exit 1
-fi
-
-# If batch size env is not mentioned, then the workload will run with the default batch size.
-if [ -z "${BATCH_SIZE}"]; then
-  BATCH_SIZE="32"
-  echo "Running with default batch size of ${BATCH_SIZE}"
-fi
-
-source "${MODEL_DIR}/quickstart/common/utils.sh"
-_command python ${MODEL_DIR}/benchmarks/launch_benchmark.py \
-  --model-name mlperf_gnmt \
-  --framework tensorflow \
-  --precision fp32 \
-  --mode inference \
-  --batch-size ${BATCH_SIZE} \
-  --socket-id 0 \
-  --data-location ${DATASET_DIR} \
-  --in-graph ${PRETRAINED_MODEL} \
-  --output-dir ${OUTPUT_DIR} \
-  --accuracy-only
diff --git a/quickstart/language_translation/tensorflow/mlperf_gnmt/inference/cpu/batch_inference.sh b/quickstart/language_translation/tensorflow/mlperf_gnmt/inference/cpu/batch_inference.sh
deleted file mode 100755
index 9834abed1..000000000
--- a/quickstart/language_translation/tensorflow/mlperf_gnmt/inference/cpu/batch_inference.sh
+++ /dev/null
@@ -1,80 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2020 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-MODEL_DIR=${MODEL_DIR-$PWD}
-
-if [ -z "${OUTPUT_DIR}" ]; then
-  echo "The required environment variable OUTPUT_DIR has not been set"
-  exit 1
-fi
-
-# Create the output directory in case it doesn't already exist
-mkdir -p ${OUTPUT_DIR}
-
-if [ -z "${DATASET_DIR}" ]; then
-  echo "The required environment variable DATASET_DIR has not been set"
-  exit 1
-fi
-
-if [ ! -d "${DATASET_DIR}" ]; then
-  echo "The DATASET_DIR '${DATASET_DIR}' does not exist"
-  exit 1
-fi
-
-# If precision env is not mentioned, then the workload will run with the default precision.
-if [ -z "${PRECISION}"]; then
-  PRECISION=fp32
-  echo "Running with default precision ${PRECISION}"
-fi
-
-if [[ $PRECISION != "fp32" ]]; then
-  echo "The specified precision '${PRECISION}' is unsupported."
-  echo "Supported precision is fp32."
-  exit 1
-fi
-
-if [ -z "${PRETRAINED_MODEL}" ]; then
-  PRETRAINED_MODEL="${MODEL_DIR}/mlperf_gnmt_fp32_pretrained_model.pb"
-
-  if [[ ! -f "${PRETRAINED_MODEL}" ]]; then
-    echo "The pretrained model could not be found. Please set the PRETRAINED_MODEL env var to point to the frozen graph file."
-    exit 1
-  fi
-elif [[ ! -f "${PRETRAINED_MODEL}" ]]; then
-  echo "The file specified by the PRETRAINED_MODEL environment variable (${PRETRAINED_MODEL}) does not exist."
-  exit 1
-fi
-
-# If batch size env is not mentioned, then the workload will run with the default batch size.
-if [ -z "${BATCH_SIZE}"]; then
-  BATCH_SIZE="32"
-  echo "Running with default batch size of ${BATCH_SIZE}"
-fi
-
-source "${MODEL_DIR}/quickstart/common/utils.sh"
-_command python ${MODEL_DIR}/benchmarks/launch_benchmark.py \
-  --model-name mlperf_gnmt \
-  --framework tensorflow \
-  --precision ${PRECISION} \
-  --mode inference \
-  --batch-size ${BATCH_SIZE} \
-  --socket-id 0 \
-  --data-location ${DATASET_DIR} \
-  --in-graph ${PRETRAINED_MODEL} \
-  --output-dir ${OUTPUT_DIR} \
-  --benchmark-only
-
diff --git a/quickstart/language_translation/tensorflow/mlperf_gnmt/inference/cpu/online_inference.sh b/quickstart/language_translation/tensorflow/mlperf_gnmt/inference/cpu/online_inference.sh
deleted file mode 100755
index 9f4bf09eb..000000000
--- a/quickstart/language_translation/tensorflow/mlperf_gnmt/inference/cpu/online_inference.sh
+++ /dev/null
@@ -1,80 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2020 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-MODEL_DIR=${MODEL_DIR-$PWD}
-
-if [ -z "${OUTPUT_DIR}" ]; then
-  echo "The required environment variable OUTPUT_DIR has not been set"
-  exit 1
-fi
-
-# Create the output directory in case it doesn't already exist
-mkdir -p ${OUTPUT_DIR}
-
-if [ -z "${DATASET_DIR}" ]; then
-  echo "The required environment variable DATASET_DIR has not been set"
-  exit 1
-fi
-
-if [ ! -d "${DATASET_DIR}" ]; then
-  echo "The DATASET_DIR '${DATASET_DIR}' does not exist"
-  exit 1
-fi
-
-# If precision env is not mentioned, then the workload will run with the default precision.
-if [ -z "${PRECISION}"]; then
-  PRECISION=fp32
-  echo "Running with default precision ${PRECISION}"
-fi
-
-if [[ $PRECISION != "fp32" ]]; then
-  echo "The specified precision '${PRECISION}' is unsupported."
-  echo "Supported precision is fp32."
-  exit 1
-fi
-
-if [ -z "${PRETRAINED_MODEL}" ]; then
-  PRETRAINED_MODEL="${MODEL_DIR}/mlperf_gnmt_fp32_pretrained_model.pb"
-
-  if [[ ! -f "${PRETRAINED_MODEL}" ]]; then
-    echo "The pretrained model could not be found. Please set the PRETRAINED_MODEL env var to point to the frozen graph file."
-    exit 1
-  fi
-elif [[ ! -f "${PRETRAINED_MODEL}" ]]; then
-  echo "The file specified by the PRETRAINED_MODEL environment variable (${PRETRAINED_MODEL}) does not exist."
-  exit 1
-fi
-
-# If batch size env is not mentioned, then the workload will run with the default batch size.
-if [ -z "${BATCH_SIZE}"]; then
-  BATCH_SIZE="1"
-  echo "Running with default batch size of ${BATCH_SIZE}"
-fi
-
-source "${MODEL_DIR}/quickstart/common/utils.sh"
-_command python ${MODEL_DIR}/benchmarks/launch_benchmark.py \
-  --model-name mlperf_gnmt \
-  --framework tensorflow \
-  --precision ${PRECISION} \
-  --mode inference \
-  --batch-size ${BATCH_SIZE} \
-  --socket-id 0 \
-  --data-location ${DATASET_DIR} \
-  --in-graph ${PRETRAINED_MODEL} \
-  --output-dir ${OUTPUT_DIR} \
-  --benchmark-only
-
diff --git a/quickstart/language_translation/tensorflow/transformer_lt_official/inference/cpu/.docs/advancedoptions.md b/quickstart/language_translation/tensorflow/transformer_lt_official/inference/cpu/.docs/advancedoptions.md
deleted file mode 100644
index aa7aa64e6..000000000
--- a/quickstart/language_translation/tensorflow/transformer_lt_official/inference/cpu/.docs/advancedoptions.md
+++ /dev/null
@@ -1,4 +0,0 @@
-<!-- 61. Advanced Options -->
-
-See the [Advanced Options for Model Packages and Containers](/quickstart/common/tensorflow/ModelPackagesAdvancedOptions.md)
-document for more advanced use cases.
diff --git a/quickstart/language_translation/tensorflow/transformer_lt_official/inference/cpu/.docs/baremetal.md b/quickstart/language_translation/tensorflow/transformer_lt_official/inference/cpu/.docs/baremetal.md
deleted file mode 100644
index fd8c77b14..000000000
--- a/quickstart/language_translation/tensorflow/transformer_lt_official/inference/cpu/.docs/baremetal.md
+++ /dev/null
@@ -1,104 +0,0 @@
-<!--- 50. AI Kit -->
-## Run the model
-
-Setup your environment using the instructions below, depending on if you are
-using [AI Kit](/docs/general/tensorflow/AIKit.md):
-
-<table>
-  <tr>
-    <th>Setup using AI Kit on Linux</th>
-    <th>Setup without AI Kit on Linux</th>
-    <th>Setup without AI Kit on Windows</th>
-  </tr>
-  <tr>
-    <td>
-      <p>To run using AI Kit on Linux you will need:</p>
-      <ul>
-        <li>Cython
-        <li>numactl
-        <li>pandas
-        <li>wget
-        <li>Activate the `tensorflow` conda environment
-        <pre>conda activate tensorflow</pre>
-      </ul>
-    </td>
-    <td>
-      <p>To run without AI Kit on Linux you will need:</p>
-      <ul>
-        <li>Python 3
-        <li>intel-tensorflow>=2.5.0
-        <li>Cython
-        <li>git
-        <li>numactl
-        <li>pandas
-        <li>wget
-        <li>A clone of the Model Zoo repo<br />
-        <pre>git clone https://github.com/IntelAI/models.git</pre>
-      </ul>
-    </td>
-    <td>
-      <p>To run without AI Kit on Windows you will need:</p>
-      <ul>
-        <li><a href="/docs/general/Windows.md">Intel Model Zoo on Windows Systems prerequisites</a>
-        <li>Cython
-        <li>pandas
-        <li>A clone of the Model Zoo repo<br />
-        <pre>git clone https://github.com/IntelAI/models.git</pre>
-      </ul>
-    </td>
-  </tr>
-</table>
-
-Once you've completed the setup, download and extract the pretrained model,
-and set the file's path to the `PRETRAINED_MODEL` environment variable.
-If you run on Windows, please use a browser to download the pretrained model using the link below. For Linux, run:
-```
-wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/transformer_lt_official_fp32_pretrained_model.tar.gz
-tar -xzf transformer_lt_official_fp32_pretrained_model.tar.gz
-export PRETRAINED_MODEL=$(pwd)/transformer_lt_official_fp32_pretrained_model/graph/fp32_graphdef.pb
-```
-
-Navigate to the model zoo directory, ensure that you have all the required
-environment variables set, then run a [quickstart script](#quick-start-scripts) on either [Linux](#run-on-linux) or [Windows](#run-on-windows) systems.
-
-### Run on Linux
-```
-# cd to your model zoo directory
-cd models
-
-# Set env vars
-export DATASET_DIR=<path to the test dataset directory>
-export PRECISION=fp32
-export PRETRAINED_MODEL=<path to the frozen graph .pb file>
-export OUTPUT_DIR=<path to the directory where the log and translation file will be written>
-# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
-export BATCH_SIZE=<customized batch size value>
-
-# Run a quickstart script
-./quickstart/language_translation/tensorflow/transformer_lt_official/inference/cpu/<script name>.sh
-```
-
-### Run on Windows
-Using `cmd.exe`, run:
-```
-# cd to your model zoo directory
-cd models
-
-# Set env vars
-set DATASET_DIR=<path to the test dataset directory>
-set PRECISION=fp32
-set PRETRAINED_MODEL=<path to the frozen graph .pb file>
-set OUTPUT_DIR=<directory where the log and translation file will be written>
-# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
-set BATCH_SIZE=<customized batch size value>
-
-# Run a quickstart script
-bash quickstart\language_translation\tensorflow\transformer_lt_official\inference\cpu\<script name>.sh
-```
-> Note: You may use `cygpath` to convert the Windows paths to Unix paths before setting the environment variables. 
-As an example, if the pretrained model path on Windows is `D:\user\transformer_lt_official_fp32_pretrained_model\graph\fp32_graphdef.pb`, convert the Windows path to Unix as shown:
-> ```
-> cygpath D:\user\transformer_lt_official_fp32_pretrained_model\graph\fp32_graphdef.pb
-> /d/user/transformer_lt_official_fp32_pretrained_model/graph/fp32_graphdef.pb
->```
->Then, set the `PRETRAINED_MODEL` environment variable `set PRETRAINED_MODEL=/d/user/transformer_lt_official_fp32_pretrained_model/graph/fp32_graphdef.pb`.
diff --git a/quickstart/language_translation/tensorflow/transformer_lt_official/inference/cpu/.docs/datasets.md b/quickstart/language_translation/tensorflow/transformer_lt_official/inference/cpu/.docs/datasets.md
deleted file mode 100644
index 7e0d97719..000000000
--- a/quickstart/language_translation/tensorflow/transformer_lt_official/inference/cpu/.docs/datasets.md
+++ /dev/null
@@ -1,16 +0,0 @@
-<!--- 30. Datasets -->
-## Dataset
-
-Download and preprocess the English-German dataset using the
-[data_download.py](https://github.com/tensorflow/models/blob/v2.2.0/official/nlp/transformer/data_download.py)
-script from the [TensorFlow Model Garden repo](https://github.com/tensorflow/models).
-The Transformer model README has a section with
-[instructions for using the script](https://github.com/tensorflow/models/tree/v2.2.0/official/nlp/transformer#download-and-preprocess-datasets).
-
-Once the script completes, you should have a dataset directory with
-the following files: `newstest2014.de`, `newstest2014.en`, and
-a vocab text file. For simplicity, rename the vocab file to `vocab.txt`.
-The path to the directory with these files should be set as the
-`DATASET_DIR` environment variable when using the
-[quickstart scripts](#quick-start-scripts).
-
diff --git a/quickstart/language_translation/tensorflow/transformer_lt_official/inference/cpu/.docs/description.md b/quickstart/language_translation/tensorflow/transformer_lt_official/inference/cpu/.docs/description.md
deleted file mode 100644
index 5247f9c8e..000000000
--- a/quickstart/language_translation/tensorflow/transformer_lt_official/inference/cpu/.docs/description.md
+++ /dev/null
@@ -1,8 +0,0 @@
-<!-- 10. Description -->
-
-This document has instructions for running
-[Transformer Language Translation](https://arxiv.org/pdf/1706.03762.pdf)
- inference using Intel-optimized TensorFlow. The original code for
-the <model name> model came from the
-[TensorFlow Model Garden repository](https://github.com/tensorflow/models/tree/v2.2.0/official/nlp/transformer).
-
diff --git a/quickstart/language_translation/tensorflow/transformer_lt_official/inference/cpu/.docs/docker.md b/quickstart/language_translation/tensorflow/transformer_lt_official/inference/cpu/.docs/docker.md
deleted file mode 100644
index fee42ae07..000000000
--- a/quickstart/language_translation/tensorflow/transformer_lt_official/inference/cpu/.docs/docker.md
+++ /dev/null
@@ -1,36 +0,0 @@
-<!-- 60. Docker -->
-## Docker
-
-The model container used in the example below includes the scripts,
-libraries, and pretrained model needed to run <model name> <mode>. To run one of the model quickstart scripts using this
-container, you'll need to provide volume mounts for the dataset and an
-output directory.
-
-```
-DATASET_DIR=<path to the test dataset directory>
-PRECISION=fp32
-OUTPUT_DIR=<directory where the log and translation file will be written>
-# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
-export BATCH_SIZE=<customized batch size value>
-
-docker run \
-  --env DATASET_DIR=${DATASET_DIR} \
-  --env PRECISION=${PRECISION} \
-  --env OUTPUT_DIR=${OUTPUT_DIR} \
-  --env BATCH_SIZE=#{BATCH_SIZE} \
-  --env http_proxy=${http_proxy} \
-  --env https_proxy=${https_proxy} \
-  --volume ${DATASET_DIR}:${DATASET_DIR} \
-  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
-  --privileged --init -t \
-  <docker image> \
-  /bin/bash quickstart/<script name>.sh
-```
-
-If you have your own pretrained model, you can specify the path to the frozen 
-graph .pb file using the `FROZEN_GRAPH` environment variable and mount the
-frozen graph's directory as a volume in the container.
-
-If you are new to docker and are running into issues with the container,
-see [this document](https://github.com/IntelAI/models/tree/master/docs/general/docker.md)
-for troubleshooting tips.
diff --git a/quickstart/language_translation/tensorflow/transformer_lt_official/inference/cpu/.docs/license.md b/quickstart/language_translation/tensorflow/transformer_lt_official/inference/cpu/.docs/license.md
deleted file mode 100644
index a60f169f4..000000000
--- a/quickstart/language_translation/tensorflow/transformer_lt_official/inference/cpu/.docs/license.md
+++ /dev/null
@@ -1,4 +0,0 @@
-<!--- 80. License -->
-## License
-
-[LICENSE](/LICENSE)
diff --git a/quickstart/language_translation/tensorflow/transformer_lt_official/inference/cpu/.docs/quickstart.md b/quickstart/language_translation/tensorflow/transformer_lt_official/inference/cpu/.docs/quickstart.md
deleted file mode 100644
index 3b3c229b5..000000000
--- a/quickstart/language_translation/tensorflow/transformer_lt_official/inference/cpu/.docs/quickstart.md
+++ /dev/null
@@ -1,7 +0,0 @@
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| [`online_inference.sh`](/quickstart/language_translation/tensorflow/transformer_lt_official/inference/cpu/online_inference.sh) | Runs online inference (batch_size=1). |
-| [`batch_inference.sh`](/quickstart/language_translation/tensorflow/transformer_lt_official/inference/cpu/batch_inference.sh) | Runs batch inference (batch_size=64). |
diff --git a/quickstart/language_translation/tensorflow/transformer_lt_official/inference/cpu/.docs/title.md b/quickstart/language_translation/tensorflow/transformer_lt_official/inference/cpu/.docs/title.md
deleted file mode 100644
index 1d6c60187..000000000
--- a/quickstart/language_translation/tensorflow/transformer_lt_official/inference/cpu/.docs/title.md
+++ /dev/null
@@ -1,2 +0,0 @@
-<!--- 0. Title -->
-# <model name> <mode>
diff --git a/quickstart/language_translation/tensorflow/transformer_mlperf/inference/cpu/README_DEV_CAT.md b/quickstart/language_translation/tensorflow/transformer_mlperf/inference/cpu/README_DEV_CAT.md
new file mode 100644
index 000000000..ca7e7700a
--- /dev/null
+++ b/quickstart/language_translation/tensorflow/transformer_mlperf/inference/cpu/README_DEV_CAT.md
@@ -0,0 +1,94 @@
+# TensorFlow Transformer Language inference
+
+## Description
+This document has instructions for running Transformer Language inference using
+Intel-optimized TensorFlow.
+
+## Pull Command
+```
+docker pull intel/language-translation:centos-tf-cpu-transformer-mlperf-inference
+```
+
+<table>
+   <thead>
+      <tr>
+         <th>Script name</th>
+         <th>Description</th>
+      </tr>
+   </thead>
+   <tbody>
+      <tr>
+         <td>inference_realtime.sh</td>
+         <td>Runs multi instance realtime inference (batch-size=1) to compute latency using 4 cores per instance for the specified precision (int8, fp32, bfloat32 or bfloat16).</td>
+      </tr>
+      <tr>
+         <td>inference_throughput.sh</td>
+         <td>Runs multi instance batch inference with batch-size=448 for precisions (int8, fp32, bfloat16 and bfloat32) to compute throughput using 1 instance per socket.</td>
+      </tr>
+      <tr>
+         <td>accuracy.sh</td>
+         <td>Measures the inference accuracy for the specified precision (int8, fp32, bfloat32 or bfloat16).</td>
+      </tr>
+   </tbody>
+</table>
+
+## Datasets
+Follow [instructions](https://github.com/IntelAI/models/tree/master/datasets/transformer_data/README.md) to download and preprocess the WMT English-German dataset.
+Set `DATASET_DIR` to point out to the location of the dataset directory.
+
+## Docker Run
+(Optional) Export related proxy into docker environment.
+```bash
+export DOCKER_RUN_ENVS="-e ftp_proxy=${ftp_proxy} \
+  -e FTP_PROXY=${FTP_PROXY} -e http_proxy=${http_proxy} \
+  -e HTTP_PROXY=${HTTP_PROXY} -e https_proxy=${https_proxy} \
+  -e HTTPS_PROXY=${HTTPS_PROXY} -e no_proxy=${no_proxy} \
+  -e NO_PROXY=${NO_PROXY} -e socks_proxy=${socks_proxy} \
+  -e SOCKS_PROXY=${SOCKS_PROXY}"
+```
+
+To run Transformer Language inference, set environment variables to specify the dataset directory, precision and mode to run, and an output directory. 
+```bash
+# Set the required environment vars
+export DATASET_DIR=<path to the dataset>
+export OUTPUT_DIR=<directory where log files will be written>
+export SCRIPT=<specify the script to run>
+export PRECISION=<specify the precision to run>
+
+# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
+export BATCH_SIZE=<customized batch size value>
+
+docker run --rm \
+  --env BATCH_SIZE=${BATCH_SIZE} \
+  --env DATASET_DIR=${DATASET_DIR} \
+  --env OUTPUT_DIR=${OUTPUT_DIR} \
+  --env PRECISION=${PRECISION} \
+  --volume ${DATASET_DIR}:${DATASET_DIR} \
+  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
+  ${DOCKER_RUN_ENVS} \
+  --privileged --init -it \
+  --shm-size 8G \
+  -w /workspace/tf-transformer-mlperf-inference \
+  intel/language-translation:centos-tf-cpu-transformer-mlperf-inference \
+  /bin/bash quickstart/${SCRIPT}
+```
+
+## Documentation and Sources
+#### Get Started​
+[Docker* Repository](https://hub.docker.com/r/intel/language-translation)
+
+[Main GitHub*](https://github.com/IntelAI/models)
+
+[Release Notes](https://github.com/IntelAI/models/releases)
+
+[Get Started Guide](https://github.com/IntelAI/models/blob/master/quickstart/language_translation/tensorflow/transformer_mlperf/inference/cpu/README_DEV_CAT.md)
+
+#### Code Sources
+[Dockerfile](https://github.com/IntelAI/models/tree/master/dockerfiles/tensorflow)
+
+[Report Issue](https://community.intel.com/t5/Intel-Optimized-AI-Frameworks/bd-p/optimized-ai-frameworks)
+
+## License Agreement
+LEGAL NOTICE: By accessing, downloading or using this software and any required dependent software (the “Software Package”), you agree to the terms and conditions of the software license agreements for the Software Package, which may also include notices, disclaimers, or license terms for third party software included with the Software Package. Please refer to the [license](https://github.com/IntelAI/models/tree/master/third_party) file for additional details.
+
+[View All Containers and Solutions 🡢](https://www.intel.com/content/www/us/en/developer/tools/software-catalog/containers.html?s=Newest)
diff --git a/quickstart/language_translation/tensorflow/transformer_mlperf/inference/cpu/accuracy.sh b/quickstart/language_translation/tensorflow/transformer_mlperf/inference/cpu/accuracy.sh
index 188e1ade5..a607c963a 100755
--- a/quickstart/language_translation/tensorflow/transformer_mlperf/inference/cpu/accuracy.sh
+++ b/quickstart/language_translation/tensorflow/transformer_mlperf/inference/cpu/accuracy.sh
@@ -83,7 +83,6 @@ fi
 export TF_PATTERN_ALLOW_CTRL_DEPENDENCIES=1 
 
 source "${MODEL_DIR}/quickstart/common/utils.sh"
-_ht_status_spr
 _command python ${MODEL_DIR}/benchmarks/launch_benchmark.py \
   --model-name transformer_mlperf \
   --precision ${PRECISION} \
diff --git a/quickstart/language_translation/tensorflow/transformer_mlperf/training/cpu/README_DEV_CAT.md b/quickstart/language_translation/tensorflow/transformer_mlperf/training/cpu/README_DEV_CAT.md
new file mode 100644
index 000000000..ae1cc9b0e
--- /dev/null
+++ b/quickstart/language_translation/tensorflow/transformer_mlperf/training/cpu/README_DEV_CAT.md
@@ -0,0 +1,85 @@
+# TensorFlow Transformer Language training
+
+## Description
+This document has instructions for running Transformer Language training
+using Intel-optimized TensorFlow.
+
+## Pull Command
+```
+docker pull intel/language-translation:centos-tf-cpu-transformer-mlperf-training
+```
+
+<table>
+   <thead>
+      <tr>
+         <th>Script name</th>
+         <th>Description</th>
+      </tr>
+   </thead>
+   <tbody>
+      <tr>
+         <td>training.sh</td>
+         <td>Uses mpirun to execute 2 processes with 1 process per socket with a batch size of 5120 for the specified precision (fp32 or bfloat16). Logs for each instance are saved to the output directory.</td>
+      </tr>
+   </tbody>
+</table>
+
+## Datasets
+Follow [instructions](https://github.com/IntelAI/models/tree/master/datasets/transformer_data/README.md) to download and preprocess the WMT English-German dataset.
+Set `DATASET_DIR` to point out to the location of the dataset directory.
+
+## Docker Run
+(Optional) Export related proxy into docker environment.
+```bash
+export DOCKER_RUN_ENVS="-e ftp_proxy=${ftp_proxy} \
+  -e FTP_PROXY=${FTP_PROXY} -e http_proxy=${http_proxy} \
+  -e HTTP_PROXY=${HTTP_PROXY} -e https_proxy=${https_proxy} \
+  -e HTTPS_PROXY=${HTTPS_PROXY} -e no_proxy=${no_proxy} \
+  -e NO_PROXY=${NO_PROXY} -e socks_proxy=${socks_proxy} \
+  -e SOCKS_PROXY=${SOCKS_PROXY}"
+```
+
+To run Transformer Language training, set environment variables to specify the dataset directory, precision to run, and an output directory. 
+```bash
+# Set the required environment vars
+export DATASET_DIR=<path to the dataset>
+export OUTPUT_DIR=<directory where log files will be written>
+export PRECISION=<specify the precision to run (fp32 or bfloat16)>
+
+# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
+export BATCH_SIZE=<customized batch size value>
+
+docker run --rm \
+  --env BATCH_SIZE=${BATCH_SIZE} \
+  --env DATASET_DIR=${DATASET_DIR} \
+  --env OUTPUT_DIR=${OUTPUT_DIR} \
+  --env PRECISION=${PRECISION} \
+  --volume ${DATASET_DIR}:${DATASET_DIR} \
+  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
+  ${DOCKER_RUN_ENVS} \
+  --privileged --init -it \
+  --shm-size 8G \
+  -w /workspace/tf-transformer-mlperf-training \
+  intel/language-translation:centos-tf-cpu-transformer-mlperf-training \
+  /bin/bash quickstart/training.sh
+```
+
+## Documentation and Sources
+#### Get Started​
+[Docker* Repository](https://hub.docker.com/r/intel/language-translation)
+
+[Main GitHub*](https://github.com/IntelAI/models)
+
+[Release Notes](https://github.com/IntelAI/models/releases)
+
+[Get Started Guide](https://github.com/IntelAI/models/blob/master/quickstart/language_translation/tensorflow/transformer_mlperf/training/cpu/README_DEV_CAT.md)
+
+#### Code Sources
+[Dockerfile](https://github.com/IntelAI/models/tree/master/dockerfiles/tensorflow)
+
+[Report Issue](https://community.intel.com/t5/Intel-Optimized-AI-Frameworks/bd-p/optimized-ai-frameworks)
+
+## License Agreement
+LEGAL NOTICE: By accessing, downloading or using this software and any required dependent software (the “Software Package”), you agree to the terms and conditions of the software license agreements for the Software Package, which may also include notices, disclaimers, or license terms for third party software included with the Software Package. Please refer to the [license](https://github.com/IntelAI/models/tree/master/third_party) file for additional details.
+
+[View All Containers and Solutions 🡢](https://www.intel.com/content/www/us/en/developer/tools/software-catalog/containers.html?s=Newest)
diff --git a/quickstart/object_detection/pytorch/maskrcnn/inference/cpu/DEVCATALOG.md b/quickstart/object_detection/pytorch/maskrcnn/inference/cpu/DEVCATALOG.md
new file mode 100644
index 000000000..0473e5be6
--- /dev/null
+++ b/quickstart/object_detection/pytorch/maskrcnn/inference/cpu/DEVCATALOG.md
@@ -0,0 +1,118 @@
+# PyTorch Mask RCNN inference
+
+## Description 
+This document has instructions for running MaskRCNN inference using Intel Extension for PyTorch. 
+
+## Pull Command
+
+Docker image based on CentOS Stream8
+```
+docker pull intel/object-detection:centos-pytorch-cpu-maskrcnn-inference
+```
+Docker image based on Ubuntu 22.04
+```
+docker pull intel/object-detection:ubuntu-pytorch-cpu-maskrcnn-inference
+```
+
+## Quick Start Scripts
+| Script name | Description |
+|-------------|-------------|
+| `inference_realtime.sh` | Runs multi instance realtime inference using 4 cores per instance for the specified precision (fp32, avx-fp32, bf16, or bf32) and mode (imperative or jit). |
+| `inference_throughput.sh` | Runs multi instance batch inference using 24 cores per instance for the specified precision (fp32, avx-fp32, bf16, or bf32) and mode (imperative or jit). |
+| `accuracy.sh` | Measures the inference accuracy for the specified precision (fp32, avx-fp32, bf16, or bf32) and mode (imperative or jit). |
+
+**Note:** The `avx-fp32` precision runs the same scripts as `fp32`, except that the `DNNL_MAX_CPU_ISA` environment variable is unset. The environment variable is otherwise set to `DNNL_MAX_CPU_ISA=AVX512_CORE_AMX`.
+
+## Datasets
+Download and extract the 2017 training/validation images and annotations from the [COCO dataset website](https://cocodataset.org/#download) to a `coco` folder and unzip the files. After extracting the zip files, your dataset directory structure should look something like this:
+```
+coco
+├── annotations
+│   ├── captions_train2017.json
+│   ├── captions_val2017.json
+│   ├── instances_train2017.json
+│   ├── instances_val2017.json
+│   ├── person_keypoints_train2017.json
+│   └── person_keypoints_val2017.json
+├── train2017
+│   ├── 000000454854.jpg
+│   ├── 000000137045.jpg
+│   ├── 000000129582.jpg
+│   └── ...
+└── val2017
+    ├── 000000000139.jpg
+    ├── 000000000285.jpg
+    ├── 000000000632.jpg
+    └── ...
+```
+
+## Pre-trained Model
+Download the pretrained model and set the CHECKPOINT_DIR environment variable to point to the directory where the weights file is downloaded.
+```
+curl -O https://download.pytorch.org/models/maskrcnn/e2e_mask_rcnn_R_50_FPN_1x.pth
+export CHECKPOINT_DIR=$(pwd)
+```
+
+## Docker Run
+(Optional) Export related proxy into docker environment.
+```bash
+export DOCKER_RUN_ENVS="-e ftp_proxy=${ftp_proxy} \
+  -e FTP_PROXY=${FTP_PROXY} -e http_proxy=${http_proxy} \
+  -e HTTP_PROXY=${HTTP_PROXY} -e https_proxy=${https_proxy} \
+  -e HTTPS_PROXY=${HTTPS_PROXY} -e no_proxy=${no_proxy} \
+  -e NO_PROXY=${NO_PROXY} -e socks_proxy=${socks_proxy} \
+  -e SOCKS_PROXY=${SOCKS_PROXY}"
+```
+
+To run MaskRCNN inference, set environment variables to specify the dataset directory, precision and mode to run, and an output directory.
+
+```bash
+# Set the required environment vars
+export OS=<provide either centos or ubuntu>
+export DATASET_DIR=<path to the dataset>
+export OUTPUT_DIR=<directory where log files will be written>
+export SCRIPT=quickstart/<specify the script to run>
+export PRECISION=<specify the precision to run>
+export MODE=<specify one of jit or imperative>
+export CHECKPOINT_DIR=<path to the downloaded pre-trained model>
+
+IMAGE_NAME=intel/object-detection:${OS}-pytorch-cpu-maskrcnn-inference
+WORKDIR=/workspace/pytorch-maskrcnn-inference
+DOCKER_ARGS="--privileged --init -it"
+
+docker run --rm \
+  --env OUTPUT_DIR=${OUTPUT_DIR} \
+  --env CHECKPOINT_DIR=${CHECKPOINT_DIR} \
+  --env DATASET_DIR=${DATASET_DIR} \
+  --env http_proxy=${http_proxy} \
+  --env https_proxy=${https_proxy} \
+  --env no_proxy=${no_proxy} \
+  --volume ${CHECKPOINT_DIR}:${CHECKPOINT_DIR} \
+  --volume ${DATASET_DIR}:${DATASET_DIR} \
+  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
+  ${DOCKER_RUN_ENVS} \
+  --shm-size 8G \
+  -w ${WORKDIR} \
+  ${DOCKER_ARGS} \
+  $IMAGE_NAME \
+  /bin/bash $SCRIPT $PRECISION $MODE
+```
+## Documentation and Sources
+#### Get Started​
+[Docker* Repository](https://hub.docker.com/r/intel/object-detection)
+
+[Main GitHub*](https://github.com/IntelAI/models)
+
+[Release Notes](https://github.com/IntelAI/models/releases)
+
+[Get Started Guide](https://github.com/IntelAI/models/blob/master/quickstart/quickstart/object_detection/pytorch/maskrcnn/inference/cpu/DEVCATALOG.md)
+
+#### Code Sources
+[Dockerfile](https://github.com/IntelAI/models/tree/master/docker/pyt-cpu)
+
+[Report Issue](https://community.intel.com/t5/Intel-Optimized-AI-Frameworks/bd-p/optimized-ai-frameworks)
+
+## License Agreement
+LEGAL NOTICE: By accessing, downloading or using this software and any required dependent software (the “Software Package”), you agree to the terms and conditions of the software license agreements for the Software Package, which may also include notices, disclaimers, or license terms for third party software included with the Software Package. Please refer to the [license](https://github.com/IntelAI/models/tree/master/third_party) file for additional details.
+
+[View All Containers and Solutions 🡢](https://www.intel.com/content/www/us/en/developer/tools/software-catalog/containers.html?s=Newest)
diff --git a/quickstart/object_detection/pytorch/maskrcnn/inference/cpu/README.md b/quickstart/object_detection/pytorch/maskrcnn/inference/cpu/README.md
index 543292c48..1a8104692 100644
--- a/quickstart/object_detection/pytorch/maskrcnn/inference/cpu/README.md
+++ b/quickstart/object_detection/pytorch/maskrcnn/inference/cpu/README.md
@@ -3,87 +3,92 @@
 ## Description
 This document has instructions for running Mask R-CNN inference.
 
-## Datasets
-
-Download the 2017 [COCO dataset](https://cocodataset.org) using the `download_dataset.sh` script.
-Export the `DATASET_DIR` environment variable to specify the directory where the dataset
-will be downloaded. This environment variable will be used again when running quickstart scripts.
-```
-cd <path to your clone of the model zoo>/quickstart/object_detection/pytorch/maskrcnn/inference/cpu
-export DATASET_DIR=<directory where the dataset will be saved>
-bash download_dataset.sh
-```
-
 ## Quick Start Scripts
+| Script name | Description |
+|-------------|-------------|
+| `inference_realtime.sh` | Runs multi instance realtime inference using 4 cores per instance for the specified precision (fp32, avx-fp32, bf16, or bf32) and mode (imperative or jit). |
+| `inference_throughput.sh` | Runs multi instance batch inference using 24 cores per instance for the specified precision (fp32, avx-fp32, bf16, or bf32) and mode (imperative or jit). |
+| `accuracy.sh` | Measures the inference accuracy for the specified precision (fp32, avx-fp32, bf16, or bf32) and mode (imperative or jit). |
 
-|  DataType   |  mode  | Throughput  |  Latency    |   Accuracy  |
-| ----------- | ------ | ----------- | ----------- | ----------- |
-| FP32        | imperative | bash inference_throughput.sh fp32 imperative | bash inference_realtime.sh fp32 imperative | bash accuracy.sh fp32 imperative |
-| BF16        | imperative | bash inference_throughput.sh bf16 imperative | bash inference_realtime.sh bf16 imperative | bash accuracy.sh bf16 imperative |
-| BF32        | imperative | bash inference_throughput.sh bf32 imperative | bash inference_realtime.sh bf32 imperative | bash accuracy.sh bf32 imperative |
-| FP32        | jit | bash inference_throughput.sh fp32 jit | bash inference_realtime.sh fp32 jit | bash accuracy.sh fp32 jit |
-| BF16        | jit | bash inference_throughput.sh bf16 jit | bash inference_realtime.sh bf16 jit | bash accuracy.sh bf16 jit |
-| BF32        | jit | bash inference_throughput.sh bf32 jit | bash inference_realtime.sh bf32 jit | bash accuracy.sh bf32 jit |
-
-Follow the instructions to setup your bare metal environment on either Linux or Windows systems. Once all the setup is done,
-the Model Zoo can be used to run a [quickstart script](#quick-start-scripts).
-Ensure that you have a clone of the [Model Zoo Github repository](https://github.com/IntelAI/models).
-```
-git clone https://github.com/IntelAI/models.git
-```
-## Run on Linux
-Follow [link](/docs/general/pytorch/BareMetalSetup.md) to install Miniconda and build Pytorch, IPEX, TorchVison and Jemalloc.
-* Install dependencies
+> Note: The `avx-fp32` precisions run the same scripts as `fp32`, except that the
+> `DNNL_MAX_CPU_ISA` environment variable is unset. The environment variable is
+> otherwise set to `DNNL_MAX_CPU_ISA=AVX512_CORE_AMX`.
+* Set ENV to use AMX:
   ```
-  pip install yacs opencv-python pycocotools defusedxml cityscapesscripts
-  conda install intel-openmp
+  export DNNL_MAX_CPU_ISA=AVX512_CORE_AMX
   ```
 
-* Install model
-  ```
-  cd models/object_detection/pytorch/maskrcnn/maskrcnn-benchmark/
-  python setup.py develop
-  ```
+## Bare Metal
+### General setup
 
-* Download pretrained model
-  ```
-  cd <path to your clone of the model zoo>/quickstart/object_detection/pytorch/maskrcnn/inference/cpu
-  export CHECKPOINT_DIR=<directory where the pretrained model will be saved>
-  bash download_model.sh
-  ```
+Follow [link](/docs/general/pytorch/BareMetalSetup.md) to install Miniconda and build Pytorch, IPEX, TorchVison and Jemalloc and TCmalloc
 
-* Set Jemalloc Preload for better performance
+### Model Specific Setup
+* Set Jemalloc and tcmalloc Preload for better performance
 
-  After [Jemalloc setup](/docs/general/pytorch/BareMetalSetup.md#build-jemalloc), set the following environment variables.
+  The jemalloc should be built from the [General setup](#general-setup) section.
   ```
-  export LD_PRELOAD="<path to the jemalloc directory>/lib/libjemalloc.so":$LD_PRELOAD
+  export LD_PRELOAD="<path to the jemalloc directory>/lib/libjemalloc.so":"path_to/tcmalloc/lib/libtcmalloc.so":$LD_PRELOAD
   export MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
   ```
 
 * Set IOMP preload for better performance
 
-  IOMP should be installed in your conda env. Set the following environment variables.
+  IOMP should be installed in your conda env from the [General setup](#general-setup) section.
   ```
+  pip install packaging intel-openmp
   export LD_PRELOAD=<path to the intel-openmp directory>/lib/libiomp5.so:$LD_PRELOAD
   ```
 
-* Set ENV to use AMX if you are using SPR
+* Follow the instructions to setup your bare metal environment on either Linux or Windows systems. Once all the setup is done,
+  the Intel® AI Reference Models can be used to run a [quickstart script](#quick-start-scripts).
+  Ensure that you have a clone of the [Intel® AI Reference Models Github repository](https://github.com/IntelAI/models) and navigate to the directory.
   ```
-  export DNNL_MAX_CPU_ISA=AVX512_CORE_AMX
+  git clone https://github.com/IntelAI/models.git
+  cd models
+  ```
+
+* Install model
+  ```
+  python models/object_detection/pytorch/maskrcnn/maskrcnn-benchmark/setup.py develop
+  ```
+
+* Download pretrained model
+  ```
+  export CHECKPOINT_DIR=<directory where the pretrained model will be saved>
+  bash quickstart/object_detection/pytorch/maskrcnn/inference/cpu/download_model.sh
+  ```
+* Datasets
+  Download the 2017 [COCO dataset](https://cocodataset.org) using the `download_dataset.sh` script.
+  Export the `DATASET_DIR` environment variable to specify the directory where the dataset
+  will be downloaded. This environment variable will be used again when running quickstart scripts.
+  ```
+  cd quickstart/object_detection/pytorch/maskrcnn/inference/cpu
+  export DATASET_DIR=<directory where the dataset will be saved>
+  bash download_dataset.sh
+  cd -
   ```
 
-* Run the model:
+## Run on Linux
   ```
+  # Navigate to Intel® AI Reference Models dir:
   cd models
 
+  # Install dependency:
+  ./quickstart/object_detection/pytorch/maskrcnn/inference/cpu/setup.sh
+  
   # Set environment variables
   export DATASET_DIR=<path to the COCO dataset>
   export CHECKPOINT_DIR=<path to the downloaded pretrained model>
   export OUTPUT_DIR=<path to an output directory>
   export MODE=<set to 'jit' or 'imperative'>
+  export PRECISION=< select from :- fp32, avx-fp32, bf16, or bf32>
+
+  # Optional environemnt variables:
+  export BATCH_SIZE=<set a value for batch size, else it will run with default batch size>
 
   # Run a quickstart script (for example, FP32 batch inference jit)
-  bash quickstart/object_detection/pytorch/maskrcnn/inference/cpu/inference_throughput.sh fp32 jit
+  ./quickstart/object_detection/pytorch/maskrcnn/inference/cpu/<script.sh>
   ```
 
 ## Run on Windows
@@ -104,9 +109,10 @@ If not already setup, please follow instructions for [environment setup on Windo
   set CHECKPOINT_DIR=<path to the downloaded pretrained model>
   set OUTPUT_DIR=<path to the directory where log files will be written>
   set MODE=<set to 'jit' or 'imperative'>
+  set PRECISION=<set to fp32, avx-fp32, bf16, or bf32>
 
-  #Run a quickstart script for fp32 precision(FP32 inference_realtime or inference_throughput or accuracy)
-  bash quickstart\object_detection\pytorch\maskrcnn\inference\cpu\batch_inference_baremetal.sh fp32 jit
+  #Run a quickstart script:
+  bash quickstart\object_detection\pytorch\maskrcnn\inference\cpu\<script.sh>
   ```
 
 
diff --git a/quickstart/object_detection/pytorch/maskrcnn/inference/cpu/accuracy.sh b/quickstart/object_detection/pytorch/maskrcnn/inference/cpu/accuracy.sh
index c3877c94f..c8bfa47c9 100755
--- a/quickstart/object_detection/pytorch/maskrcnn/inference/cpu/accuracy.sh
+++ b/quickstart/object_detection/pytorch/maskrcnn/inference/cpu/accuracy.sh
@@ -38,33 +38,43 @@ if [ ! -d "${OUTPUT_DIR}" ]; then
   exit 1
 fi
 
+if [ -z  "${PRECISION}" ]; then
+  echo "The PRECISION is not set"
+  exit 1
+fi
+
+if [ -z  "${MODE}" ]; then
+  echo "The MODE is not set"
+  exit 1
+fi
+
 ARGS=""
 
-if [[ "$1" == *"avx"* ]]; then
+if [[ "$PRECISION" == *"avx"* ]]; then
     unset DNNL_MAX_CPU_ISA
 fi
 
-if [[ "$1" == "bf16" ]]; then
+if [[ "$PRECISION" == "bf16" ]]; then
     ARGS="$ARGS --bf16"
     echo "### running bf16 datatype"
-elif [[ "$1" == "bf32" ]]; then
+elif [[ "$PRECISION" == "bf32" ]]; then
     ARGS="$ARGS --bf32"
     echo "### running bf32 datatype"
-elif [[ "$1" == "fp32" || "$1" == "avx-fp32" ]]; then
+elif [[ "$PRECISION" == "fp32" || "$PRECISION" == "avx-fp32" ]]; then
     echo "### running fp32 datatype"
 else
-    echo "The specified precision '$1' is unsupported."
+    echo "The specified precision '$PRECISION' is unsupported."
     echo "Supported precisions are: fp32, avx-fp32, bf16, and bf32."
     exit 1
 fi
 
-if [[ "$2" == "jit" ]]; then
+if [[ "$MODE" == "jit" ]]; then
     ARGS="$ARGS --jit"
     echo "### running jit mode"
-elif [[ "$2" == "imperative" ]]; then
+elif [[ "$MODE" == "imperative" ]]; then
     echo "### running imperative mode"
 else
-    echo "The specified mode '$2' is unsupported."
+    echo "The specified mode '$MODE' is unsupported."
     echo "Supported mode are: imperative and jit."
     exit 1
 fi
@@ -72,11 +82,11 @@ fi
 export DNNL_PRIMITIVE_CACHE_CAPACITY=1024
 export KMP_BLOCKTIME=1
 export KMP_AFFINITY=granularity=fine,compact,1,0
-PRECISION=$1
 
 export TRAIN=0
 
-BATCH_SIZE=112
+# If BATCH_SIZE not set, runs with default batch size
+BATCH_SIZE=${BATCH_SIZE:-112}
 
 rm -rf ${OUTPUT_DIR}/maskrcnn_${PRECISION}_accuracy*
 
@@ -104,8 +114,8 @@ source "${MODEL_DIR}/quickstart/common/utils.sh"
 _get_platform_type
 if [[ ${PLATFORM} == "linux" ]]; then
     accuracy=$(grep 'bbox AP:' ${OUTPUT_DIR}/maskrcnn_${PRECISION}_accuracy* |sed -e 's/.*Accuracy//;s/[^0-9.]//g')
-    echo ""maskrcnn";"bbox AP:";$1;${BATCH_SIZE};${accuracy}" | tee -a ${OUTPUT_DIR}/summary.log
+    echo ""maskrcnn";"bbox AP:";$PRECISION;${BATCH_SIZE};${accuracy}" | tee -a ${OUTPUT_DIR}/summary.log
     accuracy=$(grep 'segm AP:' ${OUTPUT_DIR}/maskrcnn_${PRECISION}_accuracy* |sed -e 's/.*Accuracy//;s/[^0-9.]//g')
-    echo ""maskrcnn";"segm AP:";$1;${BATCH_SIZE};${accuracy}" | tee -a ${OUTPUT_DIR}/summary.log
+    echo ""maskrcnn";"segm AP:";$PRECISION;${BATCH_SIZE};${accuracy}" | tee -a ${OUTPUT_DIR}/summary.log
 fi
 
diff --git a/quickstart/object_detection/pytorch/maskrcnn/inference/cpu/inference_realtime.sh b/quickstart/object_detection/pytorch/maskrcnn/inference/cpu/inference_realtime.sh
index c9a9dcc65..7f7a60e4f 100755
--- a/quickstart/object_detection/pytorch/maskrcnn/inference/cpu/inference_realtime.sh
+++ b/quickstart/object_detection/pytorch/maskrcnn/inference/cpu/inference_realtime.sh
@@ -38,33 +38,43 @@ if [ ! -d "${OUTPUT_DIR}" ]; then
   exit 1
 fi
 
-if [[ "$1" == *"avx"* ]]; then
+if [ -z  "${PRECISION}" ]; then
+  echo "The PRECISION is not set"
+  exit 1
+fi
+
+if [ -z  "${MODE}" ]; then
+  echo "The MODE is not set"
+  exit 1
+fi
+
+if [[ "$PRECISION" == *"avx"* ]]; then
     unset DNNL_MAX_CPU_ISA
 fi
 
 ARGS=""
 
-if [[ "$1" == "bf16" ]]; then
+if [[ "$PRECISION" == "bf16" ]]; then
     ARGS="$ARGS --bf16"
     echo "### running bf16 datatype"
-elif [[ "$1" == "bf32" ]]; then
+elif [[ "$PRECISION" == "bf32" ]]; then
     ARGS="$ARGS --bf32"
     echo "### running bf32 datatype"
-elif [[ "$1" == "fp32" || "$1" == "avx-fp32" ]]; then
+elif [[ "$PRECISION" == "fp32" || "$PRECISION" == "avx-fp32" ]]; then
     echo "### running fp32 datatype"
 else
-    echo "The specified precision '$1' is unsupported."
+    echo "The specified precision '$PRECISION' is unsupported."
     echo "Supported precisions are: fp32, avx-fp32, bf16, and bf32."
     exit 1
 fi
 
-if [[ "$2" == "jit" ]]; then
+if [[ "$MODE" == "jit" ]]; then
     ARGS="$ARGS --jit"
     echo "### running jit mode"
-elif [[ "$2" == "imperative" ]]; then
+elif [[ "$MODE" == "imperative" ]]; then
     echo "### running imperative mode"
 else
-    echo "The specified mode '$2' is unsupported."
+    echo "The specified mode '$MODE' is unsupported."
     echo "Supported mode are: imperative and jit."
     exit 1
 fi
@@ -75,8 +85,8 @@ export KMP_AFFINITY=granularity=fine,compact,1,0
 
 export TRAIN=0
 
-PRECISION=$1
-BATCH_SIZE=1
+# If BATCH_SIZE not set, runs with default batch size
+BATCH_SIZE=${BATCH_SIZE:-1}
 
 rm -rf ${OUTPUT_DIR}/maskrcnn_${PRECISION}_inference_realtime*
 
@@ -137,6 +147,6 @@ END   {
     sum = sum / i;
     printf("%.3f ms", sum);
 }')
-  echo ""maskrcnn";"latency";$1;${BATCH_SIZE};${throughput}" | tee -a ${OUTPUT_DIR}/summary.log
-  echo ""maskrcnn";"p99_latency";$1;${BATCH_SIZE};${p99_latency}" | tee -a ${OUTPUT_DIR}/summary.log
+  echo ""maskrcnn";"latency";$PRECISION;${BATCH_SIZE};${throughput}" | tee -a ${OUTPUT_DIR}/summary.log
+  echo ""maskrcnn";"p99_latency";$PRECISION;${BATCH_SIZE};${p99_latency}" | tee -a ${OUTPUT_DIR}/summary.log
 fi
diff --git a/quickstart/object_detection/pytorch/maskrcnn/inference/cpu/inference_throughput.sh b/quickstart/object_detection/pytorch/maskrcnn/inference/cpu/inference_throughput.sh
index 64cd288dd..7a0a525d5 100755
--- a/quickstart/object_detection/pytorch/maskrcnn/inference/cpu/inference_throughput.sh
+++ b/quickstart/object_detection/pytorch/maskrcnn/inference/cpu/inference_throughput.sh
@@ -38,33 +38,43 @@ if [ ! -d "${OUTPUT_DIR}" ]; then
   exit 1
 fi
 
-if [[ "$1" == *"avx"* ]]; then
+if [ -z  "${PRECISION}" ]; then
+  echo "The PRECISION is not set"
+  exit 1
+fi
+
+if [ -z  "${MODE}" ]; then
+  echo "The MODE is not set"
+  exit 1
+fi
+
+if [[ "$PRECISION" == *"avx"* ]]; then
     unset DNNL_MAX_CPU_ISA
 fi
 
 ARGS=""
 
-if [[ "$1" == "bf16" ]]; then
+if [[ "$PRECISION" == "bf16" ]]; then
     ARGS="$ARGS --bf16"
     echo "### running bf16 datatype"
-elif [[ "$1" == "bf32" ]]; then
+elif [[ "$PRECISION" == "bf32" ]]; then
     ARGS="$ARGS --bf32"
     echo "### running bf32 datatype"
-elif [[ "$1" == "fp32" || "$1" == "avx-fp32" ]]; then
+elif [[ "$PRECISION" == "fp32" || "$PRECISION" == "avx-fp32" ]]; then
     echo "### running fp32 datatype"
 else
-    echo "The specified precision '$1' is unsupported."
+    echo "The specified precision '$PRECISION' is unsupported."
     echo "Supported precisions are: fp32, avx-fp32, bf16, and bf32."
     exit 1
 fi
 
-if [[ "$2" == "jit" ]]; then
+if [[ "$MODE" == "jit" ]]; then
     ARGS="$ARGS --jit"
     echo "### running jit mode"
-elif [[ "$2" == "imperative" ]]; then
+elif [[ "$MODE" == "imperative" ]]; then
     echo "### running imperative mode"
 else
-    echo "The specified mode '$2' is unsupported."
+    echo "The specified mode '$MODE' is unsupported."
     echo "Supported mode are: imperative and jit."
     exit 1
 fi
@@ -75,8 +85,6 @@ export KMP_AFFINITY=granularity=fine,compact,1,0
 
 export TRAIN=0
 
-PRECISION=$1
-
 source "${MODEL_DIR}/quickstart/common/utils.sh"
 _get_platform_type
 
@@ -85,8 +93,9 @@ if [[ ${PLATFORM} == "windows" ]]; then
 else
   CORES=`lscpu | grep Core | awk '{print $4}'`
 fi
-BATCH_SIZE=`expr $CORES \* 2`
 
+# If BATCH_SIZE not set, runs with default batch size
+BATCH_SIZE=${BATCH_SIZE:-`expr $CORES \* 2`}
 rm -rf ${OUTPUT_DIR}/maskrcnn_${PRECISION}_inference_throughput*
 
 # check if stoch PYT or IPEX is installed on the system
@@ -125,5 +134,5 @@ if [[ ${PLATFORM} == "linux" ]]; then
           sum = sum / i;
           printf("%.3f", sum);
   }')
-  echo ""maskrcnn";"throughput";$1;${BATCH_SIZE};${throughput}" | tee -a ${OUTPUT_DIR}/summary.log
+  echo ""maskrcnn";"throughput";$PRECISION;${BATCH_SIZE};${throughput}" | tee -a ${OUTPUT_DIR}/summary.log
 fi
diff --git a/quickstart/object_detection/pytorch/maskrcnn/inference/cpu/setup.sh b/quickstart/object_detection/pytorch/maskrcnn/inference/cpu/setup.sh
new file mode 100755
index 000000000..a2d4f2306
--- /dev/null
+++ b/quickstart/object_detection/pytorch/maskrcnn/inference/cpu/setup.sh
@@ -0,0 +1,19 @@
+#!/bin/bash
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+pip install matplotlib Pillow pycocotools 
+pip install yacs opencv-python cityscapesscripts defusedxml
diff --git a/quickstart/object_detection/pytorch/maskrcnn/training/cpu/DEVCATALOG.md b/quickstart/object_detection/pytorch/maskrcnn/training/cpu/DEVCATALOG.md
new file mode 100644
index 000000000..19072d3e2
--- /dev/null
+++ b/quickstart/object_detection/pytorch/maskrcnn/training/cpu/DEVCATALOG.md
@@ -0,0 +1,105 @@
+# PyTorch Mask RCNN training
+
+## Description 
+This document has instructions for running MaskRCNN training using Intel Extension for PyTorch. 
+
+## Pull Command
+
+Docker image based on CentOS Stream8
+```
+docker pull intel/object-detection:centos-pytorch-cpu-maskrcnn-training
+```
+Docker image based on Ubuntu 22.04
+```
+docker pull intel/object-detection:ubuntu-pytorch-cpu-maskrcnn-training
+```
+
+## Quick Start Scripts
+| Script name | Description |
+|-------------|-------------|
+| `training.sh` | Runs training for the specified precision (fp32, avx-fp32，bf16, or bf32). |
+
+**Note:** The `avx-fp32` precision runs the same scripts as `fp32`, except that the `DNNL_MAX_CPU_ISA` environment variable is unset. The environment variable is otherwise set to `DNNL_MAX_CPU_ISA=AVX512_CORE_AMX`.
+
+## Datasets
+Download and extract the 2017 training/validation images and annotations from the [COCO dataset website](https://cocodataset.org/#download) to a `coco` folder and unzip the files. After extracting the zip files, your dataset directory structure should look something like this:
+```
+coco
+├── annotations
+│   ├── captions_train2017.json
+│   ├── captions_val2017.json
+│   ├── instances_train2017.json
+│   ├── instances_val2017.json
+│   ├── person_keypoints_train2017.json
+│   └── person_keypoints_val2017.json
+├── train2017
+│   ├── 000000454854.jpg
+│   ├── 000000137045.jpg
+│   ├── 000000129582.jpg
+│   └── ...
+└── val2017
+    ├── 000000000139.jpg
+    ├── 000000000285.jpg
+    ├── 000000000632.jpg
+    └── ...
+```
+
+## Docker Run
+(Optional) Export related proxy into docker environment.
+```bash
+export DOCKER_RUN_ENVS="-e ftp_proxy=${ftp_proxy} \
+  -e FTP_PROXY=${FTP_PROXY} -e http_proxy=${http_proxy} \
+  -e HTTP_PROXY=${HTTP_PROXY} -e https_proxy=${https_proxy} \
+  -e HTTPS_PROXY=${HTTPS_PROXY} -e no_proxy=${no_proxy} \
+  -e NO_PROXY=${NO_PROXY} -e socks_proxy=${socks_proxy} \
+  -e SOCKS_PROXY=${SOCKS_PROXY}"
+```
+To run MaskRCNN training, set environment variables to specify the dataset directory, precision to run, and an output directory. 
+```bash
+# Set the required environment vars
+export OS=<provide either centos or ubuntu>
+export DATASET_DIR=<path to the dataset>
+export OUTPUT_DIR=<directory where log files will be written>
+export PRECISION=<specify the precision to run>
+export SCRIPT=quickstart/training.sh 
+export BATCH_SIZE=<set batch size(default is 112)>
+
+DOCKER_ARGS="--privileged --init -it"
+WORKDIR=/workspace/pytorch-maskrcnn-training
+IMAGE_NAME=intel/object-detection:${OS}-pytorch-cpu-maskrcnn-training
+
+docker run --rm \
+  --env BATCH_SIZE=${BATCH_SIZE} \
+  --env OUTPUT_DIR=${OUTPUT_DIR} \
+  --env DATASET_DIR=${DATASET_DIR} \
+  --env http_proxy=${http_proxy} \
+  --env https_proxy=${https_proxy} \
+  --env no_proxy=${no_proxy} \
+  --volume ${DATASET_DIR}:${DATASET_DIR} \
+  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
+  ${DOCKER_RUN_ENVS} \
+  --shm-size 8G \
+  -w ${WORKDIR} \
+  ${DOCKER_ARGS} \
+  $IMAGE_NAME \
+  /bin/bash $SCRIPT $PRECISION
+  ```
+## Documentation and Sources
+#### Get Started​
+[Docker* Repository](https://hub.docker.com/r/intel/object-detection)
+
+[Main GitHub*](https://github.com/IntelAI/models)
+
+[Release Notes](https://github.com/IntelAI/models/releases)
+
+[Get Started Guide](https://github.com/IntelAI/models/blob/master/quickstart/quickstart/object_detection/pytorch/maskrcnn/training/cpu/DEVCATALOG.md)
+
+#### Code Sources
+[Dockerfile](https://github.com/IntelAI/models/tree/master/docker/pyt-cpu)
+
+[Report Issue](https://community.intel.com/t5/Intel-Optimized-AI-Frameworks/bd-p/optimized-ai-frameworks)
+
+## License Agreement
+LEGAL NOTICE: By accessing, downloading or using this software and any required dependent software (the “Software Package”), you agree to the terms and conditions of the software license agreements for the Software Package, which may also include notices, disclaimers, or license terms for third party software included with the Software Package. Please refer to the [license](https://github.com/IntelAI/models/tree/master/third_party) file for additional details.
+
+[View All Containers and Solutions 🡢](https://www.intel.com/content/www/us/en/developer/tools/software-catalog/containers.html?s=Newest)
diff --git a/quickstart/object_detection/pytorch/maskrcnn/training/cpu/README.md b/quickstart/object_detection/pytorch/maskrcnn/training/cpu/README.md
index 55137f289..154bd779a 100644
--- a/quickstart/object_detection/pytorch/maskrcnn/training/cpu/README.md
+++ b/quickstart/object_detection/pytorch/maskrcnn/training/cpu/README.md
@@ -3,30 +3,37 @@
 ## Description
 This document has instructions for running Mask R-CNN training using Intel-optimized PyTorch.
 
+## Quick Start Scripts
+| Script name | Description |
+|-------------|-------------|
+| `training.sh` | Runs training for the specified precision (fp32, avx-fp32，bf16, or bf32). |
+
+> Note: The `avx-fp32` precisions run the same scripts as `fp32`, except that the
+> `DNNL_MAX_CPU_ISA` environment variable is unset. The environment variable is
+> otherwise set to `DNNL_MAX_CPU_ISA=AVX512_CORE_AMX`.
+* Set ENV to use AMX:
+  ```
+  export DNNL_MAX_CPU_ISA=AVX512_CORE_AMX
+  ```
+
+For Distributed Training:
+|  DataType   | Throughput  |
+| ----------- | ----------- |
+| FP32        | bash training_multinode.sh fp32 |
+| BF16        | bash training_multinode.sh bf16 |
+| BF32        | bash training_multinode.sh bf32 |
+
 ## Bare Metal
 ### General setup
 
-Follow [link](/docs/general/pytorch/BareMetalSetup.md) to install Miniconda and build Pytorch, IPEX, TorchVison, Torch-CCL and Jemalloc.
+Follow [link](/docs/general/pytorch/BareMetalSetup.md) to install Miniconda and build Pytorch, IPEX, TorchVison and Jemalloc and TCmalloc
 
 ### Model Specific Setup
-
-* Install dependencies
-  ```
-  pip install yacs opencv-python pycocotools defusedxml cityscapesscripts
-  conda install intel-openmp
-  ```
-
-* Install model
-  ```
-  cd <path to your clone of the model zoo>/models/object_detection/pytorch/maskrcnn/maskrcnn-benchmark
-  python setup.py develop
-  ```
-
-* Set Jemalloc Preload for better performance
+* Set Jemalloc and tcmalloc Preload for better performance
 
   The jemalloc should be built from the [General setup](#general-setup) section.
   ```
-  export LD_PRELOAD="path/lib/libjemalloc.so":$LD_PRELOAD
+  export LD_PRELOAD="<path to the jemalloc directory>/lib/libjemalloc.so":"path_to/tcmalloc/lib/libtcmalloc.so":$LD_PRELOAD
   export MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
   ```
 
@@ -34,14 +41,32 @@ Follow [link](/docs/general/pytorch/BareMetalSetup.md) to install Miniconda and
 
   IOMP should be installed in your conda env from the [General setup](#general-setup) section.
   ```
-  export LD_PRELOAD=path/lib/libiomp5.so:$LD_PRELOAD
+  pip install packaging intel-openmp
+  export LD_PRELOAD=<path to the intel-openmp directory>/lib/libiomp5.so:$LD_PRELOAD
   ```
 
-* Set ENV to use AMX if you are using SPR
+* Follow the instructions to setup your bare metal environment on either Linux or Windows systems. Once all the setup is done,
+  the Intel® AI Reference Models can be used to run a [quickstart script](#quick-start-scripts).
+  Ensure that you have a clone of the [Intel® AI Reference Models Github repository](https://github.com/IntelAI/models) and navigate to the directory.
   ```
-  export DNNL_MAX_CPU_ISA=AVX512_CORE_AMX
+  git clone https://github.com/IntelAI/models.git
+  cd models
+  ```
+* Install model
   ```
+  python models/object_detection/pytorch/maskrcnn/maskrcnn-benchmark/setup.py develop
+  ```
+* Datasets
 
+  Download the 2017 [COCO dataset](https://cocodataset.org) using the `download_dataset.sh` script.
+  Export the `DATASET_DIR` environment variable to specify the directory where the dataset
+  will be downloaded. This environment variable will be used again when running quickstart scripts.
+  ```
+  cd quickstart/object_detection/pytorch/maskrcnn/training/cpu
+  export DATASET_DIR=<directory where the dataset will be saved>
+  bash download_dataset.sh
+  cd - 
+  ```
 * Set ENV to use multi-node distributed training (no need for single-node multi-sockets)
 
   In this case, we use data-parallel distributed training and every rank will hold same model replica. The NNODES is the number of ip in the HOSTFILE. To use multi-nodes distributed training you should firstly setup the passwordless login (you can refer to [link](https://linuxize.com/post/how-to-setup-passwordless-ssh-login/)) between these nodes. 
@@ -50,53 +75,28 @@ Follow [link](/docs/general/pytorch/BareMetalSetup.md) to install Miniconda and
   export HOSTFILE=your_ip_list_file #one ip per line
   ```
 
-## Datasets
-
-Download the 2017 [COCO dataset](https://cocodataset.org) using the `download_dataset.sh` script.
-Export the `DATASET_DIR` environment variable to specify the directory where the dataset
-will be downloaded. This environment variable will be used again when running quickstart scripts.
-```
-cd <path to your clone of the model zoo>/quickstart/object_detection/pytorch/maskrcnn/training/cpu
-export DATASET_DIR=<directory where the dataset will be saved>
-bash download_dataset.sh
-```
-
-## Quick Start Scripts
-
-|  DataType   | Throughput  |
-| ----------- | ----------- |
-| FP32        | bash training.sh fp32 |
-| BF16        | bash training.sh bf16 |
-| BF32        | bash training.sh bf32 |
-
-|               stributed Training              |
-|  DataType   | Throughput  |
-| ----------- | ----------- |
-| FP32        | bash training_multinode.sh fp32 |
-| BF16        | bash training_multinode.sh bf16 |
-| BF32        | bash training_multinode.sh bf32 |
-
 ## Run the model
-
-Follow the instructions above to setup your bare metal environment, download and
-preprocess the dataset, and do the model specific setup. Once all the setup is done,
-the Model Zoo can be used to run a [quickstart script](#quick-start-scripts).
+Once all the above setup is done,we can run a [quickstart script](#quick-start-scripts).
 Ensure that you have an enviornment variables set to point to the dataset directory
 and an output directory.
 
 ```
-# Clone the model zoo repo and set the MODEL_DIR
-git clone https://github.com/IntelAI/models.git
-cd models
+# Make sure you are inside the Intel® AI Reference Models directory
 export MODEL_DIR=$(pwd)
 
-# Env vars
+# Set the environment variable:
 export DATASET_DIR=<path to the COCO dataset>
 export OUTPUT_DIR=<path to an output directory>
+export PRECISION=< select from :- fp32, avx-fp32, bf16, or bf32>
+
+# Optional environemnt variables:
+export BATCH_SIZE=<set a value for batch size, else it will run with default batch size>
+
+# Install dependency:
+./quickstart/object_detection/pytorch/maskrcnn/training/cpu/setup.sh
 
-# Run a quickstart script (for example, FP32 training)
-cd ${MODEL_DIR}/quickstart/object_detection/pytorch/maskrcnn/training/cpu
-bash training.sh fp32
+# Run a quickstart script:
+./quickstart/object_detection/pytorch/maskrcnn/training/cpu/<quickstart_script.sh>
 
 # Run distributed training script (for example, FP32 distributed training)
 cd ${MODEL_DIR}/quickstart/object_detection/pytorch/maskrcnn/training/cpu/
diff --git a/quickstart/object_detection/pytorch/maskrcnn/training/cpu/setup.sh b/quickstart/object_detection/pytorch/maskrcnn/training/cpu/setup.sh
new file mode 100755
index 000000000..c1d3651fa
--- /dev/null
+++ b/quickstart/object_detection/pytorch/maskrcnn/training/cpu/setup.sh
@@ -0,0 +1,19 @@
+#!/bin/bash
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+pip install matplotlib Pillow pycocotools
+pip install yacs opencv-python cityscapesscripts defusedxml transformers
diff --git a/quickstart/object_detection/pytorch/maskrcnn/training/cpu/training.sh b/quickstart/object_detection/pytorch/maskrcnn/training/cpu/training.sh
index 875f813d6..17ddf5ff5 100755
--- a/quickstart/object_detection/pytorch/maskrcnn/training/cpu/training.sh
+++ b/quickstart/object_detection/pytorch/maskrcnn/training/cpu/training.sh
@@ -33,22 +33,27 @@ if [ ! -d "${OUTPUT_DIR}" ]; then
   exit 1
 fi
 
-if [[ "$1" == *"avx"* ]]; then
+if [ -z "${PRECISION}" ]; then
+  echo "The PRECISION env variable is not set"
+  exit 1
+fi
+
+if [[ "$PRECISION" == *"avx"* ]]; then
     unset DNNL_MAX_CPU_ISA
 fi
 
 ARGS=""
 
-if [[ "$1" == "bf16" ]]; then
+if [[ "$PRECISION" == "bf16" ]]; then
     ARGS="$ARGS --bf16"
     echo "### running bf16 datatype"
-elif [[ "$1" == "bf32" ]]; then
+elif [[ "$PRECISION" == "bf32" ]]; then
     ARGS="$ARGS --bf32"
     echo "### running bf32 datatype"
-elif [[ "$1" == "fp32" || "$1" == "avx-fp32" ]]; then
+elif [[ "$PRECISION" == "fp32" || "$PRECISION" == "avx-fp32" ]]; then
     echo "### running fp32 datatype"
 else
-    echo "The specified precision '$1' is unsupported."
+    echo "The specified precision '$PRECISION' is unsupported."
     echo "Supported precisions are: fp32, avx-fp32, bf16, and bf32."
     exit 1
 fi
@@ -59,7 +64,6 @@ TOTAL_CORES=`expr $CORES \* $SOCKETS`
 
 CORES_PER_INSTANCE=$CORES
 
-
 export DNNL_PRIMITIVE_CACHE_CAPACITY=1024
 export USE_IPEX=1
 export KMP_BLOCKTIME=1
@@ -67,8 +71,8 @@ export KMP_AFFINITY=granularity=fine,compact,1,0
 
 export TRAIN=1
 
-PRECISION=$1
-BATCH_SIZE=${BATCH_SIZE-112}
+# Runs with default value when batch size is not set
+BATCH_SIZE=${BATCH_SIZE:-112}
 
 rm -rf ${OUTPUT_DIR}/maskrcnn_${PRECISION}_train_throughput*
 
@@ -92,5 +96,5 @@ python -m intel_extension_for_pytorch.cpu.launch \
 wait
 
 throughput=$(grep 'Training throughput:' ${OUTPUT_DIR}/maskrcnn_${PRECISION}_train_throughput* |sed -e 's/.Trainng throughput//;s/[^0-9.]//g')
-echo ""maskrcnn";"training throughput";$1;${BATCH_SIZE};${throughput}" | tee -a ${OUTPUT_DIR}/summary.log
+echo ""maskrcnn";"training throughput";$PRECISION;${BATCH_SIZE};${throughput}" | tee -a ${OUTPUT_DIR}/summary.log
 
diff --git a/quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/.docs/baremetal.md b/quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/.docs/baremetal.md
deleted file mode 100644
index 6ae4a5b23..000000000
--- a/quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/.docs/baremetal.md
+++ /dev/null
@@ -1,46 +0,0 @@
-<!--- 50. Baremetal -->
-## Run the model
-Install the following pre-requisites:
-* Python version 3.9
-* Create and activate virtual environment.
-  ```bash
-  virtualenv -p python <virtualenv_name>
-  source <virtualenv_name>/bin/activate
-  ```
-* Install PyTorch and Intel® Extension for PyTorch for GPU (IPEX):
-  ```bash
-  python -m pip install torch==1.10.0a0 -f https://developer.intel.com/ipex-whl-stable-xpu
-  python -m pip install numpy==1.23.4
-  python -m pip install intel_extension_for_pytorch==1.10.200+gpu -f https://developer.intel.com/ipex-whl-stable-xpu
-  ```
-  To verify that PyTorch and IPEX are correctly installed:
-  ```bash
-  python -c "import torch;print(torch.device('xpu'))"  # Sample output: "xpu"
-  python -c "import intel_extension_for_pytorch as ipex;print(ipex.xpu.is_available())"  #Sample output True
-  python -c "import intel_extension_for_pytorch as ipex;print(ipex.xpu.has_onemkl())"  # Sample output: True
-  ```
-* Navigate to SSD-Mobilenet inference directory and install model specific dependencies for the workload:
-  ```bash
-  cd quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu
-  ./setup.sh
-  cd -
-  ```
-* Download the dataset label file, and set the "label" environment variable to point to where it was saved (for example: `export label=/home/<user>/voc-model-labels.txt`):
-  ```bash
-  wget https://storage.googleapis.com/models-hao/voc-model-labels.txt
-  ```
-
-This snippet shows how to run the inference quickstart script. The inference script
-will download the model weights to the directory location set in 'PRETRAINED_MODEL'.
-
-```
-### Run the model on Baremetal
-Set environment variables:
-export DATASET_DIR=<Path to the VOC2007 folder>
-export OUTPUT_DIR=<Path to save the output logs>
-export PRETRAINED_MODEL=<path to directory where the model weights will be loaded>
-export label=<path to label.txt file>
-
-Run the inference script, only int8 precision is supported:
-./quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/inference_with_dummy_data.sh
-```
diff --git a/quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/.docs/datasets.md b/quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/.docs/datasets.md
deleted file mode 100644
index dda3d9138..000000000
--- a/quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/.docs/datasets.md
+++ /dev/null
@@ -1,37 +0,0 @@
-<!--- 30. Datasets -->
-## Datasets
-
-The [VOC2007](http://host.robots.ox.ac.uk/pascal/VOC/voc2007/) validation dataset is used.
-
-Download and extract the VOC2007 dataset from http://host.robots.ox.ac.uk/pascal/VOC/voc2007/,
-After extracting the data, your folder structure should look something like this:
-
-```
-VOC2007
-├── Annotations
-│   ├── 000038.xml    
-│   ├── 000724.xml
-│   ├── 001440.xml
-│   └── ...
-├── ImageSets
-│   ├── Layout    
-│   ├── Main
-│   └── Segmentation
-├── SegmentationClass
-│   ├── 005797.png   
-│   ├── 007415.png 
-│   ├── 006581.png 
-│   └── ...
-├── SegmentationObject
-│   ├── 005797.png    
-│   ├── 006581.png
-│   ├── 007415.png
-│   └── ...
-└── JPEGImages
-    ├── 002832.jpg    
-    ├── 003558.jpg
-    ├── 004262.jpg
-    └── ...
-```
-The folder should be set as the `DATASET_DIR`
-(for example: `export DATASET_DIR=/home/<user>/VOC2007`).
diff --git a/quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/.docs/description.md b/quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/.docs/description.md
deleted file mode 100644
index 1d4003781..000000000
--- a/quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/.docs/description.md
+++ /dev/null
@@ -1,5 +0,0 @@
-<!-- 10. Description -->
-## Description
-
-This document has instructions for running <model name> <mode> using
-Intel(R) Extension for PyTorch with GPU.
diff --git a/quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/.docs/docker.md b/quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/.docs/docker.md
deleted file mode 100644
index 438e8fea4..000000000
--- a/quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/.docs/docker.md
+++ /dev/null
@@ -1,43 +0,0 @@
-<!--- 60. Docker -->
-## Run the model
-
-Requirements:
-* Host machine has Intel® Data Center GPU Flex Series.
-* Host machine has the Intel® Data Center GPU Flex Series Ubuntu driver. Please follow the [link](https://registrationcenter.intel.com/en/products/download/4125/) to download.
-* Host machine has Docker installed
-* Download and build the Intel® Extension for PyTorch (IPEX) container using the [link](https://registrationcenter.intel.com/en/products/subscription/956/).
-  (`model-zoo:pytorch-ipex-gpu`)
-
-Prior to building the <model name> <mode> container, ensure that you have
-built the IPEX container (`model-zoo:pytorch-ipex-gpu`).
-
-[Extract the package](#model-package), then use the `build.sh`
-script to build the container. After the container has been built, you can
-run the model <mode> using the `run.sh` script.
-
-The `run.sh` script will execute one of the [quickstart](#quick-start-scripts) script
-using the container that was just built. By default, the
-`inference_block_format.sh` script will be run. To run a different script,
-specify the script name of the quickstart script using the `SCRIPT`
-environment variable. See the snippet below for an example.
-
-> Note: Ensure that your system has the proxy environment variables
-> set (if needed), otherwise the container build may fail when trying to pull external
-> dependencies (like apt-get and pip installs).
-
-The inference scripts will download the model weights from https://storage.googleapis.com/models-hao/mobilenet-v1-ssd-mp-0_675.pth to the anywhere directory you choice, and set environment 'PRETRAINED_MODEL'.
-```
-# Extract the package
-tar -xzf <package name>
-
-# Navigate to the SSD-Mobilenet inference directory
-cd pytorch-gpu-ssd-mobilenet-inference
-
-# Set environment vars for the dataset and an output directory
-export DATASET_DIR=<Path to the VOC2007 folder>
-export OUTPUT_DIR=<Where_to_save_OUTPUT_DIR>
-export PRETRAINED_MODEL=<Where_to_load_path>
-
-# Run the container with default script in run.sh (inference_with_dummy_data.sh)
-./run.sh
-```
diff --git a/quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/.docs/gpu_setup.md b/quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/.docs/gpu_setup.md
deleted file mode 100644
index 46a1c07ec..000000000
--- a/quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/.docs/gpu_setup.md
+++ /dev/null
@@ -1,30 +0,0 @@
-<!--- 20. GPU Setup -->
-## Hardware Requirements:
-- Intel® Data Center GPU Flex Series
-
-## Software Requirements:
-- Ubuntu 20.04 (64-bit)
-- Intel GPU Drivers: Intel® Data Center GPU Flex Series [419.40](https://dgpu-docs.intel.com/releases/stable_419_40_20220914.html)
-
-  |Release|OS|Intel GPU|Install Intel GPU Driver|
-    |-|-|-|-|
-    |v1.0.0|Ubuntu 20.04|Intel® Data Center GPU Flex Series| Refer to the [Installation Guides](https://dgpu-docs.intel.com/installation-guides/ubuntu/ubuntu-focal-dc.html) for latest driver installation. If install the verified Intel® Data Center GPU Flex Series [419.40](https://dgpu-docs.intel.com/releases/stable_419_40_20220914.html), please append the specific version after components, such as `apt-get install intel-opencl-icd=22.28.23726.1+i419~u20.04`|
-
-- Intel® oneAPI Base Toolkit 2022.3: Need to install components of Intel® oneAPI Base Toolkit
-  - Intel® oneAPI DPC++ Compiler
-  - Intel® oneAPI Math Kernel Library (oneMKL)
-  * Download and install the verified DPC++ compiler and oneMKL in Ubuntu 20.04.
-
-    ```bash
-    wget https://registrationcenter-download.intel.com/akdlm/irc_nas/18852/l_BaseKit_p_2022.3.0.8767_offline.sh
-    # 4 components are necessary: DPC++/C++ Compiler, DPC++ Libiary, Threading Building Blocks and oneMKL
-    sh ./l_BaseKit_p_2022.3.0.8767_offline.sh
-    ```
-    For any more details, please follow the procedure in https://www.intel.com/content/www/us/en/developer/tools/oneapi/base-toolkit.html.
-
-  - Set environment variables: 
-    Default installation location {ONEAPI_ROOT} is /opt/intel/oneapi for root account, ${HOME}/intel/oneapi for other accounts
-    ```bash
-    source {ONEAPI_ROOT}/setvars.sh
-    ```
-
diff --git a/quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/.docs/license.md b/quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/.docs/license.md
deleted file mode 100644
index da5269ad2..000000000
--- a/quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/.docs/license.md
+++ /dev/null
@@ -1,4 +0,0 @@
-<!--- 50. License -->
-## License
-
-[LICENSE](/LICENSE)
diff --git a/quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/.docs/package.md b/quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/.docs/package.md
deleted file mode 100644
index c2317f9ba..000000000
--- a/quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/.docs/package.md
+++ /dev/null
@@ -1,21 +0,0 @@
-<!--- 20. Model package -->
-## Model Package
-
-The model package includes the scripts and libraries needed to
-build and run <model name> <mode> using a docker container. Note that
-this model container uses the PyTorch IPEX GPU container as it's base,
-and it requires the `model-zoo:pytorch-ipex-gpu` image to be built before
-the model container is built.
-```
-<package dir>
-├── build.sh
-├── info.txt
-├── licenses
-│   ├── LICENSE
-│   └── third_party
-├── model_packages
-│   └── <package name>
-├── <package dir>.Dockerfile
-├── README.md
-└── run.sh
-```
diff --git a/quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/.docs/quickstart.md b/quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/.docs/quickstart.md
deleted file mode 100644
index 3970f4458..000000000
--- a/quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/.docs/quickstart.md
+++ /dev/null
@@ -1,6 +0,0 @@
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| `inference_with_dummy_data.sh` | Inference with dummy data, batch size 512, for int8 blocked channel first. |
diff --git a/quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/.docs/title.md b/quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/.docs/title.md
deleted file mode 100644
index 1d6c60187..000000000
--- a/quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/.docs/title.md
+++ /dev/null
@@ -1,2 +0,0 @@
-<!--- 0. Title -->
-# <model name> <mode>
diff --git a/quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/DEVCATALOG.md b/quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/DEVCATALOG.md
deleted file mode 100644
index 173c09ca1..000000000
--- a/quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/DEVCATALOG.md
+++ /dev/null
@@ -1,113 +0,0 @@
-# Running SSD-MobileNetv1 Inference on Intel® Data Center GPU Flex Series using Intel® Extension for PyTorch*
-
-## Overview
-
-This document has instructions for running SSD-Mobilenetv1 inference using Intel® Extension for PyTorch on Intel® Flex Series GPU.
-
-## Requirements
-| Item | Detail |
-| ------ | ------- |
-| Host machine  | Intel® Data Center GPU Flex Series  |
-| Drivers | GPU-compatible drivers need to be installed: [Download Driver 602](https://dgpu-docs.intel.com/installation-guides/ubuntu/ubuntu-jammy-dc.html#step-1-add-package-repository)
-| Software | Docker* Installed |
-
-## Download Datasets
-
-The [VOC2007](http://host.robots.ox.ac.uk/pascal/VOC/voc2007/) validation dataset is used.
-
-Download and extract the VOC2007 dataset from http://host.robots.ox.ac.uk/pascal/VOC/voc2007/,
-After extracting the data, your folder structure should look something like this:
-
-```
-VOC2007
-├── Annotations
-│   ├── 000038.xml    
-│   ├── 000724.xml
-│   ├── 001440.xml
-│   └── ...
-├── ImageSets
-│   ├── Layout    
-│   ├── Main
-│   └── Segmentation
-├── SegmentationClass
-│   ├── 005797.png   
-│   ├── 007415.png 
-│   ├── 006581.png 
-│   └── ...
-├── SegmentationObject
-│   ├── 005797.png    
-│   ├── 006581.png
-│   ├── 007415.png
-│   └── ...
-└── JPEGImages
-    ├── 002832.jpg    
-    ├── 003558.jpg
-    ├── 004262.jpg
-    └── ...
-```
-The folder should be set as the `DATASET_DIR`
-(for example: `export DATASET_DIR=/home/<user>/VOC2007`).
-
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| `inference_with_dummy_data.sh` | Inference with dummy data, for int8 blocked channel first on Flex series 170 |
-| `flex_multi_card_batch_inference.sh` | Inference with dummy data,for int8 and given batch size blocked channel first on Flex series 140 |
-| `flex_multi_card_online_inference.sh` | Online Inference with dummy data,for int8 blocked channel first on Flex series 140 | 
-
-## Run Using Docker
-
-### Set up Docker Image
-
-```
-docker pull intel/object-detection:pytorch-flex-gpu-ssd-mobilenet-inference
-```
-### Run Docker Image
-The SSD-MobileNet inference container includes scripts,model and libraries need to run int8 inference. To run the `flex_multi_card_batch_inference.sh` quickstart script using this container,you will need to provide an output directory where log files will be written.The script by default runs inference on dummy data. The script also performs online INT8 calibration for which the VOC2007 dataset is required to be provided. The pre-trained model will be downloaded by the script.
-
-**Note:** The Default batch size for Flex series 140 is 256 for batch inference and 1024 for Flex series 170. Additionally, add `--cap-add=SYS_NICE` to the `docker run` command for executing `flex_multi_card_online_inference.sh` and `flex_multi_card_batch_inference.sh` on Flex series 140.
-
-```
-export PRECISION=int8
-export OUTPUT_DIR=<path to output directory>
-export DATASET_DIR=<path to the preprocessed voc2007 dataset>
-export PRETRAINED_MODEL=<path to the pretrained model folder. The code downloads the model if this folder is empty>
-export SCRIPT=quickstart/inference_with_dummy_data.sh
-export BATCH_SIZE=<inference batch size>
-export NUM_ITERATIONS=<enter number of iterations>
-export label=/workspace/pytorch-flex-series-ssd-mobilenet-inference/labels/voc-model-labels.txt
-
-DOCKER_ARGS="--rm -it"
-IMAGE_NAME=intel/object-detection:pytorch-flex-gpu-ssd-mobilenet-inference 
-
-docker run \
-  --privileged \
-  --device=/dev/dri \
-  --ipc=host \
-  --env BATCH_SIZE=${BATCH_SIZE} \
-  --env NUM_ITERATIONS=${NUM_ITERATIONS} \
-  --env PRECISION=${PRECISION} \
-  --env OUTPUT_DIR=${OUTPUT_DIR} \
-  --env DATASET_DIR=${DATASET_DIR} \
-  --env label=${label} \
-  --env http_proxy=${http_proxy} \
-  --env https_proxy=${https_proxy} \
-  --env no_proxy=${no_proxy} \
-  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
-  --volume ${DATASET_DIR}:${DATASET_DIR} \
-  ${DOCKER_ARGS} \
-  ${IMAGE_NAME} \
-  /bin/bash $SCRIPT
-  ```
-
-## Documentation and Sources
-
-[GitHub* Repository](https://github.com/IntelAI/models/tree/master/dockerfiles/model_containers)
-
-## Support
-Support for Intel® Extension for PyTorch* is found via the [Intel® AI Analytics Toolkit.](https://www.intel.com/content/www/us/en/developer/tools/oneapi/ai-analytics-toolkit.html#gs.qbretz) Additionally, the Intel® Extension for PyTorch* team tracks both bugs and enhancement requests using [GitHub issues](https://github.com/intel/intel-extension-for-pytorch/issues). Before submitting a suggestion or bug report, please search the GitHub issues to see if your issue has already been reported.
-
-## License Agreement
-
-LEGAL NOTICE: By accessing, downloading or using this software and any required dependent software (the “Software Package”), you agree to the terms and conditions of the software license agreements for the Software Package, which may also include notices, disclaimers, or license terms for third party software included with the Software Package. Please refer to the [license file](https://github.com/IntelAI/models/tree/master/third_party) for additional details.
diff --git a/quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/README.md b/quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/README.md
deleted file mode 100644
index 110b57df2..000000000
--- a/quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/README.md
+++ /dev/null
@@ -1,131 +0,0 @@
-<!--- 0. Title -->
-# SSD-Mobilenetv1 inference
-
-<!-- 10. Description -->
-## Description
-
-This document has instructions for running SSD-MobileNet inference using
-Intel® Extension for PyTorch with Intel® Data Center GPU Flex Series.
-
-<!--- 20. GPU Setup -->
-## Software Requirements:
-- Intel® Data Center GPU Flex Series
-- Follow [instructions](https://intel.github.io/intel-extension-for-pytorch/xpu/latest/tutorials/installation.html) to install the latest IPEX version and other prerequisites.
-
-- Intel® oneAPI Base Toolkit: Need to install components of Intel® oneAPI Base Toolkit
-  - Intel® oneAPI DPC++ Compiler
-  - Intel® oneAPI Threading Building Blocks (oneTBB)
-  - Intel® oneAPI Math Kernel Library (oneMKL)
-  - Follow [instructions](https://www.intel.com/content/www/us/en/developer/tools/oneapi/base-toolkit-download.html?operatingsystem=linux&distributions=offline) to download and install the latest oneAPI Base Toolkit.
-
-  - Set environment variables for Intel® oneAPI Base Toolkit: 
-    Default installation location `{ONEAPI_ROOT}` is `/opt/intel/oneapi` for root account, `${HOME}/intel/oneapi` for other accounts
-    ```bash
-    source {ONEAPI_ROOT}/compiler/latest/env/vars.sh
-    source {ONEAPI_ROOT}/mkl/latest/env/vars.sh
-    source {ONEAPI_ROOT}/tbb/latest/env/vars.sh
-    source {ONEAPI_ROOT}/mpi/latest/env/vars.sh
-    source {ONEAPI_ROOT}/ccl/latest/env/vars.sh
-    ```
-
-<!--- 30. Datasets -->
-## Datasets
-
-The [VOC2007](http://host.robots.ox.ac.uk/pascal/VOC/voc2007/) validation dataset is used.
-
-Download and extract the VOC2007 dataset from http://host.robots.ox.ac.uk/pascal/VOC/voc2007/,
-After extracting the data, your folder structure should look something like this:
-
-```
-VOC2007
-├── Annotations
-│   ├── 000038.xml    
-│   ├── 000724.xml
-│   ├── 001440.xml
-│   └── ...
-├── ImageSets
-│   ├── Layout    
-│   ├── Main
-│   └── Segmentation
-├── SegmentationClass
-│   ├── 005797.png   
-│   ├── 007415.png 
-│   ├── 006581.png 
-│   └── ...
-├── SegmentationObject
-│   ├── 005797.png    
-│   ├── 006581.png
-│   ├── 007415.png
-│   └── ...
-└── JPEGImages
-    ├── 002832.jpg    
-    ├── 003558.jpg
-    ├── 004262.jpg
-    └── ...
-```
-The folder should be set as the `DATASET_DIR`
-(for example: `export DATASET_DIR=/home/<user>/VOC2007`).
-
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| `inference_with_dummy_data.sh` | Inference with dummy data, for int8 blocked channel first on Flex series 170 |
-| `flex_multi_card_batch_inference.sh` | Inference with dummy data,for int8 and given batch size blocked channel first on Flex series 140 |
-| `flex_multi_card_online_inference.sh` | Online Inference with dummy data,for int8 blocked channel first on Flex series 140 | 
-
-
-<!--- 50. Baremetal -->
-## Run the model
-Install the following pre-requisites:
-* Create and activate virtual environment.
-  ```bash
-  virtualenv -p python <virtualenv_name>
-  source <virtualenv_name>/bin/activate
-  ```
-* Clone the Model Zoo repository:
-  ```bash
-  git clone https://github.com/IntelAI/models.git
-  ```
-
-* Navigate to SSD-Mobilenet inference directory and install model specific dependencies for the workload:
-  ```bash
-  cd models
-  ./quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/setup.sh
-  ```
-* Download the dataset label file, and set the "label" environment variable to point to where it was saved (for example: `export label=/home/<user>/voc-model-labels.txt`):
-  ```bash
-  wget https://storage.googleapis.com/models-hao/voc-model-labels.txt
-  ```
-
-This snippet shows how to run the inference quickstart script. The inference script
-will download the model weights to the directory location set in 'PRETRAINED_MODEL'.
-
-```
-### Run the model on Baremetal
-Set environment variables:
-export DATASET_DIR=<Path to the VOC2007 folder>
-export OUTPUT_DIR=<Path to save the output logs>
-export PRETRAINED_MODEL=<path to directory where the model weights will be loaded>
-export label=<path to label.txt file>
-
-Run the inference script, only int8 precision is supported:
-./quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/inference_with_dummy_data.sh
-```
-To execute `flex_multi_card_batch_inference.sh` and `flex_multi_card_online_inference.sh` on Flex series 140, install the following components 
-
-```bash
-apt-get update && \
-apt-get install -y --no-install-recommends --fix-missing parallel pciutils numactl 
-```
-Then execute the quickstart scripts. For batch inference, the default batch size for Flex 140 is 256.
-```bash
-./quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/flex_multi_card_batch_inference.sh 
-./quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/flex_multi_card_online_inference.sh
-```
-
-<!--- 80. License -->
-## License
-
-[LICENSE](/LICENSE)
diff --git a/quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/flex_multi_card_batch_inference.sh b/quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/flex_multi_card_batch_inference.sh
deleted file mode 100755
index 68b9f2283..000000000
--- a/quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/flex_multi_card_batch_inference.sh
+++ /dev/null
@@ -1,92 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2023 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-MODEL_DIR=${MODEL_DIR-$PWD}
-BATCH_SIZE=${BATCH_SIZE-256}
-NUM_ITERATIONS=${NUM_ITERATIONS:-500}
-
-echo 'label='$label
-
-if [[ -z "${DATASET_DIR}" ]]; then
-  echo "The required environment variable DATASET_DIR has not been set"
-  exit 1
-fi
-
-calculate_throughput() {
-  file_loc=$1
-  batch_size=$2
-  itr=$( cat $file_loc | awk '{print $1}' | tr -d "[]" | sort -n | tail -n1 )
-  for i in $( eval echo {1..$itr} )
-  do
-  fps=$(grep -rnw [$i] $file_loc | grep 'Inference time' | tail -400 | awk -v batch_size="$batch_size" -F' ' '{sum+=$NF;} END{print batch_size/(sum/400)} ')
-  echo 'FPS: '$fps | tee -a $file_loc
-  done
-  total_fps=$(cat $file_loc | grep FPS | awk '{print $2}' | awk '{ sum_total += $1 } END { print sum_total }' )
-  echo 'Total FPS: '$total_fps | tee -a $file_loc
-}
-# Create the model weights directory, if it doesn't already exist
-mkdir -p ${MODEL_DIR}/PRETRAINED_MODEL
-
-export OverrideDefaultFP64Settings=1 
-export IGC_EnableDPEmulation=1 
-
-export CFESingleSliceDispatchCCSMode=1
-export IPEX_ONEDNN_LAYOUT=1
-export IPEX_LAYOUT_OPT=1
-export IPEX_XPU_ONEDNN_LAYOUT=1
-
-# Download the weights file if it does not already exist
-WEIGHTS_FILE="${MODEL_DIR}/PRETRAINED_MODEL/mobilenet-v1-ssd-mp-0_675.pth"
-wget https://drive.google.com/uc?id=1pSPLnWGGNs3kV_YSxr4vsmSvDCLpUsEr -O $WEIGHTS_FILE
-
-
-# Create an array of input directories that are expected and then verify that they exist
-declare -A input_envs
-
-input_envs[label]=${label}
-
-if [[ -z $OUTPUT_DIR ]]; then
-  echo "The required environment variable OUTPUT_DIR has not been set"
-  exit 1
-fi
-
-# Create the output directory, if it doesn't already exist
-mkdir -p $OUTPUT_DIR
-
-declare -a str
-device_id=$( lspci | grep -i display | sed -n '1p' | awk '{print $7}' )
-num_devs=$(lspci | grep -i display | awk '{print $7}' | wc -l)
-
-if [[ ${device_id} == "56c1" ]]; then
-    for i in $( eval echo {0..$((num_devs-1))} )
-    do
-        str+=("ZE_AFFINITY_MASK="${i}" python -u ${MODEL_DIR}/models/object_detection/pytorch/ssd-mobilenet/inference/gpu/eval_ssd.py \
-        --net mb1-ssd \
-        --dataset ${DATASET_DIR} \
-        --trained_model ${WEIGHTS_FILE} \
-        --label_file ${label} \
-        --dummy 1 \
-        --batch_size ${BATCH_SIZE} \
-        --benchmark 1 \
-        --num-iterations ${NUM_ITERATIONS} \
-        --int8" )
-    done
-    echo "ssd-mobilenet dummy data inference  nchw on Flex Series 140"
-parallel --lb -d, --tagstring "[{#}]" ::: \
-"${str[@]}" 2>&1 | tee $OUTPUT_DIR/ssd_mobilenetv1_dummy_data_xpu_inf_c0_c1_${BATCH_SIZE}.log
-calculate_throughput $OUTPUT_DIR/ssd_mobilenetv1_dummy_data_xpu_inf_c0_c1_${BATCH_SIZE}.log ${BATCH_SIZE}
-fi
diff --git a/quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/flex_multi_card_online_inference.sh b/quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/flex_multi_card_online_inference.sh
deleted file mode 100755
index 0fcf2b2dc..000000000
--- a/quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/flex_multi_card_online_inference.sh
+++ /dev/null
@@ -1,95 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2023 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-MODEL_DIR=${MODEL_DIR-$PWD}
-BATCH_SIZE=${BATCH_SIZE-1}
-NUM_ITERATIONS=${NUM_ITERATIONS:-5000}
-echo 'label='$label
-
-if [[ -z "${DATASET_DIR}" ]]; then
-  echo "The required environment variable DATASET_DIR has not been set"
-  exit 1
-fi
-
-# Create the model weights directory, if it doesn't already exist
-mkdir -p ${MODEL_DIR}/PRETRAINED_MODEL
-
-export OverrideDefaultFP64Settings=1 
-export IGC_EnableDPEmulation=1 
-
-export CFESingleSliceDispatchCCSMode=1
-export IPEX_ONEDNN_LAYOUT=1
-export IPEX_LAYOUT_OPT=1
-export IPEX_XPU_ONEDNN_LAYOUT=1
-
-# Download the weights file if it does not already exist
-WEIGHTS_FILE="${MODEL_DIR}/PRETRAINED_MODEL/mobilenet-v1-ssd-mp-0_675.pth"
-wget https://drive.google.com/uc?id=1pSPLnWGGNs3kV_YSxr4vsmSvDCLpUsEr -O $WEIGHTS_FILE
-
-# Create an array of input directories that are expected and then verify that they exist
-declare -A input_envs
-
-input_envs[label]=${label}
-
-if [[ -z $OUTPUT_DIR ]]; then
-  echo "The required environment variable OUTPUT_DIR has not been set"
-  exit 1
-fi
-
-calculate_throughput() {
-  file_loc=$1
-  batch_size=$2
-  itr=$( cat $file_loc | awk '{print $1}' | tr -d "[]" | sort -n | tail -n1 )
-  for i in $( eval echo {1..$itr} )
-  do
-  fps=$(grep -rnw [$i] $file_loc | grep 'Inference time' | tail -4900 | awk -v batch_size="$batch_size" -F' ' '{sum+=$NF;} END{print batch_size/(sum/4900)} ')
-  echo 'FPS: '$fps | tee -a $file_loc
-  done
-  total_fps=$(cat $file_loc | grep FPS | awk '{print $2}' | awk '{ sum_total += $1 } END { print sum_total }' )
-  echo 'Total FPS: '$total_fps | tee -a $file_loc
-}
-# Create the output directory, if it doesn't already exist
-mkdir -p $OUTPUT_DIR
-
-declare -a str
-device_id=$( lspci | grep -i display | sed -n '1p' | awk '{print $7}' )
-num_devs=$(lspci | grep -i display | awk '{print $7}' | wc -l)
-num_threads=1
-k=0
-if [[ ${device_id} == "56c1" ]]; then
-    for i in $( eval echo {0..$((num_devs-1))} )
-    do
-    for j in $( eval echo {1..$num_threads} )
-    do
-            str+=("ZE_AFFINITY_MASK="${i}" numactl -C ${k} -l  python -u ${MODEL_DIR}/models/object_detection/pytorch/ssd-mobilenet/inference/gpu/eval_ssd.py \
-                  --net mb1-ssd \
-                  --dataset ${DATASET_DIR} \
-                  --trained_model ${WEIGHTS_FILE} \
-                  --label_file ${label} \
-                  --dummy 1 \
-                  --batch_size ${BATCH_SIZE} \
-                  --benchmark 1 \
-                  --num-iterations ${NUM_ITERATIONS} \
-                  --int8 ")
-    ((k=k+1))
-    done
-    done
-echo "ssd-mobilenet dummy data inference  nchw on Flex Series 140"
-parallel --lb -d, --tagstring "[{#}]" ::: \
-"${str[@]}" 2>&1 | tee $OUTPUT_DIR/ssd_mobilenetv1_dummy_data_xpu_inf_c0_c1_${BATCH_SIZE}.log
-calculate_throughput $OUTPUT_DIR/ssd_mobilenetv1_dummy_data_xpu_inf_c0_c1_${BATCH_SIZE}.log ${BATCH_SIZE}
-fi
diff --git a/quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/inference_with_dummy_data.sh b/quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/inference_with_dummy_data.sh
deleted file mode 100755
index 1a15661f4..000000000
--- a/quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/inference_with_dummy_data.sh
+++ /dev/null
@@ -1,76 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2022 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-MODEL_DIR=${MODEL_DIR-$PWD}
-BATCH_SIZE=${BATCH_SIZE-1024}
-NUM_ITERATIONS=${NUM_ITERATIONS-500}
-
-echo 'label='$label
-
-if [[ -z "${DATASET_DIR}" ]]; then
-  echo "The required environment variable DATASET_DIR has not been set"
-  exit 1
-fi
-
-# Create the model weights directory, if it doesn't already exist
-mkdir -p ${MODEL_DIR}/PRETRAINED_MODEL
-
-export OverrideDefaultFP64Settings=1 
-export IGC_EnableDPEmulation=1 
-
-export CFESingleSliceDispatchCCSMode=1
-export IPEX_ONEDNN_LAYOUT=1
-export IPEX_LAYOUT_OPT=1
-export IPEX_XPU_ONEDNN_LAYOUT=1
-
-
-calculate_throughput() {
-  file_loc=$1
-  batch_size=$2
-  fps=$(cat $file_loc | grep 'Inference time' | awk '{print $3}' | tail -400 | awk -v batch_size="$batch_size" -F' ' '{sum+=$NF;} END{print batch_size/(sum/400)} ')
-  echo 'FPS: '$fps | tee -a $file_loc
-}
-
-# Download the weights file if it does not already exist
-WEIGHTS_FILE="${MODEL_DIR}/PRETRAINED_MODEL/mobilenet-v1-ssd-mp-0_675.pth"
-wget https://drive.google.com/uc?id=1pSPLnWGGNs3kV_YSxr4vsmSvDCLpUsEr -O $WEIGHTS_FILE
-
-# Create an array of input directories that are expected and then verify that they exist
-declare -A input_envs
-
-input_envs[label]=${label}
-
-if [[ -z $OUTPUT_DIR ]]; then
-  echo "The required environment variable OUTPUT_DIR has not been set"
-  exit 1
-fi
-
-# Create the output directory, if it doesn't already exist
-mkdir -p $OUTPUT_DIR
-
-echo "ssd-mobilenet dummy data inference  nchw"
-IPEX_XPU_ONEDNN_LAYOUT=1 python -u ${MODEL_DIR}/models/object_detection/pytorch/ssd-mobilenet/inference/gpu/eval_ssd.py \
-  --net mb1-ssd \
-  --dataset ${DATASET_DIR} \
-  --trained_model ${WEIGHTS_FILE} \
-  --label_file ${label} \
-  --dummy 1 \
-  --batch_size ${BATCH_SIZE} \
-  --benchmark 1 \
-  --num-iterations ${NUM_ITERATIONS} \
-  --int8 2>&1 | tee $OUTPUT_DIR/ssd_mobilenetv1_dummy_data_xpu_${BATCH_SIZE}.log
-calculate_throughput $OUTPUT_DIR/ssd_mobilenetv1_dummy_data_xpu_${BATCH_SIZE}.log ${BATCH_SIZE}
diff --git a/quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/setup.sh b/quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/setup.sh
deleted file mode 100755
index 61d35512d..000000000
--- a/quickstart/object_detection/pytorch/ssd-mobilenet/inference/gpu/setup.sh
+++ /dev/null
@@ -1,21 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2022 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-
-pip install opencv-python-headless
-pip install pandas
-
diff --git a/quickstart/object_detection/pytorch/ssd-resnet34/inference/cpu/DEVCATALOG.md b/quickstart/object_detection/pytorch/ssd-resnet34/inference/cpu/DEVCATALOG.md
new file mode 100644
index 000000000..c5e77ff87
--- /dev/null
+++ b/quickstart/object_detection/pytorch/ssd-resnet34/inference/cpu/DEVCATALOG.md
@@ -0,0 +1,123 @@
+# PyTorch ResNet34 inference
+
+## Description 
+This document has instructions for running ResNet34 inference using Intel Extension for PyTorch. 
+
+## Pull Command
+
+Docker image based on CentOS Stream8
+```
+docker pull intel/object-detection:centos-pytorch-cpu-ssd-resnet34-inference
+```
+
+Docker image based on Ubuntu 22.04
+```
+docker pull intel/object-detection:ubuntu-pytorch-cpu-ssd-resnet34-inference
+```
+
+## Quick Start Scripts
+| Script name | Description |
+|-------------|-------------|
+| `inference_realtime.sh` | Runs multi instance realtime inference using 4 cores per instance for the specified precision (fp32, avx-fp32, int8, avx-int8, bf32 or bf16). |
+| `inference_throughput.sh` | Runs multi instance batch inference using 1 instance per socket for the specified precision (fp32, avx-fp32, int8, avx-int8, bf32 or bf16). |
+| `accuracy.sh` | Measures the inference accuracy (providing a `DATASET_DIR` environment variable is required) for the specified precision (fp32, avx-fp32, int8, avx-int8, bf32 or bf16). |
+
+**Note:** The `avx-int8` and `avx-fp32` precisions run the same scripts as `int8` and `fp32`, except that the `DNNL_MAX_CPU_ISA` environment variable is unset. The environment variable is otherwise set to `DNNL_MAX_CPU_ISA=AVX512_CORE_AMX`
+
+## Datasets
+Download and extract the 2017 training/validation images and annotations from the [COCO dataset website](https://cocodataset.org/#download) to a `coco` folder and unzip the files. After extracting the zip files, your dataset directory structure should look something like this:
+```
+coco
+├── annotations
+│   ├── captions_train2017.json
+│   ├── captions_val2017.json
+│   ├── instances_train2017.json
+│   ├── instances_val2017.json
+│   ├── person_keypoints_train2017.json
+│   └── person_keypoints_val2017.json
+├── train2017
+│   ├── 000000454854.jpg
+│   ├── 000000137045.jpg
+│   ├── 000000129582.jpg
+│   └── ...
+└── val2017
+    ├── 000000000139.jpg
+    ├── 000000000285.jpg
+    ├── 000000000632.jpg
+    └── ...
+```
+The parent of the `annotations`, `train2017`, and `val2017` directory (in this example `coco`) is the directory that should be used when setting the `image` environment variable for YOLOv4 (for example: `export image=/home/<user>/coco/val2017/000000581781.jpg`).In addition, we should also set the `size` environment to match the size of image.(for example: `export size=416`)
+
+## Pre-Trained Model
+Download the ResNet34 Pre-trained model from the following link. Set the `CHECKPOINT_DIR` to point to the model file. 
+```
+dir=$(pwd)
+CHECKPOINT_DIR=${PWD}
+mkdir -p ${CHECKPOINT_DIR}/pretrained
+cd ${CHECKPOINT_DIR}/pretrained
+wget --no-check-certificate 'https://docs.google.com/uc?export=download&id=13kWgEItsoxbVKUlkQz4ntjl1IZGk6_5Z'  -O 'resnet34-ssd1200.pth'
+cd $dir
+```
+
+## Docker Run
+(Optional) Export related proxy into docker environment.
+```bash
+export DOCKER_RUN_ENVS="-e ftp_proxy=${ftp_proxy} \
+  -e FTP_PROXY=${FTP_PROXY} -e http_proxy=${http_proxy} \
+  -e HTTP_PROXY=${HTTP_PROXY} -e https_proxy=${https_proxy} \
+  -e HTTPS_PROXY=${HTTPS_PROXY} -e no_proxy=${no_proxy} \
+  -e NO_PROXY=${NO_PROXY} -e socks_proxy=${socks_proxy} \
+  -e SOCKS_PROXY=${SOCKS_PROXY}"
+```
+
+To run ResNet34 inference, set environment variables to specify the dataset directory, precision,checkpoint directory and an output directory. 
+
+```bash
+export OS=<provide either centos or ubuntu>
+export DATASET_DIR=<path to the dataset>
+export OUTPUT_DIR=<directory where log files will be written>
+export SCRIPT=quickstart/<specify the script to run>
+export PRECISION=<specify the precision to run>
+export CHECKPOINT_DIR=${PWD}
+
+IMAGE_NAME=intel/object-detection:${OS}-pytorch-cpu-ssd-resnet34-inference
+WORKDIR=/workspace/pytorch-ssd-resnet34-inference
+DOCKER_ARGS="--privileged --init -it"
+
+docker run --rm \
+  --env DATASET_DIR=${DATASET_DIR} \
+  --env CHECKPOINT_DIR=${CHECKPOINT_DIR} \
+  --env OUTPUT_DIR=${OUTPUT_DIR} \
+  --env http_proxy=${http_proxy} \
+  --env https_proxy=${https_proxy} \
+  --env no_proxy=${no_proxy} \
+  --volume ${DATASET_DIR}:${DATASET_DIR} \
+  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
+  --volume ${CHECKPOINT_DIR}:${CHECKPOINT_DIR} \
+  ${DOCKER_RUN_ENVS} \
+  --shm-size 8G \
+  -w ${WORKDIR} \
+  ${DOCKER_ARGS} \
+  $IMAGE_NAME \
+  /bin/bash $SCRIPT $PRECISION
+```
+
+## Documentation and Sources
+#### Get Started​
+[Docker* Repository](https://hub.docker.com/r/intel/object-detection)
+
+[Main GitHub*](https://github.com/IntelAI/models)
+
+[Release Notes](https://github.com/IntelAI/models/releases)
+
+[Get Started Guide](https://github.com/IntelAI/models/blob/master/quickstart/quickstart/object_detection/pytorch/ssd-resnet34/inference/cpu/DEVCATALOG.md)
+
+#### Code Sources
+[Dockerfile](https://github.com/IntelAI/models/tree/master/docker/pyt-cpu)
+
+[Report Issue](https://community.intel.com/t5/Intel-Optimized-AI-Frameworks/bd-p/optimized-ai-frameworks)
+
+## License Agreement
+LEGAL NOTICE: By accessing, downloading or using this software and any required dependent software (the “Software Package”), you agree to the terms and conditions of the software license agreements for the Software Package, which may also include notices, disclaimers, or license terms for third party software included with the Software Package. Please refer to the [license](https://github.com/IntelAI/models/tree/master/third_party) file for additional details.
+
+[View All Containers and Solutions 🡢](https://www.intel.com/content/www/us/en/developer/tools/software-catalog/containers.html?s=Newest)
diff --git a/quickstart/object_detection/pytorch/ssd-resnet34/inference/cpu/README.md b/quickstart/object_detection/pytorch/ssd-resnet34/inference/cpu/README.md
index fe1428319..ea9ce3e25 100644
--- a/quickstart/object_detection/pytorch/ssd-resnet34/inference/cpu/README.md
+++ b/quickstart/object_detection/pytorch/ssd-resnet34/inference/cpu/README.md
@@ -3,29 +3,32 @@
 ## Description
 This document has instructions for running SSD-ResNet34 Inference using Intel-optimized PyTorch.
 
+## Quick Start Scripts
+| Script name | Description |
+|-------------|-------------|
+| `inference_realtime.sh` | Runs multi instance realtime inference using 4 cores per instance for the specified precision (fp32, avx-fp32, int8, avx-int8, bf32 or bf16). |
+| `inference_throughput.sh` | Runs multi instance batch inference using 1 instance per socket for the specified precision (fp32, avx-fp32, int8, avx-int8, bf32 or bf16). |
+| `accuracy.sh` | Measures the inference accuracy (providing a `DATASET_DIR` environment variable is required) for the specified precision (fp32, avx-fp32, int8, avx-int8, bf32 or bf16). |
+
+**Note:** The `avx-int8` and `avx-fp32` precisions run the same scripts as `int8` and `fp32`, except that the `DNNL_MAX_CPU_ISA` environment variable is unset. The environment variable is otherwise set to `DNNL_MAX_CPU_ISA=AVX512_CORE_AMX`
+* Set ENV to use AMX:
+  ```
+  export DNNL_MAX_CPU_ISA=AVX512_CORE_AMX
+  ```
+
+To do the int8 calibration bash bare_metal_int8_calibration.sh int8 <file where to save the calibrated model> <steps to run calibration>, for example bash bare_metal_int8_calibration.sh int8 test.json 10.
+
 ## Bare Metal
 ### General setup
 
-Follow [link](/docs/general/pytorch/BareMetalSetup.md) to install Miniconda and build Pytorch, IPEX, TorchVison and Jemalloc.
+Follow [link](/docs/general/pytorch/BareMetalSetup.md) to install Miniconda and build Pytorch, IPEX, TorchVison and Jemalloc and TCmalloc
 
 ### Model Specific Setup
-* Install dependencies
-  ```
-  pip install matplotlib Pillow pycocotools defusedxml
-  ```
-
-* Download pretrained model
-  ```
-  cd <path to your clone of the model zoo>/quickstart/object_detection/pytorch/ssd-resnet34/inference/cpu
-  export CHECKPOINT_DIR=<directory where to save the pretrained model>
-  bash download_model.sh
-  ```
-
-* Set Jemalloc Preload for better performance
+* Set Jemalloc and tcmalloc Preload for better performance
 
   The jemalloc should be built from the [General setup](#general-setup) section.
   ```
-  export LD_PRELOAD="<path to the jemalloc directory>/lib/libjemalloc.so":$LD_PRELOAD
+  export LD_PRELOAD="<path to the jemalloc directory>/lib/libjemalloc.so":"path_to/tcmalloc/lib/libtcmalloc.so":$LD_PRELOAD
   export MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
   ```
 
@@ -33,58 +36,49 @@ Follow [link](/docs/general/pytorch/BareMetalSetup.md) to install Miniconda and
 
   IOMP should be installed in your conda env from the [General setup](#general-setup) section.
   ```
+  pip install packaging intel-openmp
   export LD_PRELOAD=<path to the intel-openmp directory>/lib/libiomp5.so:$LD_PRELOAD
   ```
 
-* Set ENV to use AMX if you are using SPR
-  ```
-  export DNNL_MAX_CPU_ISA=AVX512_CORE_AMX
-  ```
+## Run the model
+Once all the setup is done, the Intel® AI Reference Models can be used to run a [quickstart script](#quick-start-scripts).
+Ensure that you have an enviornment variables set to point to the dataset directory,
+the downloaded pretrained model, and an output directory.
 
-## Datasets
+```
+# Clone the Intel® AI Reference Models repo and set the MODEL_DIR
+git clone https://github.com/IntelAI/models.git
+cd models
+export MODEL_DIR=$(pwd)
 
+# Dataset
 Download the 2017 [COCO dataset](https://cocodataset.org) using the `download_dataset.sh` script.
 Export the `DATASET_DIR` environment variable to specify the directory where the dataset
 will be downloaded. This environment variable will be used again when running quickstart scripts.
-```
-cd <path to your clone of the model zoo>/quickstart/object_detection/pytorch/ssd-resnet34/inference/cpu
+
+cd quickstart/object_detection/pytorch/ssd-resnet34/inference/cpu
 export DATASET_DIR=<directory where the dataset will be saved>
 bash download_dataset.sh
-```
-
-## Quick Start Scripts
+cd - 
+cd ${MODEL_DIR}
 
-|  DataType   | Throughput  |  Latency    |   Accuracy  |
-| ----------- | ----------- | ----------- | ----------- |
-| FP32        | bash inference_throughput.sh fp32 | bash inference_realtime.sh fp32 | bash accuracy.sh fp32 |
-| BF16        | bash inference_throughput.sh bf16 | bash inference_realtime.sh bf16 | bash accuracy.sh bf16 |
-| INT8        | bash inference_throughput.sh int8 | bash inference_realtime.sh int8 | bash accuracy.sh int8 |
-| BF32        | bash inference_throughput.sh bf32 | bash inference_realtime.sh bf32 | bash accuracy.sh bf32 |
+# install model specific dependencies
+./quickstart/object_detection/pytorch/ssd-resnet34/inference/cpu/setup.sh
 
-To do the int8 calibration `bash bare_metal_int8_calibration.sh int8 <file where to save the calibrated model> <steps to run calibration>`, for example `bash bare_metal_int8_calibration.sh int8 test.json 10`.
-
-## Run the model
-
-Follow the instructions above to setup your bare metal environment, download and
-preprocess the dataset, and do the model specific setup. Once all the setup is done,
-the Model Zoo can be used to run a [quickstart script](#quick-start-scripts).
-Ensure that you have an enviornment variables set to point to the dataset directory,
-the downloaded pretrained model, and an output directory.
-
-```
-# Clone the model zoo repo and set the MODEL_DIR
-git clone https://github.com/IntelAI/models.git
-cd models
-export MODEL_DIR=$(pwd)
+# Download pretrained model
+export CHECKPOINT_DIR=<directory where to save the pretrained model>
+bash quickstart/object_detection/pytorch/ssd-resnet34/inference/cpu/download_model.sh
 
 # Env vars
 export DATASET_DIR=<path to the COCO dataset>
-export CHECKPOINT_DIR=<path to the pretrained model>
 export OUTPUT_DIR=<path to an output directory>
+export PRECISION=<select from :- int8, avx-int8, fp32, avx-fp32, bf16, or bf32>
+
+# Optional environemnt variables:
+export BATCH_SIZE=<set a value for batch size, else it will run with default batch size>
 
-# Run a quickstart script (for example, FP32 batch inference)
-cd ${MODEL_DIR}/quickstart/object_detection/pytorch/ssd-resnet34/inference/cpu
-bash inference_throughput.sh fp32
+# Run a quickstart script:
+./quickstart/object_detection/pytorch/ssd-resnet34/inference/cpu/<quickstart_script.sh>
 ```
 
 <!--- 80. License -->
diff --git a/quickstart/object_detection/pytorch/ssd-resnet34/inference/cpu/accuracy.sh b/quickstart/object_detection/pytorch/ssd-resnet34/inference/cpu/accuracy.sh
index 1cd987b8b..3b88e82ea 100755
--- a/quickstart/object_detection/pytorch/ssd-resnet34/inference/cpu/accuracy.sh
+++ b/quickstart/object_detection/pytorch/ssd-resnet34/inference/cpu/accuracy.sh
@@ -38,26 +38,31 @@ if [ ! -d "${OUTPUT_DIR}" ]; then
   exit 1
 fi
 
-if [[ "$1" == *"avx"* ]]; then
+if [ -z "${PRECISION}" ]; then
+  echo "PRECISION is not set"
+  exit 1
+fi
+
+if [[ "$PRECISION" == *"avx"* ]]; then
     unset DNNL_MAX_CPU_ISA
 fi
 
 ARGS=""
-if [[ "$1" == "int8" || "$1" == "avx-int8" ]]; then
+if [[ "$PRECISION" == "int8" || "$PRECISION" == "avx-int8" ]]; then
     ARGS="$ARGS --int8"
     ARGS="$ARGS --seed 1 --threshold 0.2 --configure ${MODEL_DIR}/models/object_detection/pytorch/ssd-resnet34/inference/cpu/pytorch_default_recipe_ssd_configure.json"
     export DNNL_GRAPH_CONSTANT_CACHE=1
     echo "### running int8 datatype"
-elif [[ "$1" == "bf16" ]]; then
+elif [[ "$PRECISION" == "bf16" ]]; then
     ARGS="$ARGS --autocast"
     echo "### running bf16 datatype"
-elif [[ "$1" == "fp32" || "$1" == "avx-fp32" ]]; then
+elif [[ "$PRECISION" == "fp32" || "$PRECISION" == "avx-fp32" ]]; then
     echo "### running fp32 datatype"
-elif [[ "$1" == "bf32" ]]; then
+elif [[ "$PRECISION" == "bf32" ]]; then
     ARGS="$ARGS --bf32"
     echo "### running bf32 datatype"
 else
-    echo "The specified precision '$1' is unsupported."
+    echo "The specified precision '$PRECISION' is unsupported."
     echo "Supported precisions are: fp32, avx-fp32, bf16, int8, bf32, and avx-int8"
     exit 1
 fi
@@ -69,7 +74,6 @@ export KMP_AFFINITY=granularity=fine,compact,1,0
 
 rm -rf ${OUTPUT_DIR}/ssdresnet34_${PRECISION}_inference_accuracy*
 
-PRECISION=$1
 weight_sharing=true
 if [ -z "${WEIGHT_SHAREING}" ]; then
   weight_sharing=false
@@ -94,7 +98,8 @@ if [ "$weight_sharing" = true ]; then
     BATCH_PER_STREAM=1
     CORES_PER_STREAM=1
     STREAM_PER_INSTANCE=`expr $CORES / $CORES_PER_STREAM`
-    BATCH_SIZE=`expr $BATCH_PER_STREAM \* $STREAM_PER_INSTANCE`
+    # Runs with default value when BATCH_SIZE is not set
+    BATCH_SIZE=${BATCH_SIZE:- `expr $BATCH_PER_STREAM \* $STREAM_PER_INSTANCE`}
 
     export OMP_NUM_THREADS=$CORES_PER_STREAM
 
@@ -137,4 +142,4 @@ fi
 # For the summary of results
 
 accuracy=$(grep 'Accuracy:' ${OUTPUT_DIR}/ssdresnet34_${PRECISION}_inference_accuracy* |sed -e 's/.*Accuracy//;s/[^0-9.]//g')
-echo ""SSD-RN34";"accuracy";$1; ${BATCH_SIZE};${accuracy}" | tee -a ${OUTPUT_DIR}/summary.log
+echo ""SSD-RN34";"accuracy";$PRECISION; ${BATCH_SIZE};${accuracy}" | tee -a ${OUTPUT_DIR}/summary.log
diff --git a/quickstart/object_detection/pytorch/ssd-resnet34/inference/cpu/inference_realtime.sh b/quickstart/object_detection/pytorch/ssd-resnet34/inference/cpu/inference_realtime.sh
index b0f9dd84f..b1b95700d 100755
--- a/quickstart/object_detection/pytorch/ssd-resnet34/inference/cpu/inference_realtime.sh
+++ b/quickstart/object_detection/pytorch/ssd-resnet34/inference/cpu/inference_realtime.sh
@@ -38,26 +38,31 @@ if [ ! -d "${OUTPUT_DIR}" ]; then
   exit 1
 fi
 
-if [[ "$1" == *"avx"* ]]; then
+if [ -z "${PRECISION}" ]; then
+  echo "PRECISION is not set"
+  exit 1
+fi
+
+if [[ "$PRECISION" == *"avx"* ]]; then
     unset DNNL_MAX_CPU_ISA
 fi
 
 ARGS=""
-if [[ "$1" == "int8" || "$1" == "avx-int8" ]]; then
+if [[ "$PRECISION" == "int8" || "$PRECISION" == "avx-int8" ]]; then
     ARGS="$ARGS --int8"
     ARGS="$ARGS --seed 1 --threshold 0.2 --configure ${MODEL_DIR}/models/object_detection/pytorch/ssd-resnet34/inference/cpu/pytorch_default_recipe_ssd_configure.json"
     export DNNL_GRAPH_CONSTANT_CACHE=1
     echo "### running int8 datatype"
-elif [[ "$1" == "bf16" ]]; then
+elif [[ "$PRECISION" == "bf16" ]]; then
     ARGS="$ARGS --autocast"
     echo "### running bf16 datatype"
-elif [[ "$1" == "fp32" || "$1" == "avx-fp32" ]]; then
+elif [[ "$PRECISION" == "fp32" || "$PRECISION" == "avx-fp32" ]]; then
     echo "### running fp32 datatype"
-elif [[ "$1" == "bf32" ]]; then
+elif [[ "$PRECISION" == "bf32" ]]; then
     ARGS="$ARGS --bf32"
     echo "### running bf32 datatype"
 else
-    echo "The specified precision '$1' is unsupported."
+    echo "The specified precision '$PRECISION' is unsupported."
     echo "Supported precisions are: fp32, avx-fp32, bf16, int8, bf32, and avx-int8"
     exit 1
 fi
@@ -67,8 +72,8 @@ export USE_IPEX=1
 export KMP_BLOCKTIME=1
 export KMP_AFFINITY=granularity=fine,compact,1,0
 
-PRECISION=$1
-BATCH_SIZE=1
+# Runs with default value when BATCH_SIZE is not set:
+BATCH_SIZE=${BATCH_SIZE:- 1}
 
 rm -rf ${OUTPUT_DIR}/latency_log*
 
@@ -151,5 +156,5 @@ END   {
     sum = sum / i;
     printf("%.3f ms", sum);
 }')
-echo ""SSD-RN34";"latency";$1; ${BATCH_SIZE};${throughput}" | tee -a ${OUTPUT_DIR}/summary.log
-echo ""SSD-RN34";"p99_latency";$1; ${BATCH_SIZE};${p99_latency}" | tee -a ${OUTPUT_DIR}/summary.log
+echo ""SSD-RN34";"latency";$PRECISION; ${BATCH_SIZE};${throughput}" | tee -a ${OUTPUT_DIR}/summary.log
+echo ""SSD-RN34";"p99_latency";$PRECISION; ${BATCH_SIZE};${p99_latency}" | tee -a ${OUTPUT_DIR}/summary.log
diff --git a/quickstart/object_detection/pytorch/ssd-resnet34/inference/cpu/inference_throughput.sh b/quickstart/object_detection/pytorch/ssd-resnet34/inference/cpu/inference_throughput.sh
index 0993e4326..5743c73a5 100755
--- a/quickstart/object_detection/pytorch/ssd-resnet34/inference/cpu/inference_throughput.sh
+++ b/quickstart/object_detection/pytorch/ssd-resnet34/inference/cpu/inference_throughput.sh
@@ -38,7 +38,12 @@ if [ ! -d "${OUTPUT_DIR}" ]; then
   exit 1
 fi
 
-if [[ "$1" == *"avx"* ]]; then
+if [ -z "${PRECISION}" ]; then
+  echo "PRECISION is not set"
+  exit 1
+fi
+
+if [[ "$PRECISION" == *"avx"* ]]; then
     unset DNNL_MAX_CPU_ISA
 fi
 
@@ -46,27 +51,28 @@ CORES=`lscpu | grep Core | awk '{print $4}'`
 SOCKETS=`lscpu | grep Socket | awk '{print $2}'`
 TOTAL_CORES=`expr $CORES \* $SOCKETS`
 
-BATCH_SIZE=112
+# Runs with default value when BATCH SIZE is not set
+BATCH_SIZE=${BATCH_SIZE:-112}
 
 ARGS=""
-if [[ "$1" == "int8" || "$1" == "avx-int8" ]]; then
+if [[ "$PRECISION" == "int8" || "$PRECISION" == "avx-int8" ]]; then
     NUMA_NODES=`lscpu | grep 'NUMA node(s)' | awk '{print $3}'`
     CORES_PER_NODE=`expr $TOTAL_CORES / $NUMA_NODES`
-    BATCH_SIZE=`expr $CORES_PER_NODE \* 2`
+    BATCH_SIZE=${BATCH_SIZE:-`expr $CORES_PER_NODE \* 2`}
     ARGS="$ARGS --int8"
     ARGS="$ARGS --seed 1 --threshold 0.2 --configure ${MODEL_DIR}/models/object_detection/pytorch/ssd-resnet34/inference/cpu/pytorch_default_recipe_ssd_configure.json"
     export DNNL_GRAPH_CONSTANT_CACHE=1
     echo "### running int8 datatype"
-elif [[ "$1" == "bf16" ]]; then
+elif [[ "$PRECISION" == "bf16" ]]; then
     ARGS="$ARGS --autocast"
     echo "### running bf16 datatype"
-elif [[ "$1" == "fp32" || "$1" == "avx-fp32" ]]; then
+elif [[ "$PRECISION" == "fp32" || "$PRECISION" == "avx-fp32" ]]; then
     echo "### running fp32 datatype"
-elif [[ "$1" == "bf32" ]]; then
+elif [[ "$PRECISION" == "bf32" ]]; then
     ARGS="$ARGS --bf32"
     echo "### running bf32 datatype"
 else
-    echo "The specified precision '$1' is unsupported."
+    echo "The specified precision '$PRECISION' is unsupported."
     echo "Supported precisions are: fp32, avx-fp32, bf16, int8, bf32, and avx-int8"
     exit 1
 fi
@@ -78,7 +84,6 @@ export KMP_AFFINITY=granularity=fine,compact,1,0
 
 rm -rf ${OUTPUT_DIR}/throughput_log*
 
-PRECISION=$1
 weight_sharing=true
 if [ -z "${WEIGHT_SHAREING}" ]; then
   weight_sharing=false
@@ -164,4 +169,4 @@ END   {
 sum = sum / i;
         printf("%.3f", sum);
 }')
-echo ""SSD-RN34";"throughput";$1; ${BATCH_SIZE};${throughput}" | tee -a ${OUTPUT_DIR}/summary.log
+echo ""SSD-RN34";"throughput";$PRECISION; ${BATCH_SIZE};${throughput}" | tee -a ${OUTPUT_DIR}/summary.log
diff --git a/quickstart/object_detection/pytorch/ssd-resnet34/inference/cpu/setup.sh b/quickstart/object_detection/pytorch/ssd-resnet34/inference/cpu/setup.sh
new file mode 100755
index 000000000..de6afd05e
--- /dev/null
+++ b/quickstart/object_detection/pytorch/ssd-resnet34/inference/cpu/setup.sh
@@ -0,0 +1,18 @@
+#!/bin/bash
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+pip install matplotlib Pillow pycocotools defusedxml
diff --git a/quickstart/object_detection/pytorch/ssd-resnet34/training/cpu/DEVCATALOG.md b/quickstart/object_detection/pytorch/ssd-resnet34/training/cpu/DEVCATALOG.md
new file mode 100644
index 000000000..f810508d6
--- /dev/null
+++ b/quickstart/object_detection/pytorch/ssd-resnet34/training/cpu/DEVCATALOG.md
@@ -0,0 +1,116 @@
+# PyTorch SSD-ResNet34 training
+
+## Description 
+This document has instructions for running SSD-ResNet34 training using Intel Extension for PyTorch. 
+
+## Pull Command
+
+Docker image based on CentOS Stream8
+```
+docker pull intel/object-detection:centos-pytorch-cpu-ssd-resnet34-training
+```
+Docker image based on Ubuntu 22.04
+```
+docker pull intel/object-detection:ubuntu-pytorch-cpu-ssd-resnet34-training
+```
+
+## Quick Start Scripts
+| Script name | Description |
+|-------------|-------------|
+| `throughput.sh` | Tests the training performance for SSD-ResNet34 for the specified precision (fp32, avx-fp32, bf32 or bf16). |
+| `accuracy.sh` | Tests the training accuracy for SSD-ResNet34 for the specified precision (fp32, avx-fp32, bf32 or bf16). |
+
+**Note**: The `avx-fp32` precision runs the same scripts as `fp32`, except that the `DNNL_MAX_CPU_ISA` environment variable is unset. The environment variable is otherwise set to `DNNL_MAX_CPU_ISA=AVX512_CORE_AMX`.
+
+## Datasets
+Download and extract the 2017 training/validation images and annotations from the [COCO dataset website](https://cocodataset.org/#download) to a `coco` folder and unzip the files. After extracting the zip files, your dataset directory structure should look something like this:
+```
+coco
+├── annotations
+│   ├── captions_train2017.json
+│   ├── captions_val2017.json
+│   ├── instances_train2017.json
+│   ├── instances_val2017.json
+│   ├── person_keypoints_train2017.json
+│   └── person_keypoints_val2017.json
+├── train2017
+│   ├── 000000454854.jpg
+│   ├── 000000137045.jpg
+│   ├── 000000129582.jpg
+│   └── ...
+└── val2017
+    ├── 000000000139.jpg
+    ├── 000000000285.jpg
+    ├── 000000000632.jpg
+    └── ...
+```
+
+## Pre-trained Model
+Download the pre-trained model and set `CHECKPOINT_DIR`
+```
+CHECKPOINT_DIR=${PWD}
+mkdir -p ${CHECKPOINT_DIR}/ssd; cd ${CHECKPOINT_DIR}/ssd
+curl -O https://download.pytorch.org/models/resnet34-333f7ec4.pth
+cd -
+```
+## Docker Run
+(Optional) Export related proxy into docker environment.
+```bash
+export DOCKER_RUN_ENVS="-e ftp_proxy=${ftp_proxy} \
+  -e FTP_PROXY=${FTP_PROXY} -e http_proxy=${http_proxy} \
+  -e HTTP_PROXY=${HTTP_PROXY} -e https_proxy=${https_proxy} \
+  -e HTTPS_PROXY=${HTTPS_PROXY} -e no_proxy=${no_proxy} \
+  -e NO_PROXY=${NO_PROXY} -e socks_proxy=${socks_proxy} \
+  -e SOCKS_PROXY=${SOCKS_PROXY}"
+```
+
+To run SSD-ResNet34, set environment variables to specify the dataset directory, precision and an output directory. 
+
+```bash
+export DATASET_DIR=<path to the dataset>
+export OUTPUT_DIR=<directory where log files will be written>
+export SCRIPT=quickstart/<specify the script to run>
+export PRECISION=<specify the precision to run>
+export CHECKPOINT_DIR=<path to pre-trained model>
+
+IMAGE_NAME=intel/object-detection:${OS}-pytorch-cpu-ssd-resnet34-training
+DOCKER_ARGS="--privileged --init -it"
+WORKDIR=/workspace/pytorch-ssd-resnet34-training
+
+docker run --rm \
+  --env OUTPUT_DIR=${OUTPUT_DIR} \
+  --env DATASET_DIR=${DATASET_DIR} \
+  --env CHECKPOINT_DIR=${CHECKPOINT_DIR} \
+  --env http_proxy=${http_proxy} \
+  --env https_proxy=${https_proxy} \
+  --env no_proxy=${no_proxy} \
+  --volume ${DATASET_DIR}:${DATASET_DIR} \
+  --volume ${CHECKPOINT_DIR}:${CHECKPOINT_DIR} \
+  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
+  ${DOCKER_RUN_ENVS} \
+  --shm-size 8G \
+  -w ${WORKDIR} \
+  ${DOCKER_ARGS} \
+  $IMAGE_NAME \
+  /bin/bash $SCRIPT $PRECISION
+  ```
+
+## Documentation and Sources
+#### Get Started​
+[Docker* Repository](https://hub.docker.com/r/intel/object-detection)
+
+[Main GitHub*](https://github.com/IntelAI/models)
+
+[Release Notes](https://github.com/IntelAI/models/releases)
+
+[Get Started Guide](https://github.com/IntelAI/models/blob/master/quickstart/quickstart/object_detection/pytorch/ssd-resnet34/training/cpu/DEVCATALOG.md)
+
+#### Code Sources
+[Dockerfile](https://github.com/IntelAI/models/tree/master/docker/pyt-cpu)
+
+[Report Issue](https://community.intel.com/t5/Intel-Optimized-AI-Frameworks/bd-p/optimized-ai-frameworks)
+
+## License Agreement
+LEGAL NOTICE: By accessing, downloading or using this software and any required dependent software (the “Software Package”), you agree to the terms and conditions of the software license agreements for the Software Package, which may also include notices, disclaimers, or license terms for third party software included with the Software Package. Please refer to the [license](https://github.com/IntelAI/models/tree/master/third_party) file for additional details.
+
+[View All Containers and Solutions 🡢](https://www.intel.com/content/www/us/en/developer/tools/software-catalog/containers.html?s=Newest)
diff --git a/quickstart/object_detection/pytorch/ssd-resnet34/training/cpu/README.md b/quickstart/object_detection/pytorch/ssd-resnet34/training/cpu/README.md
index 817599030..8ef459f62 100644
--- a/quickstart/object_detection/pytorch/ssd-resnet34/training/cpu/README.md
+++ b/quickstart/object_detection/pytorch/ssd-resnet34/training/cpu/README.md
@@ -3,47 +3,47 @@
 ## Description
 This document has instructions for running SSD-ResNet34 training using Intel-optimized PyTorch.
 
-## Bare Metal
-### General setup
-
-Follow [link](/docs/general/pytorch/BareMetalSetup.md) to install Miniconda and build Pytorch, IPEX, TorchVison, Torch-CCL and Jemalloc.
+## Quick Start Scripts
+| Script name | Description |
+|-------------|-------------|
+| `throughput.sh` | Tests the training performance for SSD-ResNet34 for the specified precision (fp32, avx-fp32, bf32 or bf16). |
+| `accuracy.sh` | Tests the training accuracy for SSD-ResNet34 for the specified precision (fp32, avx-fp32, bf32 or bf16). |
 
-### Model Specific Setup
-* Install dependencies
+**Note**: The `avx-fp32` precision runs the same scripts as `fp32`, except that the `DNNL_MAX_CPU_ISA` environment variable is unset. The environment variable is otherwise set to `DNNL_MAX_CPU_ISA=AVX512_CORE_AMX`.
+* Set ENV to use AMX:
   ```
-  pip install --no-cache-dir cython
-  pip install --no-cache-dir https://github.com/mlperf/logging/archive/9ea0afa.zip
-  pip install matplotlib Pillow pycocotools defusedxml
-  pip install --no-cache-dir pytz==2018.5
+  export DNNL_MAX_CPU_ISA=AVX512_CORE_AMX
   ```
 
-* Download pretrained model
-  ```
-  cd <path to your clone of the model zoo>/quickstart/object_detection/pytorch/ssd-resnet34/training/cpu
-  export CHECKPOINT_DIR=<directory where the pretrained model will be saved>
-  bash download_model.sh
-  ```
+|           Distributed Training          |
+|  DataType   | Throughput  |   Accuracy  |
+| ----------- | ----------- | ----------- |
+| FP32        | bash throughput_dist.sh fp32 | bash accuracy_dist.sh fp32 |
+| BF16        | bash throughput_dist.sh bf16 | bash accuracy_dist.sh bf16 |
+| BF32        | bash throughput_dist.sh bf32 | bash accuracy_dist.sh bf32 |
 
-* Set Jemalloc Preload for better performance
+## Bare Metal
+### General setup
+
+Follow [link](/docs/general/pytorch/BareMetalSetup.md) to install Miniconda and build Pytorch, IPEX, TorchVison, Torch-CCL TCmalloc and Jemalloc.
+
+### Model Specific Setup
+* Set Jemalloc and tcmalloc Preload for better performance
 
   The jemalloc should be built from the [General setup](#general-setup) section.
   ```
-  export LD_PRELOAD="path/lib/libjemalloc.so":$LD_PRELOAD
+  export LD_PRELOAD="<path to the jemalloc directory>/lib/libjemalloc.so":"path_to/tcmalloc/lib/libtcmalloc.so":$LD_PRELOAD
   export MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
   ```
 
 * Set IOMP preload for better performance
 
-  IOMP should be installed in your conda env from the [General setup](#general-setup) section.
+  IOMP should be installed in your conda/virtual env from the [General setup](#general-setup) section.
   ```
+  pip install packaging intel-openmp
   export LD_PRELOAD=path/lib/libiomp5.so:$LD_PRELOAD
   ```
 
-* Set ENV to use AMX if you are using SPR
-  ```
-  export DNNL_MAX_CPU_ISA=AVX512_CORE_AMX
-  ```
-
 * Set ENV to use multi-node distributed training (no need for single-node multi-sockets)
 
   In this case, we use data-parallel distributed training and every rank will hold same model replica. The NNODES is the number of ip in the HOSTFILE. To use multi-nodes distributed training you should firstly setup the passwordless login (you can refer to [link](https://linuxize.com/post/how-to-setup-passwordless-ssh-login/)) between these nodes.
@@ -52,58 +52,46 @@ Follow [link](/docs/general/pytorch/BareMetalSetup.md) to install Miniconda and
   export HOSTFILE=your_ip_list_file #one ip per line
   ```
 
-## Datasets
+### Run the model
 
-Download the 2017 [COCO dataset](https://cocodataset.org) using the `download_dataset.sh` script.
-Export the `DATASET_DIR` environment variable to specify the directory where the dataset
-will be downloaded. This environment variable will be used again when running quickstart scripts.
-```
-cd <path to your clone of the model zoo>/quickstart/object_detection/pytorch/ssd-resnet34/training/cpu
-export DATASET_DIR=<directory where the dataset will be saved>
-bash download_dataset.sh
-```
-
-## Quick Start Scripts
-
-|  DataType   | Throughput  |   Accuracy  |
-| ----------- | ----------- | ----------- |
-| FP32        | bash throughput.sh fp32 | bash accuracy.sh fp32 |
-| BF16        | bash throughput.sh bf16 | bash accuracy.sh bf16 |
-| BF32        | bash throughput.sh bf32 | bash accuracy.sh bf32 |
-
-|           Distributed Training          |
-|  DataType   | Throughput  |   Accuracy  |
-| ----------- | ----------- | ----------- |
-| FP32        | bash throughput_dist.sh fp32 | bash accuracy_dist.sh fp32 |
-| BF16        | bash throughput_dist.sh bf16 | bash accuracy_dist.sh bf16 |
-| BF32        | bash throughput_dist.sh bf32 | bash accuracy_dist.sh bf32 |
-
-## Run the model
-
-Follow the instructions above to setup your bare metal environment, download and
-preprocess the dataset, and do the model specific setup. Once all the setup is done,
-the Model Zoo can be used to run a [quickstart script](#quick-start-scripts).
+Once all the setup is done, the Intel® AI Reference Models repo can be used to run a [quickstart script](#quick-start-scripts).
 Ensure that you have an enviornment variables set to point to the dataset directory
 and an output directory.
 
 ```
-# Clone the model zoo repo and set the MODEL_DIR
+# Clone the Intel® AI Reference Models riepo and set the MODEL_DIR
 git clone https://github.com/IntelAI/models.git
 cd models
 export MODEL_DIR=$(pwd)
 
+# Dataset
+Download the 2017 [COCO dataset](https://cocodataset.org) using the `download_dataset.sh` script.
+Export the `DATASET_DIR` environment variable to specify the directory where the dataset
+will be downloaded. This environment variable will be used again when running quickstart scripts.
+
+cd quickstart/object_detection/pytorch/ssd-resnet34/training/cpu
+export DATASET_DIR=<directory where the dataset will be saved>
+bash download_dataset.sh
+cd - 
+cd ${MODEL_DIR}
+
+# install model specific dependencies
+./quickstart/object_detection/pytorch/ssd-resnet34/training/cpu/setup.sh
+
+# Download pretrained model
+export CHECKPOINT_DIR=<directory where the pretrained model will be saved>
+bash quickstart/object_detection/pytorch/ssd-resnet34/training/cpu/download_model.sh
+
 # Env vars
 export DATASET_DIR=<path to the COCO dataset>
-export CHECKPOINT_DIR=<path to the pretrained model>
 export OUTPUT_DIR=<path to an output directory>
+export PRECISION=<select from :- fp32, avx-fp32, bf16, or bf32>
 
-# Run a quickstart script (for example, FP32 throughput inference)
-cd ${MODEL_DIR}/quickstart/object_detection/pytorch/ssd-resnet34/training/cpu
-bash throughput.sh fp32
+# Optional environemnt variables:
+export BATCH_SIZE=<set a value for batch size, else it will run with default batch size>
 
-# Run distributed training script (for example, FP32 distributed training throughput)
-cd ${MODEL_DIR}/quickstart/object_detection/pytorch/ssd-resnet34/training/cpu
-bash throughput_dist.sh fp32
+# Run a quickstart script
+./quickstart/object_detection/pytorch/ssd-resnet34/training/cpu/<script.sh>
 ```
 
 <!--- 80. License -->
diff --git a/quickstart/object_detection/pytorch/ssd-resnet34/training/cpu/accuracy.sh b/quickstart/object_detection/pytorch/ssd-resnet34/training/cpu/accuracy.sh
index 1ca2928a4..6fd62f0fb 100755
--- a/quickstart/object_detection/pytorch/ssd-resnet34/training/cpu/accuracy.sh
+++ b/quickstart/object_detection/pytorch/ssd-resnet34/training/cpu/accuracy.sh
@@ -38,27 +38,32 @@ if [ ! -d "${OUTPUT_DIR}" ]; then
   exit 1
 fi
 
+if [ -z "${PRECISION}" ]; then
+  echo "Precision is not set"
+  exit 1
+fi
+
 CORES=`lscpu | grep Core | awk '{print $4}'`
 SOCKETS=`lscpu | grep Socket | awk '{print $2}'`
 TOTAL_CORES=`expr $CORES \* $SOCKETS`
 
 CORES_PER_INSTANCE=$CORES
 
-if [[ "$1" == *"avx"* ]]; then
+if [[ "$PRECISION" == *"avx"* ]]; then
     unset DNNL_MAX_CPU_ISA
 fi
 
 ARGS=""
-if [ "$1" == "bf16" ]; then
+if [ "$PRECISION" == "bf16" ]; then
     ARGS="$ARGS --autocast"
     echo "### running bf16 datatype"
-elif [[ $1 == "fp32" || $1 == "avx-fp32" ]]; then
+elif [[ $PRECISION == "fp32" || $PRECISION == "avx-fp32" ]]; then
     echo "### running fp32 datatype"
-elif [[ "$1" == "bf32" ]]; then
+elif [[ "$PRECISION" == "bf32" ]]; then
     ARGS="$ARGS --bf32"
     echo "### running bf32 datatype"
 else
-    echo "The specified precision '$1' is unsupported."
+    echo "The specified precision '$PRECISION' is unsupported."
     echo "Supported precisions are: fp32, avx-fp32, bf32, and bf16"
     exit 1
 fi
@@ -68,8 +73,8 @@ export USE_IPEX=1
 export KMP_BLOCKTIME=1
 export KMP_AFFINITY=granularity=fine,compact,1,0
 
-PRECISION=$1
-BATCH_SIZE=100
+# Runs with default value when BATCH_SIZE is not set:
+BATCH_SIZE=${BATCH_SIZE:- 100}
 
 rm -rf ${OUTPUT_DIR}/train_ssdresnet34_${PRECISION}_accuracy*
 
@@ -93,4 +98,4 @@ python -m intel_extension_for_pytorch.cpu.launch \
 wait
 
 accuracy=$(grep 'Accuracy:' ${OUTPUT_DIR}/train_ssdresnet34_${PRECISION}_accuracy* |sed -e 's/.*Accuracy//;s/[^0-9.]//g')
-echo ""SSD-RN34";"accuracy";$1; ${BATCH_SIZE};${accuracy}" | tee -a ${OUTPUT_DIR}/summary.log
+echo ""SSD-RN34";"accuracy";$PRECISION; ${BATCH_SIZE};${accuracy}" | tee -a ${OUTPUT_DIR}/summary.log
diff --git a/quickstart/object_detection/pytorch/ssd-resnet34/training/cpu/setup.sh b/quickstart/object_detection/pytorch/ssd-resnet34/training/cpu/setup.sh
new file mode 100755
index 000000000..79b5ca351
--- /dev/null
+++ b/quickstart/object_detection/pytorch/ssd-resnet34/training/cpu/setup.sh
@@ -0,0 +1,21 @@
+#!/bin/bash
+#
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+pip install --no-cache-dir cython
+pip install --no-cache-dir https://github.com/mlperf/logging/archive/9ea0afa.zip
+pip install matplotlib Pillow pycocotools pytz==2018.5 cython && \
+pip install yacs opencv-python cityscapesscripts transformers defusedxml
diff --git a/quickstart/object_detection/pytorch/ssd-resnet34/training/cpu/throughput.sh b/quickstart/object_detection/pytorch/ssd-resnet34/training/cpu/throughput.sh
index 97bbb7906..4d03e81b4 100755
--- a/quickstart/object_detection/pytorch/ssd-resnet34/training/cpu/throughput.sh
+++ b/quickstart/object_detection/pytorch/ssd-resnet34/training/cpu/throughput.sh
@@ -38,21 +38,26 @@ if [ ! -d "${OUTPUT_DIR}" ]; then
   exit 1
 fi
 
-if [[ "$1" == *"avx"* ]]; then
+if [ -z "${PRECISION}" ]; then
+  echo "Precision is not set"
+  exit 1
+fi
+
+if [[ "$PRECISION" == *"avx"* ]]; then
     unset DNNL_MAX_CPU_ISA
 fi
 
 ARGS=""
-if [ "$1" == "bf16" ]; then
+if [ "$PRECISION" == "bf16" ]; then
     ARGS="$ARGS --autocast"
     echo "### running bf16 datatype"
-elif [[ $1 == "fp32" || $1 == "avx-fp32" ]]; then
+elif [[ $PRECISION == "fp32" || $PRECISION == "avx-fp32" ]]; then
     echo "### running fp32 datatype"
-elif [[ "$1" == "bf32" ]]; then
+elif [[ "$PRECISION" == "bf32" ]]; then
     ARGS="$ARGS --bf32"
     echo "### running bf32 datatype"
 else
-    echo "The specified precision '$1' is unsupported."
+    echo "The specified precision '$PRECISION' is unsupported."
     echo "Supported precisions are: fp32, avx-fp32, bf32, and bf16"
     exit 1
 fi
@@ -68,8 +73,8 @@ export USE_IPEX=1
 export KMP_BLOCKTIME=1
 export KMP_AFFINITY=granularity=fine,compact,1,0
 
-PRECISION=$1
-BATCH_SIZE=224
+# Runs with default value when BATCH_SIZE is not set
+BATCH_SIZE=${BATCH_SIZE:-224}
 
 rm -rf ${OUTPUT_DIR}/train_ssdresnet34_${PRECISION}_throughput*
 
@@ -107,4 +112,4 @@ END   {
 sum = sum / i;
         printf("%.3f", sum);
 }')
-echo ""SSD-RN34";"training throughput";$1; ${BATCH_SIZE};${throughput}" | tee -a ${OUTPUT_DIR}/summary.log
+echo ""SSD-RN34";"training throughput";$PRECISION; ${BATCH_SIZE};${throughput}" | tee -a ${OUTPUT_DIR}/summary.log
diff --git a/quickstart/object_detection/pytorch/ssd-resnet34/training/cpu/throughput_dist.sh b/quickstart/object_detection/pytorch/ssd-resnet34/training/cpu/throughput_dist.sh
index b7e8dc78d..9057ca09f 100755
--- a/quickstart/object_detection/pytorch/ssd-resnet34/training/cpu/throughput_dist.sh
+++ b/quickstart/object_detection/pytorch/ssd-resnet34/training/cpu/throughput_dist.sh
@@ -72,7 +72,7 @@ export KMP_BLOCKTIME=1
 export KMP_AFFINITY=granularity=fine,compact,1,0
 
 PRECISION=$1
-BATCH_SIZE=224
+BATCH_SIZE=${BATCH_SIZE:-224}
 
 rm -rf ${OUTPUT_DIR}/train_ssdresnet34_${PRECISION}_throughput_dist*
 
diff --git a/quickstart/object_detection/pytorch/yolov4/inference/gpu/.docs/baremetal.md b/quickstart/object_detection/pytorch/yolov4/inference/gpu/.docs/baremetal.md
deleted file mode 100644
index 1330ed1ba..000000000
--- a/quickstart/object_detection/pytorch/yolov4/inference/gpu/.docs/baremetal.md
+++ /dev/null
@@ -1,42 +0,0 @@
-<!--- 50. Baremetal -->
-## Run the model
-Install the following pre-requisites:
-* Python version 3.9
-* Create and activate virtual environment.
-  ```bash
-  virtualenv -p python <virtualenv_name>
-  source <virtualenv_name>/bin/activate
-  ```
-* Install PyTorch and Intel® Extension for PyTorch for GPU (IPEX):
-  ```bash
-  python -m pip install torch==1.10.0a0 -f https://developer.intel.com/ipex-whl-stable-xpu
-  python -m pip install numpy==1.23.4
-  python -m pip install intel_extension_for_pytorch==1.10.200+gpu -f https://developer.intel.com/ipex-whl-stable-xpu
-  ```
-  To verify that PyTorch and IPEX are correctly installed:
-  ```bash
-  python -c "import torch;print(torch.device('xpu'))"  # Sample output: "xpu"
-  python -c "import intel_extension_for_pytorch as ipex;print(ipex.xpu.is_available())"  #Sample output True
-  python -c "import intel_extension_for_pytorch as ipex;print(ipex.xpu.has_onemkl())"  # Sample output: True
-  ```
-* Navigate to Yolov4 inference directory and install model specific dependencies for the workload:
-  ```bash
-  cd quickstart/object_detection/pytorch/yolov4/inference/gpu
-  ./setup.sh
-  cd -
-  ```
-* Download the pretrained weights file, and set the PRETRAINED_MODEL environment variable to point to where it was saved:
-  ```bash
-  wget https://drive.google.com/open?id=1wv_LiFeCRYwtpkqREPeI13-gPELBDwuJ
-  ```
-
-### Run the model on Baremetal
-```
-Set environment variables:
-export DATASET_DIR=<path where yolov4 COCO dataset>
-export PRETRAINED_MODEL=<path to directory where the pretrained weights file was saved>
-export OUTPUT_DIR=<Path to save the output logs>
-
-Run the inference script, only int8 precision is supported:
-./quickstart/object_detection/pytorch/yolov4/inference/gpu/inference_with_dummy_data.sh
-```
diff --git a/quickstart/object_detection/pytorch/yolov4/inference/gpu/.docs/datasets.md b/quickstart/object_detection/pytorch/yolov4/inference/gpu/.docs/datasets.md
deleted file mode 100644
index 40207d49f..000000000
--- a/quickstart/object_detection/pytorch/yolov4/inference/gpu/.docs/datasets.md
+++ /dev/null
@@ -1,32 +0,0 @@
-<!--- 30. Datasets -->
-## Datasets
-
-Download and extract the 2017 training/validation images and annotations from the
-[COCO dataset website](https://cocodataset.org/#download) to a `coco` folder
-and unzip the files. After extracting the zip files, your dataset directory
-structure should look something like this:
-```
-coco
-├── annotations
-│   ├── captions_train2017.json
-│   ├── captions_val2017.json
-│   ├── instances_train2017.json
-│   ├── instances_val2017.json
-│   ├── person_keypoints_train2017.json
-│   └── person_keypoints_val2017.json
-├── train2017
-│   ├── 000000454854.jpg
-│   ├── 000000137045.jpg
-│   ├── 000000129582.jpg
-│   └── ...
-└── val2017
-    ├── 000000000139.jpg
-    ├── 000000000285.jpg
-    ├── 000000000632.jpg
-    └── ...
-```
-The parent of the `annotations`, `train2017`, and `val2017` directory (in this example `coco`)
-is the directory that should be used when setting the `image` environment
-variable for YOLOv4 (for example: `export image=/home/<user>/coco/val2017/000000581781.jpg`).
-In addition, we should also set the `size` environment to match the size of image.
-(for example: `export size=416`)
diff --git a/quickstart/object_detection/pytorch/yolov4/inference/gpu/.docs/description.md b/quickstart/object_detection/pytorch/yolov4/inference/gpu/.docs/description.md
deleted file mode 100644
index 1d4003781..000000000
--- a/quickstart/object_detection/pytorch/yolov4/inference/gpu/.docs/description.md
+++ /dev/null
@@ -1,5 +0,0 @@
-<!-- 10. Description -->
-## Description
-
-This document has instructions for running <model name> <mode> using
-Intel(R) Extension for PyTorch with GPU.
diff --git a/quickstart/object_detection/pytorch/yolov4/inference/gpu/.docs/docker.md b/quickstart/object_detection/pytorch/yolov4/inference/gpu/.docs/docker.md
deleted file mode 100644
index b648b28e4..000000000
--- a/quickstart/object_detection/pytorch/yolov4/inference/gpu/.docs/docker.md
+++ /dev/null
@@ -1,52 +0,0 @@
-<!--- 60. Docker -->
-## Docker
-
-Requirements:
-* Host machine has Intel® Data Center GPU Flex Series.
-* Host machine has the Intel® Data Center GPU Flex Series Ubuntu driver. Please follow the [link](https://registrationcenter.intel.com/en/products/download/4125/) to download.
-* Host machine has Docker installed
-* Download and build the Intel® Extension for PyTorch (IPEX) container using the [link](https://registrationcenter.intel.com/en/products/subscription/956/).
-  (`model-zoo:pytorch-ipex-gpu`)
-
-Prior to building the <model name> <mode> container, ensure that you have
-built the IPEX container (`model-zoo:pytorch-ipex-gpu`).
-
-[Extract the package](#model-package), then use the `build.sh`
-script to build the container. After the container has been built, you can
-run the model <mode> using the `run.sh` script.
-
-The `run.sh` script will execute one of the [quickstart](#quick-start-scripts) script
-using the container that was just built. By default, the
-`inference_with_dummy_data.sh` script will be run. To run a different script,
-specify the script name of the quickstart script using the `SCRIPT`
-environment variable. See the snippet below for an example.
-
-> Note: Ensure that your system has the proxy environment variables
-> set (if needed), otherwise the container build may fail when trying to pull external
-> dependencies (like apt-get and pip installs). 
-
-You need download pretrained weights from:
-yolov4.pth(https://pan.baidu.com/s/1ZroDvoGScDgtE1ja_QqJVw Extraction code:xrq9) or
-yolov4.pth(https://drive.google.com/open?id=1wv_LiFeCRYwtpkqREPeI13-gPELBDwuJ)
-to the anywhere directory you choice, and set environment 'PRETRAINED_MODEL'.
-```
-# Extract the package
-tar -xzf <package name>
-
-# Navigate to the YOLOv4 inference directory
-cd pytorch-gpu-YOLOv4-inference
-
-# Build the container
-./build.sh
-
-# Set environment vars for pretrained model
-export PRETRAINED_MODEL=<Path of the downloaded pretrained-model file on the machine >
-
-# Run the container with the default inference_with_dummy_data.sh script
-./run.sh
-
-For inference_with_dummy_data.sh environment is enough, but for inference.sh also need:
-export image=<path to the image .jpg file>
-export size=<Image size>
-export PRECISION=int8
-```
diff --git a/quickstart/object_detection/pytorch/yolov4/inference/gpu/.docs/gpu_setup.md b/quickstart/object_detection/pytorch/yolov4/inference/gpu/.docs/gpu_setup.md
deleted file mode 100644
index 46a1c07ec..000000000
--- a/quickstart/object_detection/pytorch/yolov4/inference/gpu/.docs/gpu_setup.md
+++ /dev/null
@@ -1,30 +0,0 @@
-<!--- 20. GPU Setup -->
-## Hardware Requirements:
-- Intel® Data Center GPU Flex Series
-
-## Software Requirements:
-- Ubuntu 20.04 (64-bit)
-- Intel GPU Drivers: Intel® Data Center GPU Flex Series [419.40](https://dgpu-docs.intel.com/releases/stable_419_40_20220914.html)
-
-  |Release|OS|Intel GPU|Install Intel GPU Driver|
-    |-|-|-|-|
-    |v1.0.0|Ubuntu 20.04|Intel® Data Center GPU Flex Series| Refer to the [Installation Guides](https://dgpu-docs.intel.com/installation-guides/ubuntu/ubuntu-focal-dc.html) for latest driver installation. If install the verified Intel® Data Center GPU Flex Series [419.40](https://dgpu-docs.intel.com/releases/stable_419_40_20220914.html), please append the specific version after components, such as `apt-get install intel-opencl-icd=22.28.23726.1+i419~u20.04`|
-
-- Intel® oneAPI Base Toolkit 2022.3: Need to install components of Intel® oneAPI Base Toolkit
-  - Intel® oneAPI DPC++ Compiler
-  - Intel® oneAPI Math Kernel Library (oneMKL)
-  * Download and install the verified DPC++ compiler and oneMKL in Ubuntu 20.04.
-
-    ```bash
-    wget https://registrationcenter-download.intel.com/akdlm/irc_nas/18852/l_BaseKit_p_2022.3.0.8767_offline.sh
-    # 4 components are necessary: DPC++/C++ Compiler, DPC++ Libiary, Threading Building Blocks and oneMKL
-    sh ./l_BaseKit_p_2022.3.0.8767_offline.sh
-    ```
-    For any more details, please follow the procedure in https://www.intel.com/content/www/us/en/developer/tools/oneapi/base-toolkit.html.
-
-  - Set environment variables: 
-    Default installation location {ONEAPI_ROOT} is /opt/intel/oneapi for root account, ${HOME}/intel/oneapi for other accounts
-    ```bash
-    source {ONEAPI_ROOT}/setvars.sh
-    ```
-
diff --git a/quickstart/object_detection/pytorch/yolov4/inference/gpu/.docs/license.md b/quickstart/object_detection/pytorch/yolov4/inference/gpu/.docs/license.md
deleted file mode 100644
index a60f169f4..000000000
--- a/quickstart/object_detection/pytorch/yolov4/inference/gpu/.docs/license.md
+++ /dev/null
@@ -1,4 +0,0 @@
-<!--- 80. License -->
-## License
-
-[LICENSE](/LICENSE)
diff --git a/quickstart/object_detection/pytorch/yolov4/inference/gpu/.docs/package.md b/quickstart/object_detection/pytorch/yolov4/inference/gpu/.docs/package.md
deleted file mode 100644
index c2317f9ba..000000000
--- a/quickstart/object_detection/pytorch/yolov4/inference/gpu/.docs/package.md
+++ /dev/null
@@ -1,21 +0,0 @@
-<!--- 20. Model package -->
-## Model Package
-
-The model package includes the scripts and libraries needed to
-build and run <model name> <mode> using a docker container. Note that
-this model container uses the PyTorch IPEX GPU container as it's base,
-and it requires the `model-zoo:pytorch-ipex-gpu` image to be built before
-the model container is built.
-```
-<package dir>
-├── build.sh
-├── info.txt
-├── licenses
-│   ├── LICENSE
-│   └── third_party
-├── model_packages
-│   └── <package name>
-├── <package dir>.Dockerfile
-├── README.md
-└── run.sh
-```
diff --git a/quickstart/object_detection/pytorch/yolov4/inference/gpu/.docs/quickstart.md b/quickstart/object_detection/pytorch/yolov4/inference/gpu/.docs/quickstart.md
deleted file mode 100644
index 9c0ec26c1..000000000
--- a/quickstart/object_detection/pytorch/yolov4/inference/gpu/.docs/quickstart.md
+++ /dev/null
@@ -1,6 +0,0 @@
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| `inference_with_dummy_data.sh` | Inference with int8 batch_size64 dummy data |
diff --git a/quickstart/object_detection/pytorch/yolov4/inference/gpu/.docs/title.md b/quickstart/object_detection/pytorch/yolov4/inference/gpu/.docs/title.md
deleted file mode 100644
index 1d6c60187..000000000
--- a/quickstart/object_detection/pytorch/yolov4/inference/gpu/.docs/title.md
+++ /dev/null
@@ -1,2 +0,0 @@
-<!--- 0. Title -->
-# <model name> <mode>
diff --git a/quickstart/object_detection/pytorch/yolov4/inference/gpu/DEVCATALOG.md b/quickstart/object_detection/pytorch/yolov4/inference/gpu/DEVCATALOG.md
deleted file mode 100644
index b9d0a2645..000000000
--- a/quickstart/object_detection/pytorch/yolov4/inference/gpu/DEVCATALOG.md
+++ /dev/null
@@ -1,107 +0,0 @@
-# Running YOLOv4 inference on Intel® Data Center GPU Flex Series using Intel® Extension for PyTorch*
-
-
-## Overview
-
-This document has instructions for running YOLOv4 inference using Intel® Extension for PyTorch on Intel® Flex Series GPU.
-
-## Requirements
-| Item | Detail |
-| ------ | ------- |
-| Host machine  | Intel® Data Center GPU Flex Series  |
-| Drivers | GPU-compatible drivers need to be installed:[Download Driver 602](https://dgpu-docs.intel.com/installation-guides/ubuntu/ubuntu-jammy-dc.html#step-1-add-package-repository)
-| Software | Docker* Installed |
-
-## Dataset
-
-Download and extract the 2017 training/validation images and annotations from the [COCO dataset website](https://cocodataset.org/#download) to a `coco` folder and unzip the files. After extracting the zip files, your dataset directory structure should look something like this:
-```
-coco
-├── annotations
-│   ├── captions_train2017.json
-│   ├── captions_val2017.json
-│   ├── instances_train2017.json
-│   ├── instances_val2017.json
-│   ├── person_keypoints_train2017.json
-│   └── person_keypoints_val2017.json
-├── train2017
-│   ├── 000000454854.jpg
-│   ├── 000000137045.jpg
-│   ├── 000000129582.jpg
-│   └── ...
-└── val2017
-    ├── 000000000139.jpg
-    ├── 000000000285.jpg
-    ├── 000000000632.jpg
-    └── ...
-```
-The parent of the `annotations`, `train2017`, and `val2017` directory (in this example `coco`) is the directory that should be used when setting the `IMAGE_FILE` environment
-variable for YOLOv4 (for example: `export IMAGE_FILE=/home/<user>/coco/val2017/000000581781.jpg`). In addition, we should also set the `size` environment to match the size of image.
-(for example: `export size=416`)
-
-## Pretrained Model
-
-You need to download pretrained weights from: yolov4.pth(https://pan.baidu.com/s/1ZroDvoGScDgtE1ja_QqJVw Extraction code:xrq9) or yolov4.pth(https://drive.google.com/open?id=1wv_LiFeCRYwtpkqREPeI13-gPELBDwuJ) to the any directory of your choice, and set environment `PRETRAINED_MODEL`.
-
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| `inference_block_format.sh` | Inference with int8 for specified precision(int8) and batch size on dummy data on Flex series 170 |
-| `flex_multi_card_batch_inference.sh` | Inference with int8 for specified precision(int8) and batch size on dummy data on Flex series 140 |
-| `flex_multi_card_online_inference.sh` | Online Inference with int8 for specified precision(int8) on dummy data on Flex series 140 |
-## Run Using Docker
-
-### Set up Docker Image
-
-```
-docker pull intel/object-detection:pytorch-flex-gpu-yolov4-inference
-```
-### Run Docker Image
-The Yolov4 inference container includes scripts,model and libraries need to run int8 inference. To run the `inference_block_format.sh` quickstart script using this container, the script uses dummy image data. The script also performs online INT8 Calibration on the provided pre-trained model.You will need to provide an output directory where log files will be written. 
-
-**Note:** The default batch size of Flex series 170 is 256 and the batch size for Flex series 140 is 64. Add `--cap-add=SYS_NICE` to the `docker run` command for executing `flex_multi_card_online_inference.sh` and `flex_multi_card_batch_inference.sh` on Flex series 140.
-
-```
-export PRECISION=int8
-export OUTPUT_DIR=<path to output directory>
-export SCRIPT=quickstart/inference_block_format.sh
-export PRETRAINED_MODEL=<path to downloaded yolov4 model>
-export IMAGE_FILE=<path to coco dataset image>
-export BATCH_SIZE=<inference batch size>
-export NUM_ITERATIONS=<number of iterations>
-IMAGE_NAME=intel/object-detection:pytorch-flex-gpu-yolov4-inference
-
-DOCKER_ARGS="--rm -it"
-
-docker run \
-  --privileged \
-  --device=/dev/dri \
-  --ipc=host \
-  --env BATCH_SIZE=${BATCH_SIZE} \
-  --env NUM_ITERATIONS=${NUM_ITERATIONS} \
-  --env PRECISION=${PRECISION} \
-  --env OUTPUT_DIR=${OUTPUT_DIR} \
-  --env IMAGE_FILE=${IMAGE_FILE} \
-  --env PRETRAINED_MODEL=${PRETRAINED_MODEL} \
-  --env http_proxy=${http_proxy} \
-  --env https_proxy=${https_proxy} \
-  --env no_proxy=${no_proxy} \
-  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
-  --volume ${IMAGE_FILE}:${IMAGE_FILE} \
-  --volume ${PRETRAINED_MODEL}:${PRETRAINED_MODEL} \
-  ${DOCKER_ARGS} \
-  ${IMAGE_NAME} \
-  /bin/bash $SCRIPT
-  ```
-
-## Documentation and Sources
-
-[GitHub* Repository](https://github.com/IntelAI/models/tree/master/dockerfiles/model_containers)
-
-## Support
-Support for Intel® Extension for PyTorch* is found via the [Intel® AI Analytics Toolkit.](https://www.intel.com/content/www/us/en/developer/tools/oneapi/ai-analytics-toolkit.html#gs.qbretz) Additionally, the Intel® Extension for PyTorch* team tracks both bugs and enhancement requests using [GitHub issues](https://github.com/intel/intel-extension-for-pytorch/issues). Before submitting a suggestion or bug report, please search the GitHub issues to see if your issue has already been reported.
-
-## License Agreement
-
-LEGAL NOTICE: By accessing, downloading or using this software and any required dependent software (the “Software Package”), you agree to the terms and conditions of the software license agreements for the Software Package, which may also include notices, disclaimers, or license terms for third party software included with the Software Package. Please refer to the [license file](https://github.com/IntelAI/models/tree/master/third_party) for additional details.
diff --git a/quickstart/object_detection/pytorch/yolov4/inference/gpu/README.md b/quickstart/object_detection/pytorch/yolov4/inference/gpu/README.md
deleted file mode 100644
index 284a173e3..000000000
--- a/quickstart/object_detection/pytorch/yolov4/inference/gpu/README.md
+++ /dev/null
@@ -1,123 +0,0 @@
-<!--- 0. Title -->
-# YOLOv4 inference
-
-<!-- 10. Description -->
-## Description
-
-This document has instructions for running Yolov4 inference using
-Intel® Extension for PyTorch with Intel® Data Center GPU Flex Series.
-
-<!--- 20. GPU Setup -->
-## Software Requirements:
-- Intel® Data Center GPU Flex Series
-- Follow [instructions](https://intel.github.io/intel-extension-for-pytorch/xpu/latest/tutorials/installation.html) to install the latest IPEX version and other prerequisites.
-
-- Intel® oneAPI Base Toolkit: Need to install components of Intel® oneAPI Base Toolkit
-  - Intel® oneAPI DPC++ Compiler
-  - Intel® oneAPI Threading Building Blocks (oneTBB)
-  - Intel® oneAPI Math Kernel Library (oneMKL)
-  - Follow [instructions](https://www.intel.com/content/www/us/en/developer/tools/oneapi/base-toolkit-download.html?operatingsystem=linux&distributions=offline) to download and install the latest oneAPI Base Toolkit.
-
-  - Set environment variables for Intel® oneAPI Base Toolkit: 
-    Default installation location `{ONEAPI_ROOT}` is `/opt/intel/oneapi` for root account, `${HOME}/intel/oneapi` for other accounts
-    ```bash
-    source {ONEAPI_ROOT}/compiler/latest/env/vars.sh
-    source {ONEAPI_ROOT}/mkl/latest/env/vars.sh
-    source {ONEAPI_ROOT}/tbb/latest/env/vars.sh
-    source {ONEAPI_ROOT}/mpi/latest/env/vars.sh
-    source {ONEAPI_ROOT}/ccl/latest/env/vars.sh
-    ```
-
-
-<!--- 30. Datasets -->
-## Datasets
-
-Download and extract the 2017 training/validation images and annotations from the
-[COCO dataset website](https://cocodataset.org/#download) to a `coco` folder
-and unzip the files. After extracting the zip files, your dataset directory
-structure should look something like this:
-```
-coco
-├── annotations
-│   ├── captions_train2017.json
-│   ├── captions_val2017.json
-│   ├── instances_train2017.json
-│   ├── instances_val2017.json
-│   ├── person_keypoints_train2017.json
-│   └── person_keypoints_val2017.json
-├── train2017
-│   ├── 000000454854.jpg
-│   ├── 000000137045.jpg
-│   ├── 000000129582.jpg
-│   └── ...
-└── val2017
-    ├── 000000000139.jpg
-    ├── 000000000285.jpg
-    ├── 000000000632.jpg
-    └── ...
-```
-The parent of the `annotations`, `train2017`, and `val2017` directory (in this example `coco`)
-is the directory that should be used when setting the `IMAGE_FILE` environment
-variable for YOLOv4 (for example: `export IMAGE_FILE=/home/<user>/coco/val2017/000000581781.jpg`).
-In addition, we should also set the `size` environment to match the size of image.
-(for example: `export size=416`)
-
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| `inference_block_format.sh` | Inference with int8 batch_size256 block format on Flex series 170 |
-| `flex_multi_card_batch_inference.sh` | Inference with dummy data,for int8 and given batch size blocked channel first on Flex series 140 |
-| `flex_multi_card_online_inference.sh` | Online Inference with dummy data,for int8 blocked channel first on Flex series 140 | 
-
-
-<!--- 50. Baremetal -->
-## Run the model
-Install the following pre-requisites:
-* Create and activate virtual environment.
-  ```bash
-  virtualenv -p python <virtualenv_name>
-  source <virtualenv_name>/bin/activate
-  ```
-* Clone the Model Zoo repository:
-  ```bash
-  git clone https://github.com/IntelAI/models.git
-  ```
-
-* Navigate to Yolov4 inference directory and install model specific dependencies for the workload:
-  ```bash
-  cd models
-  ./quickstart/object_detection/pytorch/yolov4/inference/gpu/setup.sh
-  ```
-* Download the pretrained weights file, and set the PRETRAINED_MODEL environment variable to point to where it was saved:
-  ```bash
-  wget https://drive.google.com/open?id=1wv_LiFeCRYwtpkqREPeI13-gPELBDwuJ
-  ```
-
-### Run the model on Baremetal
-```
-Set environment variables:
-export IMAGE_FILE<path where yolov4 COCO image>
-export PRETRAINED_MODEL=<path to directory where the pretrained weights file was saved>
-export OUTPUT_DIR=<Path to save the output logs>
-
-Run the inference script, only int8 precision is supported:
-./quickstart/object_detection/pytorch/yolov4/inference/gpu/inference_block_format.sh
-```
-To execute `flex_multi_card_batch_inference.sh` and `flex_multi_card_online_inference.sh` on Flex series 140, install the following components 
-
-```bash
-apt-get update && \
-apt-get install -y --no-install-recommends --fix-missing parallel pciutils numactl 
-```
-Then execute the quickstart scripts.
-```bash
-./quickstart/object_detection/pytorch/yolov4/inference/gpu/flex_multi_card_batch_inference.sh 
-./quickstart/object_detection/pytorch/yolov4/inference/gpu/flex_multi_card_online_inference.sh
-```
-
-<!--- 80. License -->
-## License
-
-[LICENSE](/LICENSE)
diff --git a/quickstart/object_detection/pytorch/yolov4/inference/gpu/flex_multi_card_batch_inference.sh b/quickstart/object_detection/pytorch/yolov4/inference/gpu/flex_multi_card_batch_inference.sh
deleted file mode 100755
index d2e044581..000000000
--- a/quickstart/object_detection/pytorch/yolov4/inference/gpu/flex_multi_card_batch_inference.sh
+++ /dev/null
@@ -1,60 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2023 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-MODEL_DIR=${MODEL_DIR-$PWD}
-BATCH_SIZE=${BATCH_SIZE-64}
-NUM_ITERATIONS=${NUM_ITERATIONS-500}
-
-if [[ -z "${PRETRAINED_MODEL}" ]]; then
-  echo "The required environment variable PRETRAINED_MODEL has not been set."
-  echo "Please specify a directory where the model weights were downloaded"
-  exit 1
-fi
-
-export OverrideDefaultFP64Settings=1 
-export IGC_EnableDPEmulation=1 
-export CFESingleSliceDispatchCCSMode=1
-export IPEX_ONEDNN_LAYOUT=1
-export IPEX_LAYOUT_OPT=1
-export IPEX_XPU_ONEDNN_LAYOUT=1
-
-declare -a str
-device_id=$( lspci | grep -i display | sed -n '1p' | awk '{print $7}' )
-num_devs=$(lspci | grep -i display | awk '{print $7}' | wc -l)
-
-if [[ ${device_id} == "56c1" ]]; then
- for i in $( eval echo {0..$((num_devs-1))} )
-    do
-    str+=("ZE_AFFINITY_MASK="${i}" python -u ${MODEL_DIR}/models/object_detection/pytorch/yolov4/inference/gpu/models.py \
-            -n 80 \
-            -i ${IMAGE_FILE} \
-            --weight ${PRETRAINED_MODEL} \
-            -e 416 \
-            -w 416 \
-            -d int8 \
-            --dummy 1 \
-            -b ${BATCH_SIZE} \
-            --benchmark 1 \
-            --iter ${NUM_ITERATIONS}" )
-    done
-echo "YOLOv4 dummy data int8 inference block nchw on Flex Series 140"
-parallel --lb -d, --tagstring "[{#}]" ::: \
-"${str[@]}" 2>&1 | tee $OUTPUT_DIR/YOLOv4_dummy_data_xpu_inf_c0_c1_${BATCH_SIZE}.log
-file_loc=$OUTPUT_DIR/YOLOv4_dummy_data_xpu_inf_c0_c1_${BATCH_SIZE}.log
-total_fps=$( cat $file_loc | grep FPS | awk '{print $4}' | awk '{ sum_total += $1 } END { print sum_total }' )
-echo 'Total FPS: '$total_fps | tee -a $file_loc
-fi
diff --git a/quickstart/object_detection/pytorch/yolov4/inference/gpu/flex_multi_card_online_inference.sh b/quickstart/object_detection/pytorch/yolov4/inference/gpu/flex_multi_card_online_inference.sh
deleted file mode 100755
index 50298e8b6..000000000
--- a/quickstart/object_detection/pytorch/yolov4/inference/gpu/flex_multi_card_online_inference.sh
+++ /dev/null
@@ -1,67 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2023 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-MODEL_DIR=${MODEL_DIR-$PWD}
-BATCH_SIZE=${BATCH_SIZE-1}
-NUM_ITERATIONS=${NUM_ITERATIONS-5000}
-
-if [[ -z "${PRETRAINED_MODEL}" ]]; then
-  echo "The required environment variable PRETRAINED_MODEL has not been set."
-  echo "Please specify a directory where the model weights were downloaded"
-  exit 1
-fi
-
-export OverrideDefaultFP64Settings=1 
-export IGC_EnableDPEmulation=1 
-export CFESingleSliceDispatchCCSMode=1
-export IPEX_ONEDNN_LAYOUT=1
-export IPEX_LAYOUT_OPT=1
-export IPEX_XPU_ONEDNN_LAYOUT=1
-
-declare -a str
-device_id=$( lspci | grep -i display | sed -n '1p' | awk '{print $7}' )
-num_devs=$(lspci | grep -i display | awk '{print $7}' | wc -l)
-num_threads=1
-k=0
-
-if [[ ${device_id} == "56c1" ]]; then
- for i in $( eval echo {0..$((num_devs-1))} )
-    do
-    for j in $( eval echo {1..$num_threads} )
-    do
-    str+=("ZE_AFFINITY_MASK="${i}" numactl -C ${k} -l  python -u ${MODEL_DIR}/models/object_detection/pytorch/yolov4/inference/gpu/models.py \
-          -n 80 \
-          -i ${IMAGE_FILE} \
-          --weight ${PRETRAINED_MODEL} \
-          -e 416 \
-          -w 416 \
-          -d int8 \
-          --dummy 1 \
-          -b ${BATCH_SIZE} \
-          --benchmark 1 \
-          --iter ${NUM_ITERATIONS} ")
-    ((k=k+1))
-    done
-    done
-
-echo "YOLOv4 dummy data int8 inference block nchw on Flex Series 140"
-parallel --lb -d, --tagstring "[{#}]" ::: \
-"${str[@]}" 2>&1 | tee $OUTPUT_DIR/YOLOv4_dummy_data_xpu_inf_c0_c1_${BATCH_SIZE}.log
-file_loc=$OUTPUT_DIR/YOLOv4_dummy_data_xpu_inf_c0_c1_${BATCH_SIZE}.log
-total_fps=$( cat $file_loc | grep FPS | awk '{print $4}' | awk '{ sum_total += $1 } END { print sum_total }' )
-echo 'Total FPS: '$total_fps | tee -a $file_loc
-fi
diff --git a/quickstart/object_detection/pytorch/yolov4/inference/gpu/inference_block_format.sh b/quickstart/object_detection/pytorch/yolov4/inference/gpu/inference_block_format.sh
deleted file mode 100755
index 60856bb16..000000000
--- a/quickstart/object_detection/pytorch/yolov4/inference/gpu/inference_block_format.sh
+++ /dev/null
@@ -1,60 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2022 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-MODEL_DIR=${MODEL_DIR-$PWD}
-BATCH_SIZE=${BATCH_SIZE-256}
-NUM_ITERATIONS=${NUM_ITERATIONS-500}
-
-if [[ -z "${IMAGE_FILE}" ]]; then
-  echo "Please specify coco image file variable IMAGE_FILE"
-  exit 1
-fi
-
-# if [[ -z "${LABELS_FILE}" ]]; then
-#   echo "Please specify coco names dataset file as variable LABELS_FILE"
-#   exit 1
-# fi
-
-export CFESingleSliceDispatchCCSMode=1
-export IPEX_ONEDNN_LAYOUT=1
-export IPEX_LAYOUT_OPT=1
-export IPEX_XPU_ONEDNN_LAYOUT=1
-
-
-if [[ -z "${PRETRAINED_MODEL}" ]]; then
-  echo "The required environment variable PRETRAINED_MODEL has not been set."
-  echo "Please specify a directory where the model weights were downloaded"
-  exit 1
-fi
-
-if [[ -z $OUTPUT_DIR ]]; then
-  echo "The required environment variable OUTPUT_DIR has not been set"
-  exit 1
-fi
-
-echo "YOLO(v4) Inference INT8 Block NCHW BS=256"
-python -u ${MODEL_DIR}/models/object_detection/pytorch/yolov4/inference/gpu/models.py \
-  -n 80 \
-  -i ${IMAGE_FILE} \
-  --weight ${PRETRAINED_MODEL} \
-  -e 416 \
-  -w 416 \
-  -d int8 \
-  --dummy 1 \
-  -b ${BATCH_SIZE} \
-  --benchmark 1 \
-  --iter ${NUM_ITERATIONS} 2>&1 | tee $OUTPUT_DIR/YOLOv4_int8_bs256_inf_block_nchw.log
diff --git a/quickstart/object_detection/pytorch/yolov4/inference/gpu/setup.sh b/quickstart/object_detection/pytorch/yolov4/inference/gpu/setup.sh
deleted file mode 100755
index ae532a300..000000000
--- a/quickstart/object_detection/pytorch/yolov4/inference/gpu/setup.sh
+++ /dev/null
@@ -1,26 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2022 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-
-pip install matplotlib==2.2.3
-pip install tqdm==4.43.0
-pip install easydict==1.9
-pip install Pillow>=9.3.0
-pip install scikit-image
-pip install opencv-python-headless
-pip install pycocotools
-
diff --git a/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/fp32/.docs/aikit.md b/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/fp32/.docs/aikit.md
deleted file mode 100644
index 6fc902ce7..000000000
--- a/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/fp32/.docs/aikit.md
+++ /dev/null
@@ -1,84 +0,0 @@
-<!--- 50. AI Kit -->
-## Run the model
-
-Setup your environment using the instructions below, depending on if you are
-using [AI Kit](/docs/general/tensorflow/AIKit.md):
-
-<table>
-  <tr>
-    <th>Setup using AI Kit</th>
-    <th>Setup without AI Kit</th>
-  </tr>
-  <tr>
-    <td>
-      <p>AI Kit does not currently support TF 1.15.2 models</p>
-    </td>
-    <td>
-      <p>To run without AI Kit you will need:</p>
-      <ul>
-        <li>Python 3.6 or 3.7
-        <li>git
-        <li>numactl
-        <li>wget
-        <li><a href="https://github.com/tensorflow/models/blob/v1.12.0/research/object_detection/g3doc/installation.md#protobuf-compilation">Protobuf Compilation</a>
-        <li><a href="https://pypi.org/project/intel-tensorflow/1.15.2/">intel-tensorflow==1.15.2</a>
-        <li>Cython
-        <li>contextlib2
-        <li>jupyter
-        <li>lxml
-        <li>matplotlib
-        <li>pillow>=9.3.0
-        <li>pycocotools
-        <li>A clone of the Model Zoo repo<br />
-        <pre>git clone https://github.com/IntelAI/models.git</pre>
-      </ul>
-    </td>
-  </tr>
-</table>
-
-Running <model name> <precision> <mode> also requires cloning the TensorFlow
-models repo using the tag specified below. Set the `TF_MODELS_DIR` environment
-variable to point to the TensorFlow models directory. Run the
-[protobuf-compiler](https://github.com/tensorflow/models/blob/v1.12.0/research/object_detection/g3doc/installation.md#protobuf-compilation)
-on the `research` directory.
-```
-# Clone the TF models repo
-git clone https://github.com/tensorflow/models.git tf_models
-pushd tf_models
-git checkout tags/v1.12.0
-export TF_MODELS_DIR=$(pwd)
-
-# Run the protobuf-compiler from the TF models research directory
-pushd research
-wget -O protobuf.zip https://github.com/google/protobuf/releases/download/v3.0.0/protoc-3.0.0-linux-x86_64.zip
-unzip protobuf.zip
-./bin/protoc object_detection/protos/*.proto --python_out=.
-rm protobuf.zip
-popd
-popd
-```
-
-Download and extract the pretrained model. The path to this directory
-should be set to the `PRETRAINED_MODEL` environment variable before
-running the quickstart scripts.
-```
-wget https://storage.googleapis.com/intel-optimized-tensorflow/models/faster_rcnn_resnet50_fp32_coco_pretrained_model.tar.gz
-tar -xvf faster_rcnn_resnet50_fp32_coco_pretrained_model.tar.gz
-export PRETRAINED_MODEL=$(pwd)/faster_rcnn_resnet50_fp32_coco
-```
-
-In addition to the `TF_MODELS_DIR` and `PRETRAINED_MODEL` variables from
-above, set environment variables for the path to your `DATASET_DIR` (directory
-where the `coco_val.record` TF records file is located)  and an `OUTPUT_DIR`
-where log files will be written, then run a [quickstart script](#quick-start-scripts).
-```
-# cd to your model zoo directory
-cd models
-
-export DATASET_DIR=<path to the directory that contains the coco_val.record file>
-export OUTPUT_DIR=<path to the directory where log files will be written>
-export TF_MODELS_DIR=<path to the TensorFlow models dir>
-export PRETRAINED_MODEL=<path to the extracted pretrained model dir>
-
-./quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/fp32/<script name>.sh
-```
diff --git a/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/fp32/.docs/baremetal.md b/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/fp32/.docs/baremetal.md
deleted file mode 100644
index 946f39970..000000000
--- a/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/fp32/.docs/baremetal.md
+++ /dev/null
@@ -1,45 +0,0 @@
-<!--- 50. Bare Metal -->
-## Bare Metal
-
-To run on bare metal, the following prerequisites must be installed in your environment:
-* Python 3.6 or 3.7
-* git
-* numactl
-* wget
-* [Protobuf Compilation](https://github.com/tensorflow/models/blob/v1.12.0/research/object_detection/g3doc/installation.md#protobuf-compilation)
-* [intel-tensorflow==1.15.2](https://pypi.org/project/intel-tensorflow/1.15.2/)
-* Cython
-* contextlib2
-* jupyter
-* lxml
-* matplotlib
-* pillow>=9.3.0
-* pycocotools
-
-Clone the [TensorFlow Model Garden](https://github.com/tensorflow/models)
-repository using the specified tag, and save the path to the `TF_MODELS_DIR`
-environment variable.
-```
-# Clone the TF models repo
-git clone https://github.com/tensorflow/models.git
-pushd models
-git checkout tags/v1.12.0
-export TF_MODELS_DIR=$(pwd)
-popd
-```
-
-Download and extract the model package, which includes the pretrained
-model and scripts needed to run inference. Set environment variables
-for the path to your `DATASET_DIR` (directory where the `coco_val.record`
-TF records file is located) and an `OUTPUT_DIR` where log files will be
-written, then run a [quickstart script](#quick-start-scripts).
-```
-wget <package url>
-tar -xzf <package name>
-cd <package dir>
-
-export DATASET_DIR=<path to the directory that contains the coco_val.record file>
-export OUTPUT_DIR=<directory where log files will be written>
-
-./quickstart/<script name>.sh
-```
diff --git a/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/fp32/.docs/datasets.md b/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/fp32/.docs/datasets.md
deleted file mode 100644
index 2e462d44e..000000000
--- a/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/fp32/.docs/datasets.md
+++ /dev/null
@@ -1,8 +0,0 @@
-<!--- 30. Datasets -->
-## Datasets
-
-The [COCO validation dataset](http://cocodataset.org) is used in the
-<model name> quickstart scripts. The scripts require that the dataset
-has been converted to the TF records format. See the
-[COCO dataset](/datasets/coco/README.md) for instructions on downloading
-and preprocessing the COCO validation dataset.
diff --git a/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/fp32/.docs/description.md b/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/fp32/.docs/description.md
deleted file mode 100644
index a0718bd31..000000000
--- a/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/fp32/.docs/description.md
+++ /dev/null
@@ -1,5 +0,0 @@
-<!-- 10. Description -->
-## Description
-
-This document has instructions for running <model name> <precision> <mode> using
-Intel-optimized TensorFlow.
diff --git a/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/fp32/.docs/docker.md b/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/fp32/.docs/docker.md
deleted file mode 100644
index 35389bbf8..000000000
--- a/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/fp32/.docs/docker.md
+++ /dev/null
@@ -1,27 +0,0 @@
-<!--- 60. Docker -->
-## Docker
-
-The model container includes the scripts and libraries needed to run 
-<model name> <precision> <mode> and the pretrained model. To run one of the quickstart scripts
-using this container, you'll need to provide volume mounts for the dataset 
-and an output directory.
-
-```
-DATASET_DIR=<path to the dataset>
-OUTPUT_DIR=<directory where log files will be written>
-
-docker run \
-  --env DATASET_DIR=${DATASET_DIR} \
-  --env OUTPUT_DIR=${OUTPUT_DIR} \
-  --env http_proxy=${http_proxy} \
-  --env https_proxy=${https_proxy} \
-  --volume ${DATASET_DIR}:${DATASET_DIR} \
-  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
-  --privileged --init -t \
-  <docker image> \
-  /bin/bash quickstart/<script name>.sh
-```
-
-If you are new to docker and are running into issues with the container,
-see [this document](https://github.com/IntelAI/models/tree/master/docs/general/docker.md)
-for troubleshooting tips.
diff --git a/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/fp32/.docs/download.md b/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/fp32/.docs/download.md
deleted file mode 100644
index 613926758..000000000
--- a/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/fp32/.docs/download.md
+++ /dev/null
@@ -1,4 +0,0 @@
-<!--- 20. Download link -->
-## Download link
-
-[<package name>](<package url>)
diff --git a/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/fp32/.docs/license.md b/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/fp32/.docs/license.md
deleted file mode 100644
index a60f169f4..000000000
--- a/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/fp32/.docs/license.md
+++ /dev/null
@@ -1,4 +0,0 @@
-<!--- 80. License -->
-## License
-
-[LICENSE](/LICENSE)
diff --git a/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/fp32/.docs/quickstart.md b/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/fp32/.docs/quickstart.md
deleted file mode 100644
index a97b256de..000000000
--- a/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/fp32/.docs/quickstart.md
+++ /dev/null
@@ -1,7 +0,0 @@
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| [fp32_inference.sh](/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/fp32/fp32_inference.sh) | Runs batch and online inference using the coco dataset |
-| [fp32_accuracy.sh](/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/fp32/fp32_accuracy.sh) | Runs inference and evaluates the model's accuracy |
diff --git a/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/fp32/.docs/title.md b/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/fp32/.docs/title.md
deleted file mode 100644
index 6416ff7ff..000000000
--- a/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/fp32/.docs/title.md
+++ /dev/null
@@ -1,2 +0,0 @@
-<!--- 0. Title -->
-# <model name> <precision> <mode>
diff --git a/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/fp32/README.md b/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/fp32/README.md
deleted file mode 100644
index 9f616df10..000000000
--- a/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/fp32/README.md
+++ /dev/null
@@ -1,110 +0,0 @@
-<!--- 0. Title -->
-# Faster R-CNN FP32 inference
-
-<!-- 10. Description -->
-## Description
-
-This document has instructions for running Faster R-CNN FP32 inference using
-Intel-optimized TensorFlow.
-
-<!--- 20. Download link -->
-## Download link
-
-[faster-rcnn-fp32-inference.tar.gz](https://storage.googleapis.com/intel-optimized-tensorflow/models/v2_7_0/faster-rcnn-fp32-inference.tar.gz)
-
-<!--- 30. Datasets -->
-## Datasets
-
-The [COCO validation dataset](http://cocodataset.org) is used in the
-Faster R-CNN quickstart scripts. The scripts require that the dataset
-has been converted to the TF records format. See the
-[COCO dataset](/datasets/coco/README.md) for instructions on downloading
-and preprocessing the COCO validation dataset.
-
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| [fp32_inference.sh](/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/fp32/fp32_inference.sh) | Runs batch and online inference using the coco dataset |
-| [fp32_accuracy.sh](/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/fp32/fp32_accuracy.sh) | Runs inference and evaluates the model's accuracy |
-
-<!--- 50. Bare Metal -->
-## Bare Metal
-
-To run on bare metal, the following prerequisites must be installed in your environment:
-* Python 3.6 or 3.7
-* git
-* numactl
-* wget
-* [Protobuf Compilation](https://github.com/tensorflow/models/blob/v1.12.0/research/object_detection/g3doc/installation.md#protobuf-compilation)
-* [intel-tensorflow==1.15.2](https://pypi.org/project/intel-tensorflow/1.15.2/)
-* Cython
-* contextlib2
-* jupyter
-* lxml
-* matplotlib
-* pillow>=9.3.0
-* pycocotools
-
-Clone the [TensorFlow Model Garden](https://github.com/tensorflow/models)
-repository using the specified tag, and save the path to the `TF_MODELS_DIR`
-environment variable.
-```
-# Clone the TF models repo
-git clone https://github.com/tensorflow/models.git
-pushd models
-git checkout tags/v1.12.0
-export TF_MODELS_DIR=$(pwd)
-popd
-```
-
-Download and extract the model package, which includes the pretrained
-model and scripts needed to run inference. Set environment variables
-for the path to your `DATASET_DIR` (directory where the `coco_val.record`
-TF records file is located) and an `OUTPUT_DIR` where log files will be
-written, then run a [quickstart script](#quick-start-scripts).
-```
-wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v2_7_0/faster-rcnn-fp32-inference.tar.gz
-tar -xzf faster-rcnn-fp32-inference.tar.gz
-cd faster-rcnn-fp32-inference
-
-export DATASET_DIR=<path to the directory that contains the coco_val.record file>
-export OUTPUT_DIR=<directory where log files will be written>
-
-./quickstart/<script name>.sh
-```
-
-<!--- 60. Docker -->
-## Docker
-
-The model container includes the scripts and libraries needed to run 
-Faster R-CNN FP32 inference and the pretrained model. To run one of the quickstart scripts
-using this container, you'll need to provide volume mounts for the dataset 
-and an output directory.
-
-```
-DATASET_DIR=<path to the dataset>
-OUTPUT_DIR=<directory where log files will be written>
-
-docker run \
-  --env DATASET_DIR=${DATASET_DIR} \
-  --env OUTPUT_DIR=${OUTPUT_DIR} \
-  --env http_proxy=${http_proxy} \
-  --env https_proxy=${https_proxy} \
-  --volume ${DATASET_DIR}:${DATASET_DIR} \
-  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
-  --privileged --init -t \
-  intel/object-detection:tf-1.15.2-faster-rcnn-fp32-inference \
-  /bin/bash quickstart/<script name>.sh
-```
-
-If you are new to docker and are running into issues with the container,
-see [this document](https://github.com/IntelAI/models/tree/master/docs/general/docker.md)
-for troubleshooting tips.
-
-<!--- 80. License -->
-## License
-
-[LICENSE](/LICENSE)
-
diff --git a/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/fp32/fp32_accuracy.sh b/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/fp32/fp32_accuracy.sh
deleted file mode 100755
index 2fb92ebe7..000000000
--- a/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/fp32/fp32_accuracy.sh
+++ /dev/null
@@ -1,83 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2020 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-MODEL_DIR=${MODEL_DIR-$PWD}
-
-echo 'MODEL_DIR='$MODEL_DIR
-echo 'OUTPUT_DIR='$OUTPUT_DIR
-echo 'DATASET_DIR='$DATASET_DIR
-
-if [[ -z ${OUTPUT_DIR} ]]; then
-  echo "The required environment variable OUTPUT_DIR has not been set"
-  exit 1
-fi
-
-# Create the output directory in case it doesn't already exist
-mkdir -p ${OUTPUT_DIR}
-
-if [[ -z ${DATASET_DIR} ]]; then
-  echo "The required environment variable DATASET_DIR has not been set"
-  exit 1
-fi
-
-if [[ ! -d ${DATASET_DIR} ]]; then
-  echo "The DATASET_DIR '${DATASET_DIR}' does not exist"
-  exit 1
-fi
-
-if [[ -z ${TF_MODELS_DIR} ]]; then
-  echo "The required environment variable TF_MODELS_DIR has not been set"
-  exit 1
-fi
-
-if [[ ! -d ${TF_MODELS_DIR} ]]; then
-  echo "The TF_MODELS_DIR '${TF_MODELS_DIR}' does not exist"
-  exit 1
-fi
-
-# If a path to the pretrained model dir was not provided, unzip the pretrained
-# model from the model package
-if [[ -z ${PRETRAINED_MODEL} ]]; then
-  tar -xvf ${MODEL_DIR}/faster_rcnn_resnet50_fp32_coco_pretrained_model.tar.gz
-  PRETRAINED_MODEL=${MODEL_DIR}/faster_rcnn_resnet50_fp32_coco
-fi
-
-if [[ ! ${PRETRAINED_MODEL} ]]; then
-  echo "The pretrained model directory (${PRETRAINED_MODEL}) does not exist."
-  exit 1
-fi
-
-# Replace paths in the pipeline config file
-line_128=$(sed -n '128p' ${PRETRAINED_MODEL}/pipeline.config)
-new_line_128="  label_map_path: \"$PRETRAINED_MODEL/mscoco_label_map.pbtxt\""
-sed -i.bak "128s+$line_128+$new_line_128+" ${PRETRAINED_MODEL}/pipeline.config
-line_132=$(sed -n '132p' ${PRETRAINED_MODEL}/pipeline.config)
-new_line_132="    input_path: \"$DATASET_DIR/coco_val.record\""
-sed -i.bak "132s+$line_132+$new_line_132+" ${PRETRAINED_MODEL}/pipeline.config
-
-source "${MODEL_DIR}/quickstart/common/utils.sh"
-_command python ${MODEL_DIR}/benchmarks/launch_benchmark.py \
-  --model-name faster_rcnn \
-  --mode inference \
-  --precision fp32 \
-  --framework tensorflow \
-  --model-source-dir ${TF_MODELS_DIR} \
-  --output-dir ${OUTPUT_DIR} \
-  --data-location ${DATASET_DIR} \
-  --in-graph ${PRETRAINED_MODEL}/frozen_inference_graph.pb \
-  --accuracy-only \
-  $@
diff --git a/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/fp32/fp32_inference.sh b/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/fp32/fp32_inference.sh
deleted file mode 100755
index 87eeee56c..000000000
--- a/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/fp32/fp32_inference.sh
+++ /dev/null
@@ -1,84 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2020 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-MODEL_DIR=${MODEL_DIR-$PWD}
-
-echo 'MODEL_DIR='$MODEL_DIR
-echo 'OUTPUT_DIR='$OUTPUT_DIR
-echo 'DATASET_DIR='$DATASET_DIR
-
-if [[ -z ${OUTPUT_DIR} ]]; then
-  echo "The required environment variable OUTPUT_DIR has not been set"
-  exit 1
-fi
-
-# Create the output directory in case it doesn't already exist
-mkdir -p ${OUTPUT_DIR}
-
-if [[ -z ${DATASET_DIR} ]]; then
-  echo "The required environment variable DATASET_DIR has not been set"
-  exit 1
-fi
-
-if [[ ! -d ${DATASET_DIR} ]]; then
-  echo "The DATASET_DIR '${DATASET_DIR}' does not exist"
-  exit 1
-fi
-
-if [[ -z ${TF_MODELS_DIR} ]]; then
-  echo "The required environment variable TF_MODELS_DIR has not been set"
-  exit 1
-fi
-
-if [[ ! -d ${TF_MODELS_DIR} ]]; then
-  echo "The TF_MODELS_DIR '${TF_MODELS_DIR}' does not exist"
-  exit 1
-fi
-
-# If a path to the pretrained model dir was not provided, unzip the pretrained
-# model from the model package
-if [[ -z ${PRETRAINED_MODEL} ]]; then
-  tar -xvf ${MODEL_DIR}/faster_rcnn_resnet50_fp32_coco_pretrained_model.tar.gz
-  PRETRAINED_MODEL=${MODEL_DIR}/faster_rcnn_resnet50_fp32_coco
-fi
-
-if [[ ! -d ${PRETRAINED_MODEL} ]]; then
-  echo "The pretrained model directory (${PRETRAINED_MODEL}) does not exist."
-  exit 1
-fi
-
-# Replace paths in the pipeline config file for line 128 (pretrained model) and 132 (dataset dir)
-line_128=$(sed -n '128p' ${PRETRAINED_MODEL}/pipeline.config)
-new_line_128="  label_map_path: \"$PRETRAINED_MODEL/mscoco_label_map.pbtxt\""
-sed -i.bak "128s+$line_128+$new_line_128+" ${PRETRAINED_MODEL}/pipeline.config
-line_132=$(sed -n '132p' ${PRETRAINED_MODEL}/pipeline.config)
-new_line_132="    input_path: \"$DATASET_DIR/coco_val.record\""
-sed -i.bak "132s+$line_132+$new_line_132+" ${PRETRAINED_MODEL}/pipeline.config
-
-source "${MODEL_DIR}/quickstart/common/utils.sh"
-_command python ${MODEL_DIR}/benchmarks/launch_benchmark.py \
-  --data-location ${DATASET_DIR} \
-  --output-dir ${OUTPUT_DIR} \
-  --model-source-dir ${TF_MODELS_DIR} \
-  --model-name faster_rcnn \
-  --framework tensorflow \
-  --precision fp32 \
-  --mode inference \
-  --socket-id 0 \
-  --checkpoint $PRETRAINED_MODEL \
-  $@ \
-  -- config_file=pipeline.config
diff --git a/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/int8/.docs/aikit.md b/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/int8/.docs/aikit.md
deleted file mode 100644
index 62271f8b9..000000000
--- a/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/int8/.docs/aikit.md
+++ /dev/null
@@ -1,82 +0,0 @@
-<!--- 50. AI Kit -->
-## Run the model
-
-Setup your environment using the instructions below, depending on if you are
-using [AI Kit](/docs/general/tensorflow/AIKit.md):
-
-<table>
-  <tr>
-    <th>Setup using AI Kit</th>
-    <th>Setup without AI Kit</th>
-  </tr>
-  <tr>
-    <td>
-      <p>AI Kit does not currently support TF 1.15.2 models</p>
-    </td>
-    <td>
-      <p>To run without AI Kit you will need:</p>
-      <ul>
-        <li>Python 3.6 or 3.7
-        <li>git
-        <li>numactl
-        <li>wget
-        <li><a href="https://github.com/tensorflow/models/blob/v1.12.0/research/object_detection/g3doc/installation.md#protobuf-compilation">Protobuf Compilation</a>
-        <li><a href="https://pypi.org/project/intel-tensorflow/1.15.2/">intel-tensorflow==1.15.2</a>
-        <li>Cython
-        <li>contextlib2
-        <li>jupyter
-        <li>lxml
-        <li>matplotlib
-        <li>pillow>=9.3.0
-        <li>pycocotools
-        <li>A clone of the Model Zoo repo<br />
-        <pre>git clone https://github.com/IntelAI/models.git</pre>
-      </ul>
-    </td>
-  </tr>
-</table>
-
-Running <model name> <precision> <mode> also requires cloning the TensorFlow
-models repo using the tag specified below. Set the `TF_MODELS_DIR` environment
-variable to point to the TensorFlow models directory. Run the
-[protobuf-compiler](https://github.com/tensorflow/models/blob/v1.12.0/research/object_detection/g3doc/installation.md#protobuf-compilation)
-on the `research` directory.
-```
-# Clone the TF models repo
-git clone https://github.com/tensorflow/models.git tf_models
-pushd tf_models
-git checkout tags/v1.12.0
-export TF_MODELS_DIR=$(pwd)
-
-# Run the protobuf-compiler from the TF models research directory
-pushd research
-wget -O protobuf.zip https://github.com/google/protobuf/releases/download/v3.0.0/protoc-3.0.0-linux-x86_64.zip
-unzip protobuf.zip
-./bin/protoc object_detection/protos/*.proto --python_out=.
-rm protobuf.zip
-popd
-popd
-```
-
-Download the pretrained model and set the `PRETRAINED_MODEL` environment
-variable to point to the frozen graph. The quickstart scripts will use this var.
-```
-wget https://storage.googleapis.com/intel-optimized-tensorflow/models/faster_rcnn_int8_pretrained_model.pb
-export PRETRAINED_MODEL=$(pwd)/faster_rcnn_int8_pretrained_model.pb
-```
-
-In addition to the `TF_MODELS_DIR` and `PRETRAINED_MODEL` variables from
-above, set environment variables for the path to your `DATASET_DIR` (where
-the coco TF records file is located) and an `OUTPUT_DIR` where log files
-will be written, then run a [quickstart script](#quick-start-scripts).
-```
-# cd to your model zoo directory
-cd models
-
-export DATASET_DIR=<path to the coco dataset>
-export OUTPUT_DIR=<path to the directory where log files will be written>
-export TF_MODELS_DIR=<path to the TensorFlow models dir>
-export PRETRAINED_MODEL=<path to the pretrained model frozen graph file>
-
-./quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/int8/<script name>.sh
-```
diff --git a/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/int8/.docs/baremetal.md b/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/int8/.docs/baremetal.md
deleted file mode 100644
index 1e63ea3b2..000000000
--- a/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/int8/.docs/baremetal.md
+++ /dev/null
@@ -1,47 +0,0 @@
-<!--- 50. Bare Metal -->
-## Bare Metal
-
-To run on bare metal, the following prerequisites must be installed in your environment:
-* Python 3.6 or 3.7
-* git
-* numactl
-* wget
-* [Protobuf Compilation](https://github.com/tensorflow/models/blob/v1.12.0/research/object_detection/g3doc/installation.md#protobuf-compilation)
-* [intel-tensorflow==1.15.2](https://pypi.org/project/intel-tensorflow/1.15.2/)
-* Cython
-* contextlib2
-* jupyter
-* lxml
-* matplotlib
-* pillow>=9.3.0
-* pycocotools
-
-Clone the [TensorFlow Model Garden](https://github.com/tensorflow/models)
-repository using the specified tag, and save the path to the `TF_MODELS_DIR`
-environment variable.
-```
-# Clone the TF models repo
-git clone https://github.com/tensorflow/models.git
-pushd models
-git checkout tags/v1.12.0
-export TF_MODELS_DIR=$(pwd)
-popd
-```
-
-After installing the prerequisites, download and extract the model
-package, which includes the pretrained model and scripts needed
-to run inference. Set environment variables for the path to
-your `DATASET_DIR` (where the coco TF records file is
-located) and an `OUTPUT_DIR` where log files will be written, then run a
-[quickstart script](#quick-start-scripts).
-
-```
-wget <package url>
-tar -xzf <package name>
-cd <package dir>
-
-export DATASET_DIR=<path to the coco dataset>
-export OUTPUT_DIR=<directory where log files will be written>
-
-./quickstart/<script name>.sh
-```
diff --git a/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/int8/.docs/datasets.md b/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/int8/.docs/datasets.md
deleted file mode 100644
index f05a7a05a..000000000
--- a/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/int8/.docs/datasets.md
+++ /dev/null
@@ -1,10 +0,0 @@
-<!--- 30. Datasets -->
-## Datasets
-
-The [COCO validation dataset](http://cocodataset.org) is used in the
-<model name> quickstart scripts. The scripts require that the dataset
-has been converted to the TF records format. See the
-[COCO dataset](/datasets/coco/README.md) for instructions on downloading
-and preprocessing the COCO validation dataset.
-
-Set the `DATASET_DIR` to point to this directory when running <model name>.
diff --git a/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/int8/.docs/description.md b/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/int8/.docs/description.md
deleted file mode 100644
index 10e98e2c2..000000000
--- a/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/int8/.docs/description.md
+++ /dev/null
@@ -1,5 +0,0 @@
-<!-- 10. Description -->
-## Description
-
-This document has instructions for running <model name> <precision> <mode> using
-Intel Optimized TensorFlow.
diff --git a/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/int8/.docs/docker.md b/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/int8/.docs/docker.md
deleted file mode 100644
index 3b5ad43df..000000000
--- a/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/int8/.docs/docker.md
+++ /dev/null
@@ -1,27 +0,0 @@
-<!--- 60. Docker -->
-## Docker
-
-The model container includes the scripts and libraries needed to run 
-<model name> <precision> <mode>. To run one of the quickstart scripts 
-using this container, you'll need to provide volume mounts for the dataset 
-and an output directory.
-
-```
-DATASET_DIR=<path to the dataset>
-OUTPUT_DIR=<directory where log files will be written>
-
-docker run \
-  --env DATASET_DIR=${DATASET_DIR} \
-  --env OUTPUT_DIR=${OUTPUT_DIR} \
-  --env http_proxy=${http_proxy} \
-  --env https_proxy=${https_proxy} \
-  --volume ${DATASET_DIR}:${DATASET_DIR} \
-  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
-  --privileged --init -t \
-  <docker image> \
-  /bin/bash quickstart/<script name>.sh
-```
-
-If you are new to docker and are running into issues with the container,
-see [this document](https://github.com/IntelAI/models/tree/master/docs/general/docker.md)
-for troubleshooting tips.
diff --git a/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/int8/.docs/download.md b/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/int8/.docs/download.md
deleted file mode 100644
index 613926758..000000000
--- a/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/int8/.docs/download.md
+++ /dev/null
@@ -1,4 +0,0 @@
-<!--- 20. Download link -->
-## Download link
-
-[<package name>](<package url>)
diff --git a/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/int8/.docs/license.md b/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/int8/.docs/license.md
deleted file mode 100644
index a60f169f4..000000000
--- a/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/int8/.docs/license.md
+++ /dev/null
@@ -1,4 +0,0 @@
-<!--- 80. License -->
-## License
-
-[LICENSE](/LICENSE)
diff --git a/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/int8/.docs/quickstart.md b/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/int8/.docs/quickstart.md
deleted file mode 100644
index 4b083c31b..000000000
--- a/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/int8/.docs/quickstart.md
+++ /dev/null
@@ -1,12 +0,0 @@
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| [int8_inference.sh](int8_inference.sh) | Runs batch and online inference using the raw coco dataset |
-| [int8_accuracy.sh](int8_accuracy.sh) | Runs inference and evaluates the model's accuracy with coco dataset in `coco_val.record` format |
-
-These quickstart scripts can be run in different environments:
-* [Bare Metal](#bare-metal)
-* [Docker](#docker)
-* [Kubernetes](#kubernetes)
diff --git a/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/int8/.docs/title.md b/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/int8/.docs/title.md
deleted file mode 100644
index 6416ff7ff..000000000
--- a/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/int8/.docs/title.md
+++ /dev/null
@@ -1,2 +0,0 @@
-<!--- 0. Title -->
-# <model name> <precision> <mode>
diff --git a/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/int8/README.md b/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/int8/README.md
deleted file mode 100644
index 590e03d3c..000000000
--- a/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/int8/README.md
+++ /dev/null
@@ -1,119 +0,0 @@
-<!--- 0. Title -->
-# Faster_RCNN Int8 inference
-
-<!-- 10. Description -->
-## Description
-
-This document has instructions for running Faster_RCNN Int8 inference using
-Intel Optimized TensorFlow.
-
-<!--- 20. Download link -->
-## Download link
-
-[faster_rcnn-int8-inference.tar.gz](https://storage.googleapis.com/intel-optimized-tensorflow/models/v2_7_0/faster-rcnn-int8-inference.tar.gz)
-
-<!--- 30. Datasets -->
-## Datasets
-
-The [COCO validation dataset](http://cocodataset.org) is used in the
-Faster_RCNN quickstart scripts. The scripts require that the dataset
-has been converted to the TF records format. See the
-[COCO dataset](/datasets/coco/README.md) for instructions on downloading
-and preprocessing the COCO validation dataset.
-
-Set the `DATASET_DIR` to point to this directory when running Faster_RCNN.
-
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| [int8_inference.sh](int8_inference.sh) | Runs batch and online inference using the raw coco dataset |
-| [int8_accuracy.sh](int8_accuracy.sh) | Runs inference and evaluates the model's accuracy with coco dataset in `coco_val.record` format |
-
-These quickstart scripts can be run in different environments:
-* [Bare Metal](#bare-metal)
-* [Docker](#docker)
-* [Kubernetes](#kubernetes)
-
-<!--- 50. Bare Metal -->
-## Bare Metal
-
-To run on bare metal, the following prerequisites must be installed in your environment:
-* Python 3.6 or 3.7
-* git
-* numactl
-* wget
-* [Protobuf Compilation](https://github.com/tensorflow/models/blob/v1.12.0/research/object_detection/g3doc/installation.md#protobuf-compilation)
-* [intel-tensorflow==1.15.2](https://pypi.org/project/intel-tensorflow/1.15.2/)
-* Cython
-* contextlib2
-* jupyter
-* lxml
-* matplotlib
-* pillow>=9.3.0
-* pycocotools
-
-Clone the [TensorFlow Model Garden](https://github.com/tensorflow/models)
-repository using the specified tag, and save the path to the `TF_MODELS_DIR`
-environment variable.
-```
-# Clone the TF models repo
-git clone https://github.com/tensorflow/models.git
-pushd models
-git checkout tags/v1.12.0
-export TF_MODELS_DIR=$(pwd)
-popd
-```
-
-After installing the prerequisites, download and extract the model
-package, which includes the pretrained model and scripts needed
-to run inference. Set environment variables for the path to
-your `DATASET_DIR` (where the coco TF records file is
-located) and an `OUTPUT_DIR` where log files will be written, then run a
-[quickstart script](#quick-start-scripts).
-
-```
-wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v2_7_0/faster-rcnn-int8-inference.tar.gz
-tar -xzf faster_rcnn-int8-inference.tar.gz
-cd faster_rcnn-int8-inference
-
-export DATASET_DIR=<path to the coco dataset>
-export OUTPUT_DIR=<directory where log files will be written>
-
-./quickstart/<script name>.sh
-```
-
-<!--- 60. Docker -->
-## Docker
-
-The model container includes the scripts and libraries needed to run 
-Faster_RCNN Int8 inference. To run one of the quickstart scripts 
-using this container, you'll need to provide volume mounts for the dataset 
-and an output directory.
-
-```
-DATASET_DIR=<path to the dataset>
-OUTPUT_DIR=<directory where log files will be written>
-
-docker run \
-  --env DATASET_DIR=${DATASET_DIR} \
-  --env OUTPUT_DIR=${OUTPUT_DIR} \
-  --env http_proxy=${http_proxy} \
-  --env https_proxy=${https_proxy} \
-  --volume ${DATASET_DIR}:${DATASET_DIR} \
-  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
-  --privileged --init -t \
-  intel/object-detection:tf-1.15.2-faster-rcnn-int8-inference \
-  /bin/bash quickstart/<script name>.sh
-```
-
-If you are new to docker and are running into issues with the container,
-see [this document](https://github.com/IntelAI/models/tree/master/docs/general/docker.md)
-for troubleshooting tips.
-
-<!--- 80. License -->
-## License
-
-[LICENSE](/LICENSE)
-
diff --git a/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/int8/int8_accuracy.sh b/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/int8/int8_accuracy.sh
deleted file mode 100755
index 1a972126a..000000000
--- a/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/int8/int8_accuracy.sh
+++ /dev/null
@@ -1,71 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2020 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-MODEL_DIR=${MODEL_DIR-$PWD}
-
-echo 'MODEL_DIR='$MODEL_DIR
-echo 'OUTPUT_DIR='$OUTPUT_DIR
-echo 'DATASET_DIR='$DATASET_DIR
-
-if [[ -z ${OUTPUT_DIR} ]]; then
-  echo "The required environment variable OUTPUT_DIR has not been set"
-  exit 1
-fi
-
-# Create the output directory in case it doesn't already exist
-mkdir -p ${OUTPUT_DIR}
-
-if [[ -z ${DATASET_DIR} ]]; then
-  echo "The required environment variable DATASET_DIR has not been set"
-  exit 1
-fi
-
-if [[ ! -d ${DATASET_DIR} ]]; then
-  echo "The DATASET_DIR '${DATASET_DIR}' does not exist"
-  exit 1
-fi
-
-if [[ -z ${TF_MODELS_DIR} ]]; then
-  echo "The required environment variable TF_MODELS_DIR has not been set"
-  exit 1
-fi
-
-if [[ ! -d ${TF_MODELS_DIR} ]]; then
-  echo "The TF_MODELS_DIR '${TF_MODELS_DIR}' does not exist"
-  exit 1
-fi
-
-PRETRAINED_MODEL=${PRETRAINED_MODEL-${MODEL_DIR}/faster_rcnn_int8_pretrained_model.pb}
-
-if [[ ! -f ${PRETRAINED_MODEL} ]]; then
-  echo "The pretrained model (${PRETRAINED_MODEL}) does not exist."
-  exit 1
-fi
-
-source "${MODEL_DIR}/quickstart/common/utils.sh"
-_command python ${MODEL_DIR}/benchmarks/launch_benchmark.py \
-  --model-name faster_rcnn \
-  --mode inference \
-  --precision int8 \
-  --framework tensorflow \
-  --model-source-dir ${TF_MODELS_DIR} \
-  --output-dir ${OUTPUT_DIR} \
-  --data-location ${DATASET_DIR} \
-  --in-graph ${PRETRAINED_MODEL} \
-  --socket-id 0 \
-  --accuracy-only \
-  $@
diff --git a/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/int8/int8_inference.sh b/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/int8/int8_inference.sh
deleted file mode 100755
index 2ff0ef700..000000000
--- a/quickstart/object_detection/tensorflow/faster_rcnn/inference/cpu/int8/int8_inference.sh
+++ /dev/null
@@ -1,73 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2020 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-MODEL_DIR=${MODEL_DIR-$PWD}
-
-echo 'MODEL_DIR='$MODEL_DIR
-echo 'OUTPUT_DIR='$OUTPUT_DIR
-echo 'DATASET_DIR='$DATASET_DIR
-
-if [[ -z ${OUTPUT_DIR} ]]; then
-  echo "The required environment variable OUTPUT_DIR has not been set"
-  exit 1
-fi
-
-# Create the output directory in case it doesn't already exist
-mkdir -p ${OUTPUT_DIR}
-
-if [[ -z ${DATASET_DIR} ]]; then
-  echo "The required environment variable DATASET_DIR has not been set"
-  exit 1
-fi
-
-if [[ ! -d ${DATASET_DIR} ]]; then
-  echo "The DATASET_DIR '${DATASET_DIR}' does not exist"
-  exit 1
-fi
-
-if [[ -z ${TF_MODELS_DIR} ]]; then
-  echo "The required environment variable TF_MODELS_DIR has not been set"
-  exit 1
-fi
-
-if [[ ! -d ${TF_MODELS_DIR} ]]; then
-  echo "The TF_MODELS_DIR '${TF_MODELS_DIR}' does not exist"
-  exit 1
-fi
-
-PRETRAINED_MODEL=${PRETRAINED_MODEL-${MODEL_DIR}/faster_rcnn_int8_pretrained_model.pb}
-
-if [[ ! -f ${PRETRAINED_MODEL} ]]; then
-  echo "The pretrained model (${PRETRAINED_MODEL}) does not exist."
-  exit 1
-fi
-
-source "${MODEL_DIR}/quickstart/common/utils.sh"
-_command python ${MODEL_DIR}/benchmarks/launch_benchmark.py \
-  --data-location ${DATASET_DIR} \
-  --output-dir ${OUTPUT_DIR} \
-  --model-source-dir ${TF_MODELS_DIR} \
-  --model-name faster_rcnn \
-  --framework tensorflow \
-  --precision int8 \
-  --mode inference \
-  --socket-id 0 \
-  --in-graph ${PRETRAINED_MODEL} \
-  --benchmark-only \
-  $@ \
-  -- number_of_steps=5000
-
diff --git a/quickstart/object_detection/tensorflow/rfcn/inference/cpu/.docs/baremetal.md b/quickstart/object_detection/tensorflow/rfcn/inference/cpu/.docs/baremetal.md
deleted file mode 100644
index 9f2a1b82d..000000000
--- a/quickstart/object_detection/tensorflow/rfcn/inference/cpu/.docs/baremetal.md
+++ /dev/null
@@ -1,184 +0,0 @@
-<!--- 50. AI Kit -->
-## Run the model
-
-Setup your environment using the instructions below, depending on if you are
-using [AI Kit](/docs/general/tensorflow/AIKit.md):
-
-<table>
-  <tr>
-    <th>Setup using AI Kit on Linux</th>
-    <th>Setup without AI Kit on Linux</th>
-    <th>Setup without AI Kit on Windows</th>
-  </tr>
-  <tr>
-    <td>
-      <p>To run using AI Kit on Linux you will need:</p>
-      <ul>
-        <li>git
-        <li>numactl
-        <li>wget
-        <li>build-essential
-        <li>Cython
-        <li>contextlib2
-        <li>jupyter
-        <li>lxml
-        <li>matplotlib
-        <li>pillow>=9.3.0
-        <li>protobuf-compiler
-        <li>pycocotools
-        <li>Activate the `tensorflow` conda environment
-        <pre>conda activate tensorflow</pre>
-      </ul>
-    </td>
-    <td>
-      <p>To run without AI Kit on Linux you will need:</p>
-      <ul>
-        <li>Python 3
-        <li>git
-        <li>numactl
-        <li>wget
-        <li><a href="https://pypi.org/project/intel-tensorflow/">intel-tensorflow>=2.5.0</a>
-        <li>build-essential
-        <li>Cython
-        <li>contextlib2
-        <li>jupyter
-        <li>lxml
-        <li>matplotlib
-        <li>pillow>=9.3.0
-        <li>protobuf-compiler
-        <li>pycocotools
-        <li>A clone of the Model Zoo repo<br />
-        <pre>git clone https://github.com/IntelAI/models.git</pre>
-      </ul>
-    </td>
-    <td>
-      <p>To run without AI Kit on Windows you will need:</p>
-      <ul>
-        <li><a href="/docs/general/Windows.md">Intel Model Zoo on Windows Systems prerequisites</a>
-        <li>build-essential
-        <li>Cython
-        <li>contextlib2
-        <li>jupyter
-        <li>lxml
-        <li>matplotlib
-        <li>pillow>=9.3.0
-        <li>protobuf-compiler
-        <li>pycocotools
-        <li>A clone of the Model Zoo repo<br />
-        <pre>git clone https://github.com/IntelAI/models.git</pre>
-      </ul>
-    </td>
-  </tr>
-</table>
-
-For more information on the required dependencies, see the documentation on [prerequisites](https://github.com/tensorflow/models/blob/6c21084503b27a9ab118e1db25f79957d5ef540b/research/object_detection/g3doc/installation.md#installation)
-in the TensorFlow models repo.
-
-Download and extract the pretrained model and set the `PRETRAINED_MODEL`
-environment variable to point to the frozen graph file.
-If you run on Windows, please use a browser to download the pretrained model using the link below.
-For Linux, run:
-```
-# FP32 Pretrained Model
-https://storage.googleapis.com/intel-optimized-tensorflow/models/2_10_0/rfcn_frozen_inference_graph.pb
-export PRETRAINED_MODEL=$(pwd)/rfcn_frozen_inference_graph.pb
-
-# Int8 Pretrained Model
-https://storage.googleapis.com/intel-optimized-tensorflow/models/2_10_0/rfcn_final_fused_pad_and_conv.pb
-export PRETRAINED_MODEL=$(pwd)/rfcn_final_fused_pad_and_conv.pb
-```
-
-RFCN uses the object detection code from the [TensorFlow Model Garden](https://github.com/tensorflow/models).
-Clone this repo with the SHA specified below and apply the patch from the model zoo directory.
-Set the `TF_MODELS_DIR` environment variable to the path of your clone of the TF Model Garden.
-```
-# Clone the TensorFlow Model Garden
-git clone https://github.com/tensorflow/models.git tensorflow-models
-cd tensorflow-models
-git checkout 6c21084503b27a9ab118e1db25f79957d5ef540b
-
-# Apply the TF2 patch from the model zoo repo directory
-git apply --ignore-space-change --ignore-whitespace <model zoo directory>/models/object_detection/tensorflow/rfcn/inference/tf-2.0.patch
-
-# Set the TF_MODELS_DIR env var
-export TF_MODELS_DIR=$(pwd)
-```
-
-### Run on Linux
-Download and install [Google Protobuf version 3.3.0](https://github.com/google/protobuf/releases/download/v3.3.0/protoc-3.3.0-linux-x86_64.zip), and
-run [protobuf compilation](https://github.com/tensorflow/models/blob/6c21084503b27a9ab118e1db25f79957d5ef540b/research/object_detection/g3doc/installation.md#protobuf-compilation).
-```
-cd TF_MODELS_DIR/research
-protoc object_detection/protos/*.proto --python_out=.
-cd ../..
-```
-
-Once your environment is setup, navigate back to your Model Zoo directory. Ensure that
-you have set environment variables pointing to the TensorFlow Model Garden repo, the dataset,
-and output directories, and then run a quickstart script.
-
-To run inference with performance metrics:
-```
-# cd to your model zoo directory
-cd models
-
-export DATASET_DIR=<path to the coco val2017 raw image directory (ex: /home/user/coco_dataset/val2017)>
-export PRECISION=<set the precision to "int8" or "fp32">
-export OUTPUT_DIR=<path to the directory where log files will be written>
-export TF_MODELS_DIR=<directory where TensorFlow Model Garden is cloned>
-# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
-export BATCH_SIZE=<customized batch size value>
-
-./quickstart/object_detection/tensorflow/rfcn/inference/cpu/inference.sh
-```
-
-To get accuracy metrics:
-```
-# cd to your model zoo directory
-cd models
-
-export DATASET_DIR=<path to TF record file (ex: /home/user/coco_output/coco_val.record)>
-export PRECISION=<set the precision to "int8" or "fp32">
-export OUTPUT_DIR=<path to the directory where log files will be written>
-export TF_MODELS_DIR=<directory where TensorFlow Model Garden is cloned>
-# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
-export BATCH_SIZE=<customized batch size value>
-
-./quickstart/object_detection/tensorflow/rfcn/inference/cpu/accuracy.sh
-```
-
-### Run on Windows
-* Download and install [Google Protobuf version 3.4.0]((https://github.com/protocolbuffers/protobuf/releases/tag/v3.4.0)) for Windows in addition to the above listed dependencies.
-Download and extract [protoc-3.4.0-win32.zip](https://github.com/protocolbuffers/protobuf/releases/download/v3.4.0/protoc-3.4.0-win32.zip)
-Navigate to the `research` directory in `TF_MODELS_DIR` and install Google Protobuf:
-```
-set TF_MODELS_DIR=<directory where TensorFlow Model Garden is cloned>
-cd %TF_MODELS_DIR%\research
-“C:\<user>\protoc-3.4.0-win32\bin\protoc.exe” object_detection/protos/*.proto --python_out=.
-```
-
-After installing the prerequisites and cloning the TensorFlow models repo, and downloading the pretrained model,
-set the environment variables for the paths to your `PRETRAINED_MODEL`, an `OUTPUT_DIR` where log files will be written,
-TF_MODELS_DIR, and `DATASET_DIR` for COCO raw dataset directory or tf_records file based on whether you run inference or accuracy scripts.
-Navigate to your model zoo directory and then run a [quickstart script](#quick-start-scripts).
-```
-# cd to your model zoo directory
-cd models
-
-set PRETRAINED_MODEL=<path to the frozen graph downloaded above>
-set DATASET_DIR=<path to COCO raw dataset directory or tf_records file based on whether you run inference or accuracy scripts>
-set PRECISION=<set the precision to "int8" or "fp32">
-set OUTPUT_DIR=<directory where log files will be written>
-set TF_MODELS_DIR=<directory where TensorFlow Model Garden is cloned>
-# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
-set BATCH_SIZE=<customized batch size value>
-
-bash quickstart\object_detection\tensorflow\rfcn\inference\cpu\<script name>.sh
-```
-> Note: You may use `cygpath` to convert the Windows paths to Unix paths before setting the environment variables. 
-As an example, if the dataset location on Windows is `D:\user\coco_dataset\val2017`, convert the Windows path to Unix as shown:
-> ```
-> cygpath D:\user\coco_dataset\val2017
-> /d/user/coco_dataset/val2017
->```
->Then, set the `DATASET_DIR` environment variable `set DATASET_DIR=/d/user/coco_dataset/val2017`.
diff --git a/quickstart/object_detection/tensorflow/rfcn/inference/cpu/.docs/datasets.md b/quickstart/object_detection/tensorflow/rfcn/inference/cpu/.docs/datasets.md
deleted file mode 100644
index f5b5b22d8..000000000
--- a/quickstart/object_detection/tensorflow/rfcn/inference/cpu/.docs/datasets.md
+++ /dev/null
@@ -1,10 +0,0 @@
-<!--- 30. Datasets -->
-## Dataset
-
-The [COCO validation dataset](http://cocodataset.org) is used in these
-RFCN quickstart scripts. The inference quickstart scripts use raw images,
-and the accuracy quickstart scripts require the dataset to be converted
-into the TF records format.
-See the [COCO dataset](/datasets/coco/README.md) for instructions on
-downloading and preprocessing the COCO validation dataset.
-
diff --git a/quickstart/object_detection/tensorflow/rfcn/inference/cpu/.docs/description.md b/quickstart/object_detection/tensorflow/rfcn/inference/cpu/.docs/description.md
deleted file mode 100644
index 63875edda..000000000
--- a/quickstart/object_detection/tensorflow/rfcn/inference/cpu/.docs/description.md
+++ /dev/null
@@ -1,5 +0,0 @@
-<!-- 10. Description -->
-
-This document has instructions for running <model name> <mode> using
-Intel-optimized TensorFlow.
-
diff --git a/quickstart/object_detection/tensorflow/rfcn/inference/cpu/.docs/docker.md b/quickstart/object_detection/tensorflow/rfcn/inference/cpu/.docs/docker.md
deleted file mode 100644
index 65668b34e..000000000
--- a/quickstart/object_detection/tensorflow/rfcn/inference/cpu/.docs/docker.md
+++ /dev/null
@@ -1,82 +0,0 @@
-<!-- 60. Docker -->
-## Docker
-
-When running in docker, the <model name> <mode> container includes the
-libraries and the model package, which are needed to run <model name> <mode>. To run the quickstart scripts, you'll need to provide volume mounts for the
-[COCO validation dataset](/dataset/coco/README.md) and an output directory
-where log files will be written.
-
-To run inference with performance metrics:
-```
-DATASET_DIR=<path to the coco val2017 raw image directory (ex: /home/user/coco_dataset/val2017)>
-PRECISION=<set the precision to "int8" or "fp32">
-OUTPUT_DIR=<directory where log files will be written>
-# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
-export BATCH_SIZE=<customized batch size value>
-
-docker run \
-  --env DATASET_DIR=${DATASET_DIR} \
-  --env PRECISION=${PRECISION} \
-  --env OUTPUT_DIR=${OUTPUT_DIR} \
-  --env BATCH_SIZE=${BATCH_SIZE} \
-  --env http_proxy=${http_proxy} \
-  --env https_proxy=${https_proxy} \
-  --volume ${DATASET_DIR}:${DATASET_DIR} \
-  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
-  --privileged --init -t \
-  <docker image> \
-  /bin/bash quickstart/<script_name>.sh
-```
-
-When the run completes, the log tail will note the average duration per step:
-
-```
-Avg. Duration per Step: ...
-Ran inference with batch size 1
-Log file location: ${OUTPUT_DIR}/benchmark_rfcn_inference_fp32_20200620_002239.log
-```
-
-To get accuracy metrics:
-```
-DATASET_DIR=<path to TF record file (ex: /home/user/coco_output/coco_val.record)>
-OUTPUT_DIR=<directory where log files will be written>
-
-docker run \
-  --env DATASET_DIR=${DATASET_DIR} \
-  --env OUTPUT_DIR=${OUTPUT_DIR} \
-  --env http_proxy=${http_proxy} \
-  --env https_proxy=${https_proxy} \
-  --volume ${DATASET_DIR}:${DATASET_DIR} \
-  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
-  --privileged --init -t \
-  <docker image> \
-  /bin/bash quickstart/fp32_accuracy.sh
-```
-
-Below is a sample log file tail when running for accuracy:
-
-```
-Running per image evaluation...
-Evaluate annotation type *bbox*
-DONE (t=10.41s).
-Accumulating evaluation results...
-DONE (t=1.62s).
- Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.347
- Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.532
- Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.389
- Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.347
- Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
- Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.282
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.396
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.400
- Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.400
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
- Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
-Ran inference with batch size 1
-Log file location: ${OUTPUT_DIR}/benchmark_rfcn_inference_fp32_20200620_002841.log
-```
-
-If you are new to docker and are running into issues with the container,
-see [this document](https://github.com/IntelAI/models/tree/master/docs/general/docker.md)
-for troubleshooting tips.
diff --git a/quickstart/object_detection/tensorflow/rfcn/inference/cpu/.docs/license.md b/quickstart/object_detection/tensorflow/rfcn/inference/cpu/.docs/license.md
deleted file mode 100644
index a60f169f4..000000000
--- a/quickstart/object_detection/tensorflow/rfcn/inference/cpu/.docs/license.md
+++ /dev/null
@@ -1,4 +0,0 @@
-<!--- 80. License -->
-## License
-
-[LICENSE](/LICENSE)
diff --git a/quickstart/object_detection/tensorflow/rfcn/inference/cpu/.docs/quickstart.md b/quickstart/object_detection/tensorflow/rfcn/inference/cpu/.docs/quickstart.md
deleted file mode 100644
index 75ff1a2d9..000000000
--- a/quickstart/object_detection/tensorflow/rfcn/inference/cpu/.docs/quickstart.md
+++ /dev/null
@@ -1,7 +0,0 @@
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| [`inference.sh`](/quickstart/object_detection/tensorflow/rfcn/inference/cpu/inference.sh) | Runs inference on a directory of raw images for 500 steps and outputs performance metrics. |
-| [`accuracy.sh`](/quickstart/object_detection/tensorflow/rfcn/inference/cpu/accuracy.sh) | Processes the TF records to run inference and check accuracy on the results. |
diff --git a/quickstart/object_detection/tensorflow/rfcn/inference/cpu/.docs/title.md b/quickstart/object_detection/tensorflow/rfcn/inference/cpu/.docs/title.md
deleted file mode 100644
index 1d6c60187..000000000
--- a/quickstart/object_detection/tensorflow/rfcn/inference/cpu/.docs/title.md
+++ /dev/null
@@ -1,2 +0,0 @@
-<!--- 0. Title -->
-# <model name> <mode>
diff --git a/quickstart/object_detection/tensorflow/rfcn/inference/cpu/README.md b/quickstart/object_detection/tensorflow/rfcn/inference/cpu/README.md
deleted file mode 100644
index a74b3b923..000000000
--- a/quickstart/object_detection/tensorflow/rfcn/inference/cpu/README.md
+++ /dev/null
@@ -1,116 +0,0 @@
-<!--- 0. Title -->
-# RFCN inference
-
-<!-- 10. Description -->
-
-This document has instructions for running RFCN inference using
-Intel-optimized TensorFlow.
-
-
-<!--- 30. Datasets -->
-## Dataset
-
-The [COCO validation dataset](http://cocodataset.org) is used in these
-RFCN quickstart scripts. The inference quickstart scripts use raw images,
-and the accuracy quickstart scripts require the dataset to be converted
-into the TF records format.
-See the [COCO dataset](/datasets/coco/README.md) for instructions on
-downloading and preprocessing the COCO validation dataset.
-
-
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| [`inference.sh`](/quickstart/object_detection/tensorflow/rfcn/inference/cpu/inference.sh) | Runs inference on a directory of raw images for 500 steps and outputs performance metrics. |
-| [`accuracy.sh`](/quickstart/object_detection/tensorflow/rfcn/inference/cpu/accuracy.sh) | Processes the TF records to run inference and check accuracy on the results. |
-
-<!-- 60. Docker -->
-## Docker
-
-When running in docker, the RFCN inference container includes the
-libraries and the model package, which are needed to run RFCN inference. To run the quickstart scripts, you'll need to provide volume mounts for the
-[COCO validation dataset](/dataset/coco/README.md) and an output directory
-where log files will be written.
-
-To run inference with performance metrics:
-```
-DATASET_DIR=<path to the coco val2017 raw image directory (ex: /home/user/coco_dataset/val2017)>
-PRECISION=<set the precision to "int8" or "fp32">
-OUTPUT_DIR=<directory where log files will be written>
-# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
-export BATCH_SIZE=<customized batch size value>
-
-docker run \
-  --env DATASET_DIR=${DATASET_DIR} \
-  --env PRECISION=${PRECISION} \
-  --env OUTPUT_DIR=${OUTPUT_DIR} \
-  --env BATCH_SIZE=${BATCH_SIZE} \
-  --env http_proxy=${http_proxy} \
-  --env https_proxy=${https_proxy} \
-  --volume ${DATASET_DIR}:${DATASET_DIR} \
-  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
-  --privileged --init -t \
-  intel/object-detection:tf-latest-rfcn-inference \
-  /bin/bash quickstart/<script_name>.sh
-```
-
-When the run completes, the log tail will note the average duration per step:
-
-```
-Avg. Duration per Step: ...
-Ran inference with batch size 1
-Log file location: ${OUTPUT_DIR}/benchmark_rfcn_inference_fp32_20200620_002239.log
-```
-
-To get accuracy metrics:
-```
-DATASET_DIR=<path to TF record file (ex: /home/user/coco_output/coco_val.record)>
-OUTPUT_DIR=<directory where log files will be written>
-
-docker run \
-  --env DATASET_DIR=${DATASET_DIR} \
-  --env OUTPUT_DIR=${OUTPUT_DIR} \
-  --env http_proxy=${http_proxy} \
-  --env https_proxy=${https_proxy} \
-  --volume ${DATASET_DIR}:${DATASET_DIR} \
-  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
-  --privileged --init -t \
-  intel/object-detection:tf-latest-rfcn-inference \
-  /bin/bash quickstart/fp32_accuracy.sh
-```
-
-Below is a sample log file tail when running for accuracy:
-
-```
-Running per image evaluation...
-Evaluate annotation type *bbox*
-DONE (t=10.41s).
-Accumulating evaluation results...
-DONE (t=1.62s).
- Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.347
- Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.532
- Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.389
- Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.347
- Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
- Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.282
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.396
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.400
- Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.400
- Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = -1.000
- Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = -1.000
-Ran inference with batch size 1
-Log file location: ${OUTPUT_DIR}/benchmark_rfcn_inference_fp32_20200620_002841.log
-```
-
-If you are new to docker and are running into issues with the container,
-see [this document](https://github.com/IntelAI/models/tree/master/docs/general/docker.md)
-for troubleshooting tips.
-
-<!--- 80. License -->
-## License
-
-[LICENSE](/LICENSE)
-
diff --git a/quickstart/object_detection/tensorflow/rfcn/inference/cpu/accuracy.sh b/quickstart/object_detection/tensorflow/rfcn/inference/cpu/accuracy.sh
deleted file mode 100755
index 7e68ca66b..000000000
--- a/quickstart/object_detection/tensorflow/rfcn/inference/cpu/accuracy.sh
+++ /dev/null
@@ -1,88 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2020 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-MODEL_DIR=${MODEL_DIR-$PWD}
-
-if [ -z "${OUTPUT_DIR}" ]; then
-  echo "The required environment variable OUTPUT_DIR has not been set"
-  exit 1
-fi
-
-# Create the output directory in case it doesn't already exist
-mkdir -p ${OUTPUT_DIR}
-
-if [ -z "${DATASET_DIR}" ]; then
-  echo "The required environment variable DATASET_DIR has not been set"
-  exit 1
-fi
-
-if [ -z "${PRECISION}" ]; then
-  echo "The required environment variable PRECISION has not been set"
-  echo "Please set PRECISION to fp32 or int8"
-  exit 1
-fi
-
-if [[ $PRECISION != "fp32" ]] && [[ $PRECISION != "int8" ]]; then
-  echo "The specified precision '${PRECISION}' is unsupported."
-  echo "Supported precisions are: fp32 and int8"
-  exit 1
-fi
-
-if [ -z "${TF_MODELS_DIR}" ]; then
-  echo "The required environment variable TF_MODELS_DIR has not been set"
-  exit 1
-fi
-
-if [ -z "${PRETRAINED_MODEL}" ]; then
-  if [ $PRECISION == "int8" ]; then
-    PRETRAINED_MODEL="${MODEL_DIR}/rfcn_final_fused_pad_and_conv.pb"
-  elif [[ $PRECISION == "fp32" ]]; then  
-    pretrained_model_dir="${OUTPUT_DIR}/rfcn_frozen_inference_graph.pb"
-  else
-    echo "The specified precision '${PRECISION}' is unsupported."
-    echo "Supported precisions are: fp32 and int8"
-    exit 1
-  fi  
-  if [[ ! -f "${PRETRAINED_MODEL}" ]]; then
-    echo "The pretrained model could not be found. Please set the PRETRAINED_MODEL env var to point to the frozen graph file."
-    exit 1
-  fi
-elif [[ ! -f "${PRETRAINED_MODEL}" ]]; then
-  echo "The file specified by the PRETRAINED_MODEL environment variable (${PRETRAINED_MODEL}) does not exist."
-  exit 1
-fi  
-  
-# If batch size env is not mentioned, then the workload will run with the default batch size.
-if [ -z "${BATCH_SIZE}"]; then
-  BATCH_SIZE="1"
-  echo "Running with default batch size of ${BATCH_SIZE}"
-fi
-
-source "${MODEL_DIR}/quickstart/common/utils.sh"
-_command python ${MODEL_DIR}/benchmarks/launch_benchmark.py \
-    --model-name rfcn \
-    --mode inference \
-    --precision ${PRECISION} \
-    --framework tensorflow \
-    --model-source-dir ${TF_MODELS_DIR} \
-    --data-location ${DATASET_DIR} \
-    --in-graph ${PRETRAINED_MODEL} \
-    --batch-size ${BATCH_SIZE} \
-    --accuracy-only \
-    --output-dir ${OUTPUT_DIR} \
-    $@ \
-    -- split="accuracy_message"
diff --git a/quickstart/object_detection/tensorflow/rfcn/inference/cpu/inference.sh b/quickstart/object_detection/tensorflow/rfcn/inference/cpu/inference.sh
deleted file mode 100755
index 01e12c6a1..000000000
--- a/quickstart/object_detection/tensorflow/rfcn/inference/cpu/inference.sh
+++ /dev/null
@@ -1,89 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2020 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-MODEL_DIR=${MODEL_DIR-$PWD}
-
-if [ -z "${OUTPUT_DIR}" ]; then
-  echo "The required environment variable OUTPUT_DIR has not been set"
-  exit 1
-fi
-
-# Create the output directory in case it doesn't already exist
-mkdir -p ${OUTPUT_DIR}
-
-if [ -z "${DATASET_DIR}" ]; then
-  echo "The required environment variable DATASET_DIR has not been set"
-  exit 1
-fi
-
-if [ -z "${PRECISION}" ]; then
-  echo "The required environment variable PRECISION has not been set"
-  echo "Please set PRECISION to fp32 or int8"
-  exit 1
-fi
-
-if [[ $PRECISION != "fp32" ]] && [[ $PRECISION != "int8" ]]; then
-  echo "The specified precision '${PRECISION}' is unsupported."
-  echo "Supported precisions are: fp32 and int8"
-  exit 1
-fi
-
-if [ -z "${TF_MODELS_DIR}" ]; then
-  echo "The required environment variable TF_MODELS_DIR has not been set"
-  exit 1
-fi
-
-if [ -z "${PRETRAINED_MODEL}" ]; then
-  if [ $PRECISION == "int8" ]; then
-    PRETRAINED_MODEL="${MODEL_DIR}/final_fused_pad_and_conv.pb"
-  elif [[ $PRECISION == "fp32" ]]; then  
-    pretrained_model_dir="${OUTPUT_DIR}/frozen_inference_graph.pb"
-  else
-    echo "The specified precision '${PRECISION}' is unsupported."
-    echo "Supported precisions are: fp32 and int8"
-    exit 1
-  fi  
-  if [[ ! -f "${PRETRAINED_MODEL}" ]]; then
-    echo "The pretrained model could not be found. Please set the PRETRAINED_MODEL env var to point to the frozen graph file."
-    exit 1
-  fi
-elif [[ ! -f "${PRETRAINED_MODEL}" ]]; then
-  echo "The file specified by the PRETRAINED_MODEL environment variable (${PRETRAINED_MODEL}) does not exist."
-  exit 1
-fi
-
-# If batch size env is not mentioned, then the workload will run with the default batch size.
-if [ -z "${BATCH_SIZE}"]; then
-  BATCH_SIZE="1"
-  echo "Running with default batch size of ${BATCH_SIZE}"
-fi
-
-source "${MODEL_DIR}/quickstart/common/utils.sh"
-_command python ${MODEL_DIR}/benchmarks/launch_benchmark.py \
-    --model-name rfcn \
-    --mode inference \
-    --precision ${PRECISION} \
-    --framework tensorflow \
-    --model-source-dir ${TF_MODELS_DIR} \
-    --data-location ${DATASET_DIR} \
-    --in-graph ${PRETRAINED_MODEL} \
-    --output-dir ${OUTPUT_DIR} \
-    --batch-size ${BATCH_SIZE} \
-    --benchmark-only \
-    $@ \
-    -- number_of_steps=500
-
diff --git a/quickstart/object_detection/tensorflow/ssd-mobilenet/inference/cpu/README_DEV_CAT.md b/quickstart/object_detection/tensorflow/ssd-mobilenet/inference/cpu/README_DEV_CAT.md
new file mode 100644
index 000000000..2f126f5e7
--- /dev/null
+++ b/quickstart/object_detection/tensorflow/ssd-mobilenet/inference/cpu/README_DEV_CAT.md
@@ -0,0 +1,98 @@
+# TensorFlow SSD-MobileNet inference
+
+## Description
+This document has instructions for running SSD-MobileNet inference using Intel-optimized TensorFlow.
+
+## Pull Command
+```
+docker pull intel/object-detection:centos-tf-cpu-ssd-mobilenet-inference
+```
+
+<table>
+   <thead>
+      <tr>
+         <th>Script name</th>
+         <th>Description</th>
+      </tr>
+   </thead>
+   <tbody>
+      <tr>
+         <td>inference_realtime_multi_instance.sh</td>
+         <td>A multi-instance run that uses 4 cores per instance with a batch size of 1. Uses synthetic data if no `DATASET_DIR` is set. Supported versions are (fp32, int8, bfloat16, bfloat32) </td>
+      </tr>
+      <tr>
+         <td>inference_throughput_multi_instance.sh</td>
+         <td>A multi-instance run that uses all the cores for each socket for each instance with a batch size of 448 and synthetic data. Supported versions are (fp32, int8, bfloat16, bfloat32)</td>
+      </tr>
+      <tr>
+         <td>accuracy.sh</td>
+         <td>Measures the inference accuracy (providing a `DATASET_DIR` environment variable is required). Supported versions are (fp32, int8, bfloat16, bfloat32).</td>
+      </tr>
+   </tbody>
+</table>
+
+## Datasets
+
+The [COCO validation dataset](http://cocodataset.org) is used in these
+SSD-Mobilenet quickstart scripts. The accuracy quickstart script require the dataset to be converted into the TF records format.
+See the [COCO dataset](https://github.com/IntelAI/models/tree/master/datasets/coco) for instructions on
+downloading and preprocessing the COCO validation dataset.
+
+Set the `DATASET_DIR` to point to the dataset directory that contains the TF records file `coco_val.record` when running SSD-MobileNet accuracy script.
+
+## Docker Run
+(Optional) Export related proxy into docker environment.
+```bash
+export DOCKER_RUN_ENVS="-e ftp_proxy=${ftp_proxy} \
+  -e FTP_PROXY=${FTP_PROXY} -e http_proxy=${http_proxy} \
+  -e HTTP_PROXY=${HTTP_PROXY} -e https_proxy=${https_proxy} \
+  -e HTTPS_PROXY=${HTTPS_PROXY} -e no_proxy=${no_proxy} \
+  -e NO_PROXY=${NO_PROXY} -e socks_proxy=${socks_proxy} \
+  -e SOCKS_PROXY=${SOCKS_PROXY}"
+```
+
+To run SSD-MobileNet inference, set environment variables to specify the dataset directory, precision and mode to run, and an output directory. 
+```bash
+# Set the required environment vars
+export DATASET_DIR=<path to the dataset>
+export OUTPUT_DIR=<directory where log files will be written>
+export SCRIPT=<specify the script to run>
+export PRECISION=<specify the precision to run>
+
+# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
+export BATCH_SIZE=<customized batch size value>
+
+docker run --rm \
+  --env BATCH_SIZE=${BATCH_SIZE} \
+  --env DATASET_DIR=${DATASET_DIR} \
+  --env OUTPUT_DIR=${OUTPUT_DIR} \
+  --env PRECISION=${PRECISION} \
+  --volume ${DATASET_DIR}:${DATASET_DIR} \
+  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
+  ${DOCKER_RUN_ENVS} \
+  --privileged --init -it \
+  --shm-size 8G \
+  -w /workspace/tf-ssd-mobilenet-inference \
+  intel/object-detection:centos-tf-cpu-ssd-mobilenet-inference \
+  /bin/bash quickstart/${SCRIPT}
+```
+
+## Documentation and Sources
+#### Get Started​
+[Docker* Repository](https://hub.docker.com/r/intel/object-detection)
+
+[Main GitHub*](https://github.com/IntelAI/models)
+
+[Release Notes](https://github.com/IntelAI/models/releases)
+
+[Get Started Guide](https://github.com/IntelAI/models/blob/master/quickstart/object_detection/tensorflow/ssd-mobilenet/inference/cpu/README_DEV_CAT.md)
+
+#### Code Sources
+[Dockerfile](https://github.com/IntelAI/models/tree/master/dockerfiles/tensorflow)
+
+[Report Issue](https://community.intel.com/t5/Intel-Optimized-AI-Frameworks/bd-p/optimized-ai-frameworks)
+
+## License Agreement
+LEGAL NOTICE: By accessing, downloading or using this software and any required dependent software (the “Software Package”), you agree to the terms and conditions of the software license agreements for the Software Package, which may also include notices, disclaimers, or license terms for third party software included with the Software Package. Please refer to the [license](https://github.com/IntelAI/models/tree/master/third_party) file for additional details.
+
+[View All Containers and Solutions 🡢](https://www.intel.com/content/www/us/en/developer/tools/software-catalog/containers.html?s=Newest)
diff --git a/quickstart/object_detection/tensorflow/ssd-mobilenet/inference/cpu/accuracy.sh b/quickstart/object_detection/tensorflow/ssd-mobilenet/inference/cpu/accuracy.sh
index f4050199c..98540b0b7 100755
--- a/quickstart/object_detection/tensorflow/ssd-mobilenet/inference/cpu/accuracy.sh
+++ b/quickstart/object_detection/tensorflow/ssd-mobilenet/inference/cpu/accuracy.sh
@@ -52,6 +52,7 @@ if [ -z "${PRETRAINED_MODEL}" ]; then
     PRETRAINED_MODEL="${MODEL_DIR}/pretrained_model/ssdmobilenet_fp32_pretrained_model_combinedNMS.pb"
   elif [[ $PRECISION == "int8" ]]; then
     PRETRAINED_MODEL="${MODEL_DIR}/pretrained_model/ssdmobilenet_int8_pretrained_model_combinedNMS_s8.pb"
+    INPUT_SUBGRAPH="${MODEL_DIR}/pretrained_model/ssdmobilenet_preprocess.pb"
   elif [[ $PRECISION == "bfloat16" ]]; then
     PRETRAINED_MODEL="${MODEL_DIR}/pretrained_model/ssdmobilenet_fp32_pretrained_model_combinedNMS.pb"      
   else
@@ -87,7 +88,6 @@ if [ -z "${BATCH_SIZE}"]; then
 fi
 
 source "${MODEL_DIR}/quickstart/common/utils.sh"
-_ht_status_spr
 _command python ${MODEL_DIR}/benchmarks/launch_benchmark.py \
   --model-name ssd-mobilenet \
   --precision ${PRECISION} \
diff --git a/quickstart/object_detection/tensorflow/ssd-resnet34/inference/cpu/README_DEV_CAT.md b/quickstart/object_detection/tensorflow/ssd-resnet34/inference/cpu/README_DEV_CAT.md
new file mode 100644
index 000000000..bdd035f1e
--- /dev/null
+++ b/quickstart/object_detection/tensorflow/ssd-resnet34/inference/cpu/README_DEV_CAT.md
@@ -0,0 +1,105 @@
+# TensorFlow SSD-ResNet34 inference
+
+## Description
+This document has instructions for running SSD-ResNet34 inference using Intel-optimized TensorFlow.
+
+## Pull Command
+```
+docker pull intel/object-detection:centos-tf-cpu-ssd-resnet34-inference
+```
+
+<table>
+   <thead>
+      <tr>
+         <th>Script name</th>
+         <th>Description</th>
+      </tr>
+   </thead>
+   <tbody>
+      <tr>
+         <td>multi_instance_online_inference_1200.sh</td>
+         <td>Runs multi instance realtime inference (batch-size=1) using 4 cores per instance for the specified precision (fp32, int8 or bfloat16). Uses synthetic data with an input size of 1200x1200. Waits for all instances to complete, then prints a summarized throughput value.</td>
+      </tr>
+      <tr>
+         <td>multi_instance_batch_inference_1200.sh</td>
+         <td>Runs multi instance batch inference (batch-size=16) using 1 instance per socket for the specified precision (fp32, int8 or bfloat16). Uses synthetic data with an input size of 1200x1200. Waits for all instances to complete, then prints a summarized throughput value.</td>
+      </tr>
+      <tr>
+         <td>accuracy_1200.sh</td>
+         <td>Measures the inference accuracy (providing a `DATASET_DIR` environment variable is required) for the specified precision (fp32, int8 or bfloat16) with an input size of 1200x1200.</td>
+      </tr>
+   </tbody>
+</table>
+
+## Datasets
+
+The SSD-ResNet34 accuracy script `accuracy.sh` uses the
+[COCO validation dataset](http://cocodataset.org) in the TF records
+format. See the [COCO dataset document](https://github.com/IntelAI/models/tree/master/datasets/coco) for
+instructions on downloading and preprocessing the COCO validation dataset.
+The inference scripts use synthetic data, so no dataset is required.
+
+After the script to convert the raw images to the TF records file completes, rename the tf_records file:
+```
+mv ${OUTPUT_DIR}/coco_val.record ${OUTPUT_DIR}/validation-00000-of-00001
+```
+Set the `DATASET_DIR` to the folder that has the `validation-00000-of-00001`
+file when running the accuracy test. Note that the inference performance
+test uses synthetic dataset.
+
+## Docker Run
+(Optional) Export related proxy into docker environment.
+```bash
+export DOCKER_RUN_ENVS="-e ftp_proxy=${ftp_proxy} \
+  -e FTP_PROXY=${FTP_PROXY} -e http_proxy=${http_proxy} \
+  -e HTTP_PROXY=${HTTP_PROXY} -e https_proxy=${https_proxy} \
+  -e HTTPS_PROXY=${HTTPS_PROXY} -e no_proxy=${no_proxy} \
+  -e NO_PROXY=${NO_PROXY} -e socks_proxy=${socks_proxy} \
+  -e SOCKS_PROXY=${SOCKS_PROXY}"
+```
+
+To run Mask R-CNN inference, set environment variables to specify the dataset directory, precision and mode to run, and an output directory. 
+```bash
+# Set the required environment vars
+export DATASET_DIR=<path to the dataset>
+export OUTPUT_DIR=<directory where log files will be written>
+export SCRIPT=<specify the script to run>
+export PRECISION=<specify the precision to run>
+
+# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
+export BATCH_SIZE=<customized batch size value>
+
+docker run --rm \
+  --env BATCH_SIZE=${BATCH_SIZE} \
+  --env DATASET_DIR=${DATASET_DIR} \
+  --env OUTPUT_DIR=${OUTPUT_DIR} \
+  --env PRECISION=${PRECISION} \
+  --volume ${DATASET_DIR}:${DATASET_DIR} \
+  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
+  ${DOCKER_RUN_ENVS} \
+  --privileged --init -it \
+  --shm-size 8G \
+  -w /workspace/tf-ssd-resnet34-inference \
+  intel/object-detection:centos-tf-cpu-ssd-resnet34-inference \
+  /bin/bash quickstart/${SCRIPT}
+```
+
+## Documentation and Sources
+#### Get Started​
+[Docker* Repository](https://hub.docker.com/r/intel/object-detection)
+
+[Main GitHub*](https://github.com/IntelAI/models)
+
+[Release Notes](https://github.com/IntelAI/models/releases)
+
+[Get Started Guide](https://github.com/IntelAI/models/blob/master/quickstart/object_detection/tensorflow/ssd-resnet34/inference/cpu/README_DEV_CAT.md)
+
+#### Code Sources
+[Dockerfile](https://github.com/IntelAI/models/tree/master/dockerfiles/tensorflow)
+
+[Report Issue](https://community.intel.com/t5/Intel-Optimized-AI-Frameworks/bd-p/optimized-ai-frameworks)
+
+## License Agreement
+LEGAL NOTICE: By accessing, downloading or using this software and any required dependent software (the “Software Package”), you agree to the terms and conditions of the software license agreements for the Software Package, which may also include notices, disclaimers, or license terms for third party software included with the Software Package. Please refer to the [license](https://github.com/IntelAI/models/tree/master/third_party) file for additional details.
+
+[View All Containers and Solutions 🡢](https://www.intel.com/content/www/us/en/developer/tools/software-catalog/containers.html?s=Newest)
diff --git a/quickstart/object_detection/tensorflow/ssd-resnet34/inference/cpu/accuracy_1200.sh b/quickstart/object_detection/tensorflow/ssd-resnet34/inference/cpu/accuracy_1200.sh
index d23c36138..16ae31f2a 100755
--- a/quickstart/object_detection/tensorflow/ssd-resnet34/inference/cpu/accuracy_1200.sh
+++ b/quickstart/object_detection/tensorflow/ssd-resnet34/inference/cpu/accuracy_1200.sh
@@ -91,7 +91,6 @@ if [ -z "${BATCH_SIZE}"]; then
 fi
 
 source "${MODEL_DIR}/quickstart/common/utils.sh"
-_ht_status_spr
 _command python ${MODEL_DIR}/benchmarks/launch_benchmark.py \
   --model-source-dir ${TF_MODELS_DIR} \
   --model-name ssd-resnet34 \
diff --git a/quickstart/object_detection/tensorflow/ssd-resnet34/training/cpu/README_DEV_CAT.md b/quickstart/object_detection/tensorflow/ssd-resnet34/training/cpu/README_DEV_CAT.md
new file mode 100644
index 000000000..a68b219ea
--- /dev/null
+++ b/quickstart/object_detection/tensorflow/ssd-resnet34/training/cpu/README_DEV_CAT.md
@@ -0,0 +1,84 @@
+# TensorFlow SSD-ResNet34 training
+
+## Description
+This document has instructions for running SSD-ResNet34 training
+using Intel-optimized TensorFlow.
+
+## Pull Command
+```
+docker pull intel/object-detection:centos-tf-cpu-ssd-resnet34-training
+```
+
+<table>
+   <thead>
+      <tr>
+         <th>Script name</th>
+         <th>Description</th>
+      </tr>
+   </thead>
+   <tbody>
+      <tr>
+         <td>training.sh</td>
+         <td>Uses mpirun to execute 1 processes with 1 process per socket with a batch size of 896 for the specified precision (fp32 or bfloat16 or bfloat32). Logs for each instance are saved to the output directory.</td>
+      </tr>
+   </tbody>
+</table>
+
+## Datasets
+SSD-ResNet34 training uses the COCO training dataset. Use the [instructions](https://github.com/IntelAI/models/tree/master/datasets/coco/README_train.md) to download and preprocess the dataset.
+
+## Docker Run
+(Optional) Export related proxy into docker environment.
+```bash
+export DOCKER_RUN_ENVS="-e ftp_proxy=${ftp_proxy} \
+  -e FTP_PROXY=${FTP_PROXY} -e http_proxy=${http_proxy} \
+  -e HTTP_PROXY=${HTTP_PROXY} -e https_proxy=${https_proxy} \
+  -e HTTPS_PROXY=${HTTPS_PROXY} -e no_proxy=${no_proxy} \
+  -e NO_PROXY=${NO_PROXY} -e socks_proxy=${socks_proxy} \
+  -e SOCKS_PROXY=${SOCKS_PROXY}"
+```
+
+To run Mask R-CNN training, set environment variables to specify the dataset directory, precision to run, and an output directory. 
+```bash
+# Set the required environment vars
+export DATASET_DIR=<path to the dataset>
+export OUTPUT_DIR=<directory where log files will be written>
+export PRECISION=<specify the precision to run (fp32 or bfloat16)>
+
+# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
+export BATCH_SIZE=<customized batch size value>
+
+docker run --rm \
+  --env BATCH_SIZE=${BATCH_SIZE} \
+  --env DATASET_DIR=${DATASET_DIR} \
+  --env OUTPUT_DIR=${OUTPUT_DIR} \
+  --env PRECISION=${PRECISION} \
+  --volume ${DATASET_DIR}:${DATASET_DIR} \
+  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
+  ${DOCKER_RUN_ENVS} \
+  --privileged --init -it \
+  --shm-size 8G \
+  -w /workspace/tf-ssd-resnet34-training \
+  intel/object-detection:centos-tf-cpu-ssd-resnet34-training \
+  /bin/bash quickstart/training.sh
+```
+
+## Documentation and Sources
+#### Get Started​
+[Docker* Repository](https://hub.docker.com/r/intel/object-detection)
+
+[Main GitHub*](https://github.com/IntelAI/models)
+
+[Release Notes](https://github.com/IntelAI/models/releases)
+
+[Get Started Guide](https://github.com/IntelAI/models/blob/master/quickstart/object_detection/tensorflow/ssd-resnet34/training/cpu/README_DEV_CAT.md)
+
+#### Code Sources
+[Dockerfile](https://github.com/IntelAI/models/tree/master/dockerfiles/tensorflow)
+
+[Report Issue](https://community.intel.com/t5/Intel-Optimized-AI-Frameworks/bd-p/optimized-ai-frameworks)
+
+## License Agreement
+LEGAL NOTICE: By accessing, downloading or using this software and any required dependent software (the “Software Package”), you agree to the terms and conditions of the software license agreements for the Software Package, which may also include notices, disclaimers, or license terms for third party software included with the Software Package. Please refer to the [license](https://github.com/IntelAI/models/tree/master/third_party) file for additional details.
+
+[View All Containers and Solutions 🡢](https://www.intel.com/content/www/us/en/developer/tools/software-catalog/containers.html?s=Newest)
diff --git a/quickstart/recommendation/pytorch/dlrm/inference/cpu/.docs/container_build.md b/quickstart/recommendation/pytorch/dlrm/inference/cpu/.docs/container_build.md
deleted file mode 100644
index d57bf6489..000000000
--- a/quickstart/recommendation/pytorch/dlrm/inference/cpu/.docs/container_build.md
+++ /dev/null
@@ -1,26 +0,0 @@
-## Build the container
-
-The <model name> <mode> package has scripts and a Dockerfile that are
-used to build a workload container that runs the model. This container
-uses the PyTorch/IPEX container as it's base, so ensure that you have built
-the `pytorch-ipex-spr.tar.gz` container prior to building this model container.
-
-Use `docker images` to verify that you have the base container built. For example:
-```
-$ docker images | grep pytorch-ipex-spr
-model-zoo         pytorch-ipex-spr         f5b473554295        2 hours ago         4.08GB
-```
-
-To build the <model name> <mode> container, extract the package and
-run the `build.sh` script.
-```
-# Extract the package
-tar -xzf <package name>
-cd <package dir>
-
-# Build the container
-./build.sh
-```
-
-After the build completes, you should have a container called
-`<docker image>` that will be used to run the model.
diff --git a/quickstart/recommendation/pytorch/dlrm/inference/cpu/.docs/datasets.md b/quickstart/recommendation/pytorch/dlrm/inference/cpu/.docs/datasets.md
deleted file mode 100644
index 88fc484ee..000000000
--- a/quickstart/recommendation/pytorch/dlrm/inference/cpu/.docs/datasets.md
+++ /dev/null
@@ -1,21 +0,0 @@
-## Datasets
-
-### Criteo Terabyte Dataset
-
-The [Criteo Terabyte Dataset](https://labs.criteo.com/2013/12/download-terabyte-click-logs/) is
-used to run DLRM. To download the dataset, you will need to visit the Criteo website and accept
-their terms of use:
-[https://labs.criteo.com/2013/12/download-terabyte-click-logs/](https://labs.criteo.com/2013/12/download-terabyte-click-logs/).
-Copy the download URL into the command below as the `<download url>` and
-replace the `<dir/to/save/dlrm_data>` to any path where you want to download
-and save the dataset.
-```
-export DATASET_DIR=<dir/to/save/dlrm_data>
-
-mkdir ${DATASET_DIR} && cd ${DATASET_DIR}
-curl -O <download url>/day_{$(seq -s , 0 23)}.gz
-gunzip day_*.gz
-```
-The raw data will be automatically preprocessed and saved as `day_*.npz` to
-the `DATASET_DIR` when DLRM is run for the first time. On subsequent runs, the
-scripts will automatically use the preprocessed data.
diff --git a/quickstart/recommendation/pytorch/dlrm/inference/cpu/.docs/description.md b/quickstart/recommendation/pytorch/dlrm/inference/cpu/.docs/description.md
deleted file mode 100644
index 49f899094..000000000
--- a/quickstart/recommendation/pytorch/dlrm/inference/cpu/.docs/description.md
+++ /dev/null
@@ -1,5 +0,0 @@
-<!-- 10. Description -->
-## Description
-
-This document has instructions for running <model name> <mode> using
-Intel-optimized PyTorch.
diff --git a/quickstart/recommendation/pytorch/dlrm/inference/cpu/.docs/docker_spr.md b/quickstart/recommendation/pytorch/dlrm/inference/cpu/.docs/docker_spr.md
deleted file mode 100644
index 025068dc5..000000000
--- a/quickstart/recommendation/pytorch/dlrm/inference/cpu/.docs/docker_spr.md
+++ /dev/null
@@ -1,32 +0,0 @@
-## Run the model
-
-Download the DLRM PyTorch weights (`tb00_40M.pt`, 90GB) from the
-[MLPerf repo](https://github.com/mlcommons/inference/tree/master/recommendation/dlrm/pytorch#more-information-about-the-model-weights)
-and set the `WEIGHT_PATH` to point to the weights file.
-```
-export WEIGHT_PATH=<path to the tb00_40M.pt file>
-```
-
-After you've downloaded the model weight file and followed the instructions to
-[build the container](#build-the-container) and [downloaded dataset](#datasets),
-use the `run.sh` script from the container package to run <model name> <mode>.
-Set environment variables to specify the dataset directory, precision to run, and
-an output directory. By default, the `run.sh` script will run the
-`inference_performance.sh` quickstart script. To run a different script, specify
-the name of the script using the `SCRIPT` environment variable.
-```
-# Navigate to the container package directory
-cd <package dir>
-
-# Set the required environment vars
-export PRECISION=<specify the precision to run>
-export WEIGHT_PATH=<path to the tb00_40M.pt file>
-export DATASET_DIR=<path to the dataset>
-export OUTPUT_DIR=<directory where log files will be written>
-
-# Run the container with inference_performance.sh quickstart script
-./run.sh
-
-# Specify a different quickstart script to run, for example, accuracy.sh
-SCRIPT=accuracy.sh ./run.sh
-```
diff --git a/quickstart/recommendation/pytorch/dlrm/inference/cpu/.docs/license.md b/quickstart/recommendation/pytorch/dlrm/inference/cpu/.docs/license.md
deleted file mode 100644
index e547f148d..000000000
--- a/quickstart/recommendation/pytorch/dlrm/inference/cpu/.docs/license.md
+++ /dev/null
@@ -1,4 +0,0 @@
-<!--- 80. License -->
-## License
-
-Licenses can be found in the model package, in the `licenses` directory.
diff --git a/quickstart/recommendation/pytorch/dlrm/inference/cpu/.docs/quickstart.md b/quickstart/recommendation/pytorch/dlrm/inference/cpu/.docs/quickstart.md
deleted file mode 100644
index 06de1f033..000000000
--- a/quickstart/recommendation/pytorch/dlrm/inference/cpu/.docs/quickstart.md
+++ /dev/null
@@ -1,11 +0,0 @@
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| `inference_performance.sh` | Run inference to verify performance for the specified precision (fp32, avx-fp32, int8, avx-int8, or bf16). |
-| `accuracy.sh` | Measures the inference accuracy for the specified precision (fp32, avx-fp32, int8, avx-int8, or bf16). |
-
-> Note: The `avx-int8` and `avx-fp32` precisions run the same scripts as `int8` and `fp32`, except that the
-> `DNNL_MAX_CPU_ISA` environment variable is unset. The environment variable is
-> otherwise set to `DNNL_MAX_CPU_ISA=AVX512_CORE_AMX`.
diff --git a/quickstart/recommendation/pytorch/dlrm/inference/cpu/.docs/title.md b/quickstart/recommendation/pytorch/dlrm/inference/cpu/.docs/title.md
deleted file mode 100644
index a4ccbf863..000000000
--- a/quickstart/recommendation/pytorch/dlrm/inference/cpu/.docs/title.md
+++ /dev/null
@@ -1,2 +0,0 @@
-<!--- 0. Title -->
-# PyTorch <model name> <mode>
diff --git a/quickstart/recommendation/pytorch/dlrm/inference/cpu/.docs/wrapper_package.md b/quickstart/recommendation/pytorch/dlrm/inference/cpu/.docs/wrapper_package.md
deleted file mode 100644
index 0d77199f3..000000000
--- a/quickstart/recommendation/pytorch/dlrm/inference/cpu/.docs/wrapper_package.md
+++ /dev/null
@@ -1,16 +0,0 @@
-## Model Package
-
-The model package includes the Dockerfile and scripts needed to build and
-run <model name> <mode> in a container.
-```
-<package dir>
-├── README.md
-├── build.sh
-├── licenses
-│   ├── LICENSE
-│   └── third_party
-├── model_packages
-│   └── <package name>
-├── <package dir>.Dockerfile
-└── run.sh
-```
diff --git a/quickstart/recommendation/pytorch/dlrm/inference/cpu/DEVCATALOG.md b/quickstart/recommendation/pytorch/dlrm/inference/cpu/DEVCATALOG.md
new file mode 100644
index 000000000..e87c04df4
--- /dev/null
+++ b/quickstart/recommendation/pytorch/dlrm/inference/cpu/DEVCATALOG.md
@@ -0,0 +1,102 @@
+# PyTorch DLRM inference
+
+## Description 
+This document has instructions for running DLRM Inference using Intel Extension for PyTorch. 
+
+## Pull Command
+
+Docker image based on CentOS Stream8
+```
+docker pull intel/recommendation:centos-pytorch-cpu-dlrm-inference
+```
+
+Docker image based on Ubuntu 22.04
+```
+docker pull intel/recommendation:ubuntu-pytorch-cpu-dlrm-inference
+```
+
+## Quick Start Scripts
+| Script name | Description |
+|-------------|-------------|
+| `inference_performance.sh` | Run inference to verify performance for the specified precision (fp32, int8, bf32 or bf16). |
+| `accuracy.sh` | Measures the inference accuracy for the specified precision (fp32, int8, bf32 or bf16). |
+
+## Datasets
+### Criteo Terabyte Dataset
+
+The [Criteo Terabyte Dataset](https://labs.criteo.com/2013/12/download-terabyte-click-logs/) is used to run DLRM. To download the dataset, you will need to visit the Criteo website and accept their terms of use: [https://labs.criteo.com/2013/12/download-terabyte-click-logs/](https://labs.criteo.com/2013/12/download-terabyte-click-logs/). Copy the download URL into the command below as the `<download url>` and replace the `<dir/to/save/dlrm_data>` to any path where you want to download and save the dataset.
+```
+export DATASET_DIR=<dir/to/save/dlrm_data>
+mkdir ${DATASET_DIR} && cd ${DATASET_DIR}
+curl -O <download url>/day_{$(seq -s , 0 23)}.gz
+gunzip day_*.gz
+```
+The raw data will be automatically preprocessed and saved as `day_*.npz` to the `DATASET_DIR` when DLRM is run for the first time. On subsequent runs, the scripts will automatically use the preprocessed data.
+
+## Pre-trained Model
+Download the DLRM PyTorch weights (`tb00_40M.pt`, 90GB) from the [MLPerf repo](https://github.com/mlcommons/inference/tree/master/recommendation/dlrm/pytorch#more-information-about-the-model-weights) and set the `WEIGHT_PATH` to point to the weights file.
+```
+export WEIGHT_PATH=<path to the tb00_40M.pt file>
+```
+
+## Docker Run
+(Optional) Export related proxy into docker environment.
+```bash
+export DOCKER_RUN_ENVS="-e ftp_proxy=${ftp_proxy} \
+  -e FTP_PROXY=${FTP_PROXY} -e http_proxy=${http_proxy} \
+  -e HTTP_PROXY=${HTTP_PROXY} -e https_proxy=${https_proxy} \
+  -e HTTPS_PROXY=${HTTPS_PROXY} -e no_proxy=${no_proxy} \
+  -e NO_PROXY=${NO_PROXY} -e socks_proxy=${socks_proxy} \
+  -e SOCKS_PROXY=${SOCKS_PROXY}"
+```
+To run DLRM inference, set environment variables to specify the dataset directory, precision,pre-trained model, and an output directory. 
+```bash
+export OS=<provide either centos or ubuntu>
+export DATASET_DIR=<path to the dataset>
+export OUTPUT_DIR=<directory where log files will be written>
+export SCRIPT=quickstart/<specify the script to run>
+export PRECISION=<specify the precision to run>
+export BERT_WEIGHT=<specify path to pre-trained model>
+
+IMAGE_NAME=intel/recommendation:${OS}-pytorch-cpu-dlrm-inference
+DOCKER_ARGS="--privileged --init -it"
+WORKDIR=/workspace/pytorch-dlrm-inference
+
+docker run --rm \
+  --env DATASET_DIR=${DATASET_DIR} \
+  --env PRECISION=${PRECISION} \
+  --env WEIGHT_PATH=${WEIGHT_PATH} \
+  --env OUTPUT_DIR=${OUTPUT_DIR} \
+  --env http_proxy=${http_proxy} \
+  --env https_proxy=${https_proxy} \
+  --env no_proxy=${no_proxy} \
+  --volume ${DATASET_DIR}:${DATASET_DIR} \
+  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
+  --volume ${WEIGHT_PATH}:${WEIGHT_PATH} \
+  ${DOCKER_RUN_ENVS} \
+  --shm-size 8G \
+  -w ${WORKDIR} \
+  ${DOCKER_ARGS} \
+  $IMAGE_NAME \
+  /bin/bash $SCRIPT
+  ```
+
+## Documentation and Sources
+#### Get Started​
+[Docker* Repository](https://hub.docker.com/r/intel/recommendation)
+
+[Main GitHub*](https://github.com/IntelAI/models)
+
+[Release Notes](https://github.com/IntelAI/models/releases)
+
+[Get Started Guide](https://github.com/IntelAI/models/blob/master/quickstart/quickstart/recommendation/pytorch/dlrm/inference/cpu/DEVCATALOG.md)
+
+#### Code Sources
+[Dockerfile](https://github.com/IntelAI/models/tree/master/docker/pyt-cpu)
+
+[Report Issue](https://community.intel.com/t5/Intel-Optimized-AI-Frameworks/bd-p/optimized-ai-frameworks)
+
+## License Agreement
+LEGAL NOTICE: By accessing, downloading or using this software and any required dependent software (the “Software Package”), you agree to the terms and conditions of the software license agreements for the Software Package, which may also include notices, disclaimers, or license terms for third party software included with the Software Package. Please refer to the [license](https://github.com/IntelAI/models/tree/master/third_party) file for additional details.
+
+[View All Containers and Solutions 🡢](https://www.intel.com/content/www/us/en/developer/tools/software-catalog/containers.html?s=Newest)
diff --git a/quickstart/recommendation/pytorch/dlrm/inference/cpu/README.md b/quickstart/recommendation/pytorch/dlrm/inference/cpu/README.md
index 80d717d01..fbb4e5344 100644
--- a/quickstart/recommendation/pytorch/dlrm/inference/cpu/README.md
+++ b/quickstart/recommendation/pytorch/dlrm/inference/cpu/README.md
@@ -7,97 +7,89 @@
 This document has instructions for running DLRM inference using
 Intel-optimized PyTorch for bare metal.
 
-## Bare Metal
-### General Setup
-Follow [link](/docs/general/pytorch/BareMetalSetup.md) to install Miniconda and build Pytorch, IPEX, and Jemalloc.
-
-### Model Specific Setup
-
-* Install dependencies
-  ```bash
-  cd <clone of the model zoo>/quickstart/recommendation/pytorch/dlrm
-  pip install requirements.txt
-  ```
-
-* Set Jemalloc Preload for better performance
-
-  The jemalloc should be built from the [General setup](#general-setup) section.
-  ```bash
-  export LD_PRELOAD="<path to the jemalloc directory>/lib/libjemalloc.so":$LD_PRELOAD
-  export MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
-  ```
-
-* Set IOMP preload for better performance
+## Quick Start Scripts
+| Script name | Description |
+|-------------|-------------|
+| `inference_performance.sh` | Run inference to verify performance for the specified precision (fp32, int8, bf32 or bf16). |
+| `accuracy.sh` | Measures the inference accuracy for the specified precision (fp32, int8, bf32 or bf16). |
 
-  IOMP should be installed in your conda env from the [General setup](#general-setup) section.
-  ```bash
-  export LD_PRELOAD=<path to the intel-openmp directory>/lib/libiomp5.so:$LD_PRELOAD
+> Note: The `avx-int8` and `avx-fp32` precisions run the same scripts as `int8` and `fp32`, except that the
+> `DNNL_MAX_CPU_ISA` environment variable is unset. The environment variable is
+> otherwise set to `DNNL_MAX_CPU_ISA=AVX512_CORE_AMX`.
+* Set ENV to use AMX:
   ```
-
-* Set ENV to use AMX if you are using SPR
-  ```bash
   export DNNL_MAX_CPU_ISA=AVX512_CORE_AMX
   ```
 
 ## Datasets
-
 ### Criteo Terabyte Dataset
 
-The [Criteo Terabyte Dataset](https://labs.criteo.com/2013/12/download-terabyte-click-logs/) is
-used to run DLRM. To download the dataset, you will need to visit the Criteo website and accept
-their terms of use:
-[https://labs.criteo.com/2013/12/download-terabyte-click-logs/](https://labs.criteo.com/2013/12/download-terabyte-click-logs/).
-Copy the download URL into the command below as the `<download url>` and
-replace the `<dir/to/save/dlrm_data>` to any path where you want to download
-and save the dataset.
-```bash
+The [Criteo Terabyte Dataset](https://labs.criteo.com/2013/12/download-terabyte-click-logs/) is used to run DLRM. To download the dataset, you will need to visit the Criteo website and accept their terms of use: [https://labs.criteo.com/2013/12/download-terabyte-click-logs/](https://labs.criteo.com/2013/12/download-terabyte-click-logs/). Copy the download URL into the command below as the `<download url>` and replace the `<dir/to/save/dlrm_data>` to any path where you want to download and save the dataset.
+```
 export DATASET_DIR=<dir/to/save/dlrm_data>
-
 mkdir ${DATASET_DIR} && cd ${DATASET_DIR}
 curl -O <download url>/day_{$(seq -s , 0 23)}.gz
 gunzip day_*.gz
 ```
-The raw data will be automatically preprocessed and saved as `day_*.npz` to
-the `DATASET_DIR` when DLRM is run for the first time. On subsequent runs, the
-scripts will automatically use the preprocessed data.
+The raw data will be automatically preprocessed and saved as `day_*.npz` to the `DATASET_DIR` when DLRM is run for the first time. On subsequent runs, the scripts will automatically use the preprocessed data.
 
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| `inference_performance.sh` | Run inference to verify performance for the specified precision (fp32, bf32, bf16, or int8). |
-| `accuracy.sh` | Measures the inference accuracy for the specified precision (fp32, bf32, bf16, or int8). |
-
-## Run the model
-
-Download the DLRM PyTorch weights (`tb00_40M.pt`, 90GB) from the
-[MLPerf repo](https://github.com/mlcommons/inference/tree/master/recommendation/dlrm/pytorch#more-information-about-the-model-weights)
-and set the `WEIGHT_PATH` to point to the weights file.
+## Pre-trained Model
+Download the DLRM PyTorch weights (`tb00_40M.pt`, 90GB) from the [MLPerf repo](https://github.com/mlcommons/inference/tree/master/recommendation/dlrm/pytorch#more-information-about-the-model-weights) and set the `WEIGHT_PATH` to point to the weights file.
 ```
 export WEIGHT_PATH=<path to the tb00_40M.pt file>
 ```
 
+## Bare Metal
+### General Setup
+Follow [link](/docs/general/pytorch/BareMetalSetup.md) to install Miniconda and build Pytorch, IPEX, TorchVison and Jemalloc and TCmalloc
+
+### Model Specific Setup
+* Set Jemalloc and tcmalloc Preload for better performance
+
+  The jemalloc should be built from the [General setup](#general-setup) section.
+  ```
+  export LD_PRELOAD="<path to the jemalloc directory>/lib/libjemalloc.so":"path_to/tcmalloc/lib/libtcmalloc.so":$LD_PRELOAD
+  export MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+  ```
+
+* Set IOMP preload for better performance
+
+  IOMP should be installed in your conda env from the [General setup](#general-setup) section.
+  ```
+  pip install packaging intel-openmp
+  export LD_PRELOAD=<path to the intel-openmp directory>/lib/libiomp5.so:$LD_PRELOAD
+  ```
+
+## Run the model
 Follow the instructions above to setup your bare metal environment, do the
 model-specific setup and download and prepropcess the datsaet. Once all the
-setup is done, the Model Zoo can be used to run a [quickstart script](#quick-start-scripts).
+setup is done, the Intel® AI Reference Models can be used to run a [quickstart script](#quick-start-scripts).
 Ensure that you have enviornment variables set to point to the dataset directory,
 precision, weights file, and an output directory.
 
 ```bash
-# Clone the model zoo repo and set the MODEL_DIR
+# Clone the Intel® AI Reference Models repo and set the MODEL_DIR
 git clone https://github.com/IntelAI/models.git
 cd models
 export MODEL_DIR=$(pwd)
 
+# Install dependencies
+pip install quickstart/recommendation/pytorch/dlrm/requirements.txt
+
 # Env vars
-export PRECISION=<specify the precision to run>
+export PRECISION=<specify the precision to run: int8, fp32, bf32 or bf16>
 export WEIGHT_PATH=<path to the tb00_40M.pt file> # only needed for testing accuracy
 export DATASET_DIR=<path to the dataset>
 export OUTPUT_DIR=<directory where log files will be written>
 
-# Run a quickstart script (for example, bare metal performance)
-cd ${MODEL_DIR}/quickstart/recommendation/pytorch/dlrm/inference/cpu
-bash inference_performance.sh
+# Optional environemnt variables:
+export BATCH_SIZE=<set a value for batch size, else it will run with default batch size>
+
+# [optional] Compile model with PyTorch Inductor backend 
+export TORCH_INDUCTOR=1
+
+# Run a quickstart script
+./quickstart/recommendation/pytorch/dlrm/inference/cpu/<script.sh>
 ```
 
 <!--- 80. License -->
diff --git a/quickstart/recommendation/pytorch/dlrm/inference/cpu/accuracy.sh b/quickstart/recommendation/pytorch/dlrm/inference/cpu/accuracy.sh
index 25275184a..345469200 100755
--- a/quickstart/recommendation/pytorch/dlrm/inference/cpu/accuracy.sh
+++ b/quickstart/recommendation/pytorch/dlrm/inference/cpu/accuracy.sh
@@ -48,6 +48,11 @@ if [ -z "${WEIGHT_PATH}" ]; then
   exit 1
 fi
 
+if [ -z "${PRECISION}" ]; then
+  echo "Please set PRECISION: int8, fp32, bf16, bf32"
+  exit 1
+fi
+
 # Create the output directory in case it doesn't already exist
 mkdir -p ${OUTPUT_DIR}/dlrm_inference_accuracy_log
 
@@ -81,16 +86,31 @@ numa_cmd="numactl -C 0-$((CORES-1))  "
 echo "will run on core 0-$((CORES-1)) on socket 0" 
 
 export OMP_NUM_THREADS=$CORES
-python -m intel_extension_for_pytorch.cpu.launch --node_id=0 --enable_tcmalloc $MODEL_SCRIPT \
---raw-data-file=${DATASET_DIR}/day --processed-data-file=${DATASET_DIR}/terabyte_processed.npz \
---data-set=terabyte \
---memory-map --mlperf-bin-loader --round-targets=True --learning-rate=1.0 \
---arch-mlp-bot=13-512-256-128 --arch-mlp-top=1024-1024-512-256-1 \
---arch-sparse-feature-size=128 --max-ind-range=40000000 \
---numpy-rand-seed=727  --inference-only --ipex-interaction \
---print-freq=100 --print-time --mini-batch-size=2048 --test-mini-batch-size=16384 \
---test-freq=2048 --print-auc $ARGS \
---load-model=${WEIGHT_PATH} | tee $LOG
+TORCH_INDUCTOR=${TORCH_INDUCTOR:-"0"}
+if [[ "0" == ${TORCH_INDUCTOR} ]];then
+    python -m intel_extension_for_pytorch.cpu.launch --node_id=0 --enable_tcmalloc $MODEL_SCRIPT \
+    --raw-data-file=${DATASET_DIR}/day --processed-data-file=${DATASET_DIR}/terabyte_processed.npz \
+    --data-set=terabyte \
+    --memory-map --mlperf-bin-loader --round-targets=True --learning-rate=1.0 \
+    --arch-mlp-bot=13-512-256-128 --arch-mlp-top=1024-1024-512-256-1 \
+    --arch-sparse-feature-size=128 --max-ind-range=40000000 \
+    --numpy-rand-seed=727  --inference-only --ipex-interaction \
+    --print-freq=100 --print-time --mini-batch-size=2048 --test-mini-batch-size=16384 \
+    --test-freq=2048 --print-auc $ARGS \
+    --load-model=${WEIGHT_PATH} | tee $LOG
+else
+  echo "### running with torch.compile inductor backend"
+  python -m intel_extension_for_pytorch.cpu.launch --node_id=0 --enable_tcmalloc $MODEL_SCRIPT \
+    --raw-data-file=${DATASET_DIR}/day --processed-data-file=${DATASET_DIR}/terabyte_processed.npz \
+    --data-set=terabyte \
+    --memory-map --mlperf-bin-loader --round-targets=True --learning-rate=1.0 \
+    --arch-mlp-bot=13-512-256-128 --arch-mlp-top=1024-1024-512-256-1 \
+    --arch-sparse-feature-size=128 --max-ind-range=40000000 \
+    --numpy-rand-seed=727  --inference-only --inductor \
+    --print-freq=100 --print-time --mini-batch-size=2048 --test-mini-batch-size=16384 \
+    --test-freq=2048 --print-auc $ARGS \
+    --load-model=${WEIGHT_PATH} | tee $LOG
+fi
 
 accuracy=$(grep 'Accuracy:' $LOG |sed -e 's/.*Accuracy//;s/[^0-9.]//g')
 echo ""dlrm";"auc";${PRECISION};16384;${accuracy}" | tee -a ${OUTPUT_DIR}/summary.log
diff --git a/quickstart/recommendation/pytorch/dlrm/inference/cpu/inference_performance.sh b/quickstart/recommendation/pytorch/dlrm/inference/cpu/inference_performance.sh
index 3166b695b..5457a0599 100755
--- a/quickstart/recommendation/pytorch/dlrm/inference/cpu/inference_performance.sh
+++ b/quickstart/recommendation/pytorch/dlrm/inference/cpu/inference_performance.sh
@@ -41,6 +41,10 @@ if [ ! -d "${DATASET_DIR}" ]; then
   exit 1
 fi
 
+if [ -z "${PRECISION}" ]; then
+  echo "Please set PRECISION: int8, fp32, bf16, bf32"
+  exit 1
+fi
 
 # Create the output directory in case it doesn't already exist
 mkdir -p ${OUTPUT_DIR}
@@ -54,6 +58,9 @@ NUMA_NODES=`lscpu | grep "NUMA node(s)" | awk '{print $3}'`
 NUMA_NODES_PER_SOCKETS=`expr $NUMA_NODES / $SOCKETS`
 CORES_PER_NUMA_NODE=`expr $CORES_PER_SOCKET / $NUMA_NODES_PER_SOCKETS`
 
+# Runs with default value when BATCH SIZE is not set:
+BATCH_SIZE=${BATCH_SIZE:-128}
+
 ARGS=""
 if [[ $PRECISION == "int8" ]]; then
     echo "running int8 path"
@@ -74,15 +81,29 @@ fi
 
 export OMP_NUM_THREADS=1
 LOG_0="${LOG}/throughput.log"
-python -m intel_extension_for_pytorch.cpu.launch --throughput_mode --enable_jemalloc $MODEL_SCRIPT \
---raw-data-file=${DATASET_DIR}/day --processed-data-file=${DATASET_DIR}/terabyte_processed.npz \
---data-set=terabyte \
---memory-map --mlperf-bin-loader --round-targets=True --learning-rate=1.0 \
---arch-mlp-bot=13-512-256-128 --arch-mlp-top=1024-1024-512-256-1 \
---arch-sparse-feature-size=128 --max-ind-range=40000000 --ipex-interaction \
---numpy-rand-seed=727  --inference-only --num-batches=1000 \
---print-freq=10 --print-time --test-mini-batch-size=128 --share-weight-instance=$CORES_PER_NUMA_NODE \
-$ARGS |tee $LOG_0
+TORCH_INDUCTOR=${TORCH_INDUCTOR:-"0"}
+if [[ "0" == ${TORCH_INDUCTOR} ]];then
+    python -m intel_extension_for_pytorch.cpu.launch --throughput_mode --enable_jemalloc $MODEL_SCRIPT \
+        --raw-data-file=${DATASET_DIR}/day --processed-data-file=${DATASET_DIR}/terabyte_processed.npz \
+        --data-set=terabyte \
+        --memory-map --mlperf-bin-loader --round-targets=True --learning-rate=1.0 \
+        --arch-mlp-bot=13-512-256-128 --arch-mlp-top=1024-1024-512-256-1 \
+        --arch-sparse-feature-size=128 --max-ind-range=40000000 --ipex-interaction \
+        --numpy-rand-seed=727  --inference-only --num-batches=1000 \
+        --print-freq=10 --print-time --test-mini-batch-size=${BATCH_SIZE} --share-weight-instance=$CORES_PER_NUMA_NODE \
+        $ARGS |tee $LOG_0
+else
+    echo "### running with torch.compile inductor backend"
+    python -m intel_extension_for_pytorch.cpu.launch --throughput_mode --enable_jemalloc $MODEL_SCRIPT \
+        --raw-data-file=${DATASET_DIR}/day --processed-data-file=${DATASET_DIR}/terabyte_processed.npz \
+        --data-set=terabyte \
+        --memory-map --mlperf-bin-loader --round-targets=True --learning-rate=1.0 \
+        --arch-mlp-bot=13-512-256-128 --arch-mlp-top=1024-1024-512-256-1 \
+        --arch-sparse-feature-size=128 --max-ind-range=40000000 --inductor \
+        --numpy-rand-seed=727  --inference-only --num-batches=1000 \
+        --print-freq=10 --print-time --test-mini-batch-size=${BATCH_SIZE} --share-weight-instance=$CORES_PER_NUMA_NODE \
+        $ARGS |tee $LOG_0
+fi
 wait
 
 throughput=$(grep 'Throughput:' ${LOG}/throughput.log |sed -e 's/.*Throughput//;s/[^0-9.]//g' |awk '
@@ -98,4 +119,4 @@ END   {
 sum = sum / i;
         printf("%.3f", sum);
 }')
-echo ""dlrm";"throughput";${PRECISION};128;${throughput}" | tee -a ${OUTPUT_DIR}/summary.log
+echo ""dlrm";"throughput";${PRECISION};${BATCH_SIZE};${throughput}" | tee -a ${OUTPUT_DIR}/summary.log
diff --git a/quickstart/recommendation/pytorch/dlrm/training/cpu/.docs/container_build.md b/quickstart/recommendation/pytorch/dlrm/training/cpu/.docs/container_build.md
deleted file mode 100644
index d57bf6489..000000000
--- a/quickstart/recommendation/pytorch/dlrm/training/cpu/.docs/container_build.md
+++ /dev/null
@@ -1,26 +0,0 @@
-## Build the container
-
-The <model name> <mode> package has scripts and a Dockerfile that are
-used to build a workload container that runs the model. This container
-uses the PyTorch/IPEX container as it's base, so ensure that you have built
-the `pytorch-ipex-spr.tar.gz` container prior to building this model container.
-
-Use `docker images` to verify that you have the base container built. For example:
-```
-$ docker images | grep pytorch-ipex-spr
-model-zoo         pytorch-ipex-spr         f5b473554295        2 hours ago         4.08GB
-```
-
-To build the <model name> <mode> container, extract the package and
-run the `build.sh` script.
-```
-# Extract the package
-tar -xzf <package name>
-cd <package dir>
-
-# Build the container
-./build.sh
-```
-
-After the build completes, you should have a container called
-`<docker image>` that will be used to run the model.
diff --git a/quickstart/recommendation/pytorch/dlrm/training/cpu/.docs/datasets.md b/quickstart/recommendation/pytorch/dlrm/training/cpu/.docs/datasets.md
deleted file mode 100644
index 88fc484ee..000000000
--- a/quickstart/recommendation/pytorch/dlrm/training/cpu/.docs/datasets.md
+++ /dev/null
@@ -1,21 +0,0 @@
-## Datasets
-
-### Criteo Terabyte Dataset
-
-The [Criteo Terabyte Dataset](https://labs.criteo.com/2013/12/download-terabyte-click-logs/) is
-used to run DLRM. To download the dataset, you will need to visit the Criteo website and accept
-their terms of use:
-[https://labs.criteo.com/2013/12/download-terabyte-click-logs/](https://labs.criteo.com/2013/12/download-terabyte-click-logs/).
-Copy the download URL into the command below as the `<download url>` and
-replace the `<dir/to/save/dlrm_data>` to any path where you want to download
-and save the dataset.
-```
-export DATASET_DIR=<dir/to/save/dlrm_data>
-
-mkdir ${DATASET_DIR} && cd ${DATASET_DIR}
-curl -O <download url>/day_{$(seq -s , 0 23)}.gz
-gunzip day_*.gz
-```
-The raw data will be automatically preprocessed and saved as `day_*.npz` to
-the `DATASET_DIR` when DLRM is run for the first time. On subsequent runs, the
-scripts will automatically use the preprocessed data.
diff --git a/quickstart/recommendation/pytorch/dlrm/training/cpu/.docs/description.md b/quickstart/recommendation/pytorch/dlrm/training/cpu/.docs/description.md
deleted file mode 100644
index 49f899094..000000000
--- a/quickstart/recommendation/pytorch/dlrm/training/cpu/.docs/description.md
+++ /dev/null
@@ -1,5 +0,0 @@
-<!-- 10. Description -->
-## Description
-
-This document has instructions for running <model name> <mode> using
-Intel-optimized PyTorch.
diff --git a/quickstart/recommendation/pytorch/dlrm/training/cpu/.docs/docker_spr.md b/quickstart/recommendation/pytorch/dlrm/training/cpu/.docs/docker_spr.md
deleted file mode 100644
index 08028941c..000000000
--- a/quickstart/recommendation/pytorch/dlrm/training/cpu/.docs/docker_spr.md
+++ /dev/null
@@ -1,23 +0,0 @@
-## Run the model
-
-After you've followed the instructions to [build the container](#build-the-container)
-and [downloaded dataset](#datasets), use the `run.sh` script from the container package
-to run <model name> <mode>. Set environment variables to specify the dataset directory,
-precision to run, and an output directory. The `run.sh` script will run the
-`training.sh` quickstart script in the docker container. Set the number of
-batches that are run using the `NUM_BATCH` environment variable.
-```
-# Navigate to the container package directory
-cd <package dir>
-
-# Set the required environment vars
-export PRECISION=<specify the precision to run>
-export DATASET_DIR=<path to the dataset>
-export OUTPUT_DIR=<directory where log files will be written>
-
-# Run the quickstart script to test performance
-NUM_BATCH=10000 ./run.sh
-
-# Or, run quickstart script for testing convergence trend
-NUM_BATCH=50000 ./run.sh
-```
diff --git a/quickstart/recommendation/pytorch/dlrm/training/cpu/.docs/license.md b/quickstart/recommendation/pytorch/dlrm/training/cpu/.docs/license.md
deleted file mode 100644
index e547f148d..000000000
--- a/quickstart/recommendation/pytorch/dlrm/training/cpu/.docs/license.md
+++ /dev/null
@@ -1,4 +0,0 @@
-<!--- 80. License -->
-## License
-
-Licenses can be found in the model package, in the `licenses` directory.
diff --git a/quickstart/recommendation/pytorch/dlrm/training/cpu/.docs/quickstart.md b/quickstart/recommendation/pytorch/dlrm/training/cpu/.docs/quickstart.md
deleted file mode 100644
index 845a0fe4b..000000000
--- a/quickstart/recommendation/pytorch/dlrm/training/cpu/.docs/quickstart.md
+++ /dev/null
@@ -1,10 +0,0 @@
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| `training.sh` | Run training for the specified precision (fp32, avx-fp32, or bf16). |
-
-> Note: The `avx-fp32` precision runs the same scripts as `fp32`, except that the
-> `DNNL_MAX_CPU_ISA` environment variable is unset. The environment variable is
-> otherwise set to `DNNL_MAX_CPU_ISA=AVX512_CORE_AMX`.
diff --git a/quickstart/recommendation/pytorch/dlrm/training/cpu/.docs/title.md b/quickstart/recommendation/pytorch/dlrm/training/cpu/.docs/title.md
deleted file mode 100644
index a4ccbf863..000000000
--- a/quickstart/recommendation/pytorch/dlrm/training/cpu/.docs/title.md
+++ /dev/null
@@ -1,2 +0,0 @@
-<!--- 0. Title -->
-# PyTorch <model name> <mode>
diff --git a/quickstart/recommendation/pytorch/dlrm/training/cpu/.docs/wrapper_package.md b/quickstart/recommendation/pytorch/dlrm/training/cpu/.docs/wrapper_package.md
deleted file mode 100644
index 0d77199f3..000000000
--- a/quickstart/recommendation/pytorch/dlrm/training/cpu/.docs/wrapper_package.md
+++ /dev/null
@@ -1,16 +0,0 @@
-## Model Package
-
-The model package includes the Dockerfile and scripts needed to build and
-run <model name> <mode> in a container.
-```
-<package dir>
-├── README.md
-├── build.sh
-├── licenses
-│   ├── LICENSE
-│   └── third_party
-├── model_packages
-│   └── <package name>
-├── <package dir>.Dockerfile
-└── run.sh
-```
diff --git a/quickstart/recommendation/pytorch/dlrm/training/cpu/DEVCATALOG.md b/quickstart/recommendation/pytorch/dlrm/training/cpu/DEVCATALOG.md
new file mode 100644
index 000000000..836415960
--- /dev/null
+++ b/quickstart/recommendation/pytorch/dlrm/training/cpu/DEVCATALOG.md
@@ -0,0 +1,96 @@
+# PyTorch DLRM training
+
+## Description 
+This document has instructions for running DLRM Training using Intel Extension for PyTorch. 
+
+## Pull Command
+
+Docker image based on CentOS Stream8
+```
+docker pull intel/recommendation:centos-pytorch-cpu-dlrm-training
+```
+
+Docker image based on Ubuntu 22.04
+```
+docker pull intel/recommendation:ubuntu-pytorch-cpu-dlrm-training
+```
+
+## Quick Start Scripts
+
+| Script name | Description |
+|-------------|-------------|
+| `training.sh` | Run training for the specified precision (fp32, bf32 or bf16). |
+
+
+## Datasets
+### Criteo Terabyte Dataset
+
+The [Criteo Terabyte Dataset](https://labs.criteo.com/2013/12/download-terabyte-click-logs/) is used to run DLRM. To download the dataset, you will need to visit the Criteo website and accept their terms of use: [https://labs.criteo.com/2013/12/download-terabyte-click-logs/](https://labs.criteo.com/2013/12/download-terabyte-click-logs/). Copy the download URL into the command below as the `<download url>` and replace the `<dir/to/save/dlrm_data>` to any path where you want to download and save the dataset.
+```
+export DATASET_DIR=<dir/to/save/dlrm_data>
+
+mkdir ${DATASET_DIR} && cd ${DATASET_DIR}
+curl -O <download url>/day_{$(seq -s , 0 23)}.gz
+gunzip day_*.gz
+```
+The raw data will be automatically preprocessed and saved as `day_*.npz` to the `DATASET_DIR` when DLRM is run for the first time. On subsequent runs, the scripts will automatically use the preprocessed data.
+
+## Docker Run
+(Optional) Export related proxy into docker environment.
+```bash
+export DOCKER_RUN_ENVS="-e ftp_proxy=${ftp_proxy} \
+  -e FTP_PROXY=${FTP_PROXY} -e http_proxy=${http_proxy} \
+  -e HTTP_PROXY=${HTTP_PROXY} -e https_proxy=${https_proxy} \
+  -e HTTPS_PROXY=${HTTPS_PROXY} -e no_proxy=${no_proxy} \
+  -e NO_PROXY=${NO_PROXY} -e socks_proxy=${socks_proxy} \
+  -e SOCKS_PROXY=${SOCKS_PROXY}"
+```
+To run DLRM training, set environment variables to specify the dataset directory, precision,pre-trained model, and an output directory. 
+```bash
+export OS=<provide either centos or ubuntu>
+export DATASET_DIR=<path to the dataset>
+export OUTPUT_DIR=<directory where log files will be written>
+export PRECISION=<specify the precision to run>
+export SCRIPT=quickstart/training.sh
+export NUM_BATCH=<10000 for test performance and 50000 for testing convergence trend>  
+
+IMAGE_NAME=intel/recommendation:${OS}-pytorch-cpu-dlrm-training
+DOCKER_ARGS="--privileged --init -it"
+WORKDIR=/workspace/pytorch-dlrm-training
+
+docker run --rm \
+  --env NUM_BATCH=${NUM_BATCH} \
+  --env DATASET_DIR=${DATASET_DIR} \
+  --env PRECISION=${PRECISION} \
+  --env OUTPUT_DIR=${OUTPUT_DIR} \
+  --env http_proxy=${http_proxy} \
+  --env https_proxy=${https_proxy} \
+  --env no_proxy=${no_proxy} \
+  --volume ${DATASET_DIR}:${DATASET_DIR} \
+  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
+  ${DOCKER_RUN_ENVS} \
+  --shm-size 10G \
+  -w ${WORKDIR} \
+  ${DOCKER_ARGS} \
+  $IMAGE_NAME \
+  /bin/bash $SCRIPT
+```
+## Documentation and Sources
+#### Get Started​
+[Docker* Repository](https://hub.docker.com/r/intel/recommendation)
+
+[Main GitHub*](https://github.com/IntelAI/models)
+
+[Release Notes](https://github.com/IntelAI/models/releases)
+
+[Get Started Guide](https://github.com/IntelAI/models/blob/master/quickstart/recommendation/pytorch/dlrm/training/cpu/DEVCATALOG.md)
+
+#### Code Sources
+[Dockerfile](https://github.com/IntelAI/models/tree/master/docker/pyt-cpu)
+
+[Report Issue](https://community.intel.com/t5/Intel-Optimized-AI-Frameworks/bd-p/optimized-ai-frameworks)
+
+## License Agreement
+LEGAL NOTICE: By accessing, downloading or using this software and any required dependent software (the “Software Package”), you agree to the terms and conditions of the software license agreements for the Software Package, which may also include notices, disclaimers, or license terms for third party software included with the Software Package. Please refer to the [license](https://github.com/IntelAI/models/tree/master/third_party) file for additional details.
+
+[View All Containers and Solutions 🡢](https://www.intel.com/content/www/us/en/developer/tools/software-catalog/containers.html?s=Newest)
diff --git a/quickstart/recommendation/pytorch/dlrm/training/cpu/README.md b/quickstart/recommendation/pytorch/dlrm/training/cpu/README.md
index 3f9623253..035482277 100644
--- a/quickstart/recommendation/pytorch/dlrm/training/cpu/README.md
+++ b/quickstart/recommendation/pytorch/dlrm/training/cpu/README.md
@@ -7,19 +7,17 @@
 This document has instructions for running DLRM training using
 Intel-optimized PyTorch for bare metal.
 
-### General Setup
-Follow [link](/docs/general/pytorch/BareMetalSetup.md) to install Miniconda and build Pytorch, IPEX, and Jemalloc.
+## Quick Start Scripts
 
-### Model Specific Setup
+| Script name | Description |
+|-------------|-------------|
+| `training.sh` | Run training for the specified precision (fp32, avx-fp32, bf32 or bf16). |
+| `test_convergency.sh` | Run fully convergency test for the specified precision (fp32, bf16, bf32). |
+| `distribute_training.sh` | Run distribute training on 1 node with 2 sockets for the specified precision (fp32, bf16, bf32). |
 
-* Install dependencies
-  ```bash
-  cd <clone of the model zoo>/quickstart/recommendation/pytorch/dlrm
-  pip install requirements.txt
+**Note**: The `avx-fp32` precision runs the same scripts as `fp32`, except that the `DNNL_MAX_CPU_ISA` environment variable is unset. The environment variable is otherwise set to `DNNL_MAX_CPU_ISA=AVX512_CORE_AMX`.
+* Set ENV to use AMX:
   ```
-
-* Set ENV to use AMX if you are using SPR
-  ```bash
   export DNNL_MAX_CPU_ISA=AVX512_CORE_AMX
   ```
 
@@ -45,53 +43,62 @@ The raw data will be automatically preprocessed and saved as `day_*.npz` to
 the `DATASET_DIR` when DLRM is run for the first time. On subsequent runs, the
 scripts will automatically use the preprocessed data.
 
-### Environment Setup
-```bash
+### General Setup
+Follow [link](/docs/general/pytorch/BareMetalSetup.md) to install Miniconda and build Pytorch, IPEX, TorchVison and Jemalloc and TCmalloc
 
-# set OUTPUT_DIR, PRECISION, WEIGHT, DATASET
-export PRECISION=<specify the precision to run>
-export DATASET_DIR=<path to the dataset>
-export OUTPUT_DIR=<directory where log files will be written>
+### Model Specific Setup
+* Set Jemalloc and tcmalloc Preload for better performance
 
-```
-## Quick Start Scripts
+  The jemalloc should be built from the [General setup](#general-setup) section.
+  ```
+  export LD_PRELOAD="<path to the jemalloc directory>/lib/libjemalloc.so":"path_to/tcmalloc/lib/libtcmalloc.so":$LD_PRELOAD
+  export MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+  ```
 
-| Script name | Description |
-|-------------|-------------|
-| `training.sh` | Run training for the specified precision (fp32, bf16, bf32). |
-| `test_convergency.sh` | Run fully convergency test for the specified precision (fp32, bf16, bf32). |
-| `distribute_training.sh` | Run distribute training on 1 node with 2 sockets for the specified precision (fp32, bf16, bf32). |
+* Set IOMP preload for better performance
+
+  IOMP should be installed in your conda env from the [General setup](#general-setup) section.
+  ```
+  pip install packaging intel-openmp
+  export LD_PRELOAD=<path to the intel-openmp directory>/lib/libiomp5.so:$LD_PRELOAD
+  ```
 
 ## Run the model
 
 Follow the instructions above to setup your bare metal environment, do the
 model-specific setup and download and prepropcess the datsaet. Once all the
-setup is done, the Model Zoo can be used to run a [quickstart script](#quick-start-scripts).
+setup is done, the Intel® AI Reference Models can be used to run a [quickstart script](#quick-start-scripts).
 Ensure that you have enviornment variables set to point to the dataset directory,
 precision, and an output directory. The `NUM_BATCH` environment variable
 can be set to specify the number of batches to run.
 
 ```bash
-# Clone the model zoo repo and set the MODEL_DIR
+# Clone the Intel® AI Reference Models repo and set the MODEL_DIR
 git clone https://github.com/IntelAI/models.git
 cd models
 export MODEL_DIR=$(pwd)
 
+# Install dependencies
+pip install quickstart/recommendation/pytorch/dlrm/requirements.txt
+
 # Env vars
-export PRECISION=<specify the precision to run>
+export PRECISION=<specify the precision to run: fp32, bf16 or bf32>
 export DATASET_DIR=<path to the dataset>
 export OUTPUT_DIR=<directory where log files will be written>
+export NUM_BATCH=<10000 for test performance and 50000 for testing convergence trend>  
 
-# Navigate to the DLRM training quickstart directory
-cd ${MODEL_DIR}/quickstart/recommendation/pytorch/dlrm/training/cpu
+# Optional environemnt variables:
+export BATCH_SIZE=<set a value for batch size, else it will run with default batch size>
 
-# Run the quickstart script to test performance
-NUM_BATCH=10000 bash training.sh
+# [optional] Compile model with PyTorch Inductor backend 
+export TORCH_INDUCTOR=1
 
-# Run quickstart script for testing convergence trend
-NUM_BATCH=50000 bash training.sh
+# Run quick start script:
+./quickstart/recommendation/pytorch/dlrm/training/cpu/training.sh
 
 # Or, run quickstart script for testing fully convergency
+# Navigate to the DLRM training quickstart directory
+cd ${MODEL_DIR}/quickstart/recommendation/pytorch/dlrm/training/cpu
 bash test_convergence.sh
 
 # Run quickstart to distribute training dlrm on 2 sockets
diff --git a/quickstart/recommendation/pytorch/dlrm/training/cpu/bfloat16/.docs/baremetal.md b/quickstart/recommendation/pytorch/dlrm/training/cpu/bfloat16/.docs/baremetal.md
deleted file mode 100644
index d4bd98dcc..000000000
--- a/quickstart/recommendation/pytorch/dlrm/training/cpu/bfloat16/.docs/baremetal.md
+++ /dev/null
@@ -1,21 +0,0 @@
-<!--- 40. Bare Metal -->
-## Bare Metal
-
-To run on bare metal first, follow the [instruction described here](/models/recommendation/pytorch/dlrm/training/bfloat16/README.md#1-install-anaconda-30) until section 4.
-
-After installing the prerequisites, download and untar the DLRM bf16 training model package, and run the quick start script:
-```
-wget <package url>
-tar -xvf <package name>
-```
-Set environment variables
-for the path to your `DATA_PATH`then run a 
-[quickstart script](#quick-start-scripts).
-
-```
-DATA_PATH=<path to the dataset>
-OUTPUT_DIR=<directory where log files will be written>
-
-cd <package directory>
-bash quickstart/<script name>.sh
-```
diff --git a/quickstart/recommendation/pytorch/dlrm/training/cpu/bfloat16/.docs/datasets.md b/quickstart/recommendation/pytorch/dlrm/training/cpu/bfloat16/.docs/datasets.md
deleted file mode 100644
index 7c231daae..000000000
--- a/quickstart/recommendation/pytorch/dlrm/training/cpu/bfloat16/.docs/datasets.md
+++ /dev/null
@@ -1,6 +0,0 @@
-<!--- 20. Datasets -->
-## Datasets
-
-Prepare your dataset according to the [instruction described here](/models/recommendation/pytorch/dlrm/training/bfloat16/README.md#4-prepare-dataset)
-
-Set the `DATA_PATH` to point to "<dir/to/save/dlrm_data>" directory when running DLRM.
diff --git a/quickstart/recommendation/pytorch/dlrm/training/cpu/bfloat16/.docs/description.md b/quickstart/recommendation/pytorch/dlrm/training/cpu/bfloat16/.docs/description.md
deleted file mode 100644
index 11b988c38..000000000
--- a/quickstart/recommendation/pytorch/dlrm/training/cpu/bfloat16/.docs/description.md
+++ /dev/null
@@ -1,5 +0,0 @@
-<!-- 10. Description -->
-## Description
-
-This document has instructions for running DLRM bf16 training using
-Intel-optimized PyTorch.
diff --git a/quickstart/recommendation/pytorch/dlrm/training/cpu/bfloat16/.docs/docker.md b/quickstart/recommendation/pytorch/dlrm/training/cpu/bfloat16/.docs/docker.md
deleted file mode 100644
index 9789228b5..000000000
--- a/quickstart/recommendation/pytorch/dlrm/training/cpu/bfloat16/.docs/docker.md
+++ /dev/null
@@ -1,61 +0,0 @@
-<!--- 50. Docker -->
-## Docker
-
-Use the base [PyTorch 1.9 container](https://hub.docker.com/r/intel/intel-optimized-pytorch/) 
-`intel/intel-optimized-pytorch:1.9.0` to run DLRM bfloat16 training.
-To run the quickstart scripts using the base PyTorch 1.9 container, install the model dependencies, and provide volume mounts for the DLRM bf16 training package and the dataset:
-
-```
-DATA_PATH=<path to the dataset>
-OUTPUT_DIR=<directory where log files will be written>
-
-docker run \
-  --env DATA_PATH=${DATA_PATH} \
-  --env OUTPUT_DIR=${OUTPUT_DIR} \
-  --env http_proxy=${http_proxy} \
-  --env https_proxy=${https_proxy} \
-  --volume ${DATA_PATH}:${DATA_PATH} \
-  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
-  --volume <path to the model package directory>:/dlrm-bf16-training \
-  --privileged --init -it \
-  intel/intel-optimized-pytorch:1.9.0 /bin/bash 
-```
-
-Install the model dependencies inside the base PyTorch 1.9 container:
-```
-# MLPerf logging
-git clone https://github.com/mlperf/logging.git mlperf-logging
-pip install -e mlperf-logging
-
-# MPICH
-apt-get update && apt install mpich
-
-# Model dependencies
-pip install sklearn onnx tqdm
-
-# torch-ccl:
-git clone --single-branch --branch ccl_torch1.9 https://github.com/intel/torch-ccl.git && cd torch-ccl 
-git submodule sync 
-git submodule update --init --recursive
-python setup.py install
-
-# tcmalloc
-apt-get update && apt-get install wget
-wget https://github.com/gperftools/gperftools/releases/download/gperftools-2.7.90/gperftools-2.7.90.tar.gz && \
-tar -xzf gperftools-2.7.90.tar.gz && \
-cd gperftools-2.7.90 && \
-mkdir -p /workspace/lib/ && \
-./configure --prefix=/workspace/lib/tcmalloc/ && \
-make && \
-make install
-```
-
-Run DLRM bfloat16 training script:
-```
-cd /dlrm-bf16-training
-bash quickstart/<script name>.sh
-```
-
-If you are new to docker and are running into issues with the container,
-see [this document](https://github.com/IntelAI/models/tree/master/docs/general/docker.md)
-for troubleshooting tips.
diff --git a/quickstart/recommendation/pytorch/dlrm/training/cpu/bfloat16/.docs/license.md b/quickstart/recommendation/pytorch/dlrm/training/cpu/bfloat16/.docs/license.md
deleted file mode 100644
index a089c69b8..000000000
--- a/quickstart/recommendation/pytorch/dlrm/training/cpu/bfloat16/.docs/license.md
+++ /dev/null
@@ -1,4 +0,0 @@
-<!--- 70. License -->
-## License
-
-[LICENSE](/LICENSE)
diff --git a/quickstart/recommendation/pytorch/dlrm/training/cpu/bfloat16/.docs/quickstart.md b/quickstart/recommendation/pytorch/dlrm/training/cpu/bfloat16/.docs/quickstart.md
deleted file mode 100644
index df5b4acaa..000000000
--- a/quickstart/recommendation/pytorch/dlrm/training/cpu/bfloat16/.docs/quickstart.md
+++ /dev/null
@@ -1,10 +0,0 @@
-<!--- 30. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| [`train_single_node.sh`](train_single_node.sh) | Run 32K global BS with 4 ranks on 1 node (1 CPX6-4s Node). | 
-
-These quickstart scripts can be run in different environments:
-* [Bare Metal](#bare-metal)
-* [Docker](#docker)
diff --git a/quickstart/recommendation/pytorch/dlrm/training/cpu/bfloat16/.docs/title.md b/quickstart/recommendation/pytorch/dlrm/training/cpu/bfloat16/.docs/title.md
deleted file mode 100644
index d2c2252a7..000000000
--- a/quickstart/recommendation/pytorch/dlrm/training/cpu/bfloat16/.docs/title.md
+++ /dev/null
@@ -1,2 +0,0 @@
-<!--- 0. Title -->
-# DLRM bf16 training
diff --git a/quickstart/recommendation/pytorch/dlrm/training/cpu/bfloat16/README.md b/quickstart/recommendation/pytorch/dlrm/training/cpu/bfloat16/README.md
deleted file mode 100644
index 80e4b7eee..000000000
--- a/quickstart/recommendation/pytorch/dlrm/training/cpu/bfloat16/README.md
+++ /dev/null
@@ -1,115 +0,0 @@
-<!--- 0. Title -->
-# DLRM bf16 training
-
-<!-- 10. Description -->
-## Description
-
-This document has instructions for running DLRM bf16 training using
-Intel-optimized PyTorch.
-
-<!--- 20. Datasets -->
-## Datasets
-
-Prepare your dataset according to the [instruction described here](/models/recommendation/pytorch/dlrm/training/bfloat16/README.md#4-prepare-dataset)
-
-Set the `DATA_PATH` to point to "<dir/to/save/dlrm_data>" directory when running DLRM.
-
-<!--- 30. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| [`train_single_node.sh`](train_single_node.sh) | Run 32K global BS with 4 ranks on 1 node (1 CPX6-4s Node). | 
-
-These quickstart scripts can be run in different environments:
-* [Bare Metal](#bare-metal)
-* [Docker](#docker)
-
-<!--- 40. Bare Metal -->
-## Bare Metal
-
-To run on bare metal first, follow the [instruction described here](/models/recommendation/pytorch/dlrm/training/bfloat16/README.md#1-install-anaconda-30) until section 4.
-
-After installing the prerequisites, download and untar the DLRM bf16 training model package, and run the quick start script:
-```
-wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v2_7_0/dlrm-bfloat16-training.tar.gz
-tar -xvf dlrm-bfloat16-training.tar.gz
-```
-Set environment variables
-for the path to your `DATA_PATH`then run a 
-[quickstart script](#quick-start-scripts).
-
-```
-DATA_PATH=<path to the dataset>
-OUTPUT_DIR=<directory where log files will be written>
-
-cd dlrm-bfloat16-training
-bash quickstart/<script name>.sh
-```
-
-<!--- 50. Docker -->
-## Docker
-
-Use the base [PyTorch 1.9 container](https://hub.docker.com/r/intel/intel-optimized-pytorch/) 
-`intel/intel-optimized-pytorch:1.9.0` to run DLRM bfloat16 training.
-To run the quickstart scripts using the base PyTorch 1.9 container, install the model dependencies, and provide volume mounts for the DLRM bf16 training package and the dataset:
-
-```
-DATA_PATH=<path to the dataset>
-OUTPUT_DIR=<directory where log files will be written>
-
-docker run \
-  --env DATA_PATH=${DATA_PATH} \
-  --env OUTPUT_DIR=${OUTPUT_DIR} \
-  --env http_proxy=${http_proxy} \
-  --env https_proxy=${https_proxy} \
-  --volume ${DATA_PATH}:${DATA_PATH} \
-  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
-  --volume <path to the model package directory>:/dlrm-bf16-training \
-  --privileged --init -it \
-  intel/intel-optimized-pytorch:1.9.0 /bin/bash 
-```
-
-Install the model dependencies inside the base PyTorch 1.9 container:
-```
-# MLPerf logging
-git clone https://github.com/mlperf/logging.git mlperf-logging
-pip install -e mlperf-logging
-
-# MPICH
-apt-get update && apt install mpich
-
-# Model dependencies
-pip install sklearn onnx tqdm
-
-# torch-ccl:
-git clone --single-branch --branch ccl_torch1.9 https://github.com/intel/torch-ccl.git && cd torch-ccl
-git submodule sync 
-git submodule update --init --recursive
-python setup.py install
-
-# tcmalloc
-apt-get update && apt-get install wget
-wget https://github.com/gperftools/gperftools/releases/download/gperftools-2.7.90/gperftools-2.7.90.tar.gz && \
-tar -xzf gperftools-2.7.90.tar.gz && \
-cd gperftools-2.7.90 && \
-mkdir -p /workspace/lib/ && \
-./configure --prefix=/workspace/lib/tcmalloc/ && \
-make && \
-make install
-```
-
-Run DLRM bfloat16 training script:
-```
-cd /dlrm-bf16-training
-bash quickstart/<script name>.sh
-```
-
-If you are new to docker and are running into issues with the container,
-see [this document](https://github.com/IntelAI/models/tree/master/docs/general/docker.md)
-for troubleshooting tips.
-
-<!--- 70. License -->
-## License
-
-[LICENSE](/LICENSE)
diff --git a/quickstart/recommendation/pytorch/dlrm/training/cpu/bfloat16/train_single_node.sh b/quickstart/recommendation/pytorch/dlrm/training/cpu/bfloat16/train_single_node.sh
deleted file mode 100644
index a66902cd4..000000000
--- a/quickstart/recommendation/pytorch/dlrm/training/cpu/bfloat16/train_single_node.sh
+++ /dev/null
@@ -1,42 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2020 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-export MODEL_DIR=${MODEL_DIR-$PWD}
-
-if [ -z "${OUTPUT_DIR}" ]; then
-  echo "The required environment variable OUTPUT_DIR has not been set"
-  exit 1
-fi
-
-# Create the output directory in case it doesn't already exist
-mkdir -p ${OUTPUT_DIR}
-
-if [ -z "${DATA_PATH}" ]; then
-  echo "The required environment variable DATA_PATH has not been set"
-  exit 1
-fi
-
-if [ ! -d "${DATA_PATH}" ]; then
-  echo "The DATA_PATH'${DATA_PATH}' does not exist"
-  exit 1
-fi
-
-# TODO: Fill in the launch_benchmark.py command with the recommended args
-
-${MODEL_DIR}/models/recommendation/pytorch/dlrm/training/bfloat16/bench/cleanup.sh
-${MODEL_DIR}/models/recommendation/pytorch/dlrm/training/bfloat16/bench/dlrm_mlperf_4s_1n_cpx.sh
-
diff --git a/quickstart/recommendation/pytorch/dlrm/training/cpu/distribute_training.sh b/quickstart/recommendation/pytorch/dlrm/training/cpu/distribute_training.sh
index 50f29e5c6..06c714964 100644
--- a/quickstart/recommendation/pytorch/dlrm/training/cpu/distribute_training.sh
+++ b/quickstart/recommendation/pytorch/dlrm/training/cpu/distribute_training.sh
@@ -103,18 +103,35 @@ LOCAL_BATCH_SIZE=$((BATCHSIZE / NODE))
 LOCAL_BATCH_SIZE=$((LOCAL_BATCH_SIZE / 2))
 
 LOG_0="${LOG}/socket.log"
-python -m intel_extension_for_pytorch.cpu.launch --enable_tcmalloc --logical_core_for_ccl --ccl_worker_count $NUM_CCL_WORKER --distributed --hostfile $HOSTFILE --nnodes $NODE \
-$MODEL_SCRIPT \
-  --raw-data-file=${DATASET_DIR}/day --processed-data-file=${DATASET_DIR}/terabyte_processed.npz \
-  --data-set=terabyte \
-  --memory-map --mlperf-bin-loader --mlperf-bin-shuffle --round-targets=True --learning-rate=18.0 \
-  --arch-mlp-bot=13-512-256-128 --arch-mlp-top=1024-1024-512-256-1 \
-  --arch-sparse-feature-size=128 --max-ind-range=40000000 \
-  --numpy-rand-seed=${seed_num} --print-auc --mlperf-auc-threshold=0.8025 \
-  --lr-num-warmup-steps=8000   --lr-decay-start-step=70000 --lr-num-decay-steps=30000\
-  --local-batch-size=${LOCAL_BATCH_SIZE} --print-freq=100 --print-time --ipex-interaction \
-  --test-mini-batch-size=65536 --ipex-merged-emb --should-test --test-freq 6400\
-  $ARGS |tee $LOG_0
+TORCH_INDUCTOR=${TORCH_INDUCTOR:-"0"}
+if [[ "0" == ${TORCH_INDUCTOR} ]];then
+    python -m intel_extension_for_pytorch.cpu.launch --enable_tcmalloc --logical_core_for_ccl --ccl_worker_count $NUM_CCL_WORKER --distributed --hostfile $HOSTFILE --nnodes $NODE \
+    $MODEL_SCRIPT \
+      --raw-data-file=${DATASET_DIR}/day --processed-data-file=${DATASET_DIR}/terabyte_processed.npz \
+      --data-set=terabyte \
+      --memory-map --mlperf-bin-loader --mlperf-bin-shuffle --round-targets=True --learning-rate=18.0 \
+      --arch-mlp-bot=13-512-256-128 --arch-mlp-top=1024-1024-512-256-1 \
+      --arch-sparse-feature-size=128 --max-ind-range=40000000 \
+      --numpy-rand-seed=${seed_num} --print-auc --mlperf-auc-threshold=0.8025 \
+      --lr-num-warmup-steps=8000   --lr-decay-start-step=70000 --lr-num-decay-steps=30000\
+      --local-batch-size=${LOCAL_BATCH_SIZE} --print-freq=100 --print-time --ipex-interaction \
+      --test-mini-batch-size=65536 --ipex-merged-emb --should-test --test-freq 6400\
+      $ARGS |tee $LOG_0
+else
+    export TORCHINDUCTOR_FREEZING=1
+    python -m intel_extension_for_pytorch.cpu.launch --enable_tcmalloc --logical_core_for_ccl --ccl_worker_count $NUM_CCL_WORKER --distributed --hostfile $HOSTFILE --nnodes $NODE \
+    $MODEL_SCRIPT \
+      --raw-data-file=${DATASET_DIR}/day --processed-data-file=${DATASET_DIR}/terabyte_processed.npz \
+      --data-set=terabyte \
+      --memory-map --mlperf-bin-loader --mlperf-bin-shuffle --round-targets=True --learning-rate=18.0 \
+      --arch-mlp-bot=13-512-256-128 --arch-mlp-top=1024-1024-512-256-1 \
+      --arch-sparse-feature-size=128 --max-ind-range=40000000 \
+      --numpy-rand-seed=${seed_num} --print-auc --mlperf-auc-threshold=0.8025 \
+      --lr-num-warmup-steps=8000   --lr-decay-start-step=70000 --lr-num-decay-steps=30000\
+      --local-batch-size=${LOCAL_BATCH_SIZE} --print-freq=100 --print-time \
+      --test-mini-batch-size=65536 --should-test --test-freq 6400 --inductor\ 
+      $ARGS |tee $LOG_0
+
 wait
 
 throughput=$(grep 'Throughput:' ${LOG}/socket* |sed -e 's/.*Throughput//;s/[^0-9.]//g' |awk '
diff --git a/quickstart/recommendation/pytorch/dlrm/training/cpu/training.sh b/quickstart/recommendation/pytorch/dlrm/training/cpu/training.sh
index 64fba8392..7e93b81d6 100755
--- a/quickstart/recommendation/pytorch/dlrm/training/cpu/training.sh
+++ b/quickstart/recommendation/pytorch/dlrm/training/cpu/training.sh
@@ -41,6 +41,11 @@ if [ ! -d "${DATASET_DIR}" ]; then
   exit 1
 fi
 
+if [-z "${PRECISION}" ]; then
+  echo "Please set PRECISION: fp32, bf16, bf32"
+  exit 1
+fi
+
 export TEST_FULLY_CONVERGENCE=0
 if [ -z "${NUM_BATCH}" ]; then
   echo "The required environment variable NUM_BATCH has not been set"
@@ -73,20 +78,42 @@ fi
 
 CORES=`lscpu | grep Core | awk '{print $4}'`
 # BATCHSIZE=$((128*CORES))
-BATCHSIZE=32768 # a converged BS and have better performance on SPR
-export OMP_NUM_THREADS=$CORES
+# a converged BS and have better performance on SPR
+BATCHSIZE=${BATCHSIZE:-32768}
+
+CORES_PER_SOCKET=`lscpu | grep "Core(s) per socket" | awk '{print $4}'`
+SOCKETS=`lscpu | grep "Socket(s)" | awk '{print $2}'`
+NUMA_NODES=`lscpu | grep "NUMA node(s)" | awk '{print $3}'`
+NUMA_NODES_PER_SOCKETS=`expr $NUMA_NODES / $SOCKETS`
+CORES_PER_NUMA_NODE=`expr $CORES_PER_SOCKET / $NUMA_NODES_PER_SOCKETS`
+
+export OMP_NUM_THREADS=$CORES_PER_NUMA_NODE
 
 LOG_0="${LOG}/socket_0"
-python -m intel_extension_for_pytorch.cpu.launch --node_id=0 --enable_tcmalloc $MODEL_SCRIPT \
-  --raw-data-file=${DATASET_DIR}/day --processed-data-file=${DATASET_DIR}/terabyte_processed.npz \
-  --data-set=terabyte \
-  --memory-map --mlperf-bin-loader --round-targets=True --learning-rate=1.0 \
-  --arch-mlp-bot=13-512-256-128 --arch-mlp-top=1024-1024-512-256-1 \
-  --arch-sparse-feature-size=128 --max-ind-range=40000000 \
-  --numpy-rand-seed=727 --print-auc --mlperf-auc-threshold=0.8025 \
-  --mini-batch-size=${BATCHSIZE} --print-freq=100 --print-time --ipex-interaction \
-  --test-mini-batch-size=16384 --ipex-merged-emb \
-  $ARGS |tee $LOG_0
+TORCH_INDUCTOR=${TORCH_INDUCTOR:-"0"}
+if [[ "0" == ${TORCH_INDUCTOR} ]];then
+    python -m intel_extension_for_pytorch.cpu.launch --node_id=0 --enable_tcmalloc $MODEL_SCRIPT \
+      --raw-data-file=${DATASET_DIR}/day --processed-data-file=${DATASET_DIR}/terabyte_processed.npz \
+      --data-set=terabyte \
+      --memory-map --mlperf-bin-loader --round-targets=True --learning-rate=1.0 \
+      --arch-mlp-bot=13-512-256-128 --arch-mlp-top=1024-1024-512-256-1 \
+      --arch-sparse-feature-size=128 --max-ind-range=40000000 \
+      --numpy-rand-seed=727 --print-auc --mlperf-auc-threshold=0.8025 \
+      --mini-batch-size=${BATCHSIZE} --print-freq=100 --print-time --ipex-interaction \
+      --test-mini-batch-size=16384 --ipex-merged-emb \
+      $ARGS |tee $LOG_0
+else
+    python -m intel_extension_for_pytorch.cpu.launch --node_id=0 --enable_tcmalloc $MODEL_SCRIPT \
+      --raw-data-file=${DATASET_DIR}/day --processed-data-file=${DATASET_DIR}/terabyte_processed.npz \
+      --data-set=terabyte \
+      --memory-map --mlperf-bin-loader --round-targets=True --learning-rate=1.0 \
+      --arch-mlp-bot=13-512-256-128 --arch-mlp-top=1024-1024-512-256-1 \
+      --arch-sparse-feature-size=128 --max-ind-range=40000000 \
+      --numpy-rand-seed=727 --print-auc --mlperf-auc-threshold=0.8025 \
+      --mini-batch-size=${BATCHSIZE} --print-freq=100 --print-time \
+      --test-mini-batch-size=16384 --inductor \
+      $ARGS |tee $LOG_0
+fi
 wait
 
 throughput=$(grep 'Throughput:' ${LOG}/socket* |sed -e 's/.*Throughput//;s/[^0-9.]//g' |awk '
diff --git a/quickstart/recommendation/pytorch/torchrec_dlrm/inference/cpu/README.md b/quickstart/recommendation/pytorch/torchrec_dlrm/inference/cpu/README.md
index 7589059ea..33ee43dbf 100644
--- a/quickstart/recommendation/pytorch/torchrec_dlrm/inference/cpu/README.md
+++ b/quickstart/recommendation/pytorch/torchrec_dlrm/inference/cpu/README.md
@@ -39,6 +39,11 @@ Follow [link](/docs/general/pytorch/BareMetalSetup.md) to install Miniconda and
   export DNNL_MAX_CPU_ISA=AVX512_CORE_AMX
   ```
 
+* [optional] Compile model with PyTorch Inductor backend
+  ```shell
+  export TORCH_INDUCTOR=1
+  ```
+
 ## Datasets
 The dataset can be downloaded and preprocessed by following https://github.com/mlcommons/training/tree/master/recommendation_v2/torchrec_dlrm#create-the-synthetic-multi-hot-dataset.
 We also provided a preprocessed scripts based on the instruction above. `preprocess_raw_dataset.sh`.
diff --git a/quickstart/recommendation/pytorch/torchrec_dlrm/inference/cpu/inference_performance.sh b/quickstart/recommendation/pytorch/torchrec_dlrm/inference/cpu/inference_performance.sh
index 7ec3df9f0..351e41cbe 100644
--- a/quickstart/recommendation/pytorch/torchrec_dlrm/inference/cpu/inference_performance.sh
+++ b/quickstart/recommendation/pytorch/torchrec_dlrm/inference/cpu/inference_performance.sh
@@ -36,22 +36,37 @@ LOG=${OUTPUT_DIR}/dlrm_inference_performance_log/${PRECISION}
 rm -rf ${LOG}
 mkdir -p ${LOG}
 
+export EXTRA_ARGS=""
+if [ -z "${DATASET_DIR}" ]; then
+  echo "DATASET_DIR are not set, will use dummy generated dataset"
+  EXTRA_ARGS="$EXTRA_ARGS --multi_hot_distribution_type uniform "
+  EXTRA_ARGS="$EXTRA_ARGS --multi_hot_sizes 3,2,1,2,6,1,1,1,1,7,3,8,1,6,9,5,1,1,1,12,100,27,10,3,1,1 "
+else
+  EXTRA_ARGS="$EXTRA_ARGS --synthetic_multi_hot_criteo_path $DATASET_DIR "
+fi
+
+TORCH_INDUCTOR=${TORCH_INDUCTOR:-"0"}
+
 ARGS=""
 if [[ $PRECISION == "bf16" ]]; then
-    ARGS="$ARGS --dtype bf16 --ipex-merged-emb-cat"
+    ARGS="$ARGS --dtype bf16"
     echo "running bf16 path"
 elif [[ $PRECISION == "fp32" ]]; then
     echo "running fp32 path"
-    ARGS="$ARGS --dtype fp32 --ipex-merged-emb-cat"
+    ARGS="$ARGS --dtype fp32"
 elif [[ $PRECISION == "bf32" ]]; then
     echo "running bf32 path"
-    ARGS="$ARGS --dtype bf32 --ipex-merged-emb-cat"
+    ARGS="$ARGS --dtype bf32"
 elif [[ $PRECISION == "fp16" ]]; then
     echo "running fp16 path"
-    ARGS="$ARGS --dtype fp16 --ipex-merged-emb-cat"
+    ARGS="$ARGS --dtype fp16"
 elif [[ $PRECISION == "int8" ]]; then
+    if [[ "0" == ${TORCH_INDUCTOR} ]];then
+      echo "prepare int8 weight"
+      bash ${MODEL_DIR}/quickstart/recommendation/pytorch/torchrec_dlrm/inference/cpu/prepare_int8.sh
+    fi
     echo "running int8 path"
-    ARGS="$ARGS --dtype int8 --ipex-merged-emb-cat --int8-configure-dir ${INT8_CONFIG}"
+    ARGS="$ARGS --dtype int8 --int8-configure-dir ${INT8_CONFIG}"
 else
     echo "The specified PRECISION '${PRECISION}' is unsupported."
     echo "Supported PRECISIONs are: fp32, fp16, bf16, bf32, int8"
@@ -68,15 +83,12 @@ fi
 
 LOG_0="${LOG}/throughput.log"
 
-if [ -z "${BATCH_SIZE}" ]; then
-  BATCH_SIZE=512
-fi
 
-export launcher_arg="-m intel_extension_for_pytorch.cpu.launch --throughput_mode --enable_jemalloc"
+export launcher_cmd="-m intel_extension_for_pytorch.cpu.launch --throughput-mode --enable_jemalloc"
 if [[ $PLOTMEM == "true" ]]; then
 pip install memory_profiler matplotlib
 export mrun_cmd="mprof run --python -o ${MEMLOG}"
-unset launcher_arg
+unset launcher_cmd
 fi
 
 if [[ $ENABLE_TORCH_PROFILE == "true" ]]; then
@@ -90,27 +102,30 @@ NUMA_NODES_PER_SOCKETS=`expr $NUMA_NODES / $SOCKETS`
 CORES_PER_NUMA_NODE=`expr $CORES_PER_SOCKET / $NUMA_NODES_PER_SOCKETS`
 
 export OMP_NUM_THREADS=1
+COMMON_ARGS=" --embedding_dim 128 \
+              --dense_arch_layer_sizes 512,256,128 \
+              --over_arch_layer_sizes 1024,1024,512,256,1 \
+              --num_embeddings_per_feature 40000000,39060,17295,7424,20265,3,7122,1543,63,40000000,3067956,405282,10,2209,11938,155,4,976,14,40000000,40000000,40000000,590152,12973,108,36 \
+              --epochs 1 \
+              --pin_memory \
+              --mmap_mode \
+              --batch_size $BATCH_SIZE \
+              --interaction_type=dcn \
+              --dcn_num_layers=3 \
+              --dcn_low_rank_dim=512 \
+              --limit_val_batches 300 \
+              --log-freq 10 \
+              --inference-only \
+              --benchmark \
+              --share-weight-instance=$CORES_PER_NUMA_NODE \
+              $EXTRA_ARGS $ARGS "
+
 
-$mrun_cmd python $launcher_arg $MODEL_SCRIPT \
-    --embedding_dim 128 \
-    --dense_arch_layer_sizes 512,256,128 \
-    --over_arch_layer_sizes 1024,1024,512,256,1 \
-    --num_embeddings_per_feature 40000000,39060,17295,7424,20265,3,7122,1543,63,40000000,3067956,405282,10,2209,11938,155,4,976,14,40000000,40000000,40000000,590152,12973,108,36 \
-    --epochs 1 \
-    --pin_memory \
-    --mmap_mode \
-    --batch_size $BATCH_SIZE \
-    --interaction_type=dcn \
-    --dcn_num_layers=3 \
-    --dcn_low_rank_dim=512 \
-    --limit_val_batches 1000 \
-    --ipex-optimize \
-    --log-freq 10 \
-    --jit \
-    --inference-only \
-    --benchmark \
-    --share-weight-instance=$CORES_PER_NUMA_NODE \
-    $ARGS 2>&1 | tee $LOG_0
+if [[ "0" == ${TORCH_INDUCTOR} ]];then
+  $mrun_cmd python $launcher_cmd $MODEL_SCRIPT $COMMON_ARGS --ipex-optimize --jit --ipex-merged-emb-cat 2>&1 | tee $LOG_0
+else
+  $mrun_cmd python $launcher_cmd $MODEL_SCRIPT $COMMON_ARGS --inductor 2>&1 | tee $LOG_0
+fi
 wait
 
 if [[ $PLOTMEM == "true" ]]; then
diff --git a/quickstart/recommendation/pytorch/torchrec_dlrm/inference/cpu/test_accuracy.sh b/quickstart/recommendation/pytorch/torchrec_dlrm/inference/cpu/test_accuracy.sh
index 7d8f68275..7fe74483b 100644
--- a/quickstart/recommendation/pytorch/torchrec_dlrm/inference/cpu/test_accuracy.sh
+++ b/quickstart/recommendation/pytorch/torchrec_dlrm/inference/cpu/test_accuracy.sh
@@ -45,23 +45,23 @@ mkdir -p ${OUTPUT_DIR}
 LOG=${OUTPUT_DIR}/dlrm_inference_accuarcy_log/${PRECISION}
 rm -rf ${LOG}
 mkdir -p ${LOG}
-
+TORCH_INDUCTOR=${TORCH_INDUCTOR:-"0"}
 ARGS=""
 if [[ $PRECISION == "bf16" ]]; then
-    ARGS="$ARGS --dtype bf16 --ipex-merged-emb-cat"
+    ARGS="$ARGS --dtype bf16 "
     echo "running bf16 path"
 elif [[ $PRECISION == "fp32" ]]; then
     echo "running fp32 path"
-    ARGS="$ARGS --dtype fp32 --ipex-merged-emb-cat"
+    ARGS="$ARGS --dtype fp32"
 elif [[ $PRECISION == "bf32" ]]; then
     echo "running bf32 path"
-    ARGS="$ARGS --dtype bf32 --ipex-merged-emb-cat"
+    ARGS="$ARGS --dtype bf32"
 elif [[ $PRECISION == "fp16" ]]; then
     echo "running fp16 path"
-    ARGS="$ARGS --dtype fp16 --ipex-merged-emb-cat"
+    ARGS="$ARGS --dtype fp16"
 elif [[ $PRECISION == "int8" ]]; then
     echo "running int8 path"
-    ARGS="$ARGS --dtype int8 --ipex-merged-emb-cat --int8-configure-dir ${INT8_CONFIG}"
+    ARGS="$ARGS --dtype int8 --int8-configure-dir ${INT8_CONFIG}"
 else
     echo "The specified PRECISION '${PRECISION}' is unsupported."
     echo "Supported PRECISIONs are: fp32, bf32, fp16, bf16, int8"
@@ -76,25 +76,29 @@ pip install memory_profiler
 export mrun_cmd="mprof run --python -o ${MEMLOG}"
 fi
 
+COMMON_ARGS=" --embedding_dim 128 \
+              --dense_arch_layer_sizes 512,256,128 \
+              --over_arch_layer_sizes 1024,1024,512,256,1 \
+              --num_embeddings_per_feature 40000000,39060,17295,7424,20265,3,7122,1543,63,40000000,3067956,405282,10,2209,11938,155,4,976,14,40000000,40000000,40000000,590152,12973,108,36 \
+              --epochs 1 \
+              --pin_memory \
+              --mmap_mode \
+              --batch_size $BATCH_SIZE \
+              --interaction_type=dcn \
+              --dcn_num_layers=3 \
+              --dcn_low_rank_dim=512 \
+              --limit_val_batches 100 \
+              --log-freq 10 \
+              --inference-only \
+              $EXTRA_ARGS $ARGS "
+
 # Do not need to use launcher to bind memory for accuracy test
-$mrun_cmd python $MODEL_SCRIPT \
-    --snapshot-dir $WEIGHT_DIR \
-    --embedding_dim 128 \
-    --dense_arch_layer_sizes 512,256,128 \
-    --over_arch_layer_sizes 1024,1024,512,256,1 \
-    --num_embeddings_per_feature 40000000,39060,17295,7424,20265,3,7122,1543,63,40000000,3067956,405282,10,2209,11938,155,4,976,14,40000000,40000000,40000000,590152,12973,108,36 \
-    --epochs 1 \
-    --pin_memory \
-    --mmap_mode \
-    --batch_size $BATCH_SIZE \
-    --interaction_type=dcn \
-    --dcn_num_layers=3 \
-    --dcn_low_rank_dim=512 \
-    --ipex-optimize \
-    --inference-only \
-    --synthetic_multi_hot_criteo_path $DATASET_DIR \
-    --test_auroc \
-    $ARGS 2>&1 | tee $LOG_0
+if [[ "0" == ${TORCH_INDUCTOR} ]];then
+  $mrun_cmd python $launcher_cmd $MODEL_SCRIPT $COMMON_ARGS --ipex-optimize --jit --ipex-merged-emb-cat 2>&1 | tee $LOG_0
+else
+  $mrun_cmd python $launcher_cmd $MODEL_SCRIPT $COMMON_ARGS --inductor 2>&1 | tee $LOG_0
+fi
+wait
 
 if [[ $PLOTMEM == "true" ]]; then
 mprof plot ${MEMLOG} -o ${MEMPIC}
diff --git a/quickstart/recommendation/pytorch/torchrec_dlrm/training/cpu/README.md b/quickstart/recommendation/pytorch/torchrec_dlrm/training/cpu/README.md
index c5816c492..0851f06e7 100644
--- a/quickstart/recommendation/pytorch/torchrec_dlrm/training/cpu/README.md
+++ b/quickstart/recommendation/pytorch/torchrec_dlrm/training/cpu/README.md
@@ -39,6 +39,11 @@ Follow [link](/docs/general/pytorch/BareMetalSetup.md) to install Miniconda and
   export DNNL_MAX_CPU_ISA=AVX512_CORE_AMX
   ```
 
+* [optional] Compile model with PyTorch Inductor backend
+  ```shell
+  export TORCH_INDUCTOR=1
+  ```
+
 ## Datasets
 
 Use random dataset.
@@ -62,8 +67,33 @@ export PRECISION=<specify the precision to run>
 
 # Run a quickstart script (for example, bare metal performance)
 cd ${MODEL_DIR}/quickstart/recommendation/pytorch/torchrec_dlrm/training/cpu
+# we recommend to use real dataset to evaluate performance data since the performance will highly related with input data distribution
+# do not set ONE_HOT_DATASET_DIR will use dummy data
+export ONE_HOT_DATASET_DIR=<your one hot dataset dir>
+# Normal Training
 bash training_performance.sh
+# Distributed training
+DIST=1 bash training_performance.sh
+# Convergence test for Normal Training
+CONVERGENCE=1 bash training_performance.sh
+# Convergence test for Distributed Training
+DIST=1 CONVERGENCE=1 bash training_performance.sh
+```
+
+## Distributed training advance setting
+The default behaviour for `DIST=1 bash training_performance.sh`
+(1) Will run 2 instance on numa node `0, 1` within non-SNC mode.
+(2) Will run n instance on socket 0 on sub numa node `0, 1, ..., n` with SNC-N mode. For example, will run 3 instance on socket 0, sub numa node `0, 1, 2` with SNC3.
+(3) You can further setting it by setting `NODE_LIST` and `NODE` and `PROC_PER_NODE`. For example, if you wish to run 4 instance on 2 nodes, and run on numa nodes `2, 3` for each nodes
+You can run
+```bash
+export NODE_LIST=2,3
+export NODE=2
+export PROC_PER_NODE=2
+DIST=1 bash training_performance.sh
 ```
+Please remember to config your IP address in `hostfile1`, 1 address per line.
+
 
 <!--- 80. License -->
 ## License
diff --git a/quickstart/recommendation/pytorch/torchrec_dlrm/training/cpu/distributed_setup.sh b/quickstart/recommendation/pytorch/torchrec_dlrm/training/cpu/distributed_setup.sh
new file mode 100644
index 000000000..23b49cf0f
--- /dev/null
+++ b/quickstart/recommendation/pytorch/torchrec_dlrm/training/cpu/distributed_setup.sh
@@ -0,0 +1,64 @@
+# Copyright (c) 2023 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+oneccl_bindings_for_pytorch_path=$(python -c "import torch; import oneccl_bindings_for_pytorch; import os;  print(os.path.abspath(os.path.dirname(oneccl_bindings_for_pytorch.__file__)))")
+source $oneccl_bindings_for_pytorch_path/env/setvars.sh
+export NODE=${NODE:-1}
+export NUM_CCL_WORKER=${NUM_CCL_WORKER:-1}
+export NUM_CCL_WORKER=${NUM_CCL_WORKER:-1}
+export HOSTFILE=${HOSTFILE:-hostfile1}
+export TORCH_CPP_LOG_LEVEL=INFO
+export TORCH_DISTRIBUTED_DEBUG=INFO
+export CCL_LOG_LEVEL=info
+export CCL_ALLREDUCE=rabenseifner
+export LOGICAL_CORE_CCL=${LOGICAL_CORE_CCL:-0}
+CORES_PER_SOCKET=`lscpu | grep "Core(s) per socket" | awk '{print $4}'`
+SOCKETS=`lscpu | grep "Socket(s)" | awk '{print $2}'`
+NUMA_NODES=`lscpu | grep "NUMA node(s)" | awk '{print $3}'`
+NUMA_NODES_PER_SOCKETS=`expr $NUMA_NODES / $SOCKETS`
+CORES_PER_NUMA_NODE=`expr $CORES_PER_SOCKET / $NUMA_NODES_PER_SOCKETS`
+
+NODE_LIST=${NODE_LIST:-"0"}
+if [[ "0" == ${NODE_LIST} ]];then
+    if [ $NUMA_NODES_PER_SOCKETS -eq 1 ]
+    then 
+        NODE_LIST="0,1"
+        PROC_PER_NODE=${PROC_PER_NODE:-"2"}
+    else
+        for i in $(seq 1 `expr $NUMA_NODES_PER_SOCKETS - 1`)
+        do
+            NODE_LIST="${NODE_LIST},$i"
+        done
+        PROC_PER_NODE=${PROC_PER_NODE:-"$NUMA_NODES_PER_SOCKETS"}
+    fi
+fi
+
+export launcher_dist_args="--nodes-list ${NODE_LIST} --nprocs-per-node=$PROC_PER_NODE --ccl_worker_count $NUM_CCL_WORKER --distributed --hostfile $HOSTFILE --nnodes $NODE "
+if [[ "1" == ${NODE_LIST} ]];then
+export launcher_dist_args = "${launcher_dist_args} --logical_core_for_ccl"
+fi
+export W_SIZE=`expr $PROC_PER_NODE \* $NODE`
+export MASTER_ADDR=${MASTER_ADDR:-127.0.0.1}
+export MASTER_PORT=${MASTER_PORT:-29550}
+
+echo "NODE: $NODE"
+echo "NODE_LIST: $NODE_LIST"
+echo "PROC_PER_NODE: $PROC_PER_NODE"
+echo "W_SIZE: $W_SIZE"
+echo "MASTER_ADDR: $MASTER_ADDR"
+echo "MASTER_PORT: $MASTER_PORT"
+echo "NUM_CCL_WORKER: $NUM_CCL_WORKER"
+echo "HOSTFILE: $HOSTFILE"
+echo "launcher_dist_args: $launcher_dist_args"
\ No newline at end of file
diff --git a/quickstart/recommendation/pytorch/torchrec_dlrm/training/cpu/hostfile1 b/quickstart/recommendation/pytorch/torchrec_dlrm/training/cpu/hostfile1
new file mode 100644
index 000000000..e56ea71e3
--- /dev/null
+++ b/quickstart/recommendation/pytorch/torchrec_dlrm/training/cpu/hostfile1
@@ -0,0 +1 @@
+127.0.0.1
\ No newline at end of file
diff --git a/quickstart/recommendation/pytorch/torchrec_dlrm/training/cpu/training_performance.sh b/quickstart/recommendation/pytorch/torchrec_dlrm/training/cpu/training_performance.sh
index fee8de5a9..26f31cc1e 100644
--- a/quickstart/recommendation/pytorch/torchrec_dlrm/training/cpu/training_performance.sh
+++ b/quickstart/recommendation/pytorch/torchrec_dlrm/training/cpu/training_performance.sh
@@ -58,36 +58,63 @@ if [[ $ENABLE_TORCH_PROFILE == "true" ]]; then
   ARGS="$ARGS --profile"
 fi
 
-export launcher_arg="-m intel_extension_for_pytorch.cpu.launch --throughput_mode --enable_jemalloc"
+BATCH_SIZE=${BATCH_SIZE:-5120}
+if [[ $DIST == "1" ]]; then
+  source ${MODEL_DIR}/quickstart/recommendation/pytorch/torchrec_dlrm/training/cpu/distributed_setup.sh
+  launcher_arg=$launcher_dist_args
+  ARGS="$ARGS --ipex-dist-merged-emb-adagrad --distributed-training "
+  BATCH_SIZE=`expr $BATCH_SIZE \* 3`
+else
+  NODE_LIST=${NODE_LIST:-"0"}
+  launcher_arg=" --nodes-list $NODE_LIST "
+  ARGS="$ARGS --ipex-merged-emb-adagrad"
+fi
+
+export launcher_cmd="-m intel_extension_for_pytorch.cpu.launch --enable_tcmalloc ${launcher_arg}"
 if [[ $PLOTMEM == "true" ]]; then
 pip install memory_profiler matplotlib
 export mrun_cmd="mprof run --python -o ${MEMLOG}"
 unset launcher_arg
 fi
 
+if [[ $CONVERGENCE == "1" ]]; then
+  export TOTAL_TRAINING_SAMPLES=4195197692
+  export VAL_FREQ=$((TOTAL_TRAINING_SAMPLES / (BATCH_SIZE * 20)))
+  ARGS="$ARGS --validation_auroc 0.80275 --validation_freq_within_epoch $VAL_FREQ --log-freq 100 --print_progress "
+  if [ -z "${ONE_HOT_DATASET_DIR}" ]; then
+    echo "CONVERGENCE test need set ONE_HOT_DATASET_DIR"
+    exit 1
+  fi
+else
+ ARGS="$ARGS --limit_train_batches 300 --log-freq 10 "
+fi
+
+COMMON_ARGS=" --embedding_dim 128 \
+              --dense_arch_layer_sizes 512,256,128 \
+              --over_arch_layer_sizes 1024,1024,512,256,1 \
+              --num_embeddings_per_feature 40000000,39060,17295,7424,20265,3,7122,1543,63,40000000,3067956,405282,10,2209,11938,155,4,976,14,40000000,40000000,40000000,590152,12973,108,36 \
+              --epochs 1 \
+              --pin_memory \
+              --mmap_mode \
+              --batch_size $BATCH_SIZE \
+              --interaction_type=dcn \
+              --dcn_num_layers=3 \
+              --dcn_low_rank_dim=512 \
+              --adagrad \
+              --learning_rate 0.004 \
+              --multi_hot_distribution_type uniform \
+              --multi_hot_sizes 3,2,1,2,6,1,1,1,1,7,3,8,1,6,9,5,1,1,1,12,100,27,10,3,1,1 \
+              $ARGS "
+
 LOG_0="${LOG}/throughput.log"
-export BATCH_SIZE=16
-$mrun_cmd python $launcher_arg $MODEL_SCRIPT \
-    --embedding_dim 128 \
-    --dense_arch_layer_sizes 512,256,128 \
-    --over_arch_layer_sizes 1024,1024,512,256,1 \
-    --num_embeddings_per_feature 40000000,39060,17295,7424,20265,3,7122,1543,63,40000000,3067956,405282,10,2209,11938,155,4,976,14,40000000,40000000,40000000,590152,12973,108,36 \
-    --epochs 1 \
-    --pin_memory \
-    --mmap_mode \
-    --batch_size $BATCH_SIZE \
-    --interaction_type=dcn \
-    --dcn_num_layers=3 \
-    --dcn_low_rank_dim=512 \
-    --adagrad \
-    --learning_rate 0.005 \
-    --multi_hot_distribution_type uniform \
-    --multi_hot_sizes 3,2,1,2,6,1,1,1,1,7,3,8,1,6,9,5,1,1,1,12,100,27,10,3,1,1 \
-    --limit_train_batches 100 \
-    --ipex-optimize \
-    --log-freq 10 \
-    --benchmark \
-    $ARGS 2>&1 | tee $LOG_0
+
+BATCH_SIZE=${BATCH_SIZE:-32768}
+TORCH_INDUCTOR=${TORCH_INDUCTOR:-"0"}
+if [[ "0" == ${TORCH_INDUCTOR} ]];then
+  $mrun_cmd python $launcher_cmd $MODEL_SCRIPT $COMMON_ARGS --ipex-optimize 2>&1 | tee $LOG_0
+else
+  $mrun_cmd python $launcher_cmd $MODEL_SCRIPT $COMMON_ARGS --inductor 2>&1 | tee $LOG_0
+fi
 wait
 
 if [[ $PLOTMEM == "true" ]]; then
diff --git a/quickstart/recommendation/tensorflow/dien/inference/cpu/README_DEV_CAT.md b/quickstart/recommendation/tensorflow/dien/inference/cpu/README_DEV_CAT.md
new file mode 100644
index 000000000..4fefbff57
--- /dev/null
+++ b/quickstart/recommendation/tensorflow/dien/inference/cpu/README_DEV_CAT.md
@@ -0,0 +1,106 @@
+# TensorFlow DIEN inference
+
+## Description
+This document has instructions for running DIEN inference using Intel-optimized TensorFlow.
+
+## Pull Command
+```
+docker pull intel/recommendation:centos-tf-cpu-dien-inference
+```
+
+<table>
+   <thead>
+      <tr>
+         <th>Script name</th>
+         <th>Description</th>
+      </tr>
+   </thead>
+   <tbody>
+      <tr>
+         <td>inference_realtime.sh</td>
+         <td>Runs multi instance realtime inference using 4 cores per instance for the specified precision (fp32, bfloat16 or bfloat32) with a default`batch_size=16`. Waits for all instances to complete, then prints a summarized throughput value.</td>
+      </tr>
+      <tr>
+         <td>inference_throughput.sh</td>
+         <td>Runs multi instance batch inference using 1 instance per socket for the specified precision (fp32, bfloat16 or bfloat32) with a default `batch_size=65536`. Waits for all instances to complete, then prints a summarized throughput value.</td>
+      </tr>
+      <tr>
+         <td>accuracy.sh</td>
+         <td>Measures the inference accuracy for the specified precision (fp32, bfloat16 or bfloat32) with a default `batch_size=128`.</td>
+      </tr>
+   </tbody>
+</table>
+
+## Datasets
+Use [prepare_data.sh](https://github.com/alibaba/ai-matrix/blob/master/macro_benchmark/DIEN_TF2/prepare_data.sh) to get [a subset of the Amazon book reviews data](http://snap.stanford.edu/data/amazon/productGraph/categoryFiles/) and process it.
+Or download and extract the preprocessed data files directly:
+```
+wget https://zenodo.org/record/3463683/files/data.tar.gz
+wget https://zenodo.org/record/3463683/files/data1.tar.gz
+wget https://zenodo.org/record/3463683/files/data2.tar.gz
+
+tar -jxvf data.tar.gz
+mv data/* .
+tar -jxvf data1.tar.gz
+mv data1/* .
+tar -jxvf data2.tar.gz
+mv data2/* .
+```
+Set the `DATASET_DIR` to point to the directory with the dataset files when running DIEN.
+
+## Docker Run
+(Optional) Export related proxy into docker environment.
+```bash
+export DOCKER_RUN_ENVS="-e ftp_proxy=${ftp_proxy} \
+  -e FTP_PROXY=${FTP_PROXY} -e http_proxy=${http_proxy} \
+  -e HTTP_PROXY=${HTTP_PROXY} -e https_proxy=${https_proxy} \
+  -e HTTPS_PROXY=${HTTPS_PROXY} -e no_proxy=${no_proxy} \
+  -e NO_PROXY=${NO_PROXY} -e socks_proxy=${socks_proxy} \
+  -e SOCKS_PROXY=${SOCKS_PROXY}"
+```
+
+To run DIEN inference, set environment variables to specify the dataset directory, precision and mode to run, and an output directory. 
+```bash
+# Set the required environment vars
+export DATASET_DIR=<path to the dataset>
+export OUTPUT_DIR=<directory where log files will be written>
+export SCRIPT=<specify the script to run>
+export PRECISION=<specify the precision to run>
+
+# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
+export BATCH_SIZE=<customized batch size value>
+
+docker run --rm \
+  --env BATCH_SIZE=${BATCH_SIZE} \
+  --env DATASET_DIR=${DATASET_DIR} \
+  --env OUTPUT_DIR=${OUTPUT_DIR} \
+  --env PRECISION=${PRECISION} \
+  --volume ${DATASET_DIR}:${DATASET_DIR} \
+  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
+  ${DOCKER_RUN_ENVS} \
+  --privileged --init -it \
+  --shm-size 8G \
+  -w /workspace/tf-dien-inference \
+  intel/recommendation:centos-tf-cpu-dien-inference \
+  /bin/bash quickstart/${SCRIPT}
+```
+
+## Documentation and Sources
+#### Get Started​
+[Docker* Repository](https://hub.docker.com/r/intel/recommendation)
+
+[Main GitHub*](https://github.com/IntelAI/models)
+
+[Release Notes](https://github.com/IntelAI/models/releases)
+
+[Get Started Guide](https://github.com/IntelAI/models/blob/master/quickstart/recommendation/tensorflow/dien/inference/cpu/README_DEV_CAT.md)
+
+#### Code Sources
+[Dockerfile](https://github.com/IntelAI/models/tree/master/dockerfiles/tensorflow)
+
+[Report Issue](https://community.intel.com/t5/Intel-Optimized-AI-Frameworks/bd-p/optimized-ai-frameworks)
+
+## License Agreement
+LEGAL NOTICE: By accessing, downloading or using this software and any required dependent software (the “Software Package”), you agree to the terms and conditions of the software license agreements for the Software Package, which may also include notices, disclaimers, or license terms for third party software included with the Software Package. Please refer to the [license](https://github.com/IntelAI/models/tree/master/third_party) file for additional details.
+
+[View All Containers and Solutions 🡢](https://www.intel.com/content/www/us/en/developer/tools/software-catalog/containers.html?s=Newest)
diff --git a/quickstart/recommendation/tensorflow/dien/inference/cpu/accuracy.sh b/quickstart/recommendation/tensorflow/dien/inference/cpu/accuracy.sh
index 414b83ed8..d33f8f7f1 100755
--- a/quickstart/recommendation/tensorflow/dien/inference/cpu/accuracy.sh
+++ b/quickstart/recommendation/tensorflow/dien/inference/cpu/accuracy.sh
@@ -80,7 +80,6 @@ if [ -z "${BATCH_SIZE}"]; then
 fi
 
 source "${MODEL_DIR}/quickstart/common/utils.sh"
-_ht_status_spr
 _command python ${MODEL_DIR}/benchmarks/launch_benchmark.py \
   --model-name=dien \
   --precision ${PRECISION} \
diff --git a/quickstart/recommendation/tensorflow/ncf/inference/cpu/fp32/.docs/aikit.md b/quickstart/recommendation/tensorflow/ncf/inference/cpu/fp32/.docs/aikit.md
deleted file mode 100644
index b06b0fb01..000000000
--- a/quickstart/recommendation/tensorflow/ncf/inference/cpu/fp32/.docs/aikit.md
+++ /dev/null
@@ -1,78 +0,0 @@
-<!--- 50. AI Kit -->
-## Run the model
-
-Setup your environment using the instructions below, depending on if you are
-using [AI Kit](/docs/general/tensorflow/AIKit.md):
-
-<table>
-  <tr>
-    <th>Setup using AI Kit</th>
-    <th>Setup without AI Kit</th>
-  </tr>
-  <tr>
-    <td>
-      <p>AI Kit does not currently support TF 1.15.2 models</p>
-    </td>
-    <td>
-      <p>To run without AI Kit you will need:</p>
-      <ul>
-        <li>Python 3
-        <li><a href="https://pypi.org/project/intel-tensorflow/1.15.2/">intel-tensorflow==1.15.2</a>
-        <li>numactl
-        <li>git
-        <li>google-api-python-client==1.6.7
-        <li>google-cloud-bigquery==0.31.0
-        <li>kaggle==1.3.9
-        <li>numpy==1.16.3
-        <li>oauth2client==4.1.2
-        <li>pandas
-        <li>psutil>=5.6.7
-        <li>py-cpuinfo==3.3.0
-        <li>tar
-        <li>typing
-        <li>wget
-        <li>A clone of the Model Zoo repo<br />
-        <pre>git clone https://github.com/IntelAI/models.git</pre>
-      </ul>
-    </td>
-  </tr>
-</table>
-
-
-Running <model name> also requires a clone of the
-[TensorFlow models repository](https://github.com/tensorflow/models) with
-at the `v1.11` tag. Set the `TF_MODELS_DIR` env var to the path of your clone.
-```
-git clone https://github.com/tensorflow/models.git tf_models
-cd tf_models
-git checkout v1.11
-export TF_MODELS_DIR=$(pwd)
-cd ..
-```
-
-Download and extract the pretrained model and set the path to the
-`PRETRAINED_MODEL` env var.
-```
-wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_5/ncf_fp32_pretrained_model.tar.gz
-tar -xzvf ncf_fp32_pretrained_model.tar.gz
-export PRETRAINED_MODEL=$(pwd)/ncf_trained_movielens_1m
-```
-
-After your environment is setup, set environment variables to the `DATASET_DIR`
-and an `OUTPUT_DIR` where log files will be written. Ensure that you already have
-the `TF_MODELS_DIR` and `PRETRAINED_MODEL` paths set from the previous commands.
-Once the environment variables are all set, you can run a
-[quickstart script](#quick-start-scripts).
-```
-# cd to your model zoo directory
-cd models
-
-export DATASET_DIR=<path to the dataset>
-export OUTPUT_DIR=<path to the directory where log files will be written>
-export TF_MODELS_DIR=<path to the TensorFlow models directory tf_models>
-export PRETRAINED_MODEL=<path to the pretrained model>
-# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
-export BATCH_SIZE=<customized batch size value>
-
-./quickstart/recommendation/tensorflow/ncf/inference/cpu/fp32/<script name>.sh
-```
diff --git a/quickstart/recommendation/tensorflow/ncf/inference/cpu/fp32/.docs/baremetal.md b/quickstart/recommendation/tensorflow/ncf/inference/cpu/fp32/.docs/baremetal.md
deleted file mode 100644
index 921954d9a..000000000
--- a/quickstart/recommendation/tensorflow/ncf/inference/cpu/fp32/.docs/baremetal.md
+++ /dev/null
@@ -1,41 +0,0 @@
-<!--- 50. Bare Metal -->
-## Bare Metal
-
-To run on bare metal, the following prerequisites must be installed in your environment:
-* Python 3.6 or 3.7
-* [intel-tensorflow==1.15.2](https://pypi.org/project/intel-tensorflow/1.15.2/)
-* numactl
-* google-api-python-client==1.6.7
-* google-cloud-bigquery==0.31.0
-* kaggle==1.3.9
-* numpy==1.16.3
-* oauth2client==4.1.2
-* pandas
-* 'psutil>=5.6.7'
-* py-cpuinfo==3.3.0
-* typing
-* TensorFlow models, clone the official `tensorflow/models` repository with  tag `v1.11`:
-```
-git clone https://github.com/tensorflow/models.git tf_models
-cd tf_models
-git checkout v1.11
-```
-
-After installing the prerequisites, download and untar the model package.
-Set environment variables for the path to your `DATASET_DIR` and an
-`OUTPUT_DIR` where log files will be written, then run a 
-[quickstart script](#quick-start-scripts).
-
-```
-DATASET_DIR=<path to the dataset>
-OUTPUT_DIR=<directory where log files will be written>
-TF_MODELS_DIR=<path to the TensorFlow models directory tf_models>
-# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
-export BATCH_SIZE=<customized batch size value>
-
-wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v2_7_0/ncf-fp32-inference.tar.gz
-tar -xzf ncf-fp32-inference.tar.gz
-cd ncf-fp32-inference
-
-./quickstart/<script name>.sh
-```
diff --git a/quickstart/recommendation/tensorflow/ncf/inference/cpu/fp32/.docs/datasets.md b/quickstart/recommendation/tensorflow/ncf/inference/cpu/fp32/.docs/datasets.md
deleted file mode 100644
index 7216e02ee..000000000
--- a/quickstart/recommendation/tensorflow/ncf/inference/cpu/fp32/.docs/datasets.md
+++ /dev/null
@@ -1,9 +0,0 @@
-<!--- 30. Datasets -->
-## Datasets
-
-Download [movielens 1M dataset](https://grouplens.org/datasets/movielens/1m/)
-```
-wget http://files.grouplens.org/datasets/movielens/ml-1m.zip
-unzip ml-1m.zip
-```
-Set the `DATASET_DIR` to point to this directory when running <model name>.
diff --git a/quickstart/recommendation/tensorflow/ncf/inference/cpu/fp32/.docs/description.md b/quickstart/recommendation/tensorflow/ncf/inference/cpu/fp32/.docs/description.md
deleted file mode 100644
index 2fa6d4b66..000000000
--- a/quickstart/recommendation/tensorflow/ncf/inference/cpu/fp32/.docs/description.md
+++ /dev/null
@@ -1,5 +0,0 @@
-<!-- 10. Description -->
-## Description
-
-This document has instructions for running Neural Collaborative Filtering (NCF)
-<precision> <mode> using Intel-optimized TensorFlow.
diff --git a/quickstart/recommendation/tensorflow/ncf/inference/cpu/fp32/.docs/docker.md b/quickstart/recommendation/tensorflow/ncf/inference/cpu/fp32/.docs/docker.md
deleted file mode 100644
index de922ae00..000000000
--- a/quickstart/recommendation/tensorflow/ncf/inference/cpu/fp32/.docs/docker.md
+++ /dev/null
@@ -1,30 +0,0 @@
-<!--- 60. Docker -->
-## Docker
-
-The model container includes the scripts and libraries needed to run 
-<model name> <precision> <mode>. To run one of the quickstart scripts 
-using this container, you'll need to provide volume mounts for the dataset 
-and an output directory.
-
-```
-DATASET_DIR=<path to the dataset>
-OUTPUT_DIR=<directory where log files will be written>
-# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
-export BATCH_SIZE=<customized batch size value>
-
-docker run \
-  --env DATASET_DIR=${DATASET_DIR} \
-  --env OUTPUT_DIR=${OUTPUT_DIR} \
-  --env BATCH_SIZE=${BATCH_SIZE} \
-  --env http_proxy=${http_proxy} \
-  --env https_proxy=${https_proxy} \
-  --volume ${DATASET_DIR}:${DATASET_DIR} \
-  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
-  --privileged --init -t \
-  <docker image> \
-  /bin/bash quickstart/<script name>.sh
-```
-
-If you are new to docker and are running into issues with the container,
-see [this document](https://github.com/IntelAI/models/tree/master/docs/general/docker.md)
-for troubleshooting tips.
diff --git a/quickstart/recommendation/tensorflow/ncf/inference/cpu/fp32/.docs/download.md b/quickstart/recommendation/tensorflow/ncf/inference/cpu/fp32/.docs/download.md
deleted file mode 100644
index 613926758..000000000
--- a/quickstart/recommendation/tensorflow/ncf/inference/cpu/fp32/.docs/download.md
+++ /dev/null
@@ -1,4 +0,0 @@
-<!--- 20. Download link -->
-## Download link
-
-[<package name>](<package url>)
diff --git a/quickstart/recommendation/tensorflow/ncf/inference/cpu/fp32/.docs/license.md b/quickstart/recommendation/tensorflow/ncf/inference/cpu/fp32/.docs/license.md
deleted file mode 100644
index a60f169f4..000000000
--- a/quickstart/recommendation/tensorflow/ncf/inference/cpu/fp32/.docs/license.md
+++ /dev/null
@@ -1,4 +0,0 @@
-<!--- 80. License -->
-## License
-
-[LICENSE](/LICENSE)
diff --git a/quickstart/recommendation/tensorflow/ncf/inference/cpu/fp32/.docs/quickstart.md b/quickstart/recommendation/tensorflow/ncf/inference/cpu/fp32/.docs/quickstart.md
deleted file mode 100644
index 73386bb57..000000000
--- a/quickstart/recommendation/tensorflow/ncf/inference/cpu/fp32/.docs/quickstart.md
+++ /dev/null
@@ -1,8 +0,0 @@
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| [`fp32_online_inference.sh`](fp32_online_inference.sh) | Runs online inference (batch_size=1). |
-| [`fp32_batch_inference.sh`](fp32_batch_inference.sh) | Runs batch inference (batch_size=256). |
-| [`fp32_accuracy.sh`](fp32_accuracy.sh) | Measures the model accuracy (batch_size=256). |
diff --git a/quickstart/recommendation/tensorflow/ncf/inference/cpu/fp32/.docs/title.md b/quickstart/recommendation/tensorflow/ncf/inference/cpu/fp32/.docs/title.md
deleted file mode 100644
index 6416ff7ff..000000000
--- a/quickstart/recommendation/tensorflow/ncf/inference/cpu/fp32/.docs/title.md
+++ /dev/null
@@ -1,2 +0,0 @@
-<!--- 0. Title -->
-# <model name> <precision> <mode>
diff --git a/quickstart/recommendation/tensorflow/ncf/inference/cpu/fp32/README.md b/quickstart/recommendation/tensorflow/ncf/inference/cpu/fp32/README.md
deleted file mode 100644
index 89827d568..000000000
--- a/quickstart/recommendation/tensorflow/ncf/inference/cpu/fp32/README.md
+++ /dev/null
@@ -1,111 +0,0 @@
-<!--- 0. Title -->
-# NCF FP32 inference
-
-<!-- 10. Description -->
-## Description
-
-This document has instructions for running Neural Collaborative Filtering (NCF)
-FP32 inference using Intel-optimized TensorFlow.
-
-<!--- 20. Download link -->
-## Download link
-
-[ncf-fp32-inference.tar.gz](https://storage.googleapis.com/intel-optimized-tensorflow/models/v2_7_0/ncf-fp32-inference.tar.gz)
-
-<!--- 30. Datasets -->
-## Datasets
-
-Download [movielens 1M dataset](https://grouplens.org/datasets/movielens/1m/)
-```
-wget http://files.grouplens.org/datasets/movielens/ml-1m.zip
-unzip ml-1m.zip
-```
-Set the `DATASET_DIR` to point to this directory when running NCF.
-
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| [`fp32_online_inference.sh`](fp32_online_inference.sh) | Runs online inference (batch_size=1). |
-| [`fp32_batch_inference.sh`](fp32_batch_inference.sh) | Runs batch inference (batch_size=256). |
-| [`fp32_accuracy.sh`](fp32_accuracy.sh) | Measures the model accuracy (batch_size=256). |
-
-<!--- 50. Bare Metal -->
-## Bare Metal
-
-To run on bare metal, the following prerequisites must be installed in your environment:
-* Python 3.6 or 3.7
-* [intel-tensorflow==1.15.2](https://pypi.org/project/intel-tensorflow/1.15.2/)
-* numactl
-* google-api-python-client==1.6.7
-* google-cloud-bigquery==0.31.0
-* kaggle==1.3.9
-* numpy==1.16.3
-* oauth2client==4.1.2
-* pandas
-* 'psutil>=5.6.7'
-* py-cpuinfo==3.3.0
-* typing
-* TensorFlow models, clone the official `tensorflow/models` repository with  tag `v1.11`:
-```
-git clone https://github.com/tensorflow/models.git tf_models
-cd tf_models
-git checkout v1.11
-```
-
-After installing the prerequisites, download and untar the model package.
-Set environment variables for the path to your `DATASET_DIR` and an
-`OUTPUT_DIR` where log files will be written, then run a 
-[quickstart script](#quick-start-scripts).
-
-```
-DATASET_DIR=<path to the dataset>
-OUTPUT_DIR=<directory where log files will be written>
-TF_MODELS_DIR=<path to the TensorFlow models directory tf_models>
-# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
-export BATCH_SIZE=<customized batch size value>
-
-wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v2_7_0/ncf-fp32-inference.tar.gz
-tar -xzf ncf-fp32-inference.tar.gz
-cd ncf-fp32-inference
-
-./quickstart/<script name>.sh
-```
-
-<!--- 60. Docker -->
-## Docker
-
-The model container includes the scripts and libraries needed to run 
-NCF FP32 inference. To run one of the quickstart scripts 
-using this container, you'll need to provide volume mounts for the dataset 
-and an output directory.
-
-```
-DATASET_DIR=<path to the dataset>
-OUTPUT_DIR=<directory where log files will be written>
-# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
-export BATCH_SIZE=<customized batch size value>
-
-docker run \
-  --env DATASET_DIR=${DATASET_DIR} \
-  --env OUTPUT_DIR=${OUTPUT_DIR} \
-  --env BATCH_SIZE=${BATCH_SIZE} \
-  --env http_proxy=${http_proxy} \
-  --env https_proxy=${https_proxy} \
-  --volume ${DATASET_DIR}:${DATASET_DIR} \
-  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
-  --privileged --init -t \
-  intel/recommendation:tf-1.15.2-ncf-fp32-inference \
-  /bin/bash quickstart/<script name>.sh
-```
-
-If you are new to docker and are running into issues with the container,
-see [this document](https://github.com/IntelAI/models/tree/master/docs/general/docker.md)
-for troubleshooting tips.
-
-<!--- 80. License -->
-## License
-
-[LICENSE](/LICENSE)
-
diff --git a/quickstart/recommendation/tensorflow/ncf/inference/cpu/fp32/fp32_accuracy.sh b/quickstart/recommendation/tensorflow/ncf/inference/cpu/fp32/fp32_accuracy.sh
deleted file mode 100755
index 240605904..000000000
--- a/quickstart/recommendation/tensorflow/ncf/inference/cpu/fp32/fp32_accuracy.sh
+++ /dev/null
@@ -1,104 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2020 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-MODEL_DIR=${MODEL_DIR-$PWD}
-
-echo 'MODEL_DIR='$MODEL_DIR
-echo 'OUTPUT_DIR='$OUTPUT_DIR
-echo 'DATASET_DIR='$DATASET_DIR
-
-if [ -z "${OUTPUT_DIR}" ]; then
-  echo "The required environment variable OUTPUT_DIR has not been set"
-  exit 1
-fi
-
-# Create the output directory in case it doesn't already exist
-mkdir -p ${OUTPUT_DIR}
-
-# Check for TF_MODELS_DIR
-if [ -d "/tensorflow/models" ]; then # if true assume running in docker
-   TF_MODELS_DIR=/tensorflow/models
-elif [ -z ${TF_MODELS_DIR} ]; then
-    echo "Please set TF_MODELS_DIR or run in docker mode." >&2
-    exit 1
-fi
-
-echo 'TF_MODELS_DIR='$TF_MODELS_DIR
-# Commenting out a line that causes a crash in the model script
-sed -i.bak 's/atexit.register/# atexit.register/g' ${TF_MODELS_DIR}/official/recommendation/data_async_generation.py
-
-if [ -z "${PRETRAINED_MODEL}" ]; then
-  # Unzip pretrained model files
-  pretrained_model_dir="pretrained_model/ncf_trained_movielens_1m"
-  if [ ! -d "${pretrained_model_dir}" ]; then
-    if [ -f pretrained_model/ncf_fp32_pretrained_model.tar.gz ]; then
-      tar -C pretrained_model/ -xvf pretrained_model/ncf_fp32_pretrained_model.tar.gz
-    else
-      echo "The pretrained model could not be found. Please set the PRETRAINED_MODEL env var to point to the pretrained model directory."
-      exit 1
-    fi
-  fi
-  CHECKPOINT_DIR="${MODEL_DIR}/${pretrained_model_dir}"
-else
-  if [ ! -d "${PRETRAINED_MODEL}" ]; then
-    echo "The PRETRAINED_MODEL directory (${PRETRAINED_MODEL}) does not exist."
-    exit 1
-  else
-    CHECKPOINT_DIR=${PRETRAINED_MODEL}
-  fi
-fi
-
-# Create an array of input directories that are expected and then verify that they exist
-declare -A input_dirs
-input_dirs[CHECKPOINT_DIR]=${CHECKPOINT_DIR}
-input_dirs[DATASET_DIR]=${DATASET_DIR}
-
-for i in "${!input_dirs[@]}"; do
-  var_name=$i
-  dir_path=${input_dirs[$i]}
-
-  if [[ -z $dir_path ]]; then
-    echo "The required environment variable $var_name is empty" >&2
-    exit 1
-  fi
-
-  if [[ ! -d $dir_path ]]; then
-    echo "The $var_name path '$dir_path' does not exist" >&2
-    exit 1
-  fi
-done
-
-# If batch size env is not mentioned, then the workload will run with the default batch size.
-if [ -z "${BATCH_SIZE}"]; then
-  BATCH_SIZE="256"
-  echo "Running with default batch size of ${BATCH_SIZE}"
-fi
-
-source "${MODEL_DIR}/quickstart/common/utils.sh"
-_command python ${MODEL_DIR}/benchmarks/launch_benchmark.py \
-      --checkpoint ${CHECKPOINT_DIR} \
-      --data-location ${DATASET_DIR} \
-      --model-source-dir ${TF_MODELS_DIR} \
-      --model-name ncf \
-      --socket-id 0 \
-      --batch-size ${BATCH_SIZE} \
-      --framework tensorflow \
-      --precision fp32 \
-      --mode inference \
-      --accuracy-only \
-      --output-dir ${OUTPUT_DIR} \
-      $@
diff --git a/quickstart/recommendation/tensorflow/ncf/inference/cpu/fp32/fp32_batch_inference.sh b/quickstart/recommendation/tensorflow/ncf/inference/cpu/fp32/fp32_batch_inference.sh
deleted file mode 100755
index 51744c3f2..000000000
--- a/quickstart/recommendation/tensorflow/ncf/inference/cpu/fp32/fp32_batch_inference.sh
+++ /dev/null
@@ -1,103 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2020 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-MODEL_DIR=${MODEL_DIR-$PWD}
-
-echo 'MODEL_DIR='$MODEL_DIR
-echo 'OUTPUT_DIR='$OUTPUT_DIR
-echo 'DATASET_DIR='$DATASET_DIR
-
-if [ -z "${OUTPUT_DIR}" ]; then
-  echo "The required environment variable OUTPUT_DIR has not been set"
-  exit 1
-fi
-
-# Create the output directory in case it doesn't already exist
-mkdir -p ${OUTPUT_DIR}
-
-# Check for TF_MODELS_DIR
-if [ -d "/tensorflow/models" ]; then # if true assume running in docker
-   TF_MODELS_DIR=/tensorflow/models
-elif [ -z ${TF_MODELS_DIR} ]; then
-    echo "Please set TF_MODELS_DIR or run in docker mode." >&2
-    exit 1
-fi
-
-echo 'TF_MODELS_DIR='$TF_MODELS_DIR
-# Commenting out a line that causes a crash in the model script
-sed -i.bak 's/atexit.register/# atexit.register/g' ${TF_MODELS_DIR}/official/recommendation/data_async_generation.py
-
-if [ -z "${PRETRAINED_MODEL}" ]; then
-  # Unzip pretrained model files
-  pretrained_model_dir="pretrained_model/ncf_trained_movielens_1m"
-  if [ ! -d "${pretrained_model_dir}" ]; then
-    if [ -f pretrained_model/ncf_fp32_pretrained_model.tar.gz ]; then
-      tar -C pretrained_model/ -xvf pretrained_model/ncf_fp32_pretrained_model.tar.gz
-    else
-      echo "The pretrained model could not be found. Please set the PRETRAINED_MODEL env var to point to the pretrained model directory."
-      exit 1
-    fi
-  fi
-  CHECKPOINT_DIR="${MODEL_DIR}/${pretrained_model_dir}"
-else
-  if [ ! -d "${PRETRAINED_MODEL}" ]; then
-    echo "The PRETRAINED_MODEL directory (${PRETRAINED_MODEL}) does not exist."
-    exit 1
-  else
-    CHECKPOINT_DIR=${PRETRAINED_MODEL}
-  fi
-fi
-
-# Create an array of input directories that are expected and then verify that they exist
-declare -A input_dirs
-input_dirs[CHECKPOINT_DIR]=${CHECKPOINT_DIR}
-input_dirs[DATASET_DIR]=${DATASET_DIR}
-
-for i in "${!input_dirs[@]}"; do
-  var_name=$i
-  dir_path=${input_dirs[$i]}
-
-  if [[ -z $dir_path ]]; then
-    echo "The required environment variable $var_name is empty" >&2
-    exit 1
-  fi
-
-  if [[ ! -d $dir_path ]]; then
-    echo "The $var_name path '$dir_path' does not exist" >&2
-    exit 1
-  fi
-done
-
-# If batch size env is not mentioned, then the workload will run with the default batch size.
-if [ -z "${BATCH_SIZE}"]; then
-  BATCH_SIZE="256"
-  echo "Running with default batch size of ${BATCH_SIZE}"
-fi
-
-source "${MODEL_DIR}/quickstart/common/utils.sh"
-_command python ${MODEL_DIR}/benchmarks/launch_benchmark.py \
-      --checkpoint ${CHECKPOINT_DIR} \
-      --data-location ${DATASET_DIR} \
-      --model-source-dir ${TF_MODELS_DIR} \
-      --model-name ncf \
-      --socket-id 0 \
-      --batch-size ${BATCH_SIZE} \
-      --framework tensorflow \
-      --precision fp32 \
-      --mode inference \
-      --output-dir ${OUTPUT_DIR} \
-      $@
diff --git a/quickstart/recommendation/tensorflow/ncf/inference/cpu/fp32/fp32_online_inference.sh b/quickstart/recommendation/tensorflow/ncf/inference/cpu/fp32/fp32_online_inference.sh
deleted file mode 100755
index 2a117c467..000000000
--- a/quickstart/recommendation/tensorflow/ncf/inference/cpu/fp32/fp32_online_inference.sh
+++ /dev/null
@@ -1,103 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2020 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-MODEL_DIR=${MODEL_DIR-$PWD}
-
-echo 'MODEL_DIR='$MODEL_DIR
-echo 'OUTPUT_DIR='$OUTPUT_DIR
-echo 'DATASET_DIR='$DATASET_DIR
-
-if [ -z "${OUTPUT_DIR}" ]; then
-  echo "The required environment variable OUTPUT_DIR has not been set"
-  exit 1
-fi
-
-# Create the output directory in case it doesn't already exist
-mkdir -p ${OUTPUT_DIR}
-
-# Check for TF_MODELS_DIR
-if [ -d "/tensorflow/models" ]; then # if true assume running in docker
-   TF_MODELS_DIR=/tensorflow/models
-elif [ -z ${TF_MODELS_DIR} ]; then
-    echo "Please set TF_MODELS_DIR or run in docker mode." >&2
-    exit 1
-fi
-
-echo 'TF_MODELS_DIR='$TF_MODELS_DIR
-# Commenting out a line that causes a crash in the model script
-sed -i.bak 's/atexit.register/# atexit.register/g' ${TF_MODELS_DIR}/official/recommendation/data_async_generation.py
-
-if [ -z "${PRETRAINED_MODEL}" ]; then
-  # Unzip pretrained model files
-  pretrained_model_dir="pretrained_model/ncf_trained_movielens_1m"
-  if [ ! -d "${pretrained_model_dir}" ]; then
-    if [ -f pretrained_model/ncf_fp32_pretrained_model.tar.gz ]; then
-      tar -C pretrained_model/ -xvf pretrained_model/ncf_fp32_pretrained_model.tar.gz
-    else
-      echo "The pretrained model could not be found. Please set the PRETRAINED_MODEL env var to point to the pretrained model directory."
-      exit 1
-    fi
-  fi
-  CHECKPOINT_DIR="${MODEL_DIR}/${pretrained_model_dir}"
-else
-  if [ ! -d "${PRETRAINED_MODEL}" ]; then
-    echo "The PRETRAINED_MODEL directory (${PRETRAINED_MODEL}) does not exist."
-    exit 1
-  else
-    CHECKPOINT_DIR=${PRETRAINED_MODEL}
-  fi
-fi
-
-# Create an array of input directories that are expected and then verify that they exist
-declare -A input_dirs
-input_dirs[CHECKPOINT_DIR]=${CHECKPOINT_DIR}
-input_dirs[DATASET_DIR]=${DATASET_DIR}
-
-for i in "${!input_dirs[@]}"; do
-  var_name=$i
-  dir_path=${input_dirs[$i]}
-
-  if [[ -z $dir_path ]]; then
-    echo "The required environment variable $var_name is empty" >&2
-    exit 1
-  fi
-
-  if [[ ! -d $dir_path ]]; then
-    echo "The $var_name path '$dir_path' does not exist" >&2
-    exit 1
-  fi
-done
-
-# If batch size env is not mentioned, then the workload will run with the default batch size.
-if [ -z "${BATCH_SIZE}"]; then
-  BATCH_SIZE="1"
-  echo "Running with default batch size of ${BATCH_SIZE}"
-fi
-
-source "${MODEL_DIR}/quickstart/common/utils.sh"
-_command python ${MODEL_DIR}/benchmarks/launch_benchmark.py \
-      --checkpoint ${CHECKPOINT_DIR} \
-      --data-location ${DATASET_DIR} \
-      --model-source-dir ${TF_MODELS_DIR} \
-      --model-name ncf \
-      --socket-id 0 \
-      --batch-size ${BATCH_SIZE} \
-      --framework tensorflow \
-      --precision fp32 \
-      --mode inference \
-      --output-dir ${OUTPUT_DIR} \
-      $@
diff --git a/quickstart/recommendation/tensorflow/wide_deep/inference/cpu/.docs/advancedoptions.md b/quickstart/recommendation/tensorflow/wide_deep/inference/cpu/.docs/advancedoptions.md
deleted file mode 100644
index aa7aa64e6..000000000
--- a/quickstart/recommendation/tensorflow/wide_deep/inference/cpu/.docs/advancedoptions.md
+++ /dev/null
@@ -1,4 +0,0 @@
-<!-- 61. Advanced Options -->
-
-See the [Advanced Options for Model Packages and Containers](/quickstart/common/tensorflow/ModelPackagesAdvancedOptions.md)
-document for more advanced use cases.
diff --git a/quickstart/recommendation/tensorflow/wide_deep/inference/cpu/.docs/baremetal.md b/quickstart/recommendation/tensorflow/wide_deep/inference/cpu/.docs/baremetal.md
deleted file mode 100644
index 7e0aeba57..000000000
--- a/quickstart/recommendation/tensorflow/wide_deep/inference/cpu/.docs/baremetal.md
+++ /dev/null
@@ -1,109 +0,0 @@
-<!--- 50. AI Kit -->
-## Run the model
-
-Setup your environment using the instructions below, depending on if you are
-using [AI Kit](/docs/general/tensorflow/AIKit.md):
-
-<table>
-  <tr>
-    <th>Setup using AI Kit on Linux</th>
-    <th>Setup without AI Kit on Linux</th>
-    <th>Setup without AI Kit on Windows</th>
-  </tr>
-  <tr>
-    <td>
-      <p>To run using AI Kit on Linux you will need:</p>
-      <ul>
-        <li>git
-        <li>numactl
-        <li>wget
-        <li>Activate the `tensorflow` conda environment
-        <pre>conda activate tensorflow</pre>
-      </ul>
-    </td>
-    <td>
-      <p>To run without AI Kit on Linux you will need:</p>
-      <ul>
-        <li>Python 3
-        <li><a href="https://pypi.org/project/intel-tensorflow/">intel-tensorflow>=2.5.0</a>
-        <li>git
-        <li>numactl
-        <li>wget
-        <li>A clone of the Model Zoo repo<br />
-        <pre>git clone https://github.com/IntelAI/models.git</pre>
-      </ul>
-    </td>
-    <td>
-      <p>To run without AI Kit on Windows you will need:</p>
-      <ul>
-        <li><a href="/docs/general/Windows.md">Intel Model Zoo on Windows Systems prerequisites</a>
-        <li>git
-        <li>A clone of the Model Zoo repo<br />
-        <pre>git clone https://github.com/IntelAI/models.git</pre>
-      </ul>
-    </td>
-  </tr>
-</table>
-
-After you've completed the setup above, download and extract the pretrained
-model. If you run on Windows, please use a browser to download the pretrained model using the link below.
-Set the directory path to the `PRETRAINED_MODEL` environment variable.
-```
-wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/wide_deep_fp32_pretrained_model.tar.gz
-tar -xzvf wide_deep_fp32_pretrained_model.tar.gz
-export PRETRAINED_MODEL=$(pwd)/wide_deep_fp32_pretrained_model
-```
-
-Wide & Deep inference also uses code from the [TensorFlow models repo](https://github.com/tensorflow/models).
-Clone the repo using the PR in the snippet below and save the directory path
-to the `TF_MODELS_DIR` environment variable.
-```
-# We going to use a branch based on older version of the tensorflow model repo.
-# Since, we need to to use logs utils on that branch, which were removed from
-# the latest master
-git clone https://github.com/tensorflow/models.git tensorflow-models
-cd tensorflow-models
-git fetch origin pull/7461/head:wide-deep-tf2
-git checkout wide-deep-tf2
-```
-
-Set the environment variables and run quickstart script on either Linux or Windows systems. See the list of quickstart scripts for details on the different options.
-
-### Run on Linux
-```
-# cd to your model zoo directory
-cd models
-
-export DATASET_DIR=<path to the Wide & Deep dataset directory>
-export PRECISION=fp32
-export OUTPUT_DIR=<path to the directory where log files will be written>
-export PRETRAINED_MODEL=<pretrained model directory>
-export TF_MODELS_DIR=<path to tensorflow-models directory>
-# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
-export BATCH_SIZE=<customized batch size value>
-
-./quickstart/recommendation/tensorflow/wide_deep/inference/cpu/fp32/<script name>.sh
-```
-### Run on Windows
-Using cmd.exe, run:
-```
-# cd to your model zoo directory
-cd models
-
-set PRETRAINED_MODEL=<pretrained model directory>
-set DATASET_DIR=<path to the Wide & Deep dataset directory>
-set PRECISION=fp32
-set OUTPUT_DIR=<directory where log files will be written>
-set TF_MODELS_DIR=<path to tensorflow-models directory>
-# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
-set BATCH_SIZE=<customized batch size value>
-
-bash quickstart\recommendation\tensorflow\wide_deep\inference\cpu\fp32\<script name>.sh
-```
-> Note: You may use `cygpath` to convert the Windows paths to Unix paths before setting the environment variables. 
-As an example, if the dataset location on Windows is `D:\<user>\widedeep_dataset`, convert the Windows path to Unix as shown:
-> ```
-> cygpath D:\<user>\widedeep_dataset
-> /d/<user>/widedeep_dataset
->```
->Then, set the `DATASET_DIR` environment variable `set DATASET_DIR=/d/<user>/widedeep_dataset`.
diff --git a/quickstart/recommendation/tensorflow/wide_deep/inference/cpu/.docs/datasets.md b/quickstart/recommendation/tensorflow/wide_deep/inference/cpu/.docs/datasets.md
deleted file mode 100644
index a8c115659..000000000
--- a/quickstart/recommendation/tensorflow/wide_deep/inference/cpu/.docs/datasets.md
+++ /dev/null
@@ -1,12 +0,0 @@
-<!--- 30. Datasets -->
-## Dataset
-Download and preprocess the [income census data](https://archive.ics.uci.edu/ml/datasets/Census+Income) by running 
-following python script, which is a standalone version of [census_dataset.py](https://github.com/tensorflow/models/blob/v2.2.0/official/r1/wide_deep/census_dataset.py)
-Please note that below program requires `requests` module to be installed. You can install it using `pip install requests`.
-Dataset will be downloaded in directory provided using `--data_dir`. If you are behind corporate proxy, then you can provide proxy URLs
-using `--http_proxy` and `--https_proxy` arguments.
-```
-git clone https://github.com/IntelAI/models.git
-cd models
-python ./benchmarks/recommendation/tensorflow/wide_deep/inference/fp32/data_download.py --data_dir /home/<user>/widedeep_dataset
-```
diff --git a/quickstart/recommendation/tensorflow/wide_deep/inference/cpu/.docs/description.md b/quickstart/recommendation/tensorflow/wide_deep/inference/cpu/.docs/description.md
deleted file mode 100644
index 21cf18b5b..000000000
--- a/quickstart/recommendation/tensorflow/wide_deep/inference/cpu/.docs/description.md
+++ /dev/null
@@ -1,4 +0,0 @@
-<!-- 10. Description -->
-
-This document has instructions for running <model name> <mode> using
-Intel-optimized TensorFlow.
diff --git a/quickstart/recommendation/tensorflow/wide_deep/inference/cpu/.docs/docker.md b/quickstart/recommendation/tensorflow/wide_deep/inference/cpu/.docs/docker.md
deleted file mode 100644
index 6ed448284..000000000
--- a/quickstart/recommendation/tensorflow/wide_deep/inference/cpu/.docs/docker.md
+++ /dev/null
@@ -1,31 +0,0 @@
-<!-- 60. Docker -->
-### Docker
-
-When running in docker, the <model name> <precision> <mode> container includes the model package and TensorFlow model source repo,
-which is needed to run inference. To run the quickstart scripts, you'll need to provide volume mounts for the dataset and
-an output directory where log files will be written.
-
-```
-DATASET_DIR=<path to the Wide & Deep dataset directory>
-PRECISION=fp32
-OUTPUT_DIR=<directory where log files will be written>
-# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
-export BATCH_SIZE=<customized batch size value>
-
-docker run \
---env DATASET_DIR=${DATASET_DIR} \
---env PRECISION=${PRECISION} \
---env OUTPUT_DIR=${OUTPUT_DIR} \
---env BATCH_SIZE=${BATCH_SIZE} \
---env http_proxy=${http_proxy} \
---env https_proxy=${https_proxy} \
---volume ${DATASET_DIR}:${DATASET_DIR} \
---volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
---privileged --init -t \
-<docker image> \
-/bin/bash quickstart/<script name>.sh
-```
-
-If you are new to docker and are running into issues with the container,
-see [this document](https://github.com/IntelAI/models/tree/master/docs/general/docker.md)
-for troubleshooting tips.
diff --git a/quickstart/recommendation/tensorflow/wide_deep/inference/cpu/.docs/license.md b/quickstart/recommendation/tensorflow/wide_deep/inference/cpu/.docs/license.md
deleted file mode 100644
index a60f169f4..000000000
--- a/quickstart/recommendation/tensorflow/wide_deep/inference/cpu/.docs/license.md
+++ /dev/null
@@ -1,4 +0,0 @@
-<!--- 80. License -->
-## License
-
-[LICENSE](/LICENSE)
diff --git a/quickstart/recommendation/tensorflow/wide_deep/inference/cpu/.docs/quickstart.md b/quickstart/recommendation/tensorflow/wide_deep/inference/cpu/.docs/quickstart.md
deleted file mode 100644
index 4e1b5e1c1..000000000
--- a/quickstart/recommendation/tensorflow/wide_deep/inference/cpu/.docs/quickstart.md
+++ /dev/null
@@ -1,7 +0,0 @@
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| [`inference_online.sh`](/quickstart/recommendation/tensorflow/wide_deep/inference/cpu/inference_online.sh) | Runs wide & deep model inference online mode (batch size = 1)|
-| [`inference_batch.sh`](/quickstart/recommendation/tensorflow/wide_deep/inference/cpu/inference_batch.sh) | Runs wide & deep model inference in batch mode (batch size = 1024)|
diff --git a/quickstart/recommendation/tensorflow/wide_deep/inference/cpu/.docs/title.md b/quickstart/recommendation/tensorflow/wide_deep/inference/cpu/.docs/title.md
deleted file mode 100644
index 1d6c60187..000000000
--- a/quickstart/recommendation/tensorflow/wide_deep/inference/cpu/.docs/title.md
+++ /dev/null
@@ -1,2 +0,0 @@
-<!--- 0. Title -->
-# <model name> <mode>
diff --git a/quickstart/recommendation/tensorflow/wide_deep_large_ds/inference/cpu/.docs/baremetal.md b/quickstart/recommendation/tensorflow/wide_deep_large_ds/inference/cpu/.docs/baremetal.md
deleted file mode 100644
index 080af1ee4..000000000
--- a/quickstart/recommendation/tensorflow/wide_deep_large_ds/inference/cpu/.docs/baremetal.md
+++ /dev/null
@@ -1,63 +0,0 @@
-<!--- 50. AI Kit -->
-## Run the model
-
-Setup your environment using the instructions below, depending on if you are
-using [AI Kit](/docs/general/tensorflow/AIKit.md):
-
-<table>
-  <tr>
-    <th>Setup using AI Kit</th>
-    <th>Setup without AI Kit</th>
-  </tr>
-  <tr>
-    <td>
-      <p>AI Kit does not currently support wide and deep large ds</p>
-    </td>
-    <td>
-      <p>To run without AI Kit you will need:</p>
-      <ul>
-        <li>Python 3.7 or newer
-        <li><a href="https://pypi.org/project/intel-tensorflow">intel-tensorflow>=2.5.0</a>
-        <li>numactl
-        <li>git
-        <li>wget
-        <li>A clone of the Model Zoo repo<br />
-        <pre>git clone https://github.com/IntelAI/models.git</pre>
-      </ul>
-    </td>
-  </tr>
-</table>
-
-After completing the setup, download the pretrained model frozen graph
-and save the path to the PRETRAINED_MODEL envionment variable.
-```
-# FP32 Pretrained Model
-wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/wide_deep_fp32_pretrained_model.pb
-export PRETRAINED_MODEL=$(pwd)/wide_deep_fp32_pretrained_model.pb
-
-# Int8 Pretrained Model
-wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v1_8/wide_deep_int8_pretrained_model.pb
-export PRETRAINED_MODEL=$(pwd)/wide_deep_int8_pretrained_model.pb
-```
-
-Set the environment variables and run quickstart script. See the list of quickstart scripts for details on the different options.
-
-```
-# cd to your model zoo directory
-cd models
-
-export DATASET_DIR=<path to TF records file>
-export PRECISION=<set the precision to "int8" or "fp32">
-export OUTPUT_DIR=<path to the directory where log files will be written>
-export PRETRAINED_MODEL=<path to the frozen graph>
-# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
-export BATCH_SIZE=<customized batch size value>
-
-# Run inference with an accuracy check
-./quickstart/recommendation/tensorflow/wide_deep_large_ds/inference/cpu/accuracy.sh
-
-# Run online inference and set NUM_OMP_THREADS
-export NUM_OMP_THREADS=1
-./quickstart/recommendation/tensorflow/wide_deep_large_ds/inference/cpu/online_inference.sh \
---num-intra-threads 1 --num-inter-threads 1
-```
diff --git a/quickstart/recommendation/tensorflow/wide_deep_large_ds/inference/cpu/.docs/datasets.md b/quickstart/recommendation/tensorflow/wide_deep_large_ds/inference/cpu/.docs/datasets.md
deleted file mode 100644
index 439876313..000000000
--- a/quickstart/recommendation/tensorflow/wide_deep_large_ds/inference/cpu/.docs/datasets.md
+++ /dev/null
@@ -1,10 +0,0 @@
-<!--- 30. Datasets -->
-## Datasets
-
-Follow [instructions](https://github.com/IntelAI/models/tree/master/datasets/large_kaggle_advertising_challenge/README.md)
-to download and preprocess the Large Kaggle Display Advertising Challenge Dataset.
-
-Then, set the `DATASET_DIR` to point to this directory when running <model name>:
-```
-export DATASET_DIR=/home/<user>/dataset/eval_preprocessed_eval.tfrecords
-```
diff --git a/quickstart/recommendation/tensorflow/wide_deep_large_ds/inference/cpu/.docs/description.md b/quickstart/recommendation/tensorflow/wide_deep_large_ds/inference/cpu/.docs/description.md
deleted file mode 100644
index 106b45dbe..000000000
--- a/quickstart/recommendation/tensorflow/wide_deep_large_ds/inference/cpu/.docs/description.md
+++ /dev/null
@@ -1,5 +0,0 @@
-<!-- 10. Description -->
-## Description
-
-This document has instructions for running <model name> <mode> using
-Intel-optimized TensorFlow.
diff --git a/quickstart/recommendation/tensorflow/wide_deep_large_ds/inference/cpu/.docs/docker.md b/quickstart/recommendation/tensorflow/wide_deep_large_ds/inference/cpu/.docs/docker.md
deleted file mode 100644
index 6c47141d3..000000000
--- a/quickstart/recommendation/tensorflow/wide_deep_large_ds/inference/cpu/.docs/docker.md
+++ /dev/null
@@ -1,62 +0,0 @@
-<!--- 60. Docker -->
-## Docker
-
-The model container includes the scripts and libraries needed to run 
-<model name> <mode>. To run one of the quickstart scripts 
-using this container, you'll need to provide volume mounts for the dataset
-and an output directory.
-
-* Running inference to check accuracy:
-```
-DATASET_DIR=<path to the dataset>
-PRECISION=<set the precision to "int8" or "fp32">
-OUTPUT_DIR=<directory where log files will be written>
-# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
-export BATCH_SIZE=<customized batch size value>
-
-docker run \
-  --env DATASET_DIR=${DATASET_DIR} \
-  --env PRECISION=${PRECISION} \
-  --env OUTPUT_DIR=${OUTPUT_DIR} \
-  --env BATCH_SIZE=${BATCH_SIZE} \
-  --env http_proxy=${http_proxy} \
-  --env https_proxy=${https_proxy} \
-  --volume ${DATASET_DIR}:${DATASET_DIR} \
-  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
-  --privileged --init -t \
-  <docker image> \
-  /bin/bash quickstart/fp32_accuracy.sh
-  
-  
-```
-
-* Running online inference:
-Set `NUM_OMP_THREADS` for tunning the hyperparameter `num_omp_threads`.
-
-```
-DATASET_DIR=<path to the dataset>
-PRECISION=<set the precision to "int8" or "fp32">
-OUTPUT_DIR=<directory where log files will be written>
-NUM_OMP_THREADS=1
-# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
-export BATCH_SIZE=<customized batch size value>
-
-docker run \
-  --env DATASET_DIR=${DATASET_DIR} \
-  --env PRECISION=${PRECISION} \
-  --env OUTPUT_DIR=${OUTPUT_DIR} \
-  --env BATCH_SIZE=${BATCH_SIZE} \
-  --env NUM_OMP_THREADS=${NUM_OMP_THREADS} \
-  --env http_proxy=${http_proxy} \
-  --env https_proxy=${https_proxy} \
-  --volume ${DATASET_DIR}:${DATASET_DIR} \
-  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
-  --privileged --init -t \
-  <docker image> \
-  /bin/bash quickstart/fp32_online_inference.sh \
-  --num-intra-threads 1 --num-inter-threads 1
-```
-
-If you are new to docker and are running into issues with the container,
-see [this document](https://github.com/IntelAI/models/tree/master/docs/general/docker.md)
-for troubleshooting tips.
diff --git a/quickstart/recommendation/tensorflow/wide_deep_large_ds/inference/cpu/.docs/license.md b/quickstart/recommendation/tensorflow/wide_deep_large_ds/inference/cpu/.docs/license.md
deleted file mode 100644
index a60f169f4..000000000
--- a/quickstart/recommendation/tensorflow/wide_deep_large_ds/inference/cpu/.docs/license.md
+++ /dev/null
@@ -1,4 +0,0 @@
-<!--- 80. License -->
-## License
-
-[LICENSE](/LICENSE)
diff --git a/quickstart/recommendation/tensorflow/wide_deep_large_ds/inference/cpu/.docs/quickstart.md b/quickstart/recommendation/tensorflow/wide_deep_large_ds/inference/cpu/.docs/quickstart.md
deleted file mode 100644
index d7c7caf4f..000000000
--- a/quickstart/recommendation/tensorflow/wide_deep_large_ds/inference/cpu/.docs/quickstart.md
+++ /dev/null
@@ -1,7 +0,0 @@
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| [`online_inference.sh`](/quickstart/recommendation/tensorflow/wide_deep_large_ds/inference/cpu/online_inference.sh) | Runs online inference (`batch_size=1`). The `NUM_OMP_THREADS` environment variable and the hyperparameters `num-intra-threads`, `num-inter-threads` can be tuned for best performance. If `NUM_OMP_THREADS` is not set, it will default to `1`. |
-| [`accuracy.sh`](/quickstart/recommendation/tensorflow/wide_deep_large_ds/inference/cpu/accuracy.sh) | Measures the model accuracy (`batch_size=1000`). |
diff --git a/quickstart/recommendation/tensorflow/wide_deep_large_ds/inference/cpu/.docs/title.md b/quickstart/recommendation/tensorflow/wide_deep_large_ds/inference/cpu/.docs/title.md
deleted file mode 100644
index 1d6c60187..000000000
--- a/quickstart/recommendation/tensorflow/wide_deep_large_ds/inference/cpu/.docs/title.md
+++ /dev/null
@@ -1,2 +0,0 @@
-<!--- 0. Title -->
-# <model name> <mode>
diff --git a/quickstart/recommendation/tensorflow/wide_deep_large_ds/inference/cpu/accuracy.sh b/quickstart/recommendation/tensorflow/wide_deep_large_ds/inference/cpu/accuracy.sh
index 40728ea99..c478bafd4 100755
--- a/quickstart/recommendation/tensorflow/wide_deep_large_ds/inference/cpu/accuracy.sh
+++ b/quickstart/recommendation/tensorflow/wide_deep_large_ds/inference/cpu/accuracy.sh
@@ -35,11 +35,28 @@ if [ ! -f "${DATASET_DIR}" ]; then
   exit 1
 fi
 
+<<<<<<<< HEAD:quickstart/language_translation/tensorflow/mlperf_gnmt/inference/cpu/batch_inference.sh
+# If precision env is not mentioned, then the workload will run with the default precision.
+if [ -z "${PRECISION}"]; then
+  PRECISION=fp32
+  echo "Running with default precision ${PRECISION}"
+fi
+
+if [[ $PRECISION != "fp32" ]]; then
+  echo "The specified precision '${PRECISION}' is unsupported."
+  echo "Supported precision is fp32."
+  exit 1
+fi
+
+if [ -z "${PRETRAINED_MODEL}" ]; then
+  PRETRAINED_MODEL="${MODEL_DIR}/mlperf_gnmt_fp32_pretrained_model.pb"
+========
 if [ -z "${PRECISION}" ]; then
   echo "The required environment variable PRECISION has not been set"
   echo "Please set PRECISION to fp32 or int8"
   exit 1
 fi
+>>>>>>>> r3.1:quickstart/recommendation/tensorflow/wide_deep_large_ds/inference/cpu/accuracy.sh
 
 if [[ $PRECISION != "fp32" ]] && [[ $PRECISION != "int8" ]]; then
   echo "The specified precision '${PRECISION}' is unsupported."
@@ -68,9 +85,28 @@ fi
 
 # If batch size env is not mentioned, then the workload will run with the default batch size.
 if [ -z "${BATCH_SIZE}"]; then
+<<<<<<<< HEAD:quickstart/language_translation/tensorflow/mlperf_gnmt/inference/cpu/batch_inference.sh
+  BATCH_SIZE="32"
+  echo "Running with default batch size of ${BATCH_SIZE}"
+fi
+
+source "${MODEL_DIR}/quickstart/common/utils.sh"
+_command python ${MODEL_DIR}/benchmarks/launch_benchmark.py \
+  --model-name mlperf_gnmt \
+  --framework tensorflow \
+  --precision ${PRECISION} \
+  --mode inference \
+  --batch-size ${BATCH_SIZE} \
+  --socket-id 0 \
+  --data-location ${DATASET_DIR} \
+  --in-graph ${PRETRAINED_MODEL} \
+  --output-dir ${OUTPUT_DIR} \
+  --benchmark-only
+========
   BATCH_SIZE="1000"
   echo "Running with default batch size of ${BATCH_SIZE}"
 fi
+>>>>>>>> r3.1:quickstart/recommendation/tensorflow/wide_deep_large_ds/inference/cpu/accuracy.sh
 
 # Run wide and deep large dataset inference
 source "$MODEL_DIR/quickstart/common/utils.sh"
diff --git a/quickstart/recommendation/tensorflow/wide_deep_large_ds/training/cpu/.docs/advancedoptions.md b/quickstart/recommendation/tensorflow/wide_deep_large_ds/training/cpu/.docs/advancedoptions.md
deleted file mode 100644
index aa7aa64e6..000000000
--- a/quickstart/recommendation/tensorflow/wide_deep_large_ds/training/cpu/.docs/advancedoptions.md
+++ /dev/null
@@ -1,4 +0,0 @@
-<!-- 61. Advanced Options -->
-
-See the [Advanced Options for Model Packages and Containers](/quickstart/common/tensorflow/ModelPackagesAdvancedOptions.md)
-document for more advanced use cases.
diff --git a/quickstart/recommendation/tensorflow/wide_deep_large_ds/training/cpu/.docs/baremetal.md b/quickstart/recommendation/tensorflow/wide_deep_large_ds/training/cpu/.docs/baremetal.md
deleted file mode 100644
index 3e8cb641c..000000000
--- a/quickstart/recommendation/tensorflow/wide_deep_large_ds/training/cpu/.docs/baremetal.md
+++ /dev/null
@@ -1,56 +0,0 @@
-<!--- 50. AI Kit -->
-## Run the model
-
-Setup your environment using the instructions below, depending on if you are
-using [AI Kit](/docs/general/tensorflow/AIKit.md):
-
-<table>
-  <tr>
-    <th>Setup using AI Kit</th>
-    <th>Setup without AI Kit</th>
-  </tr>
-  <tr>
-    <td>
-      <p>To run using AI Kit you will need:</p>
-      <ul>
-        <li>numactl
-        <li>wget
-        <li>Activate the `tensorflow` conda environment
-        <pre>conda activate tensorflow</pre>
-      </ul>
-    </td>
-    <td>
-      <p>To run without AI Kit you will need:</p>
-      <ul>
-        <li>Python 3
-        <li>intel-tensorflow>=2.5.0
-        <li>numactl
-        <li>git
-        <li>wget
-        <li>A clone of the Model Zoo repo<br />
-        <pre>git clone https://github.com/IntelAI/models.git</pre>
-      </ul>
-    </td>
-  </tr>
-</table>
-
-After the setup is complete, set environment variables for the path to your
-dataset directory and an output directory where logs will be written. You can
-optionally provide a directory where checkpoint files will be read and
-written. Navigate to your model zoo directory, then select a
-[quickstart script](#quick-start-scripts) to run. Note that some quickstart
-scripts might use other environment variables in addition to the ones below,
-like `STEPS` and `TARGET_ACCURACY` for the `fp32_training_check_accuracy.sh` script.
-```
-# cd to your model zoo directory
-cd models
-
-export DATASET_DIR=<path to the dataset directory>
-export PRECISION=fp32
-export OUTPUT_DIR=<path to the directory where the logs and the saved model will be written>
-export CHECKPOINT_DIR=<Optional directory where checkpoint files will be read and written>
-# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
-export BATCH_SIZE=<customized batch size value>
-
-./quickstart/recommendation/tensorflow/wide_deep_large_ds/training/cpu/<script name>.sh
-```
diff --git a/quickstart/recommendation/tensorflow/wide_deep_large_ds/training/cpu/.docs/datasets.md b/quickstart/recommendation/tensorflow/wide_deep_large_ds/training/cpu/.docs/datasets.md
deleted file mode 100644
index 35ae88d7e..000000000
--- a/quickstart/recommendation/tensorflow/wide_deep_large_ds/training/cpu/.docs/datasets.md
+++ /dev/null
@@ -1,17 +0,0 @@
-<!--- 30. Datasets -->
-## Dataset
-
-The Large [Kaggle Display Advertising Challenge Dataset](https://www.kaggle.com/c/criteo-display-ad-challenge/data)
-will be used for training Wide and Deep. The [data](https://www.kaggle.com/c/criteo-display-ad-challenge/data) is from
-[Criteo](https://www.criteo.com) and has a field indicating if an ad was
-clicked (1) or not (0), along with integer and categorical features.
-
-Download the Large Kaggle Display Advertising Challenge Dataset from [Criteo Labs](http://labs.criteo.com/2014/02/kaggle-display-advertising-challenge-dataset/) in `$DATASET_DIR`.
-If the evaluation/train dataset were not available in the above link, it can be downloaded as follow:
-   ```
-    export DATASET_DIR=<location where dataset files will be saved>
-    mkdir $DATASET_DIR && cd $DATASET_DIR
-    wget https://storage.googleapis.com/dataset-uploader/criteo-kaggle/large_version/eval.csv
-    wget https://storage.googleapis.com/dataset-uploader/criteo-kaggle/large_version/train.csv
-   ```
-The `DATASET_DIR` environment variable will be used as the dataset directory when running [quickstart scripts](#quick-start-scripts).
diff --git a/quickstart/recommendation/tensorflow/wide_deep_large_ds/training/cpu/.docs/description.md b/quickstart/recommendation/tensorflow/wide_deep_large_ds/training/cpu/.docs/description.md
deleted file mode 100644
index 708765e56..000000000
--- a/quickstart/recommendation/tensorflow/wide_deep_large_ds/training/cpu/.docs/description.md
+++ /dev/null
@@ -1,5 +0,0 @@
-<!-- 10. Description -->
-
-This document has instructions for training [Wide and Deep](https://arxiv.org/pdf/1606.07792.pdf)
-using a large dataset using Intel-optimized TensorFlow.
-
diff --git a/quickstart/recommendation/tensorflow/wide_deep_large_ds/training/cpu/.docs/docker.md b/quickstart/recommendation/tensorflow/wide_deep_large_ds/training/cpu/.docs/docker.md
deleted file mode 100644
index f40b5068d..000000000
--- a/quickstart/recommendation/tensorflow/wide_deep_large_ds/training/cpu/.docs/docker.md
+++ /dev/null
@@ -1,38 +0,0 @@
-<!-- 60. Docker -->
-## Docker
-
-The model container used in the example below includes the scripts and
-libraries needed to run <model name> <mode>. To run one of the
-model quickstart scripts using this container, you'll need to provide
-volume mounts for the [dataset](#dataset), checkpoints, and an output
-directory where logs and the saved model will be written.
-```
-DATASET_DIR=<path to the dataset directory>
-PRECISION=fp32
-OUTPUT_DIR=<directory where the logs and the saved model will be written>
-CHECKPOINT_DIR=<directory where checkpoint files will be read and written>
-# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
-export BATCH_SIZE=<customized batch size value>
-
-docker run \
-  --env DATASET_DIR=${DATASET_DIR} \
-  --env PRECISION=fp32 \
-  --env OUTPUT_DIR=${OUTPUT_DIR} \
-  --env BATCH_SIZE=${BATCH_SIZE} \
-  --env CHECKPOINT_DIR=${CHECKPOINT_DIR} \
-  --env http_proxy=${http_proxy} \
-  --env https_proxy=${https_proxy} \
-  --volume ${DATASET_DIR}:${DATASET_DIR} \
-  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
-  --volume ${CHECKPOINT_DIR}:${CHECKPOINT_DIR} \
-  --privileged --init -t \
-  <docker image> \
-  /bin/bash quickstart/<script name>.sh
-```
-
-The script will write a log file and the saved model to the `OUTPUT_DIR`
-and checkpoints will be written to the `CHECKPOINT_DIR`.
-
-If you are new to docker and are running into issues with the container,
-see [this document](https://github.com/IntelAI/models/tree/master/docs/general/docker.md)
-for troubleshooting tips.
diff --git a/quickstart/recommendation/tensorflow/wide_deep_large_ds/training/cpu/.docs/license.md b/quickstart/recommendation/tensorflow/wide_deep_large_ds/training/cpu/.docs/license.md
deleted file mode 100644
index a60f169f4..000000000
--- a/quickstart/recommendation/tensorflow/wide_deep_large_ds/training/cpu/.docs/license.md
+++ /dev/null
@@ -1,4 +0,0 @@
-<!--- 80. License -->
-## License
-
-[LICENSE](/LICENSE)
diff --git a/quickstart/recommendation/tensorflow/wide_deep_large_ds/training/cpu/.docs/quickstart.md b/quickstart/recommendation/tensorflow/wide_deep_large_ds/training/cpu/.docs/quickstart.md
deleted file mode 100644
index 36a4501e9..000000000
--- a/quickstart/recommendation/tensorflow/wide_deep_large_ds/training/cpu/.docs/quickstart.md
+++ /dev/null
@@ -1,8 +0,0 @@
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| [`training_check_accuracy.sh`](/quickstart/recommendation/tensorflow/wide_deep_large_ds/training/cpu/training_check_accuracy.sh) | Trains the model for a specified number of steps (default is 500) and then compare the accuracy against the accuracy specified in the `TARGET_ACCURACY` env var (ex: `export TARGET_ACCURACY=0.75`). If the accuracy is not met, then script exits with error code 1. The `CHECKPOINT_DIR` environment variable can optionally be defined to start training based on previous set of checkpoints. |
-| [`training.sh`](/quickstart/recommendation/tensorflow/wide_deep_large_ds/training/cpu/training.sh) | Trains the model for 10 epochs. The `CHECKPOINT_DIR` environment variable can optionally be defined to start training based on previous set of checkpoints. |
-| [`training_demo.sh`](/quickstart/recommendation/tensorflow/wide_deep_large_ds/training/cpu/training_demo.sh) | A short demo run that trains the model for 100 steps. |
diff --git a/quickstart/recommendation/tensorflow/wide_deep_large_ds/training/cpu/.docs/title.md b/quickstart/recommendation/tensorflow/wide_deep_large_ds/training/cpu/.docs/title.md
deleted file mode 100644
index 1d6c60187..000000000
--- a/quickstart/recommendation/tensorflow/wide_deep_large_ds/training/cpu/.docs/title.md
+++ /dev/null
@@ -1,2 +0,0 @@
-<!--- 0. Title -->
-# <model name> <mode>
diff --git a/quickstart/recommendation/tensorflow/wide_deep_large_ds/training/cpu/README.md b/quickstart/recommendation/tensorflow/wide_deep_large_ds/training/cpu/README.md
deleted file mode 100644
index 460c8239b..000000000
--- a/quickstart/recommendation/tensorflow/wide_deep_large_ds/training/cpu/README.md
+++ /dev/null
@@ -1,85 +0,0 @@
-<!--- 0. Title -->
-# Wide and Deep Large Dataset training
-
-<!-- 10. Description -->
-
-This document has instructions for training [Wide and Deep](https://arxiv.org/pdf/1606.07792.pdf)
-using a large dataset using Intel-optimized TensorFlow.
-
-
-<!--- 30. Datasets -->
-## Dataset
-
-The Large [Kaggle Display Advertising Challenge Dataset](https://www.kaggle.com/c/criteo-display-ad-challenge/data)
-will be used for training Wide and Deep. The [data](https://www.kaggle.com/c/criteo-display-ad-challenge/data) is from
-[Criteo](https://www.criteo.com) and has a field indicating if an ad was
-clicked (1) or not (0), along with integer and categorical features.
-
-Download the Large Kaggle Display Advertising Challenge Dataset from [Criteo Labs](http://labs.criteo.com/2014/02/kaggle-display-advertising-challenge-dataset/) in `$DATASET_DIR`.
-If the evaluation/train dataset were not available in the above link, it can be downloaded as follow:
-   ```
-    export DATASET_DIR=<location where dataset files will be saved>
-    mkdir $DATASET_DIR && cd $DATASET_DIR
-    wget https://storage.googleapis.com/dataset-uploader/criteo-kaggle/large_version/eval.csv
-    wget https://storage.googleapis.com/dataset-uploader/criteo-kaggle/large_version/train.csv
-   ```
-The `DATASET_DIR` environment variable will be used as the dataset directory when running [quickstart scripts](#quick-start-scripts).
-
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| [`training_check_accuracy.sh`](/quickstart/recommendation/tensorflow/wide_deep_large_ds/training/cpu/training_check_accuracy.sh) | Trains the model for a specified number of steps (default is 500) and then compare the accuracy against the accuracy specified in the `TARGET_ACCURACY` env var (ex: `export TARGET_ACCURACY=0.75`). If the accuracy is not met, then script exits with error code 1. The `CHECKPOINT_DIR` environment variable can optionally be defined to start training based on previous set of checkpoints. |
-| [`training.sh`](/quickstart/recommendation/tensorflow/wide_deep_large_ds/training/cpu/training.sh) | Trains the model for 10 epochs. The `CHECKPOINT_DIR` environment variable can optionally be defined to start training based on previous set of checkpoints. |
-| [`training_demo.sh`](/quickstart/recommendation/tensorflow/wide_deep_large_ds/training/cpu/training_demo.sh) | A short demo run that trains the model for 100 steps. |
-
-<!-- 60. Docker -->
-## Docker
-
-The model container used in the example below includes the scripts and
-libraries needed to run Wide and Deep Large Dataset training. To run one of the
-model quickstart scripts using this container, you'll need to provide
-volume mounts for the [dataset](#dataset), checkpoints, and an output
-directory where logs and the saved model will be written.
-```
-DATASET_DIR=<path to the dataset directory>
-PRECISION=fp32
-OUTPUT_DIR=<directory where the logs and the saved model will be written>
-CHECKPOINT_DIR=<directory where checkpoint files will be read and written>
-# For a custom batch size, set env var `BATCH_SIZE` or it will run with a default value.
-export BATCH_SIZE=<customized batch size value>
-
-docker run \
-  --env DATASET_DIR=${DATASET_DIR} \
-  --env PRECISION=fp32 \
-  --env OUTPUT_DIR=${OUTPUT_DIR} \
-  --env BATCH_SIZE=${BATCH_SIZE} \
-  --env CHECKPOINT_DIR=${CHECKPOINT_DIR} \
-  --env http_proxy=${http_proxy} \
-  --env https_proxy=${https_proxy} \
-  --volume ${DATASET_DIR}:${DATASET_DIR} \
-  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
-  --volume ${CHECKPOINT_DIR}:${CHECKPOINT_DIR} \
-  --privileged --init -t \
-  intel/recommendation:tf-latest-wide-deep-large-ds-training \
-  /bin/bash quickstart/<script name>.sh
-```
-
-The script will write a log file and the saved model to the `OUTPUT_DIR`
-and checkpoints will be written to the `CHECKPOINT_DIR`.
-
-If you are new to docker and are running into issues with the container,
-see [this document](https://github.com/IntelAI/models/tree/master/docs/general/docker.md)
-for troubleshooting tips.
-
-<!-- 61. Advanced Options -->
-
-See the [Advanced Options for Model Packages and Containers](/quickstart/common/tensorflow/ModelPackagesAdvancedOptions.md)
-document for more advanced use cases.
-
-<!--- 80. License -->
-## License
-
-[LICENSE](/LICENSE)
-
diff --git a/quickstart/recommendation/tensorflow/wide_deep_large_ds/training/cpu/training.sh b/quickstart/recommendation/tensorflow/wide_deep_large_ds/training/cpu/training.sh
deleted file mode 100755
index cd79e11c8..000000000
--- a/quickstart/recommendation/tensorflow/wide_deep_large_ds/training/cpu/training.sh
+++ /dev/null
@@ -1,76 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2020 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-MODEL_DIR=${MODEL_DIR-$PWD}
-
-if [ -z "${OUTPUT_DIR}" ]; then
-  echo "The required environment variable OUTPUT_DIR has not been set"
-  exit 1
-fi
-
-# Create the output directory in case it doesn't already exist
-mkdir -p ${OUTPUT_DIR}
-
-if [ -z "${DATASET_DIR}" ]; then
-  echo "The required environment variable DATASET_DIR has not been set"
-  exit 1
-fi
-
-if [ ! -d "${DATASET_DIR}" ]; then
-  echo "The DATASET_DIR '${DATASET_DIR}' does not exist"
-  exit 1
-fi
-
-# If precision env is not mentioned, then the workload will run with the default precision.
-if [ -z "${PRECISION}"]; then
-  PRECISION=fp32
-  echo "Running with default precision ${PRECISION}"
-fi
-
-if [[ $PRECISION != "fp32" ]]; then
-  echo "The specified precision '${PRECISION}' is unsupported."
-  echo "Supported precision is fp32."
-  exit 1
-fi
-
-# Define a checkpoint arg, if CHECKPOINT_DIR was provided
-CHECKPOINT_ARG=""
-if [ ! -z "${CHECKPOINT_DIR}" ]; then
-  # If a checkpoint dir was provided, ensure that it exists, then setup the arg
-  mkdir -p ${CHECKPOINT_DIR}
-  CHECKPOINT_ARG="--checkpoint=${CHECKPOINT_DIR}"
-fi
-
-# If batch size env is not mentioned, then the workload will run with the default batch size.
-if [ -z "${BATCH_SIZE}"]; then
-  BATCH_SIZE="512"
-  echo "Running with default batch size of ${BATCH_SIZE}"
-fi
-
-# Run wide and deep large dataset training
-source "$MODEL_DIR/quickstart/common/utils.sh"
-_command python ${MODEL_DIR}/benchmarks/launch_benchmark.py \
-   --model-name wide_deep_large_ds \
-   --precision ${PRECISION} \
-   --mode training  \
-   --framework tensorflow \
-   --batch-size ${BATCH_SIZE} \
-   --data-location $DATASET_DIR \
-   $CHECKPOINT_ARG \
-   --output-dir $OUTPUT_DIR \
-   $@
-
diff --git a/quickstart/recommendation/tensorflow/wide_deep_large_ds/training/cpu/training_check_accuracy.sh b/quickstart/recommendation/tensorflow/wide_deep_large_ds/training/cpu/training_check_accuracy.sh
deleted file mode 100755
index e43df0046..000000000
--- a/quickstart/recommendation/tensorflow/wide_deep_large_ds/training/cpu/training_check_accuracy.sh
+++ /dev/null
@@ -1,124 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2020 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-# This script trains the model for a specified number of steps, then compares
-# the accuracy that is printed in the log file against a target accuracy. If
-# the evaluated accuracy is less than the target, the script will fail.
-#
-# Define the target accuracy using the TARGET_ACCURACY environment variable
-# (for example TARGET_ACCURACY=0.75). Set the number of steps to run using the
-# STEPS environment variable (this will default to run 500 steps, if the
-# variable has not been set).
-
-MODEL_DIR=${MODEL_DIR-$PWD}
-
-if [ -z "${OUTPUT_DIR}" ]; then
-  echo "The required environment variable OUTPUT_DIR has not been set"
-  exit 1
-fi
-
-# Create the output directory in case it doesn't already exist
-mkdir -p ${OUTPUT_DIR}
-
-if [ -z "${DATASET_DIR}" ]; then
-  echo "The required environment variable DATASET_DIR has not been set"
-  exit 1
-fi
-
-if [ ! -d "${DATASET_DIR}" ]; then
-  echo "The DATASET_DIR '${DATASET_DIR}' does not exist"
-  exit 1
-fi
-
-# If precision env is not mentioned, then the workload will run with the default precision.
-if [ -z "${PRECISION}"]; then
-  PRECISION=fp32
-  echo "Running with default precision ${PRECISION}"
-fi
-
-if [[ $PRECISION != "fp32" ]]; then
-  echo "The specified precision '${PRECISION}' is unsupported."
-  echo "Supported precision is fp32."
-  exit 1
-fi
-
-if [ ! -z "${TARGET_ACCURACY}" ]; then
-  echo "Target Accuracy: ${TARGET_ACCURACY}"
-fi
-
-# Define a checkpoint arg, if CHECKPOINT_DIR was provided
-CHECKPOINT_ARG=""
-if [ ! -z "${CHECKPOINT_DIR}" ]; then
-  # If a checkpoint dir was provided, ensure that it exists, then setup the arg
-  mkdir -p ${CHECKPOINT_DIR}
-  CHECKPOINT_ARG="--checkpoint=${CHECKPOINT_DIR}"
-fi
-
-# Set default number of steps
-STEPS=${STEPS:-500}
-
-# If batch size env is not mentioned, then the workload will run with the default batch size.
-if [ -z "${BATCH_SIZE}"]; then
-  BATCH_SIZE="512"
-  echo "Running with default batch size of ${BATCH_SIZE}"
-fi
-
-# Run wide and deep large ds training while saving output to a temp file
-temp_output_file=$(mktemp /tmp/output.XXXXXXXXX)
-source "$MODEL_DIR/quickstart/common/utils.sh"
-_command python ${MODEL_DIR}/benchmarks/launch_benchmark.py \
- --model-name wide_deep_large_ds \
- --precision ${PRECISION} \
- --mode training  \
- --framework tensorflow \
- --batch-size ${BATCH_SIZE} \
- --data-location $DATASET_DIR \
- $CHECKPOINT_ARG \
- --output-dir $OUTPUT_DIR \
- $@ \
- -- steps=$STEPS 2>&1 | tee $temp_output_file
-
-# If a target accuracy has been specified, find the accuracy from the log file
-if [ ! -z "${TARGET_ACCURACY}" ]; then
-  # Get the accuracy output from the training script
-  accuracy_output=$(grep 'Accuracy:' $temp_output_file)
-  accuracy_output=$(echo $accuracy_output | sed 's/^Accuracy: //')
-fi
-
-# Remove the temp output file
-if [[ -f $temp_output_file ]]; then
-  rm $temp_output_file
-fi
-
-# If a target accuracy has been specified, compare the actual accuracy to the
-# target, and fail if the accuracy target has not been met
-if [ ! -z "${TARGET_ACCURACY}" ]; then
-  if [[ -z $accuracy_output ]]; then
-    echo "Unable to find accuracy metric in the log file"
-    exit 1
-  fi
-
-  # Compare the accuracy to the target accuracy - 1 means the accuracy target was met
-  accuracy_met=$(python -c "print(int($accuracy_output >= $TARGET_ACCURACY))")
-  if [[ $accuracy_met == 1 ]]; then
-    echo "Accuracy of $accuracy_output met the target accuracy of $TARGET_ACCURACY"
-    exit 0
-  else
-    echo "Accuracy of $accuracy_output did not meet the target accuracy of $TARGET_ACCURACY"
-    exit 1
-  fi
-fi
diff --git a/quickstart/recommendation/tensorflow/wide_deep_large_ds/training/cpu/training_demo.sh b/quickstart/recommendation/tensorflow/wide_deep_large_ds/training/cpu/training_demo.sh
deleted file mode 100755
index f58ea4644..000000000
--- a/quickstart/recommendation/tensorflow/wide_deep_large_ds/training/cpu/training_demo.sh
+++ /dev/null
@@ -1,80 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2020 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-MODEL_DIR=${MODEL_DIR-$PWD}
-
-if [ -z "${OUTPUT_DIR}" ]; then
-  echo "The required environment variable OUTPUT_DIR has not been set"
-  exit 1
-fi
-
-# Create the output directory in case it doesn't already exist
-mkdir -p ${OUTPUT_DIR}
-
-if [ -z "${DATASET_DIR}" ]; then
-  echo "The required environment variable DATASET_DIR has not been set"
-  exit 1
-fi
-
-if [ ! -d "${DATASET_DIR}" ]; then
-  echo "The DATASET_DIR '${DATASET_DIR}' does not exist"
-  exit 1
-fi
-
-# If precision env is not mentioned, then the workload will run with the default precision.
-if [ -z "${PRECISION}"]; then
-  PRECISION=fp32
-  echo "Running with default precision ${PRECISION}"
-fi
-
-if [[ $PRECISION != "fp32" ]]; then
-  echo "The specified precision '${PRECISION}' is unsupported."
-  echo "Supported precision is fp32."
-  exit 1
-fi
-
-# Define a checkpoint arg, if CHECKPOINT_DIR was provided
-CHECKPOINT_ARG=""
-if [ ! -z "${CHECKPOINT_DIR}" ]; then
-  # If a checkpoint dir was provided, ensure that it exists, then setup the arg
-  mkdir -p ${CHECKPOINT_DIR}
-  CHECKPOINT_ARG="--checkpoint=${CHECKPOINT_DIR}"
-fi
-
-# Use for 100 steps for a demo training run
-STEPS=${STEPS-100}
-
-# If batch size env is not mentioned, then the workload will run with the default batch size.
-if [ -z "${BATCH_SIZE}"]; then
-  BATCH_SIZE="512"
-  echo "Running with default batch size of ${BATCH_SIZE}"
-fi
-
-# Run wide and deep large dataset training
-source "$MODEL_DIR/quickstart/common/utils.sh"
-_command python ${MODEL_DIR}/benchmarks/launch_benchmark.py \
- --model-name wide_deep_large_ds \
- --precision ${PRECISION} \
- --mode training  \
- --framework tensorflow \
- --batch-size ${BATCH_SIZE} \
- --data-location $DATASET_DIR \
-$CHECKPOINT_ARG \
- --output-dir $OUTPUT_DIR \
- $@ \
- -- steps=${STEPS}
-
diff --git a/quickstart/spr_base/pytorch/.docs/container_package.md b/quickstart/spr_base/pytorch/.docs/container_package.md
deleted file mode 100644
index 6f658dd56..000000000
--- a/quickstart/spr_base/pytorch/.docs/container_package.md
+++ /dev/null
@@ -1,15 +0,0 @@
-## Container Package
-
-The container package includes wheels for PyTorch and the Intel PyTorch
-Extension (IPEX), a Dockerfile, and a script to build the container.
-
-```
-<package dir>
-├── README.md
-├── build.sh
-├── licenses
-│   ├── LICENSE
-│   └── third_party
-├── pytorch-ipex-spr.Dockerfile
-└── whls
-```
diff --git a/quickstart/spr_base/pytorch/.docs/description.md b/quickstart/spr_base/pytorch/.docs/description.md
deleted file mode 100644
index 820ca82cd..000000000
--- a/quickstart/spr_base/pytorch/.docs/description.md
+++ /dev/null
@@ -1,6 +0,0 @@
-<!-- 10. Description -->
-## Description
-
-This document has instructions for building and running the Intel-optimized
-PyTorch container using the container package. This container is used as the
-base for the PyTorch model containers.
diff --git a/quickstart/spr_base/pytorch/.docs/docker.md b/quickstart/spr_base/pytorch/.docs/docker.md
deleted file mode 100644
index a5e04af46..000000000
--- a/quickstart/spr_base/pytorch/.docs/docker.md
+++ /dev/null
@@ -1,49 +0,0 @@
-## Docker
-
-### Build the container
-
-Extract the `<package name>` package and then run the `build.sh` script
-to build the <docker image> conatiner. Note that the size of the conatainer
-after it's built is about 8.31GB. The container is based on CentOS 8.3.2011
-and includes anaconda, PyTorch, Intel Extensions for PyTorch, MKL, and other
-dependencies needed to run PyTorch models.
-```
-tar -xzf <package name>
-cd <package dir>
-
-./build.sh
-```
-
-### Running the container
-
-The following command can be used to interactively run the PyTorch/IPEX
-container. You can optionally use the `-v` (or `--volume`) option to mount
-your local directory into the container.
-```
-docker run \
-    -v <your-local-dir>:/workspace \
-    -e http_proxy=${http_proxy} \
-    -e https_proxy=${https_proxy} \
-    -e no_proxy=${no_proxy} \
-    -it model-zoo:pytorch-ipex-spr /bin/bash
-```
-
-Once you're in the container, activate the pytorch environment:
-```
-# List the available conda environments
-conda env list
-
-# Activate the pytorch environment
-source activate pytorch
-```
-
-Run the following command to verify that you're able to import PyTorch and
-IPEX from python:
-```
-python -c "import torch"
-
-python -c "import intel_extension_for_pytorch"
-```
-
-After verifying that PyTorch and IPEX are available, you can run your own
-script in the container.
diff --git a/quickstart/spr_base/pytorch/.docs/license.md b/quickstart/spr_base/pytorch/.docs/license.md
deleted file mode 100644
index 53614fdc5..000000000
--- a/quickstart/spr_base/pytorch/.docs/license.md
+++ /dev/null
@@ -1,4 +0,0 @@
-<!--- 80. License -->
-## License
-
-Licenses can be found in the container package, in the `licenses` directory.
diff --git a/quickstart/spr_base/pytorch/.docs/title.md b/quickstart/spr_base/pytorch/.docs/title.md
deleted file mode 100644
index ced6c8fe2..000000000
--- a/quickstart/spr_base/pytorch/.docs/title.md
+++ /dev/null
@@ -1,2 +0,0 @@
-<!--- 0. Title -->
-# Intel PyTorch Extension (IPEX) tools container
diff --git a/quickstart/spr_base/pytorch/README.md b/quickstart/spr_base/pytorch/README.md
deleted file mode 100644
index f2433a172..000000000
--- a/quickstart/spr_base/pytorch/README.md
+++ /dev/null
@@ -1,81 +0,0 @@
-<!--- 0. Title -->
-# Intel PyTorch Extension (IPEX) tools container
-
-<!-- 10. Description -->
-## Description
-
-This document has instructions for building and running the Intel-optimized
-PyTorch container using the container package. This container is used as the
-base for the PyTorch model containers.
-
-## Container Package
-
-The container package includes wheels for PyTorch and the Intel PyTorch
-Extension (IPEX), a Dockerfile, and a script to build the container.
-
-```
-pytorch-ipex-spr
-├── README.md
-├── build.sh
-├── licenses
-│   ├── LICENSE
-│   └── third_party
-├── pytorch-ipex-spr.Dockerfile
-└── whls
-```
-
-## Docker
-
-### Build the container
-
-Extract the `pytorch-ipex-spr.tar.gz` package and then run the `build.sh` script
-to build the model-zoo:pytorch-ipex-spr conatiner. Note that the size of the conatainer
-after it's built is about 8.31GB. The container is based on CentOS 8.3.2011
-and includes anaconda, PyTorch, Intel Extensions for PyTorch, MKL, and other
-dependencies needed to run PyTorch models.
-```
-tar -xzf pytorch-ipex-spr.tar.gz
-cd pytorch-ipex-spr
-
-./build.sh
-```
-
-### Running the container
-
-The following command can be used to interactively run the PyTorch/IPEX
-container. You can optionally use the `-v` (or `--volume`) option to mount
-your local directory into the container.
-```
-docker run \
-    -v <your-local-dir>:/workspace \
-    -e http_proxy=${http_proxy} \
-    -e https_proxy=${https_proxy} \
-    -e no_proxy=${no_proxy} \
-    -it model-zoo:pytorch-ipex-spr /bin/bash
-```
-
-Once you're in the container, activate the pytorch environment:
-```
-# List the available conda environments
-conda env list
-
-# Activate the pytorch environment
-source activate pytorch
-```
-
-Run the following command to verify that you're able to import PyTorch and
-IPEX from python:
-```
-python -c "import torch"
-
-python -c "import intel_extension_for_pytorch"
-```
-
-After verifying that PyTorch and IPEX are available, you can run your own
-script in the container.
-
-<!--- 80. License -->
-## License
-
-Licenses can be found in the container package, in the `licenses` directory.
-
diff --git a/quickstart/spr_base/pytorch/build.sh b/quickstart/spr_base/pytorch/build.sh
deleted file mode 100755
index d46f1106d..000000000
--- a/quickstart/spr_base/pytorch/build.sh
+++ /dev/null
@@ -1,25 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2021 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-set -e
-
-IMAGE_NAME=model-zoo:pytorch-ipex-spr
-docker build --build-arg http_proxy=$http_proxy \
-             --build-arg https_proxy=$https_proxy  \
-             --build-arg no_proxy=$no_proxy \
-             -t $IMAGE_NAME \
-             -f pytorch-ipex-spr.Dockerfile .
diff --git a/quickstart/spr_base/tensorflow/.docs/container_package.md b/quickstart/spr_base/tensorflow/.docs/container_package.md
deleted file mode 100644
index 213baa986..000000000
--- a/quickstart/spr_base/tensorflow/.docs/container_package.md
+++ /dev/null
@@ -1,15 +0,0 @@
-## Container Package
-
-The container package includes wheel for TensorFlow, a Dockerfile, and a script to build the container.
-
-```
-<package dir>
-├── README.md
-├── build.sh
-├── licenses
-│   ├── LICENSE
-│   └── third_party
-├── <package dir>.Dockerfile
-└── whls
-    └── <wheel name>
-```
diff --git a/quickstart/spr_base/tensorflow/.docs/description.md b/quickstart/spr_base/tensorflow/.docs/description.md
deleted file mode 100644
index 5c7ec91ba..000000000
--- a/quickstart/spr_base/tensorflow/.docs/description.md
+++ /dev/null
@@ -1,5 +0,0 @@
-<!-- 10. Description -->
-## Description
-
-This document has instructions for building and running the Intel-optimized
-TensorFlow container using the container package.
diff --git a/quickstart/spr_base/tensorflow/.docs/docker.md b/quickstart/spr_base/tensorflow/.docs/docker.md
deleted file mode 100644
index ee5c8632e..000000000
--- a/quickstart/spr_base/tensorflow/.docs/docker.md
+++ /dev/null
@@ -1,36 +0,0 @@
-## Docker
-
-### Build the container
-
-Extract the `<package name>` package and then run the `build.sh` script
-to build the `<docker image>` container:
-
-```
-tar -xzf <package name>
-cd <package dir>
-./build.sh
-```
-
-### Running the container
-
-The following command can be used to interactively run the TensorFlow
-container. You can optionally use the `-v` (or `--volume`) option to mount
-your local directory into the container.
-```
-docker run \
-    -v <your-local-dir>:/workspace \
-    -e http_proxy=${http_proxy} \
-    -e https_proxy=${https_proxy} \
-    -e no_proxy=${no_proxy} \
-    -it <docker image> /bin/bash
-```
-
-Once you're in the container, Run the following command to verify that you're able to import tensorflow
-and check tensorflow version:
-
-```
-python -c "import tensorflow as tf; print(tf.__version__)"
-```
-
-After verifying that TensorFlow is available, you can run your own
-script in the container.
diff --git a/quickstart/spr_base/tensorflow/.docs/license.md b/quickstart/spr_base/tensorflow/.docs/license.md
deleted file mode 100644
index 53614fdc5..000000000
--- a/quickstart/spr_base/tensorflow/.docs/license.md
+++ /dev/null
@@ -1,4 +0,0 @@
-<!--- 80. License -->
-## License
-
-Licenses can be found in the container package, in the `licenses` directory.
diff --git a/quickstart/spr_base/tensorflow/.docs/title.md b/quickstart/spr_base/tensorflow/.docs/title.md
deleted file mode 100644
index 89860ea66..000000000
--- a/quickstart/spr_base/tensorflow/.docs/title.md
+++ /dev/null
@@ -1,2 +0,0 @@
-<!--- 0. Title -->
-# Intel TensorFlow tools container
diff --git a/quickstart/spr_base/tensorflow/3dunet_base_README.md b/quickstart/spr_base/tensorflow/3dunet_base_README.md
deleted file mode 100644
index f8c4f8891..000000000
--- a/quickstart/spr_base/tensorflow/3dunet_base_README.md
+++ /dev/null
@@ -1,67 +0,0 @@
-<!--- 0. Title -->
-# Intel TensorFlow tools container
-
-<!-- 10. Description -->
-## Description
-
-This document has instructions for building and running the Intel-optimized
-TensorFlow container using the container package.
-
-## Container Package
-
-The container package includes wheel for TensorFlow, a Dockerfile, and a script to build the container.
-
-```
-tensorflow-spr-3dunet
-├── README.md
-├── build.sh
-├── licenses
-│   ├── LICENSE
-│   └── third_party
-├── tensorflow-spr-3dunet.Dockerfile
-└── whls
-    └── tf_nightly-2.9.0.202212-cp38-cp38-linux_x86_64.whl
-```
-
-## Docker
-
-### Build the container
-
-Extract the `tensorflow-spr-3dunet.tar.gz` package and then run the `build.sh` script
-to build the `model-zoo:tensorflow-spr-3dunet` container:
-
-```
-tar -xzf tensorflow-spr-3dunet.tar.gz
-cd tensorflow-spr-3dunet
-./build.sh
-```
-
-### Running the container
-
-The following command can be used to interactively run the TensorFlow
-container. You can optionally use the `-v` (or `--volume`) option to mount
-your local directory into the container.
-```
-docker run \
-    -v <your-local-dir>:/workspace \
-    -e http_proxy=${http_proxy} \
-    -e https_proxy=${https_proxy} \
-    -e no_proxy=${no_proxy} \
-    -it model-zoo:tensorflow-spr-3dunet /bin/bash
-```
-
-Once you're in the container, Run the following command to verify that you're able to import tensorflow
-and check tensorflow version:
-
-```
-python -c "import tensorflow as tf; print(tf.__version__)"
-```
-
-After verifying that TensorFlow is available, you can run your own
-script in the container.
-
-<!--- 80. License -->
-## License
-
-Licenses can be found in the container package, in the `licenses` directory.
-
diff --git a/quickstart/spr_base/tensorflow/README.md b/quickstart/spr_base/tensorflow/README.md
deleted file mode 100644
index 1a9d72795..000000000
--- a/quickstart/spr_base/tensorflow/README.md
+++ /dev/null
@@ -1,67 +0,0 @@
-<!--- 0. Title -->
-# Intel TensorFlow tools container
-
-<!-- 10. Description -->
-## Description
-
-This document has instructions for building and running the Intel-optimized
-TensorFlow container using the container package.
-
-## Container Package
-
-The container package includes wheel for TensorFlow, a Dockerfile, and a script to build the container.
-
-```
-tensorflow-spr
-├── README.md
-├── build.sh
-├── licenses
-│   ├── LICENSE
-│   └── third_party
-├── tensorflow-spr.Dockerfile
-└── whls
-    └── tf_nightly-2.10.0.202218-cp38-cp38-linux_x86_64.whl
-```
-
-## Docker
-
-### Build the container
-
-Extract the `tensorflow-spr.tar.gz` package and then run the `build.sh` script
-to build the `model-zoo:tensorflow-spr` container:
-
-```
-tar -xzf tensorflow-spr.tar.gz
-cd tensorflow-spr
-./build.sh
-```
-
-### Running the container
-
-The following command can be used to interactively run the TensorFlow
-container. You can optionally use the `-v` (or `--volume`) option to mount
-your local directory into the container.
-```
-docker run \
-    -v <your-local-dir>:/workspace \
-    -e http_proxy=${http_proxy} \
-    -e https_proxy=${https_proxy} \
-    -e no_proxy=${no_proxy} \
-    -it model-zoo:tensorflow-spr /bin/bash
-```
-
-Once you're in the container, Run the following command to verify that you're able to import tensorflow
-and check tensorflow version:
-
-```
-python -c "import tensorflow as tf; print(tf.__version__)"
-```
-
-After verifying that TensorFlow is available, you can run your own
-script in the container.
-
-<!--- 80. License -->
-## License
-
-Licenses can be found in the container package, in the `licenses` directory.
-
diff --git a/quickstart/spr_base/tensorflow/build.sh b/quickstart/spr_base/tensorflow/build.sh
deleted file mode 100755
index 0bc74d617..000000000
--- a/quickstart/spr_base/tensorflow/build.sh
+++ /dev/null
@@ -1,25 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2021 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-set -e
-
-IMAGE_NAME=model-zoo:tensorflow-spr
-docker build --build-arg http_proxy=$http_proxy \
-             --build-arg https_proxy=$https_proxy  \
-             --build-arg no_proxy=$no_proxy \
-             -t $IMAGE_NAME \
-             -f tensorflow-spr.Dockerfile .
diff --git a/quickstart/spr_base/tensorflow/build_3dunet_base.sh b/quickstart/spr_base/tensorflow/build_3dunet_base.sh
deleted file mode 100755
index 6f93d9ae1..000000000
--- a/quickstart/spr_base/tensorflow/build_3dunet_base.sh
+++ /dev/null
@@ -1,25 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2021 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-set -e
-
-IMAGE_NAME=model-zoo:tensorflow-spr-3dunet
-docker build --build-arg http_proxy=$http_proxy \
-             --build-arg https_proxy=$https_proxy  \
-             --build-arg no_proxy=$no_proxy \
-             -t $IMAGE_NAME \
-             -f tensorflow-spr-3dunet.Dockerfile .
diff --git a/quickstart/text_to_speech/tensorflow/wavenet/inference/cpu/fp32/.docs/aikit.md b/quickstart/text_to_speech/tensorflow/wavenet/inference/cpu/fp32/.docs/aikit.md
deleted file mode 100644
index 221bb5bcd..000000000
--- a/quickstart/text_to_speech/tensorflow/wavenet/inference/cpu/fp32/.docs/aikit.md
+++ /dev/null
@@ -1,68 +0,0 @@
-<!--- 50. AI Kit -->
-## Run the model
-
-Setup your environment using the instructions below, depending on if you are
-using [AI Kit](/docs/general/tensorflow/AIKit.md):
-
-<table>
-  <tr>
-    <th>Setup using AI Kit</th>
-    <th>Setup without AI Kit</th>
-  </tr>
-  <tr>
-    <td>
-      <p>AI Kit does not currently support TF 1.15.2 models</p>
-    </td>
-    <td>
-      <p>To run without AI Kit you will need:</p>
-      <ul>
-        <li>Python 3.6 or 3.7
-        <li>git
-        <li>numactl
-        <li>wget
-        <li><a href="https://pypi.org/project/intel-tensorflow/1.15.2/">intel-tensorflow==1.15.2</a>
-        <li>librosa==0.5
-        <li>A clone of the Model Zoo repo<br />
-        <pre>git clone https://github.com/IntelAI/models.git</pre>
-      </ul>
-    </td>
-  </tr>
-</table>
-
-In addition to the requirements specified above, you will also need a clone
-of the [tensorflow-wavenet](https://github.com/ibab/tensorflow-wavenet)
-repo with pull request #352 for the CPU optimizations.  The path to
-the cloned repo needs to be set to the `TF_WAVENET_DIR` environment variable
-before running a quickstart script.
-```
-git clone https://github.com/ibab/tensorflow-wavenet.git
-cd tensorflow-wavenet/
-
-git fetch origin pull/352/head:cpu_optimized
-git checkout cpu_optimized
-export TF_WAVENET_DIR=$(pwd)
-
-cd ..
-```
-
-Download and extract the pretrained model checkpoint files.
-```
-wget https://storage.googleapis.com/intel-optimized-tensorflow/models/wavenet_fp32_pretrained_model.tar.gz
-tar -xvf wavenet_fp32_pretrained_model.tar.gz
-export PRETRAINED_MODEL=$(pwd)/wavenet_checkpoints
-```
-
-Navigate to the your model zoo directory, then set an environment variable
-for an `OUTPUT_DIR` directory where logs will be written and ensure that you
-have the `TF_WAVENET_DIR` and `PRETRAINED_MODEL` variables set. Once this
-setup is done, you can run the `fp32_inference.sh` [quickstart script](#quick-start-scripts).
-```
-# cd to your model zoo directory
-cd models
-
-export OUTPUT_DIR=<path to the directory where log files will be written>
-export TF_WAVENET_DIR=<tensorflow-wavenet directory>
-export PRETRAINED_MODEL=<path to the downloaded and extracted checkpoints>
-
-./quickstart/text_to_speech/tensorflow/wavenet/inference/cpu/fp32/fp32_inference.sh
-```
diff --git a/quickstart/text_to_speech/tensorflow/wavenet/inference/cpu/fp32/.docs/baremetal.md b/quickstart/text_to_speech/tensorflow/wavenet/inference/cpu/fp32/.docs/baremetal.md
deleted file mode 100644
index 039c7236c..000000000
--- a/quickstart/text_to_speech/tensorflow/wavenet/inference/cpu/fp32/.docs/baremetal.md
+++ /dev/null
@@ -1,36 +0,0 @@
-<!--- 50. Bare Metal -->
-## Bare Metal
-
-To run on bare metal, the following prerequisites must be installed in your environment:
-* Python 3.6 or 3.7
-* [intel-tensorflow==1.15.2](https://pypi.org/project/intel-tensorflow/1.15.2/)
-* librosa==0.5
-* numactl
-* Clone the [tensorflow-wavenet](https://github.com/ibab/tensorflow-wavenet)
-repo and get pull request #352 for the CPU optimizations.  The path to
-the cloned repo will be passed as the model source directory when
-running the launch script.
-
-```
-git clone https://github.com/ibab/tensorflow-wavenet.git
-cd tensorflow-wavenet/
-
-git fetch origin pull/352/head:cpu_optimized
-git checkout cpu_optimized
-```
-
-After installing the prerequisites, download and untar the model package.
-Set environment variables for the path to your `TF_WAVENET_DIR` and an
-`OUTPUT_DIR` where log files will be written, then run a 
-[quickstart script](#quick-start-scripts).
-
-```
-export TF_WAVENET_DIR=<tensorflow-wavenet directory>
-export OUTPUT_DIR=<directory where log files will be written>
-
-wget <package url>
-tar -xzf <package name>
-cd <package dir>
-
-./quickstart/fp32_inference.sh
-```
diff --git a/quickstart/text_to_speech/tensorflow/wavenet/inference/cpu/fp32/.docs/description.md b/quickstart/text_to_speech/tensorflow/wavenet/inference/cpu/fp32/.docs/description.md
deleted file mode 100644
index a0718bd31..000000000
--- a/quickstart/text_to_speech/tensorflow/wavenet/inference/cpu/fp32/.docs/description.md
+++ /dev/null
@@ -1,5 +0,0 @@
-<!-- 10. Description -->
-## Description
-
-This document has instructions for running <model name> <precision> <mode> using
-Intel-optimized TensorFlow.
diff --git a/quickstart/text_to_speech/tensorflow/wavenet/inference/cpu/fp32/.docs/docker.md b/quickstart/text_to_speech/tensorflow/wavenet/inference/cpu/fp32/.docs/docker.md
deleted file mode 100644
index c98d02e50..000000000
--- a/quickstart/text_to_speech/tensorflow/wavenet/inference/cpu/fp32/.docs/docker.md
+++ /dev/null
@@ -1,23 +0,0 @@
-<!--- 60. Docker -->
-## Docker
-
-The model container <docker image> includes the scripts and libraries needed to run 
-<model name> <precision> <mode>. To run one of the quickstart scripts 
-using this container, you'll need to provide volume mounts for the output directory.
-
-```
-OUTPUT_DIR=<directory where log files will be written>
-
-docker run \
-  --env OUTPUT_DIR=${OUTPUT_DIR} \
-  --env http_proxy=${http_proxy} \
-  --env https_proxy=${https_proxy} \
-  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
-  --privileged --init -t \
-  <docker image> \
-  /bin/bash quickstart/fp32_inference.sh
-```
-
-If you are new to docker and are running into issues with the container,
-see [this document](https://github.com/IntelAI/models/tree/master/docs/general/docker.md)
-for troubleshooting tips.
diff --git a/quickstart/text_to_speech/tensorflow/wavenet/inference/cpu/fp32/.docs/download.md b/quickstart/text_to_speech/tensorflow/wavenet/inference/cpu/fp32/.docs/download.md
deleted file mode 100644
index 613926758..000000000
--- a/quickstart/text_to_speech/tensorflow/wavenet/inference/cpu/fp32/.docs/download.md
+++ /dev/null
@@ -1,4 +0,0 @@
-<!--- 20. Download link -->
-## Download link
-
-[<package name>](<package url>)
diff --git a/quickstart/text_to_speech/tensorflow/wavenet/inference/cpu/fp32/.docs/license.md b/quickstart/text_to_speech/tensorflow/wavenet/inference/cpu/fp32/.docs/license.md
deleted file mode 100644
index a60f169f4..000000000
--- a/quickstart/text_to_speech/tensorflow/wavenet/inference/cpu/fp32/.docs/license.md
+++ /dev/null
@@ -1,4 +0,0 @@
-<!--- 80. License -->
-## License
-
-[LICENSE](/LICENSE)
diff --git a/quickstart/text_to_speech/tensorflow/wavenet/inference/cpu/fp32/.docs/quickstart.md b/quickstart/text_to_speech/tensorflow/wavenet/inference/cpu/fp32/.docs/quickstart.md
deleted file mode 100644
index eebc199c3..000000000
--- a/quickstart/text_to_speech/tensorflow/wavenet/inference/cpu/fp32/.docs/quickstart.md
+++ /dev/null
@@ -1,6 +0,0 @@
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| [fp32_inference.sh](/quickstart/text_to_speech/tensorflow/wavenet/inference/cpu/fp32/fp32_inference.sh) | Runs inference with a pretrained model |
diff --git a/quickstart/text_to_speech/tensorflow/wavenet/inference/cpu/fp32/.docs/title.md b/quickstart/text_to_speech/tensorflow/wavenet/inference/cpu/fp32/.docs/title.md
deleted file mode 100644
index 6416ff7ff..000000000
--- a/quickstart/text_to_speech/tensorflow/wavenet/inference/cpu/fp32/.docs/title.md
+++ /dev/null
@@ -1,2 +0,0 @@
-<!--- 0. Title -->
-# <model name> <precision> <mode>
diff --git a/quickstart/text_to_speech/tensorflow/wavenet/inference/cpu/fp32/README.md b/quickstart/text_to_speech/tensorflow/wavenet/inference/cpu/fp32/README.md
deleted file mode 100644
index 6f66de6a7..000000000
--- a/quickstart/text_to_speech/tensorflow/wavenet/inference/cpu/fp32/README.md
+++ /dev/null
@@ -1,87 +0,0 @@
-<!--- 0. Title -->
-# WaveNet FP32 inference
-
-<!-- 10. Description -->
-## Description
-
-This document has instructions for running WaveNet FP32 inference using
-Intel-optimized TensorFlow.
-
-<!--- 20. Download link -->
-## Download link
-
-[wavenet-fp32-inference.tar.gz](https://storage.googleapis.com/intel-optimized-tensorflow/models/v2_7_0/wavenet-fp32-inference.tar.gz)
-
-<!--- 40. Quick Start Scripts -->
-## Quick Start Scripts
-
-| Script name | Description |
-|-------------|-------------|
-| [fp32_inference.sh](/quickstart/text_to_speech/tensorflow/wavenet/inference/cpu/fp32/fp32_inference.sh) | Runs inference with a pretrained model |
-
-<!--- 50. Bare Metal -->
-## Bare Metal
-
-To run on bare metal, the following prerequisites must be installed in your environment:
-* Python 3.6 or 3.7
-* [intel-tensorflow==1.15.2](https://pypi.org/project/intel-tensorflow/1.15.2/)
-* librosa==0.5
-* numactl
-* Clone the [tensorflow-wavenet](https://github.com/ibab/tensorflow-wavenet)
-repo and get pull request #352 for the CPU optimizations.  The path to
-the cloned repo will be passed as the model source directory when
-running the launch script.
-
-```
-git clone https://github.com/ibab/tensorflow-wavenet.git
-cd tensorflow-wavenet/
-
-git fetch origin pull/352/head:cpu_optimized
-git checkout cpu_optimized
-```
-
-After installing the prerequisites, download and untar the model package.
-Set environment variables for the path to your `TF_WAVENET_DIR` and an
-`OUTPUT_DIR` where log files will be written, then run a 
-[quickstart script](#quick-start-scripts).
-
-```
-export TF_WAVENET_DIR=<tensorflow-wavenet directory>
-export OUTPUT_DIR=<directory where log files will be written>
-
-wget https://storage.googleapis.com/intel-optimized-tensorflow/models/v2_7_0/wavenet-fp32-inference.tar.gz
-tar -xzf wavenet-fp32-inference.tar.gz
-cd wavenet-fp32-inference
-
-./quickstart/fp32_inference.sh
-```
-
-<!--- 60. Docker -->
-## Docker
-
-The model container intel/text-to-speech:tf-1.15.2-wavenet-fp32-inference includes the scripts and libraries needed to run 
-WaveNet FP32 inference. To run one of the quickstart scripts 
-using this container, you'll need to provide volume mounts for the output directory.
-
-```
-OUTPUT_DIR=<directory where log files will be written>
-
-docker run \
-  --env OUTPUT_DIR=${OUTPUT_DIR} \
-  --env http_proxy=${http_proxy} \
-  --env https_proxy=${https_proxy} \
-  --volume ${OUTPUT_DIR}:${OUTPUT_DIR} \
-  --privileged --init -t \
-  intel/text-to-speech:tf-1.15.2-wavenet-fp32-inference \
-  /bin/bash quickstart/fp32_inference.sh
-```
-
-If you are new to docker and are running into issues with the container,
-see [this document](https://github.com/IntelAI/models/tree/master/docs/general/docker.md)
-for troubleshooting tips.
-
-<!--- 80. License -->
-## License
-
-[LICENSE](/LICENSE)
-
diff --git a/quickstart/text_to_speech/tensorflow/wavenet/inference/cpu/fp32/fp32_inference.sh b/quickstart/text_to_speech/tensorflow/wavenet/inference/cpu/fp32/fp32_inference.sh
deleted file mode 100755
index 9bbdb0e97..000000000
--- a/quickstart/text_to_speech/tensorflow/wavenet/inference/cpu/fp32/fp32_inference.sh
+++ /dev/null
@@ -1,86 +0,0 @@
-#!/usr/bin/env bash
-#
-# Copyright (c) 2020 Intel Corporation
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#    http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-MODEL_DIR=${MODEL_DIR-$PWD}
-
-echo 'MODEL_DIR='$MODEL_DIR
-echo 'OUTPUT_DIR='$OUTPUT_DIR
-
-if [ -z "${OUTPUT_DIR}" ]; then
-  echo "The required environment variable OUTPUT_DIR has not been set"
-  exit 1
-fi
-
-# Create the output directory in case it doesn't already exist
-mkdir -p ${OUTPUT_DIR}
-
-# Check for TF_WAVENET_DIR
-if [ -d "/tensorflow-wavenet" ]; then # if true assume running in docker
-   TF_WAVENET_DIR=/tensorflow-wavenet
-elif [ -z ${TF_WAVENET_DIR} ]; then
-    echo "Please set TF_WAVENET_DIR or run in docker mode." >&2
-    exit 1
-fi
-echo 'TF_WAVENET_DIR='$TF_WAVENET_DIR
-
-# Unzip pretrained model files
-if [ -z ${PRETRAINED_MODEL} ]; then
-  pretrained_model_dir="pretrained_model/wavenet_checkpoints"
-  if [ ! -d "${pretrained_model_dir}" ]; then
-    if [[ -f "pretrained_model/wavenet_fp32_pretrained_model.tar.gz" ]]; then
-      tar -C pretrained_model/ -xvf pretrained_model/wavenet_fp32_pretrained_model.tar.gz
-    else
-      echo "The PRETRAINED_MODEL environment variable was not defined and the checkpoints could not be found."
-      exit 1
-    fi
-  fi
-  PRETRAINED_MODEL="${MODEL_DIR}/${pretrained_model_dir}"
-fi
-
-# Create an array of input directories that are expected and then verify that they exist
-declare -A input_dirs
-input_dirs[PRETRAINED_MODEL]=${PRETRAINED_MODEL}
-
-for i in "${!input_dirs[@]}"; do
-  var_name=$i
-  dir_path=${input_dirs[$i]}
-
-  if [[ -z $dir_path ]]; then
-    echo "The required environment variable $var_name is empty" >&2
-    exit 1
-  fi
-
-  if [[ ! -d $dir_path ]]; then
-    echo "The $var_name path '$dir_path' does not exist" >&2
-    exit 1
-  fi
-done
-
-source "${MODEL_DIR}/quickstart/common/utils.sh"
-_command python ${MODEL_DIR}/benchmarks/launch_benchmark.py \
-      --checkpoint ${PRETRAINED_MODEL} \
-      --model-source-dir ${TF_WAVENET_DIR} \
-      --model-name wavenet \
-      --socket-id 0 \
-      --num-cores 1 \
-      --framework tensorflow \
-      --precision fp32 \
-      --mode inference \
-      --output-dir ${OUTPUT_DIR} \
-      $@ \
-      -- checkpoint_name=model.ckpt-99 sample=8510
-
diff --git a/tests/cicd/IPEX-XPU/setup.sh b/tests/cicd/IPEX-XPU/setup.sh
new file mode 100755
index 000000000..3a58aaa96
--- /dev/null
+++ b/tests/cicd/IPEX-XPU/setup.sh
@@ -0,0 +1,37 @@
+#!/bin/bash
+set -e
+
+echo "Setup IPEX-XPU enivornment"
+
+FRAMEWORK_VERSION=$1
+FRAMEWORK_EXTENSION_VERSION=$2
+TORCHVISION_VERSION=$3
+is_lkg_drop=$4
+AIKIT_RELEASE=$5
+
+if [[ "${is_lkg_drop}" == "true" ]]; then
+  if [ ! -d "${GITHUB_WORKSPACE}/miniconda3" ]; then
+    cd ${GITHUB_WORKSPACE}
+    curl https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -o Miniconda3-latest-Linux-x86_64.sh
+    rm -rf miniconda3
+    chmod +x Miniconda3-latest-Linux-x86_64.sh
+    ./Miniconda3-latest-Linux-x86_64.sh -b -f -p miniconda3
+  fi
+  rm -rf ${GITHUB_WORKSPACE}/pytorch_setup
+  if [ ! -d "${GITHUB_WORKSPACE}/pytorch_setup" ]; then
+    mkdir -p ${GITHUB_WORKSPACE}/pytorch_setup
+    cd ${GITHUB_WORKSPACE}/oneapi_drop_tool
+    git submodule update --init --remote --recursive
+    python -m pip install -r requirements.txt
+    python cdt.py --username=tf_qa_prod --password ${TF_QA_PROD} download --product ipytorch --release ${AIKIT_RELEASE} -c l_drop_installer --download-dir ${GITHUB_WORKSPACE}/pytorch_setup
+    cd ${GITHUB_WORKSPACE}/pytorch_setup
+    chmod +x IPEX_installer-*
+    ./IPEX_installer-* -b -u -p ${GITHUB_WORKSPACE}/pytorch_setup
+  fi
+else
+  pip install --upgrade pip
+  echo "Installing pytorch"
+  export no_proxy=“” 
+  export NO_PROXY=“”
+  python -m pip install torch==${FRAMEWORK_VERSION} torchvision==${TORCHVISION_VERSION} intel-extension-for-pytorch==${FRAMEWORK_EXTENSION_VERSION} --extra-index-url https://pytorch-extension.intel.com/release-whl-aitools/
+fi
diff --git a/tests/cicd/IPEX-XPU/stable_diffusion-inference/run_test.sh b/tests/cicd/IPEX-XPU/stable_diffusion-inference/run_test.sh
new file mode 100755
index 000000000..2afc4c578
--- /dev/null
+++ b/tests/cicd/IPEX-XPU/stable_diffusion-inference/run_test.sh
@@ -0,0 +1,31 @@
+#!/bin/bash
+set -e
+
+echo "Setup IPEX Test Enviroment for Stable Diffusion Inference"
+
+PRECISION=$1
+OUTPUT_DIR=${OUTPUT_DIR-"$(pwd)/tests/cicd/output/IPEX-XPU/stable_diffusion-inference/${PRECISION}"}
+is_lkg_drop=$2
+MULTI_TILE=$3
+PLATFORM=$4
+
+# Create the output directory in case it doesn't already exist
+mkdir -p ${OUTPUT_DIR}
+
+if [[ "${is_lkg_drop}" == "true" ]]; then
+  source ${WORKSPACE}/pytorch_setup/bin/activate pytorch
+else
+  source /oneapi/compiler/latest/env/vars.sh
+  source /oneapi/mpi/latest/env/vars.sh
+  source /oneapi/mkl/latest/env/vars.sh
+  source /oneapi/tbb/latest/env/vars.sh
+  source /oneapi/ccl/latest/env/vars.sh
+fi
+
+# run following script
+cd models_v2/pytorch/stable_diffusion/inference/gpu
+./setup.sh
+
+# Run script
+OUTPUT_DIR=${OUTPUT_DIR} MULTI_TILE=${MULTI_TILE} PRECISION=${PRECISION} PLATFORM=${PLATFORM} ./run_model.sh
+cd -
diff --git a/tests/cicd/ITEX-XPU/3d_unet-training/run_test.sh b/tests/cicd/ITEX-XPU/3d_unet-training/run_test.sh
new file mode 100755
index 000000000..46d97032d
--- /dev/null
+++ b/tests/cicd/ITEX-XPU/3d_unet-training/run_test.sh
@@ -0,0 +1,32 @@
+#!/bin/bash
+set -e
+
+echo "Setup ITEX-XPU Test Enviroment for 3D Unet Training"
+
+PRECISION=$1
+OUTPUT_DIR=${OUTPUT_DIR-"$(pwd)/tests/cicd/output/ITEX-XPU/3d_unet-training/${PRECISION}"}
+is_lkg_drop=$2
+DATASET=$3
+MULTI_TILE=$4
+
+# Create the output directory in case it doesn't already exist
+mkdir -p ${OUTPUT_DIR}
+
+if [[ "${is_lkg_drop}" == "true" ]]; then
+  source ${WORKSPACE}/tensorflow_setup/bin/activate tensorflow
+else
+  source /oneapi/compiler/latest/env/vars.sh
+  source /oneapi/mpi/latest/env/vars.sh
+  source /oneapi/mkl/latest/env/vars.sh
+  source /oneapi/tbb/latest/env/vars.sh
+  source /oneapi/ccl/latest/env/vars.sh
+fi
+
+# run following script
+cd models_v2/tensorflow/3d_unet/training/gpu
+./setup.sh
+pip uninstall horovod 
+pip install intel-optimization-for-horovod
+
+OUTPUT_DIR=${OUTPUT_DIR} PRECISION=${PRECISION} MULTI_TILE=${MULTI_TILE} DATASET_DIR=${DATASET} ./run_model.sh
+cd - 
diff --git a/tests/cicd/ITEX-XPU/bert_large-training/run_test.sh b/tests/cicd/ITEX-XPU/bert_large-training/run_test.sh
new file mode 100755
index 000000000..1184f73c9
--- /dev/null
+++ b/tests/cicd/ITEX-XPU/bert_large-training/run_test.sh
@@ -0,0 +1,26 @@
+#!/bin/bash
+set -e
+
+echo "Setup ITEX-XPU Test Enviroment for Bert Large Training"
+
+PRECISION=$1
+OUTPUT_DIR=${OUTPUT_DIR-"$(pwd)/tests/cicd/output/ITEX-XPU/bert_large-training/${PRECISION}"}
+is_lkg_drop=$2
+DATASET=$3
+MULTI_TILE=$4
+
+# Create the output directory in case it doesn't already exist
+mkdir -p ${OUTPUT_DIR}
+
+if [[ "${is_lkg_drop}" == "true" ]]; then
+  source ${WORKSPACE}/tensorflow_setup/bin/activate tensorflow
+fi
+
+# run following script
+cd models_v2/tensorflow/bert_large/training/gpu
+./setup.sh
+pip uninstall horovod 
+pip install intel-optimization-for-horovod
+
+RESULTS_DIR=${OUTPUT_DIR} DATATYPE=${PRECISION} MULTI_TILE=${MULTI_TILE} DATA_DIR=${DATASET} ./run_model.sh
+cd - 
diff --git a/tests/cicd/ITEX-XPU/efficientnet-inference/run_test.sh b/tests/cicd/ITEX-XPU/efficientnet-inference/run_test.sh
new file mode 100755
index 000000000..742189203
--- /dev/null
+++ b/tests/cicd/ITEX-XPU/efficientnet-inference/run_test.sh
@@ -0,0 +1,22 @@
+#!/bin/bash
+set -e
+
+echo "Setup ITEX-XPU Test Enviroment for EfficientNet Inference"
+
+MODEL_NAME=$1
+OUTPUT_DIR=${OUTPUT_DIR-"$(pwd)/tests/cicd/output/ITEX-XPU/efficientnet-inference/${MODEL_NAME}"}
+is_lkg_drop=$2
+
+# Create the output directory in case it doesn't already exist
+mkdir -p ${OUTPUT_DIR}
+
+if [[ "${is_lkg_drop}" == "true" ]]; then
+  source ${WORKSPACE}/tensorflow_setup/bin/activate tensorflow
+fi
+
+# run following script
+cd models_v2/tensorflow/efficientnet/inference/gpu
+./setup.sh
+
+OUTPUT_DIR=${OUTPUT_DIR} MODEL_NAME=${MODEL_NAME} ./run_model.sh
+cd - 
diff --git a/tests/cicd/ITEX-XPU/maskrcnn-inference/run_test.sh b/tests/cicd/ITEX-XPU/maskrcnn-inference/run_test.sh
new file mode 100755
index 000000000..da70b97c2
--- /dev/null
+++ b/tests/cicd/ITEX-XPU/maskrcnn-inference/run_test.sh
@@ -0,0 +1,32 @@
+#!/bin/bash
+set -e
+
+echo "Setup ITEX-XPU Test Enviroment for MaskRCNN Inference"
+
+PRECISION=$1
+OUTPUT_DIR=${OUTPUT_DIR-"$(pwd)/tests/cicd/output/ITEX-XPU/maskrcnn-inference/${PRECISION}"}
+is_lkg_drop=$2
+DATASET=$3
+
+# Create the output directory in case it doesn't already exist
+mkdir -p ${OUTPUT_DIR}
+
+if [[ "${is_lkg_drop}" == "true" ]]; then
+  source ${WORKSPACE}/tensorflow_setup/bin/activate tensorflow
+else
+  source /oneapi/compiler/latest/env/vars.sh
+  source /oneapi/mpi/latest/env/vars.sh
+  source /oneapi/mkl/latest/env/vars.sh
+  source /oneapi/tbb/latest/env/vars.sh
+  source /oneapi/ccl/latest/env/vars.sh
+fi
+
+# run following script
+cd models_v2/tensorflow/maskrcnn/inference/gpu
+./setup.sh
+pushd .
+cd ./DeepLearningExamples/TensorFlow2/Segmentation/MaskRCNN
+python scripts/download_weights.py --save_dir=./weights
+popd
+OUTPUT_DIR=${OUTPUT_DIR} PRECISION=${PRECISION} PRETRAINED_DIR="DeepLearningExamples/TensorFlow2/Segmentation/MaskRCNN/weights" DATASET_DIR=${DATASET} ./run_model.sh
+cd - 
diff --git a/tests/cicd/ITEX-XPU/maskrcnn-training/run_test.sh b/tests/cicd/ITEX-XPU/maskrcnn-training/run_test.sh
new file mode 100755
index 000000000..404b89e65
--- /dev/null
+++ b/tests/cicd/ITEX-XPU/maskrcnn-training/run_test.sh
@@ -0,0 +1,33 @@
+#!/bin/bash
+set -e
+
+echo "Setup ITEX-XPU Test Enviroment for MaskRCNN Training"
+
+PRECISION=$1
+OUTPUT_DIR=${OUTPUT_DIR-"$(pwd)/tests/cicd/output/ITEX-XPU/maskrcnn-training/${PRECISION}"}
+is_lkg_drop=$2
+DATASET=$3
+
+# Create the output directory in case it doesn't already exist
+mkdir -p ${OUTPUT_DIR}
+
+if [[ "${is_lkg_drop}" == "true" ]]; then
+  source ${WORKSPACE}/tensorflow_setup/bin/activate tensorflow
+else
+  source /oneapi/compiler/latest/env/vars.sh
+  source /oneapi/mpi/latest/env/vars.sh
+  source /oneapi/mkl/latest/env/vars.sh
+  source /oneapi/tbb/latest/env/vars.sh
+  source /oneapi/ccl/latest/env/vars.sh
+fi
+
+# run following script
+cd models_v2/tensorflow/maskrcnn/training/gpu
+./setup.sh
+pushd .
+cd ./DeepLearningExamples/TensorFlow2/Segmentation/MaskRCNN
+python scripts/download_weights.py --save_dir=./weights
+popd
+pip install intel-optimization-for-horovod
+OUTPUT_DIR=${OUTPUT_DIR} PRECISION=${PRECISION} DATASET_DIR=${DATASET} MULTI_TILE=False ./run_model.sh
+cd - 
diff --git a/tests/cicd/ITEX-XPU/resnet50v1_5-inference/run_test.sh b/tests/cicd/ITEX-XPU/resnet50v1_5-inference/run_test.sh
new file mode 100755
index 000000000..7859996bc
--- /dev/null
+++ b/tests/cicd/ITEX-XPU/resnet50v1_5-inference/run_test.sh
@@ -0,0 +1,38 @@
+#!/bin/bash
+set -e
+
+echo "Setup ITEX-XPU Test Enviroment for ResNet50v1.5 Inference"
+
+PRECISION=$1
+OUTPUT_DIR=${OUTPUT_DIR-"$(pwd)/tests/cicd/output/ITEX-XPU/resnet50v1_5-inference/${PRECISION}"}
+is_lkg_drop=$2
+DATASET=$3
+
+# Create the output directory in case it doesn't already exist
+mkdir -p ${OUTPUT_DIR}
+
+if [[ "${is_lkg_drop}" == "true" ]]; then
+  source ${WORKSPACE}/tensorflow_setup/bin/activate tensorflow
+else
+  source /oneapi/compiler/latest/env/vars.sh
+  source /oneapi/mpi/latest/env/vars.sh
+  source /oneapi/mkl/latest/env/vars.sh
+  source /oneapi/tbb/latest/env/vars.sh
+  source /oneapi/ccl/latest/env/vars.sh
+fi
+
+# run following script
+cd models_v2/tensorflow/resnet50v1_5/inference/gpu
+./setup.sh
+
+# Download PB file:
+if [[ "${PRECISION}" == "int8" ]]; then
+  wget https://storage.googleapis.com/intel-optimized-tensorflow/models/3_1/resnet50_v1_int8.pb
+  PB_FILE=$(pwd)/resnet50_v1_int8.pb
+elif [[ "${PRECISION}" == "float32" || "${PRECISION}" == "tensorflow32" || "${PRECISION}" == "float16" || "${PRECISION}" == "bfloat16" ]]; then
+  wget https://storage.googleapis.com/intel-optimized-tensorflow/models/3_1/resnet50_v1.pb
+  PB_FILE=$(pwd)/resnet50_v1.pb
+fi
+
+OUTPUT_DIR=${OUTPUT_DIR} DTYPE=${PRECISION} PB_FILE_PATH=${PB_FILE} DATASET_DIR=${DATASET} TEST_MODE=inference ./run_model.sh
+cd - 
diff --git a/tests/cicd/ITEX-XPU/resnet50v1_5-training/run_test.sh b/tests/cicd/ITEX-XPU/resnet50v1_5-training/run_test.sh
new file mode 100755
index 000000000..728b422b3
--- /dev/null
+++ b/tests/cicd/ITEX-XPU/resnet50v1_5-training/run_test.sh
@@ -0,0 +1,34 @@
+#!/bin/bash
+set -e
+
+echo "Setup ITEX-XPU Test Enviroment for ResNet50v1.5 Training"
+
+CONFIG_FILE=$1
+OUTPUT_DIR=${OUTPUT_DIR-"$(pwd)/tests/cicd/output/ITEX-XPU/resnet50v1_5-training/${PRECISION}"}
+is_lkg_drop=$2
+DATASET=$3
+MULTI_TILE=$4
+
+# Create the output directory in case it doesn't already exist
+mkdir -p ${OUTPUT_DIR}
+
+if [[ "${is_lkg_drop}" == "true" ]]; then
+  source ${WORKSPACE}/tensorflow_setup/bin/activate tensorflow
+else
+  source /oneapi/compiler/latest/env/vars.sh
+  source /oneapi/mpi/latest/env/vars.sh
+  source /oneapi/mkl/latest/env/vars.sh
+  source /oneapi/tbb/latest/env/vars.sh
+  source /oneapi/ccl/latest/env/vars.sh
+fi
+
+# run following script
+cd models_v2/tensorflow/resnet50v1_5/training/gpu
+./setup.sh
+
+apt-get update 
+
+pip install intel-optimization-for-horovod
+
+OUTPUT_DIR=${OUTPUT_DIR} CONFIG_FILE=$(pwd)/${CONFIG_FILE} DATASET_DIR=${DATASET} MULTI_TILE=${MULTI_TILE} ./run_model.sh
+cd - 
diff --git a/tests/cicd/ITEX-XPU/setup.sh b/tests/cicd/ITEX-XPU/setup.sh
new file mode 100755
index 000000000..9807302a5
--- /dev/null
+++ b/tests/cicd/ITEX-XPU/setup.sh
@@ -0,0 +1,36 @@
+#!/bin/bash
+set -e
+
+echo "Setup ITEX-XPU enivornment"
+
+TF_VERSION=$1
+TF_EXTENSION_VERSION=$2
+is_lkg_drop=$3
+WORKSPACE=$4
+AIKIT_RELEASE=$5
+
+if [[ "${is_lkg_drop}" == "true" ]]; then
+  if [ ! -d "${WORKSPACE}/miniconda3" ]; then
+    cd ${WORKSPACE}
+    curl https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -o Miniconda3-latest-Linux-x86_64.sh
+    rm -rf miniconda3
+    chmod +x Miniconda3-latest-Linux-x86_64.sh
+    ./Miniconda3-latest-Linux-x86_64.sh -b -f -p miniconda3
+  fi
+  rm -rf ${WORKSPACE}/tensorflow_setup
+  if [ ! -d "${WORKSPACE}/tensorflow_setup" ]; then
+    mkdir -p ${WORKSPACE}/tensorflow_setup
+    cd ${WORKSPACE}/oneapi_drop_tool
+    git submodule update --init --remote --recursive
+    python -m pip install -r requirements.txt
+    python cdt.py --username=tf_qa_prod --password ${TF_QA_PROD} download --product tensorflow --release ${AIKIT_RELEASE} -c l_drop_installer --download-dir ${WORKSPACE}/tensorflow_setup
+    cd ${WORKSPACE}/tensorflow_setup
+    chmod +x ITEX_installer-*
+    ./ITEX_installer-* -b -u -p ${WORKSPACE}/tensorflow_setup
+  fi
+else
+  pip install --upgrade pip
+  echo "Installing tensorflow and ITEX"
+  pip install tensorflow==$1
+  pip install --upgrade intel-extension-for-tensorflow[xpu]==$2
+fi
diff --git a/tests/cicd/ITEX-XPU/stable_diffusion-inference/run_test.sh b/tests/cicd/ITEX-XPU/stable_diffusion-inference/run_test.sh
new file mode 100755
index 000000000..2adca4875
--- /dev/null
+++ b/tests/cicd/ITEX-XPU/stable_diffusion-inference/run_test.sh
@@ -0,0 +1,22 @@
+#!/bin/bash
+set -e
+
+echo "Setup ITEX-XPU Test Enviroment for Stable Diffusion Inference"
+
+PRECISION=$1
+OUTPUT_DIR=${OUTPUT_DIR-"$(pwd)/tests/cicd/output/ITEX-XPU/stable_diffusion-inference/${PRECISION}"}
+is_lkg_drop=$2
+
+# Create the output directory in case it doesn't already exist
+mkdir -p ${OUTPUT_DIR}
+
+if [[ "${is_lkg_drop}" == "true" ]]; then
+  source ${WORKSPACE}/tensorflow_setup/bin/activate tensorflow
+fi
+
+# run following script
+cd models_v2/tensorflow/stable_diffusion/inference/gpu
+./setup.sh
+
+OUTPUT_DIR=${OUTPUT_DIR} PRECISION=${PRECISION} ./run_model.sh
+cd - 
diff --git a/tests/cicd/ITEX-XPU/wide_deep_large_ds-inference/run_test.sh b/tests/cicd/ITEX-XPU/wide_deep_large_ds-inference/run_test.sh
new file mode 100755
index 000000000..a61ffba46
--- /dev/null
+++ b/tests/cicd/ITEX-XPU/wide_deep_large_ds-inference/run_test.sh
@@ -0,0 +1,26 @@
+#!/bin/bash
+set -e
+
+echo "Setup ITEX-XPU Test Enviroment for Wide Deep Large Inference"
+
+OUTPUT_DIR=${OUTPUT_DIR-"$(pwd)/tests/cicd/output/ITEX-XPU/wide_dep_large_ds-inference/${PRECISION}"}
+is_lkg_drop=$2
+DATASET=$1
+
+# Create the output directory in case it doesn't already exist
+mkdir -p ${OUTPUT_DIR}
+
+if [[ "${is_lkg_drop}" == "true" ]]; then
+  source ${WORKSPACE}/tensorflow_setup/bin/activate tensorflow
+fi
+
+# run following script
+cd models_v2/tensorflow/wide_deep_large_ds/inference/gpu
+./setup.sh
+
+# Download the PB File:
+wget https://storage.googleapis.com/intel-optimized-tensorflow/models/3_0/wide_deep_fp16_pretrained_model.pb
+PB_FILE_PATH=$(pwd)/wide_deep_fp16_pretrained_model.pb
+
+OUTPUT_DIR=${OUTPUT_DIR} PB_FILE_PATH=${PB_FILE_PATH} DATASET_PATH=${DATASET} ./run_model.sh
+cd - 
diff --git a/tests/cicd/PyTorch/bert-large-inference/run_test.sh b/tests/cicd/PyTorch/bert-large-inference/run_test.sh
new file mode 100755
index 000000000..665d5090a
--- /dev/null
+++ b/tests/cicd/PyTorch/bert-large-inference/run_test.sh
@@ -0,0 +1,46 @@
+#!/bin/bash
+set -e
+
+echo "Setup PyTorch Test Enviroment for BERT LARGE Inference"
+
+PRECISION=$1
+SCRIPT=$2
+OUTPUT_DIR=${OUTPUT_DIR-"$(pwd)/tests/cicd/output/PyTorch/bert-large-inference/${SCRIPT}/${PRECISION}"}
+WORKSPACE=$3
+is_lkg_drop=$4
+DATASET=$5
+
+# Create the output directory in case it doesn't already exist
+mkdir -p ${OUTPUT_DIR}
+
+if [[ "${is_lkg_drop}" == "true" ]]; then
+  source ${WORKSPACE}/pytorch_setup/bin/activate pytorch
+fi
+
+export LD_PRELOAD="${WORKSPACE}/jemalloc/lib/libjemalloc.so":"${WORKSPACE}/tcmalloc/lib/libtcmalloc.so":"/usr/local/lib/libiomp5.so":$LD_PRELOAD 
+export MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+export DNNL_MAX_CPU_ISA=AVX512_CORE_AMX
+
+# Install dependency
+./quickstart/language_modeling/pytorch/bert_large/inference/cpu/setup.sh
+
+# Get EVAL_DATA_FILE:
+if [ -d "dev-v1.1.json" ]; then
+  echo "The eval data file exists. Skipping download."
+else
+  wget https://rajpurkar.github.io/SQuAD-explorer/dataset/dev-v1.1.json
+fi
+export EVAL_DATA_FILE=$(pwd)/dev-v1.1.json
+
+# Get Pretrained model:
+if [ -d "bert_squad_model" ]; then
+  echo "The pretrained model exists. Skipping download."
+else
+  mkdir bert_squad_model
+  wget https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-whole-word-masking-finetuned-squad-config.json -O bert_squad_model/config.json
+  wget https://cdn.huggingface.co/bert-large-uncased-whole-word-masking-finetuned-squad-pytorch_model.bin  -O bert_squad_model/pytorch_model.bin
+fi
+export FINETUNED_MODEL=$(pwd)/bert_squad_model
+
+# Run script
+OUTPUT_DIR=${OUTPUT_DIR} PRECISION=${PRECISION} FINETUNED_MODEL=${FINETUNED_MODEL} EVAL_DATA_FILE=${EVAL_DATA_FILE} ./quickstart/language_modeling/pytorch/bert_large/inference/cpu/${SCRIPT}
diff --git a/tests/cicd/PyTorch/bert-large-training/run_test.sh b/tests/cicd/PyTorch/bert-large-training/run_test.sh
new file mode 100755
index 000000000..06b6d2934
--- /dev/null
+++ b/tests/cicd/PyTorch/bert-large-training/run_test.sh
@@ -0,0 +1,35 @@
+#!/bin/bash
+set -e
+
+echo "Setup PyTorch Test Enviroment for BERT LARGE Training"
+
+PRECISION=$1
+SCRIPT=$2
+OUTPUT_DIR=${OUTPUT_DIR-"$(pwd)/tests/cicd/output/PyTorch/bert-large-training/${SCRIPT}/${PRECISION}"}
+WORKSPACE=$3
+is_lkg_drop=$4
+DATASET=$5
+
+# Create the output directory in case it doesn't already exist
+mkdir -p ${OUTPUT_DIR}
+
+if [[ "${is_lkg_drop}" == "true" ]]; then
+  source ${WORKSPACE}/pytorch_setup/bin/activate pytorch
+fi
+
+export LD_PRELOAD="${WORKSPACE}/jemalloc/lib/libjemalloc.so":"${WORKSPACE}/tcmalloc/lib/libtcmalloc.so":"/usr/local/lib/libiomp5.so":$LD_PRELOAD 
+export MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+export DNNL_MAX_CPU_ISA=AVX512_CORE_AMX
+
+# Install dependency
+./quickstart/language_modeling/pytorch/bert_large/training/cpu/setup.sh
+
+# For phase 1 get the bert config from [here](https://drive.google.com/drive/folders/1oQF4diVHNPCclykwdvQJw8n_VIWwV0PT)
+export BERT_MODEL_CONFIG=$current_directory/models/language_modeling/pytorch/bert_large/training/gpu/bert_config.json
+
+export CHECKPOINT_DIR=$(pwd)/checkpoint_phase1_dir
+
+# Run script
+OUTPUT_DIR=${OUTPUT_DIR} PRECISION=${PRECISION} CHECKPOINT_DIR=${CHECKPOINT_DIR} DATASET_DIR=${DATASET} BERT_MODEL_CONFIG=${BERT_MODEL_CONFIG} ./quickstart/language_modeling/pytorch/bert_large/training/cpu/run_bert_pretrain_phase1.sh
+
+OUTPUT_DIR=${OUTPUT_DIR} PRECISION=${PRECISION} PRETRAINED_MODEL=${CHECKPOINT_DIR} DATASET_DIR=${DATASET} ./quickstart/language_modeling/pytorch/bert_large/training/cpu/run_bert_pretrain_phase2.sh
diff --git a/tests/cicd/PyTorch/distilbert-inference/run_test.sh b/tests/cicd/PyTorch/distilbert-inference/run_test.sh
new file mode 100755
index 000000000..2b6ece1b8
--- /dev/null
+++ b/tests/cicd/PyTorch/distilbert-inference/run_test.sh
@@ -0,0 +1,40 @@
+#!/bin/bash
+set -e
+
+echo "Setup PyTorch Test Enviroment for DistilBERT Inference"
+
+PRECISION=$1
+SCRIPT=$2
+OUTPUT_DIR=${OUTPUT_DIR-"$(pwd)/tests/cicd/output/PyTorch/distilbert-inference/${SCRIPT}/${PRECISION}"}
+WORKSPACE=$3
+is_lkg_drop=$4
+DATASET=$5
+
+# Create the output directory in case it doesn't already exist
+mkdir -p ${OUTPUT_DIR}
+
+if [[ "${is_lkg_drop}" == "true" ]]; then
+  source ${WORKSPACE}/pytorch_setup/bin/activate pytorch
+fi
+
+export LD_PRELOAD="${WORKSPACE}/jemalloc/lib/libjemalloc.so":"${WORKSPACE}/tcmalloc/lib/libtcmalloc.so":"/usr/local/lib/libiomp5.so":$LD_PRELOAD 
+export MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+export DNNL_MAX_CPU_ISA=AVX512_CORE_AMX
+
+# Install dependency
+./quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/setup.sh
+
+export HF_DATASETS_OFFLINE=0
+export SEQUENCE_LENGTH=128 
+export CORE_PER_INSTANCE=4
+
+# Install pre-trained model:
+if [ -d "distilbert-base-uncased-finetuned-sst-2-english" ]; then
+  echo "The fine tuned model exists. Skipping download."
+else
+  git clone https://huggingface.co/distilbert-base-uncased-finetuned-sst-2-english
+fi
+export FINETUNED_MODEL=$(pwd)/distilbert-base-uncased-finetuned-sst-2-english
+
+# Run script
+OUTPUT_DIR=${OUTPUT_DIR} DATASET_DIR=${DATASET} PRECISION=${PRECISION} FINETUNED_MODEL=${FINETUNED_MODELL} HF_DATASETS_OFFLINE=${HF_DATASETS_OFFLINE} SEQUENCE_LENGTH=${SEQUENCE_LENGTH} CORE_PER_INSTANCE=${CORE_PER_INSTANCE} ./quickstart/language_modeling/pytorch/distilbert_base/inference/cpu/${SCRIPT}
diff --git a/tests/cicd/PyTorch/dlrm-inference/run_test.sh b/tests/cicd/PyTorch/dlrm-inference/run_test.sh
new file mode 100755
index 000000000..5acad18c3
--- /dev/null
+++ b/tests/cicd/PyTorch/dlrm-inference/run_test.sh
@@ -0,0 +1,28 @@
+#!/bin/bash
+set -e
+
+echo "Setup PyTorch Test Enviroment for DLRM Inference"
+
+PRECISION=$1
+SCRIPT=$2
+OUTPUT_DIR=${OUTPUT_DIR-"$(pwd)/tests/cicd/output/PyTorch/dlrm-inference/${SCRIPT}/${PRECISION}"}
+WORKSPACE=$3
+is_lkg_drop=$4
+DATASET=$5
+
+# Create the output directory in case it doesn't already exist
+mkdir -p ${OUTPUT_DIR}
+
+if [[ "${is_lkg_drop}" == "true" ]]; then
+  source ${WORKSPACE}/pytorch_setup/bin/activate pytorch
+fi
+
+export LD_PRELOAD="${WORKSPACE}/jemalloc/lib/libjemalloc.so":"${WORKSPACE}/tcmalloc/lib/libtcmalloc.so":"/usr/local/lib/libiomp5.so":$LD_PRELOAD 
+export MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+export DNNL_MAX_CPU_ISA=AVX512_CORE_AMX
+
+# Install model dependencies:
+pip install -r quickstart/recommendation/pytorch/dlrm/requirements.txt 
+
+# Run script
+OUTPUT_DIR=${OUTPUT_DIR} DATASET_DIR=${DATASET} PRECISION=${PRECISION} WEIGHT_PATH=${WEIGHT_PATH} ./quickstart/recommendation/pytorch/dlrm/inference/cpu/${SCRIPT}
diff --git a/tests/cicd/PyTorch/dlrm-training/run_test.sh b/tests/cicd/PyTorch/dlrm-training/run_test.sh
new file mode 100755
index 000000000..29bef8a90
--- /dev/null
+++ b/tests/cicd/PyTorch/dlrm-training/run_test.sh
@@ -0,0 +1,28 @@
+#!/bin/bash
+set -e
+
+echo "Setup PyTorch Test Enviroment for DLRM Training"
+
+PRECISION=$1
+SCRIPT=$2
+OUTPUT_DIR=${OUTPUT_DIR-"$(pwd)/tests/cicd/output/PyTorch/dlrm-training/${SCRIPT}/${PRECISION}"}
+WORKSPACE=$3
+is_lkg_drop=$4
+DATASET=$5
+
+# Create the output directory in case it doesn't already exist
+mkdir -p ${OUTPUT_DIR}
+
+if [[ "${is_lkg_drop}" == "true" ]]; then
+  source ${WORKSPACE}/pytorch_setup/bin/activate pytorch
+fi
+
+export LD_PRELOAD="${WORKSPACE}/jemalloc/lib/libjemalloc.so":"${WORKSPACE}/tcmalloc/lib/libtcmalloc.so":"/usr/local/lib/libiomp5.so":$LD_PRELOAD 
+export MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+export DNNL_MAX_CPU_ISA=AVX512_CORE_AMX
+
+# Install model dependencies:
+pip install -r quickstart/recommendation/pytorch/dlrm/requirements.txt
+
+# Run script
+OUTPUT_DIR=${OUTPUT_DIR} DATASET_DIR=${DATASET} PRECISION=${PRECISION} NUM_BATCH=10000 ./quickstart/recommendation/pytorch/dlrm/training/cpu/${SCRIPT}
diff --git a/tests/cicd/PyTorch/maskrcnn-inference/run_test.sh b/tests/cicd/PyTorch/maskrcnn-inference/run_test.sh
new file mode 100755
index 000000000..187bb5696
--- /dev/null
+++ b/tests/cicd/PyTorch/maskrcnn-inference/run_test.sh
@@ -0,0 +1,38 @@
+#!/bin/bash
+set -e
+
+echo "Setup PyTorch Test Enviroment for Mask R-CNN Inference"
+
+PRECISION=$1
+SCRIPT=$2
+OUTPUT_DIR=${OUTPUT_DIR-"$(pwd)/tests/cicd/output/PyTorch/maskrcnn-inference/${SCRIPT}/${PRECISION}"}
+WORKSPACE=$3
+is_lkg_drop=$4
+DATASET=$5
+MODE='jit'
+
+# Create the output directory in case it doesn't already exist
+mkdir -p ${OUTPUT_DIR}
+
+if [[ "${is_lkg_drop}" == "true" ]]; then
+  source ${WORKSPACE}/pytorch_setup/bin/activate pytorch
+fi
+
+export LD_PRELOAD="${WORKSPACE}/jemalloc/lib/libjemalloc.so":"${WORKSPACE}/tcmalloc/lib/libtcmalloc.so":"/usr/local/lib/libiomp5.so":$LD_PRELOAD 
+export MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+export DNNL_MAX_CPU_ISA=AVX512_CORE_AMX
+
+# Install dependency:
+./quickstart/object_detection/pytorch/maskrcnn/inference/cpu/setup.sh
+
+# Install model:
+cd models/object_detection/pytorch/maskrcnn/maskrcnn-benchmark/
+python setup.py develop
+cd - 
+
+# Install pre-trained model:
+export CHECKPOINT_DIR=$(pwd)/tests/cicd/output/PyTorch/maskrcnn-inference/${PRECISION}
+bash quickstart/object_detection/pytorch/maskrcnn/inference/cpu/download_model.sh
+
+# Run script
+OUTPUT_DIR=${OUTPUT_DIR} DATASET_DIR=${DATASET} PRECISION=${PRECISION} CHECKPOINT_DIR=${CHECKPOINT_DIR} MODE=${MODE} ./quickstart/object_detection/pytorch/maskrcnn/inference/cpu/${SCRIPT}
diff --git a/tests/cicd/PyTorch/maskrcnn-training/run_test.sh b/tests/cicd/PyTorch/maskrcnn-training/run_test.sh
new file mode 100755
index 000000000..82219892e
--- /dev/null
+++ b/tests/cicd/PyTorch/maskrcnn-training/run_test.sh
@@ -0,0 +1,33 @@
+#!/bin/bash
+set -e
+
+echo "Setup PyTorch Test Enviroment for Mask R-CNN Training"
+
+PRECISION=$1
+SCRIPT=$2
+OUTPUT_DIR=${OUTPUT_DIR-"$(pwd)/tests/cicd/output/PyTorch/maskrcnn-training/${SCRIPT}/${PRECISION}"}
+WORKSPACE=$3
+is_lkg_drop=$4
+DATASET=$5
+
+# Create the output directory in case it doesn't already exist
+mkdir -p ${OUTPUT_DIR}
+
+if [[ "${is_lkg_drop}" == "true" ]]; then
+  source ${WORKSPACE}/pytorch_setup/bin/activate pytorch
+fi
+
+export LD_PRELOAD="${WORKSPACE}/jemalloc/lib/libjemalloc.so":"${WORKSPACE}/tcmalloc/lib/libtcmalloc.so":"/usr/local/lib/libiomp5.so":$LD_PRELOAD 
+export MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+export DNNL_MAX_CPU_ISA=AVX512_CORE_AMX
+
+# Install dependency:
+./quickstart/object_detection/pytorch/maskrcnn/training/cpu/setup.sh
+
+# Install model:
+cd models/object_detection/pytorch/maskrcnn/maskrcnn-benchmark/
+python setup.py develop
+cd - 
+
+# Run script
+OUTPUT_DIR=${OUTPUT_DIR} DATASET_DIR=${DATASET} PRECISION=${PRECISION} ./quickstart/object_detection/pytorch/maskrcnn/training/cpu/${SCRIPT}
diff --git a/tests/cicd/PyTorch/resnet50-inference/run_test.sh b/tests/cicd/PyTorch/resnet50-inference/run_test.sh
index 1bba7a070..40ef7ca4b 100755
--- a/tests/cicd/PyTorch/resnet50-inference/run_test.sh
+++ b/tests/cicd/PyTorch/resnet50-inference/run_test.sh
@@ -14,15 +14,12 @@ DATASET=$5
 mkdir -p ${OUTPUT_DIR}
 
 if [[ "${is_lkg_drop}" == "true" ]]; then
-  #export PATH=${WORKSPACE}/miniconda3/bin:$PATH
-  #source ${WORKSPACE}/pytorch_setup/setvars.sh
-  #source ${WORKSPACE}/pytorch_setup/compiler/latest/env/vars.sh
-  #source ${WORKSPACE}/pytorch_setup/mkl/latest/env/vars.sh
-  #source ${WORKSPACE}/pytorch_setup/tbb/latest/env/vars.sh
-  #source ${WORKSPACE}/pytorch_setup/mpi/latest/env/vars.sh
   source ${WORKSPACE}/pytorch_setup/bin/activate pytorch
-  #conda activate pytorch
 fi
 
+export LD_PRELOAD="${WORKSPACE}/jemalloc/lib/libjemalloc.so":"${WORKSPACE}/tcmalloc/lib/libtcmalloc.so":"/usr/local/lib/libiomp5.so":$LD_PRELOAD 
+export MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+export DNNL_MAX_CPU_ISA=AVX512_CORE_AMX
+
 # Run script
-OUTPUT_DIR=${OUTPUT_DIR} DATASET_DIR=${DATASET} ./quickstart/image_recognition/pytorch/resnet50/inference/cpu/${PRECISION}/${SCRIPT}
+OUTPUT_DIR=${OUTPUT_DIR} DATASET_DIR=${DATASET} PRECISION=${PRECISION} ./quickstart/image_recognition/pytorch/resnet50/inference/cpu/${SCRIPT}
diff --git a/tests/cicd/PyTorch/resnet50-training/run_test.sh b/tests/cicd/PyTorch/resnet50-training/run_test.sh
new file mode 100755
index 000000000..e2c01948c
--- /dev/null
+++ b/tests/cicd/PyTorch/resnet50-training/run_test.sh
@@ -0,0 +1,26 @@
+#!/bin/bash
+set -e
+
+echo "Setup PyTorch Test Enviroment for ResNet50 Training"
+
+PRECISION=$1
+SCRIPT=$2
+OUTPUT_DIR=${OUTPUT_DIR-"$(pwd)/tests/cicd/output/PyTorch/resnet50-training/${SCRIPT}/${PRECISION}"}
+WORKSPACE=$3
+is_lkg_drop=$4
+DATASET=$5
+TRAINING_EPOCHS="1"
+
+# Create the output directory in case it doesn't already exist
+mkdir -p ${OUTPUT_DIR}
+
+if [[ "${is_lkg_drop}" == "true" ]]; then
+  source ${WORKSPACE}/pytorch_setup/bin/activate pytorch
+fi
+
+export LD_PRELOAD="${WORKSPACE}/jemalloc/lib/libjemalloc.so":"${WORKSPACE}/tcmalloc/lib/libtcmalloc.so":"/usr/local/lib/libiomp5.so":$LD_PRELOAD 
+export MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+export DNNL_MAX_CPU_ISA=AVX512_CORE_AMX
+
+# Run script
+OUTPUT_DIR=${OUTPUT_DIR} DATASET_DIR=${DATASET} PRECISION=${PRECISION} TRAINING_EPOCHS=${TRAINING_EPOCHS} ./quickstart/image_recognition/pytorch/resnet50/training/cpu/${SCRIPT}
diff --git a/tests/cicd/PyTorch/resnext32x16d-inference/run_test.sh b/tests/cicd/PyTorch/resnext32x16d-inference/run_test.sh
new file mode 100755
index 000000000..27c64728b
--- /dev/null
+++ b/tests/cicd/PyTorch/resnext32x16d-inference/run_test.sh
@@ -0,0 +1,25 @@
+#!/bin/bash
+set -e
+
+echo "Setup PyTorch Test Enviroment for ResNext32x16d Inference"
+
+PRECISION=$1
+SCRIPT=$2
+OUTPUT_DIR=${OUTPUT_DIR-"$(pwd)/tests/cicd/output/PyTorch/resnext32x16d-inference/${SCRIPT}/${PRECISION}"}
+WORKSPACE=$3
+is_lkg_drop=$4
+DATASET=$5
+
+# Create the output directory in case it doesn't already exist
+mkdir -p ${OUTPUT_DIR}
+
+if [[ "${is_lkg_drop}" == "true" ]]; then
+  source ${WORKSPACE}/pytorch_setup/bin/activate pytorch
+fi
+
+export LD_PRELOAD="${WORKSPACE}/jemalloc/lib/libjemalloc.so":"${WORKSPACE}/tcmalloc/lib/libtcmalloc.so":"/usr/local/lib/libiomp5.so":$LD_PRELOAD 
+export MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+export DNNL_MAX_CPU_ISA=AVX512_CORE_AMX
+
+# Run script
+OUTPUT_DIR=${OUTPUT_DIR} DATASET_DIR=${DATASET} PRECISION=${PRECISION} ./quickstart/image_recognition/pytorch/resnext-32x16d/inference/cpu/${SCRIPT}
diff --git a/tests/cicd/PyTorch/rnnt-inference/run_test.sh b/tests/cicd/PyTorch/rnnt-inference/run_test.sh
new file mode 100755
index 000000000..5336e3e8d
--- /dev/null
+++ b/tests/cicd/PyTorch/rnnt-inference/run_test.sh
@@ -0,0 +1,40 @@
+#!/bin/bash
+set -e
+
+echo "Setup PyTorch Test Enviroment for RNN-T Inference"
+
+PRECISION=$1
+SCRIPT=$2
+OUTPUT_DIR=${OUTPUT_DIR-"$(pwd)/tests/cicd/output/PyTorch/rnnt-inference/${SCRIPT}/${PRECISION}"}
+WORKSPACE=$3
+is_lkg_drop=$4
+DATASET=$5
+
+# Create the output directory in case it doesn't already exist
+mkdir -p ${OUTPUT_DIR}
+
+if [[ "${is_lkg_drop}" == "true" ]]; then
+  source ${WORKSPACE}/pytorch_setup/bin/activate pytorch
+fi
+ 
+export LD_PRELOAD="${WORKSPACE}/jemalloc/lib/libjemalloc.so":"${WORKSPACE}/tcmalloc/lib/libtcmalloc.so":"/usr/local/lib/libiomp5.so":$LD_PRELOAD 
+export MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+export DNNL_MAX_CPU_ISA=AVX512_CORE_AMX
+
+# Install model:
+./quickstart/language_modeling/pytorch/rnnt/inference/cpu/install_dependency_baremetal.sh
+
+# Install pre-trained model:
+export CHECKPOINT_DIR=$(pwd)/tests/cicd/output/PyTorch/rnnt-inference/${PRECISION}
+bash quickstart/language_modeling/pytorch/rnnt/inference/cpu/download_model.sh
+
+# Download dataset
+if [ -z ${DATASET} ];then
+  export DATASET_DIR=$(pwd)/tests/cicd/PyTorch/rnnt-inference/
+  ./quickstart/language_modeling/pytorch/rnnt/inference/cpu/download_dataset.sh
+else
+  DATASET_DIR=${DATASET}
+fi
+
+# Run script
+OUTPUT_DIR=${OUTPUT_DIR} DATASET_DIR=${DATASET} PRECISION=${PRECISION} CHECKPOINT_DIR=${CHECKPOINT_DIR} ./quickstart/language_modeling/pytorch/rnnt/inference/cpu/${SCRIPT}
diff --git a/tests/cicd/PyTorch/rnnt-training/run_test.sh b/tests/cicd/PyTorch/rnnt-training/run_test.sh
new file mode 100755
index 000000000..646ec7db6
--- /dev/null
+++ b/tests/cicd/PyTorch/rnnt-training/run_test.sh
@@ -0,0 +1,37 @@
+#!/bin/bash
+set -e
+
+echo "Setup PyTorch Test Enviroment for RNN-T Training"
+
+PRECISION=$1
+SCRIPT=$2
+OUTPUT_DIR=${OUTPUT_DIR-"$(pwd)/tests/cicd/output/PyTorch/rnnt-training/${SCRIPT}/${PRECISION}"}
+WORKSPACE=$3
+is_lkg_drop=$4
+DATASET=$5
+
+# Create the output directory in case it doesn't already exist
+mkdir -p ${OUTPUT_DIR}
+
+if [[ "${is_lkg_drop}" == "true" ]]; then
+  source ${WORKSPACE}/pytorch_setup/bin/activate pytorch
+fi
+
+#Build torch-ccl:
+git clone https://github.com/intel-innersource/frameworks.ai.pytorch.torch-ccl.git
+# torch-ccl branch refer to https://github.com/intel-innersource/frameworks.ai.pytorch.ipex-cpu/blob/cpu-device/dependency_version.yml
+cd frameworks.ai.pytorch.torch-ccl
+git submodule sync 
+git submodule update --init --recursive
+python setup.py install 
+cd - 
+
+export LD_PRELOAD="${WORKSPACE}/jemalloc/lib/libjemalloc.so":"${WORKSPACE}/tcmalloc/lib/libtcmalloc.so":"/usr/local/lib/libiomp5.so":$LD_PRELOAD 
+export MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+export DNNL_MAX_CPU_ISA=AVX512_CORE_AMX
+
+# Install model dependencies:
+./quickstart/language_modeling/pytorch/rnnt/training/cpu/install_dependency.sh
+
+# Run script
+OUTPUT_DIR=${OUTPUT_DIR} DATASET_DIR=${DATASET} PRECISION=${PRECISION} ./quickstart/language_modeling/pytorch/rnnt/training/cpu/${SCRIPT}
diff --git a/tests/cicd/PyTorch/setup.sh b/tests/cicd/PyTorch/setup.sh
index 70e10a761..384bf5a3d 100755
--- a/tests/cicd/PyTorch/setup.sh
+++ b/tests/cicd/PyTorch/setup.sh
@@ -24,12 +24,91 @@ if [[ "${is_lkg_drop}" == "true" ]]; then
     python -m pip install -r requirements.txt
     python cdt.py --username=tf_qa_prod --password ${TF_QA_PROD} download --product ipytorch --release ${AIKIT_RELEASE} -c l_drop_installer --download-dir ${WORKSPACE}/pytorch_setup
     cd ${WORKSPACE}/pytorch_setup
-    chmod +x IPEX_installer-2024.0.0.*
-    ./IPEX_installer-2024.0.0.* -b -u -p ${WORKSPACE}/pytorch_setup
+    chmod +x IPEX_installer-*
+    ./IPEX_installer-* -b -u -p ${WORKSPACE}/pytorch_setup
   fi
 else
   pip install --upgrade pip
   echo "Installing pytorch"
   pip install intel-extension-for-pytorch==${PY_VERSION}
-  python -m pip install torch==$1 torchvision==0.15.1
+  python -m pip install torch==$1 torchvision==0.16.0
 fi
+
+# Check the operating system type
+os_type=$(awk -F= '/^NAME/{print $2}' /etc/os-release)
+
+# Install model specific dependencies:
+if [[ "$os_name" == *"CentOS"* ]]; then
+  echo "CentOS detected. Using yum for package management."
+  yum update -y
+  yum install -y \
+    ca-certificates \
+    git \
+    cmake>=3.19.6 \
+    make \
+    autoconf \
+    bzip2 \
+    tar
+  yum install -y \
+    numactl \
+    mesa-libGL 
+  yum install -y libsndfile
+  yum clean all
+  yum install mesa-libGL
+elif [[ "$os_name" == *"Ubuntu"* ]]; then
+  echo "Ubuntu detected. Using apt-get for package management."
+  apt-get update
+  apt-get install --no-install-recommends --fix-missing -y \
+    build-essential \
+    ca-certificates \
+    git \
+    wget \
+    make \
+    cmake \
+    autoconf \
+    bzip2 \
+    tar
+  apt-get install cmake
+  apt-get install --no-install-recommends --fix-missing -y \
+    numactl \
+    libgl1 \
+    libglib2.0-0 \
+    libegl1-mesa 
+  apt-get install -y python3-dev
+  apt-get install -y gcc python3.10-dev
+  apt-get install -y libgl1-mesa-glx
+fi
+
+cd ${WORKSPACE}
+if [ -d "jemalloc" ]; then
+  echo "Repository already exists. Skipping clone."
+else
+  unset LD_PRELOAD
+  unset MALLOC_CONF
+  git clone https://github.com/jemalloc/jemalloc.git
+  cd jemalloc
+  git checkout c8209150f9d219a137412b06431c9d52839c7272
+  ./autogen.sh
+  ./configure --prefix=${WORKSPACE}/
+  make
+  make install
+  cd -
+fi
+
+pip install packaging intel-openmp
+
+if [ -d "gperftools-2.7.90" ]; then
+  echo "The gperftools directory exists. Skipping download and extraction."
+else
+  wget https://github.com/gperftools/gperftools/releases/download/gperftools-2.7.90/gperftools-2.7.90.tar.gz
+  tar -xzf gperftools-2.7.90.tar.gz
+  cd gperftools-2.7.90
+  ./configure --prefix=${WORKSPACE}/tcmalloc
+  make
+  make install
+  cd -
+fi
+
+export LD_PRELOAD="${WORKSPACE}/jemalloc/lib/libjemalloc.so":"${WORKSPACE}/tcmalloc/lib/libtcmalloc.so":"/usr/local/lib/libiomp5.so":$LD_PRELOAD 
+export MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+export DNNL_MAX_CPU_ISA=AVX512_CORE_AMX
diff --git a/tests/cicd/PyTorch/ssd-resnet34-inference/run_test.sh b/tests/cicd/PyTorch/ssd-resnet34-inference/run_test.sh
index dc9a8623a..61a9af152 100755
--- a/tests/cicd/PyTorch/ssd-resnet34-inference/run_test.sh
+++ b/tests/cicd/PyTorch/ssd-resnet34-inference/run_test.sh
@@ -14,27 +14,27 @@ DATASET=$5
 mkdir -p ${OUTPUT_DIR}
 
 if [[ "${is_lkg_drop}" == "true" ]]; then
-  #export PATH=${WORKSPACE}/miniconda3/bin:$PATH
-  #source ${WORKSPACE}/pytorch_setup/setvars.sh
-  #source ${WORKSPACE}/pytorch_setup/compiler/latest/env/vars.sh
-  #source ${WORKSPACE}/pytorch_setup/mkl/latest/env/vars.sh
-  #source ${WORKSPACE}/pytorch_setup/tbb/latest/env/vars.sh
-  #source ${WORKSPACE}/pytorch_setup/mpi/latest/env/vars.sh
   source ${WORKSPACE}/pytorch_setup/bin/activate pytorch
-  #conda activate pytorch
 fi
 
-# run model specific dependencies:
-pip install matplotlib Pillow defusedxml
-pip install pycocotools-fix
+export LD_PRELOAD="${WORKSPACE}/jemalloc/lib/libjemalloc.so":"${WORKSPACE}/tcmalloc/lib/libtcmalloc.so":"/usr/local/lib/libiomp5.so":$LD_PRELOAD 
+export MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+export DNNL_MAX_CPU_ISA=AVX512_CORE_AMX
+
+# Install dependenicies:
+./quickstart/object_detection/pytorch/ssd-resnet34/inference/cpu/setup.sh
 
 # Download Pretrained Model:
-export CHECKPOINT_DIR=$(pwd)/tests/cicd/PyTorch/ssd-resnet34-inference/
+export CHECKPOINT_DIR=$(pwd)/tests/cicd/PyTorch/ssd-resnet34-inference/${PRECISION}
 ./quickstart/object_detection/pytorch/ssd-resnet34/inference/cpu/download_model.sh
 
 # Download dataset
-export DATASET_DIR=$(pwd)/tests/cicd/PyTorch/ssd-resnet34-inference/
-./quickstart/object_detection/pytorch/ssd-resnet34/inference/cpu/download_dataset.sh
+if [ -z ${DATASET} ];then
+  export DATASET_DIR=$(pwd)/tests/cicd/PyTorch/ssd-resnet34-inference/
+  ./quickstart/object_detection/pytorch/ssd-resnet34/inference/cpu/download_dataset.sh
+else
+  DATASET_DIR=${DATASET}
+fi
 
 # Run script
-OUTPUT_DIR=${OUTPUT_DIR} CHECKPOINT_DIR=${CHECKPOINT_DIR} DATASET_DIR=${DATASET_DIR} ./quickstart/object_detection/pytorch/ssd-resnet34/inference/cpu/${SCRIPT} ${PRECISION}
+OUTPUT_DIR=${OUTPUT_DIR} CHECKPOINT_DIR=${CHECKPOINT_DIR} DATASET_DIR=${DATASET_DIR} PRECISION=${PRECISION} ./quickstart/object_detection/pytorch/ssd-resnet34/inference/cpu/${SCRIPT}
diff --git a/tests/cicd/PyTorch/ssd-resnet34-training/run_test.sh b/tests/cicd/PyTorch/ssd-resnet34-training/run_test.sh
new file mode 100755
index 000000000..2ab7e274e
--- /dev/null
+++ b/tests/cicd/PyTorch/ssd-resnet34-training/run_test.sh
@@ -0,0 +1,40 @@
+#!/bin/bash
+set -e
+
+echo "Setup PyTorch Test Enviroment for SSD-ResNet34 Training"
+
+PRECISION=$1
+SCRIPT=$2
+OUTPUT_DIR=${OUTPUT_DIR-"$(pwd)/tests/cicd/output/PyTorch/ssd-resnet34-training/${SCRIPT}/${PRECISION}"}
+WORKSPACE=$3
+is_lkg_drop=$4
+DATASET=$5
+
+# Create the output directory in case it doesn't already exist
+mkdir -p ${OUTPUT_DIR}
+
+if [[ "${is_lkg_drop}" == "true" ]]; then
+  source ${WORKSPACE}/pytorch_setup/bin/activate pytorch
+fi
+
+export LD_PRELOAD="${WORKSPACE}/jemalloc/lib/libjemalloc.so":"${WORKSPACE}/tcmalloc/lib/libtcmalloc.so":"/usr/local/lib/libiomp5.so":$LD_PRELOAD 
+export MALLOC_CONF="oversize_threshold:1,background_thread:true,metadata_thp:auto,dirty_decay_ms:9000000000,muzzy_decay_ms:9000000000"
+export DNNL_MAX_CPU_ISA=AVX512_CORE_AMX
+
+# Install dependenicies:
+./quickstart/object_detection/pytorch/ssd-resnet34/training/cpu/setup.sh
+
+# Download Pretrained Model:
+export CHECKPOINT_DIR=$(pwd)/tests/cicd/PyTorch/ssd-resnet34-training/${PRECISION}
+./quickstart/object_detection/pytorch/ssd-resnet34/training/cpu/download_model.sh
+
+# Download dataset
+if [ -z ${DATASET} ];then
+  export DATASET_DIR=$(pwd)/tests/cicd/PyTorch/ssd-resnet34-inference/
+  ./quickstart/object_detection/pytorch/ssd-resnet34/training/cpu/download_dataset.sh
+else
+  DATASET_DIR=${DATASET}
+fi
+
+# Run script
+OUTPUT_DIR=${OUTPUT_DIR} CHECKPOINT_DIR=${CHECKPOINT_DIR} DATASET_DIR=${DATASET_DIR} PRECISION=${PRECISION} ./quickstart/object_detection/pytorch/ssd-resnet34/training/cpu/${SCRIPT}
diff --git a/tests/cicd/TensorFlow/setup.sh b/tests/cicd/TensorFlow/setup.sh
index f7423d937..503eb08f9 100755
--- a/tests/cicd/TensorFlow/setup.sh
+++ b/tests/cicd/TensorFlow/setup.sh
@@ -24,8 +24,8 @@ if [[ "${is_lkg_drop}" == "true" ]]; then
     python -m pip install -r requirements.txt
     python cdt.py --username=tf_qa_prod --password ${TF_QA_PROD} download --product tensorflow --release ${AIKIT_RELEASE} -c l_drop_installer --download-dir ${WORKSPACE}/tensorflow_setup
     cd ${WORKSPACE}/tensorflow_setup
-    chmod +x ITEX_installer-2024.0.0.*
-    ./ITEX_installer-2024.0.0.* -b -u -p ${WORKSPACE}/tensorflow_setup
+    chmod +x ITEX_installer-*
+    ./ITEX_installer-* -b -u -p ${WORKSPACE}/tensorflow_setup
   fi
 else
   pip install --upgrade pip
diff --git a/tests/unit/common/tensorflow/tf_model_args/tf_faster_rcnn_args.json b/tests/unit/common/tensorflow/tf_model_args/tf_faster_rcnn_args.json
deleted file mode 100644
index a50701953..000000000
--- a/tests/unit/common/tensorflow/tf_model_args/tf_faster_rcnn_args.json
+++ /dev/null
@@ -1,32 +0,0 @@
-[
-  { "_comment": "FP32 accuracy command",
-    "input": "run_tf_benchmark.py --framework=tensorflow --use-case=object_detection --model-name=faster_rcnn --precision=fp32 --mode=inference --model-source-dir=/workspace/models --intelai-models=/workspace/intelai_models --num-cores=-1 --batch-size=1 --socket-id=-1 --accuracy-only  --verbose --checkpoint=/checkpoints         --data-location=/dataset         --in-graph=/in_graph/frozen_inference_graph.pb",
-    "output": "bash /workspace/intelai_models/inference/fp32/coco_accuracy.sh /in_graph/frozen_inference_graph.pb /dataset/coco_val.record /workspace/models",
-    "cpuset": "0-111"},
-
-  { "_comment": "FP32 benchmark command",
-    "input": "run_tf_benchmark.py --framework tensorflow --use-case object_detection --precision fp32 --mode inference --model-name faster_rcnn --checkpoint /checkpoints --intelai-models . --model-source-dir . --socket-id 0 --verbose --config_file=pipeline.config",
-    "output": "numactl --cpunodebind=0 --membind=0 python ./inference/fp32/eval.py --num_inter_threads 1 --num_intra_threads 28 --pipeline_config_path /checkpoints/pipeline.config --checkpoint_dir /checkpoints --eval_dir ./research/object_detection/log/eval",
-    "cpuset": "0-111"},
-
-  { "_comment": "FP32 benchmark command with custom --num_inter_threads 4 --num_intra_threads 16",
-    "input": "run_tf_benchmark.py --framework tensorflow --use-case object_detection --precision fp32 --mode inference --model-name faster_rcnn --checkpoint /checkpoints --intelai-models . --model-source-dir . --socket-id 0 --verbose --config_file=pipeline.config --num-inter-threads 4 --num-intra-threads 16",
-    "output": "numactl --cpunodebind=0 --membind=0 python ./inference/fp32/eval.py --num_inter_threads 4 --num_intra_threads 16 --pipeline_config_path /checkpoints/pipeline.config --checkpoint_dir /checkpoints --eval_dir ./research/object_detection/log/eval",
-    "cpuset": "0-111"},
-
-  { "_comment": "Int8 command for throughput benchmark with --number-of-steps enabled.",
-    "input": "run_tf_benchmark.py --framework=tensorflow --use-case=object_detection --model-name=faster_rcnn --precision=int8 --mode=inference --model-source-dir=/workspace/models --intelai-models=/workspace/intelai_models --num-cores=-1 --batch-size=-1 --socket-id=-1  --benchmark-only --verbose --in-graph=/in_graph/pretrained_int8_faster_rcnn_model.pb --data-location=/dataset --number-of-steps=500",
-    "output": "LD_PRELOAD=/usr/lib/libtcmalloc.so.4.2.6 python /workspace/intelai_models/inference/int8/run_frozen_graph_rcnn.py -g /in_graph/pretrained_int8_faster_rcnn_model.pb -n 500 -d /dataset --num-inter-threads 2 --num-intra-threads 56",
-    "cpuset": "0-111"},
-
-  { "_comment": "Int8 accuracy command",
-    "input": "run_tf_benchmark.py --framework=tensorflow --use-case=object_detection --model-name=faster_rcnn --precision=int8 --mode=inference --model-source-dir=/workspace/models --intelai-models=/workspace/intelai_models --num-cores=-1 --batch-size=-1 --socket-id=-1 --accuracy-only  --verbose --in-graph=/in_graph/pretrained_int8_faster_rcnn_model.pb --data-location=/dataset",
-    "output": "LD_PRELOAD=/usr/lib/libtcmalloc.so.4.2.6 /workspace/intelai_models/inference/int8/coco_int8.sh /in_graph/pretrained_int8_faster_rcnn_model.pb /dataset /workspace/models",
-    "cpuset": "0-111"},
-
-  { "_comment": "FP32 benchmark command",
-    "input": "run_tf_benchmark.py --framework=tensorflow --use-case=object_detection --model-name=faster_rcnn --precision=int8 --mode=inference --model-source-dir=/workspace/models --intelai-models=/workspace/intelai_models --num-cores=-1 --batch-size=-1 --socket-id=-1  --benchmark-only --verbose --in-graph=/in_graph/pretrained_int8_faster_rcnn_model.pb --data-location=/dataset",
-    "output": "LD_PRELOAD=/usr/lib/libtcmalloc.so.4.2.6 python /workspace/intelai_models/inference/int8/run_frozen_graph_rcnn.py -g /in_graph/pretrained_int8_faster_rcnn_model.pb -d /dataset --num-inter-threads 2 --num-intra-threads 56",
-    "cpuset": "0-111"}
-]
-
diff --git a/tests/unit/common/tensorflow/tf_model_args/tf_gnmt_args.json b/tests/unit/common/tensorflow/tf_model_args/tf_gnmt_args.json
deleted file mode 100644
index a054ab315..000000000
--- a/tests/unit/common/tensorflow/tf_model_args/tf_gnmt_args.json
+++ /dev/null
@@ -1,16 +0,0 @@
-[
-  { "_comment": "gnmt_fp32_latency",
-    "input": "run_tf_benchmark.py --framework=tensorflow --use-case=language_translation --model-name=mlperf_gnmt --precision=fp32 --mode=inference --model-source-dir=/workspace/models --benchmark-dir=/workspace/benchmarks --intelai-models=/workspace/intelai_models --num-cores=-1 --batch-size=1 --socket-id=0 --output-dir=/workspace/benchmarks/common/tensorflow/logs --benchmark-only --data-location=/dataset --in-graph=workspace/mlperf_gnmt_fp32_pretrained_model.pb",
-    "output": "numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/fp32/run_inference.py --in_graph=workspace/mlperf_gnmt_fp32_pretrained_model.pb --batch_size=1 --num_inter_threads=1 --num_intra_threads=28 --src_vocab_file=/dataset/vocab.bpe.32000.en --tgt_vocab_file=/dataset/vocab.bpe.32000.de --inference_input_file=/dataset/newstest2014.tok.bpe.32000.en --inference_ref_file=/dataset/newstest2014.tok.bpe.32000.de",
-    "cpuset": "0-111"},
-
-  { "_comment": "gnmt_fp32_throughput",
-    "input": "run_tf_benchmark.py --framework=tensorflow --use-case=language_translation --model-name=mlperf_gnmt --precision=fp32 --mode=inference --model-source-dir=/workspace/models --benchmark-dir=/workspace/benchmarks --intelai-models=/workspace/intelai_models --num-cores=-1 --batch-size=32 --socket-id=0 --output-dir=/workspace/benchmarks/common/tensorflow/logs --benchmark-only --data-location=/dataset --in-graph=workspace/mlperf_gnmt_fp32_pretrained_model.pb",
-    "output": "numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/fp32/run_inference.py --in_graph=workspace/mlperf_gnmt_fp32_pretrained_model.pb --batch_size=32 --num_inter_threads=1 --num_intra_threads=28 --src_vocab_file=/dataset/vocab.bpe.32000.en --tgt_vocab_file=/dataset/vocab.bpe.32000.de --inference_input_file=/dataset/newstest2014.tok.bpe.32000.en --inference_ref_file=/dataset/newstest2014.tok.bpe.32000.de",
-    "cpuset": "0-111"},
-
-  { "_comment": "gnmt_fp32_accuracy",
-    "input": "run_tf_benchmark.py --framework=tensorflow --use-case=language_translation --model-name=mlperf_gnmt --precision=fp32 --mode=inference --model-source-dir=/workspace/models --benchmark-dir=/workspace/benchmarks --intelai-models=/workspace/intelai_models --num-cores=-1 --batch-size=32 --socket-id=0 --output-dir=/workspace/benchmarks/common/tensorflow/logs --accuracy-only --data-location=/dataset --in-graph=workspace/mlperf_gnmt_fp32_pretrained_model.pb",
-    "output": "numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/fp32/run_inference.py --in_graph=workspace/mlperf_gnmt_fp32_pretrained_model.pb --batch_size=32 --num_inter_threads=1 --num_intra_threads=28 --src_vocab_file=/dataset/vocab.bpe.32000.en --tgt_vocab_file=/dataset/vocab.bpe.32000.de --inference_input_file=/dataset/newstest2014.tok.bpe.32000.en --inference_ref_file=/dataset/newstest2014.tok.bpe.32000.de",
-    "cpuset": "0-111"}
-]
diff --git a/tests/unit/common/tensorflow/tf_model_args/tf_gpt_j_args.json b/tests/unit/common/tensorflow/tf_model_args/tf_gpt_j_args.json
deleted file mode 100644
index ae31914d6..000000000
--- a/tests/unit/common/tensorflow/tf_model_args/tf_gpt_j_args.json
+++ /dev/null
@@ -1,17 +0,0 @@
-[
-    { "_comment": "gpt_j_fp32_accuracy",
-      "input": "run_tf_benchmark.py --framework=tensorflow --use-case=language_modeling --model-name=gpt_j --precision=fp32 --mode=inference --benchmark-dir=/workspace/intelai-models/benchmarks --intelai-models=/workspace/intelai-models/benchmarks/../models/language_modeling/tensorflow/gpt_j --num-cores=-1 --batch-size=-1 --socket-id=-1 --output-dir=/workspace/all_test_log/internal_GPT-J_fp32_bs100_Accuracy_instance2_accuracy --num-train-steps=1  --accuracy-only     --verbose  --checkpoint=/localdisk/gpt-j/checkpoint --num-inter-threads=1 --num-intra-threads=64 --disable-tcmalloc=True",
-      "output": "python /workspace/intelai-models/benchmarks/../models/language_modeling/tensorflow/gpt_j/inference/run_eval.py --model_name_or_path EleutherAI/gpt-j-6B --dataset_name EleutherAI/lambada_openai --precision fp32 --checkpoint=/localdisk/gpt-j/checkpoint --output_dir=/workspace/all_test_log/internal_GPT-J_fp32_bs100_Accuracy_instance2_accuracy",
-      "cpuset": "0-111"},
-  
-    { "_comment": "gpt_j_bf16_accuracy",
-      "input": "run_tf_benchmark.py --framework=tensorflow --use-case=language_modeling --model-name=gpt_j --precision=bfloat16 --mode=inference --benchmark-dir=/workspace/intelai-models/benchmarks --intelai-models=/workspace/intelai-models/benchmarks/../models/language_modeling/tensorflow/gpt_j --num-cores=-1 --batch-size=-1 --socket-id=-1 --output-dir=/workspace/all_test_log/internal_GPT-J_bfloat16_bs100_Accuracy_instance2_accuracy --num-train-steps=1  --accuracy-only     --verbose  --checkpoint=/localdisk/gpt-j/checkpoint --num-inter-threads=1 --num-intra-threads=64 --disable-tcmalloc=True",
-      "output": "python /workspace/intelai-models/benchmarks/../models/language_modeling/tensorflow/gpt_j/inference/run_eval.py --model_name_or_path EleutherAI/gpt-j-6B --dataset_name EleutherAI/lambada_openai --precision bfloat16 --checkpoint=/localdisk/gpt-j/checkpoint --output_dir=/workspace/all_test_log/internal_GPT-J_bfloat16_bs100_Accuracy_instance2_accuracy",
-      "cpuset": "0-111"},
-  
-    { "_comment": "gpt_j_fp16_accuracy",
-      "input": "run_tf_benchmark.py --framework=tensorflow --use-case=language_modeling --model-name=gpt_j --precision=fp16 --mode=inference --benchmark-dir=/workspace/intelai-models/benchmarks --intelai-models=/workspace/intelai-models/benchmarks/../models/language_modeling/tensorflow/gpt_j --num-cores=-1 --batch-size=-1 --socket-id=-1 --output-dir=/workspace/all_test_log/internal_GPT-J_fp16_bs100_Accuracy_instance2_accuracy --num-train-steps=1  --accuracy-only     --verbose  --checkpoint=/localdisk/gpt-j/checkpoint --num-inter-threads=1 --num-intra-threads=64 --disable-tcmalloc=True",
-      "output": "python /workspace/intelai-models/benchmarks/../models/language_modeling/tensorflow/gpt_j/inference/run_eval.py --model_name_or_path EleutherAI/gpt-j-6B --dataset_name EleutherAI/lambada_openai --precision fp16 --checkpoint=/localdisk/gpt-j/checkpoint --output_dir=/workspace/all_test_log/internal_GPT-J_fp16_bs100_Accuracy_instance2_accuracy",
-      "cpuset": "0-111"}
-  
-  ]
diff --git a/tests/unit/common/tensorflow/tf_model_args/tf_graphsage_args.json b/tests/unit/common/tensorflow/tf_model_args/tf_graphsage_args.json
deleted file mode 100644
index cd892d446..000000000
--- a/tests/unit/common/tensorflow/tf_model_args/tf_graphsage_args.json
+++ /dev/null
@@ -1,31 +0,0 @@
-[
-  { "_comment": "graphsage_fp32_accuracy",
-    "input": "run_tf_benchmark.py --framework=tensorflow --use-case=graph_networks --model-name=graphsage --precision=fp32 --mode=inference --intelai-models=/workspace/intelai_models --batch-size 128 --socket-id 0 --accuracy-only  --verbose --accuracy-only --data-location=/dataset --pretrained-model=model/",
-    "output": "numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/eval_ppi_graphsage.py --precision=fp32 --num-inter-threads=1 --num-intra-threads=28 --batch-size=128  --data-location=/dataset --accuracy-only --pretrained-model=model/ --steps=20",
-    "cpuset": "0-111"},
-
-  { "_comment": "graphsage_inference_benchmark_fp32",
-    "input": "run_tf_benchmark.py --framework tensorflow --use-case graph_networks --precision fp32 --mode inference --model-name graphsage --batch-size 128 --data-location=/dataset --pretrained-model=model/ --intelai-models=/workspace/intelai_models --socket-id 0 --verbose",
-    "output": "numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/eval_ppi_graphsage.py --precision=fp32 --num-inter-threads=1 --num-intra-threads=28 --data-location=/dataset --pretrained-model=model/ --batch-size=128 --steps=20",
-    "cpuset": "0-111"},
-
-  { "_comment": "graphsage_bfloat16_accuracy",
-    "input": "run_tf_benchmark.py --framework=tensorflow --use-case=graph_networks --model-name=graphsage --precision=bfloat16 --mode=inference --intelai-models=/workspace/intelai_models --batch-size 128 --socket-id 0 --accuracy-only  --verbose --accuracy-only --data-location=/dataset --pretrained-model=model/",
-    "output": "numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/eval_ppi_graphsage.py --precision=bfloat16 --num-inter-threads=1 --num-intra-threads=28 --batch-size=128  --data-location=/dataset --accuracy-only --pretrained-model=model/ --steps=20",
-    "cpuset": "0-111"},
-
-  { "_comment": "graphsage_inference_benchmark_bfloat16",
-    "input": "run_tf_benchmark.py --framework tensorflow --use-case graph_networks --precision bfloat16 --mode inference --model-name graphsage --batch-size 128 --data-location=/dataset --pretrained-model=model/ --intelai-models=/workspace/intelai_models --socket-id 0 --verbose",
-    "output": "numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/eval_ppi_graphsage.py --precision=bfloat16 --num-inter-threads=1 --num-intra-threads=28 --data-location=/dataset --pretrained-model=model/ --batch-size=128 --steps=20",
-    "cpuset": "0-111"},
-
-  { "_comment": "graphsage_fp16_accuracy",
-    "input": "run_tf_benchmark.py --framework=tensorflow --use-case=graph_networks --model-name=graphsage --precision=fp16 --mode=inference --intelai-models=/workspace/intelai_models --batch-size 128 --socket-id 0 --accuracy-only  --verbose --accuracy-only --data-location=/dataset --pretrained-model=model/",
-    "output": "numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/eval_ppi_graphsage.py --precision=fp16 --num-inter-threads=1 --num-intra-threads=28 --batch-size=128  --data-location=/dataset --accuracy-only --pretrained-model=model/ --steps=20",
-    "cpuset": "0-111"},
-
-  { "_comment": "graphsage_inference_benchmark_fp16",
-    "input": "run_tf_benchmark.py --framework tensorflow --use-case graph_networks --precision fp16 --mode inference --model-name graphsage --batch-size 128 --data-location=/dataset --pretrained-model=model/ --intelai-models=/workspace/intelai_models --socket-id 0 --verbose",
-    "output": "numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/eval_ppi_graphsage.py --precision=fp16 --num-inter-threads=1 --num-intra-threads=28 --data-location=/dataset --pretrained-model=model/ --batch-size=128 --steps=20",
-    "cpuset": "0-111"}
-]
diff --git a/tests/unit/common/tensorflow/tf_model_args/tf_inceptionv4_args.json b/tests/unit/common/tensorflow/tf_model_args/tf_inceptionv4_args.json
deleted file mode 100644
index b8a009dcb..000000000
--- a/tests/unit/common/tensorflow/tf_model_args/tf_inceptionv4_args.json
+++ /dev/null
@@ -1,26 +0,0 @@
-[
-  { "_comment": "inceptionv4_int8_accuracy",
-    "input": "run_tf_benchmark.py --framework=tensorflow --use-case=image_recognition --precision=int8 --mode=inference --model-name=inceptionv4 --model-source-dir=/workspace/models --intelai-models=/workspace/intelai_models --num-cores=-1 --batch-size=240 --socket-id 0 --accuracy-only --verbose --in-graph=/in_graph/inceptionv4_int8_pretrained_model.pb --data-location=/dataset",
-    "output": "LD_PRELOAD=/usr/lib/libtcmalloc.so.4.2.6 numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/accuracy.py --batch_size=240 --num_inter_threads=2 --input_graph=/in_graph/inceptionv4_int8_pretrained_model.pb --num_intra_threads=28 --data_location=/dataset",
-    "cpuset": "0-111"},
-
-  { "_comment": "inceptionv4_int8_latency_default_inter_intra",
-    "input": "run_tf_benchmark.py --framework=tensorflow --use-case=image_recognition --precision=int8 --mode=inference --model-name=inceptionv4 --model-source-dir=/workspace/models --intelai-models=/workspace/intelai_models --num-cores=-1 --batch-size=1 --socket-id 0 --benchmark-only --verbose --in-graph=/in_graph/inceptionv4_int8_pretrained_model.pb --data-location=/dataset",
-    "output": "LD_PRELOAD=/usr/lib/libtcmalloc.so.4.2.6 numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/benchmark.py --batch_size=1 --num_inter_threads=2 --input_graph=/in_graph/inceptionv4_int8_pretrained_model.pb --num_intra_threads=28",
-    "cpuset": "0-111"},
-
-  { "_comment": "inceptionv4_int8_throughput",
-    "input": "run_tf_benchmark.py --framework=tensorflow --use-case=image_recognition --precision=int8 --mode=inference --model-name=inceptionv4 --model-source-dir=/workspace/models --intelai-models=/workspace/intelai_models --num-cores=-1 --batch-size=240 --socket-id 0 --benchmark-only --verbose --in-graph=/in_graph/inceptionv4_int8_pretrained_model.pb --data-location=/dataset",
-    "output": "LD_PRELOAD=/usr/lib/libtcmalloc.so.4.2.6 numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/benchmark.py --batch_size=240 --num_inter_threads=2 --input_graph=/in_graph/inceptionv4_int8_pretrained_model.pb --num_intra_threads=28",
-    "cpuset": "0-111"},
-
-  { "_comment": "inceptionv4_fp32_accuracy",
-    "input": "run_tf_benchmark.py --framework=tensorflow --use-case=image_recognition --precision=fp32 --mode=inference --model-name=inceptionv4 --model-source-dir=/workspace/models --intelai-models=/workspace/intelai_models --num-cores=-1 --batch-size=240 --socket-id 0 --accuracy-only --verbose --in-graph=/in_graph/inceptionv4_fp32_pretrained_model.pb --data-location=/dataset",
-    "output": "numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/accuracy.py --batch_size=240 --num_inter_threads=2 --input_graph=/in_graph/inceptionv4_fp32_pretrained_model.pb --num_intra_threads=28 --data_location=/dataset",
-    "cpuset": "0-111"},
-
-  { "_comment": "inceptionv4_fp32_batch_inf",
-    "input": "run_tf_benchmark.py --framework=tensorflow --use-case=image_recognition --precision=fp32 --mode=inference --model-name=inceptionv4 --model-source-dir=/workspace/models --intelai-models=/workspace/intelai_models --num-cores=-1 --batch-size=240 --socket-id 0 --benchmark-only --in-graph=/in_graph/inceptionv4_fp32_pretrained_model.pb --data-location=/dataset --steps=200",
-    "output": "numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/benchmark.py --batch_size=240 --num_intra_threads=28 --num_inter_threads=2 --input_graph=/in_graph/inceptionv4_fp32_pretrained_model.pb --warmup_steps=10 --steps=200",
-    "cpuset": "0-111"}
-]
diff --git a/tests/unit/common/tensorflow/tf_model_args/tf_mmoe_args.json b/tests/unit/common/tensorflow/tf_model_args/tf_mmoe_args.json
deleted file mode 100644
index aa0175426..000000000
--- a/tests/unit/common/tensorflow/tf_model_args/tf_mmoe_args.json
+++ /dev/null
@@ -1,46 +0,0 @@
-[
-  { "_comment": "mmoe_fp32_accuracy",
-    "input": "run_tf_benchmark.py --framework=tensorflow --use-case=recommendation --model-name=mmoe --precision=fp32 --mode=inference --model-source-dir=/workspace/models --intelai-models=/workspace/intelai_models --batch-size 256 --socket-id 0 --accuracy-only  --verbose --in-graph=/in_graph/saved_model.pb --accuracy-only --data-location=/dataset",
-    "output": "numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/eval_census_income_dataset.py --precision=fp32 --input-graph=/in_graph/saved_model.pb --num-inter-threads=1 --num-intra-threads=28 --batch-size=256  --warmup-steps=20 --steps=200 --data-location=/dataset --accuracy-only",
-    "cpuset": "0-111"},
-
-  { "_comment": "mmoe_inference_benchmark_fp32",
-    "input": "run_tf_benchmark.py --framework tensorflow --use-case recommendation --precision fp32 --mode inference --model-name mmoe --batch-size 256 --in-graph /saved_model.pb --intelai-models . --socket-id 0 --verbose",
-    "output": "numactl --cpunodebind=0 --membind=0 python ./inference/eval_census_income_dataset.py --precision=fp32 --input-graph=/saved_model.pb --num-inter-threads=1 --num-intra-threads=28 --batch-size=256 --warmup-steps=20 --steps=200",
-    "cpuset": "0-111"},
-
-  { "_comment": "mmoe_bf16_accuracy",
-  "input": "run_tf_benchmark.py --framework=tensorflow --use-case=recommendation --model-name=mmoe --precision=bfloat16 --mode=inference --model-source-dir=/workspace/models --intelai-models=/workspace/intelai_models --batch-size 256 --socket-id 0 --accuracy-only  --verbose --in-graph=/in_graph/saved_model.pb --accuracy-only --data-location=/dataset",
-  "output": "numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/eval_census_income_dataset.py --precision=bfloat16 --input-graph=/in_graph/saved_model.pb --num-inter-threads=1 --num-intra-threads=28 --batch-size=256  --warmup-steps=20 --steps=200 --data-location=/dataset --accuracy-only",
-  "cpuset": "0-111"},
-
-  { "_comment": "mmoe_inference_benchmark_bf16",
-    "input": "run_tf_benchmark.py --framework tensorflow --use-case recommendation --precision bfloat16 --mode inference --model-name mmoe --batch-size 256 --in-graph /saved_model.pb --intelai-models . --socket-id 0 --verbose",
-    "output": "numactl --cpunodebind=0 --membind=0 python ./inference/eval_census_income_dataset.py --precision=bfloat16 --input-graph=/saved_model.pb --num-inter-threads=1 --num-intra-threads=28 --batch-size=256 --warmup-steps=20 --steps=200",
-    "cpuset": "0-111"},
-
-  { "_comment": "mmoe_fp16_accuracy",
-  "input": "run_tf_benchmark.py --framework=tensorflow --use-case=recommendation --model-name=mmoe --precision=fp16 --mode=inference --model-source-dir=/workspace/models --intelai-models=/workspace/intelai_models --batch-size 256 --socket-id 0 --accuracy-only  --verbose --in-graph=/in_graph/saved_model.pb --accuracy-only --data-location=/dataset",
-  "output": "numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/eval_census_income_dataset.py --precision=fp16 --input-graph=/in_graph/saved_model.pb --num-inter-threads=1 --num-intra-threads=28 --batch-size=256  --warmup-steps=20 --steps=200 --data-location=/dataset --accuracy-only",
-  "cpuset": "0-111"},
-
-  { "_comment": "mmoe_inference_benchmark_fp16",
-    "input": "run_tf_benchmark.py --framework tensorflow --use-case recommendation --precision fp16 --mode inference --model-name mmoe --batch-size 256 --in-graph /saved_model.pb --intelai-models . --socket-id 0 --verbose",
-    "output": "numactl --cpunodebind=0 --membind=0 python ./inference/eval_census_income_dataset.py --precision=fp16 --input-graph=/saved_model.pb --num-inter-threads=1 --num-intra-threads=28 --batch-size=256 --warmup-steps=20 --steps=200",
-    "cpuset": "0-111"},
-
-  { "_comment": "mmoe_training_fp32",
-  "input": "run_tf_benchmark.py --framework=tensorflow --use-case=recommendation --model-name=mmoe --precision=fp32 --mode=training --intelai-models=/workspace/intelai_models --batch-size=256 --output-dir=/workspace/checkpoints --data-location=/dataset --train-epochs=6",
-  "output": "python /workspace/intelai_models/training/train.py --batch-size=256 --precision=fp32 --train-epochs=6 --data-location=/dataset --output-dir=/workspace/checkpoints",
-  "cpuset": "0-111"},
-
-  { "_comment": "mmoe_training_bfloat16",
-  "input": "run_tf_benchmark.py --framework=tensorflow --use-case=recommendation --model-name=mmoe --precision=bfloat16 --mode=training --intelai-models=/workspace/intelai_models --batch-size=256 --output-dir=/workspace/checkpoints --data-location=/dataset --train-epochs=6",
-  "output": "python /workspace/intelai_models/training/train.py --batch-size=256 --precision=bfloat16 --train-epochs=6 --data-location=/dataset --output-dir=/workspace/checkpoints",
-  "cpuset": "0-111"},
-
-  { "_comment": "mmoe_training_fp16",
-    "input": "run_tf_benchmark.py --framework=tensorflow --use-case=recommendation --model-name=mmoe --precision=fp16 --mode=training --intelai-models=/workspace/intelai_models --batch-size=256 --output-dir=/workspace/checkpoints --data-location=/dataset --train-epochs=6",
-    "output": "python /workspace/intelai_models/training/train.py --batch-size=256 --precision=fp16 --train-epochs=6 --data-location=/dataset --output-dir=/workspace/checkpoints",
-    "cpuset": "0-111"}
-]
diff --git a/tests/unit/common/tensorflow/tf_model_args/tf_ncf_args.json b/tests/unit/common/tensorflow/tf_model_args/tf_ncf_args.json
deleted file mode 100644
index 815ab35ac..000000000
--- a/tests/unit/common/tensorflow/tf_model_args/tf_ncf_args.json
+++ /dev/null
@@ -1,26 +0,0 @@
-[
-  { "_comment": "FP32 latency benchmark",
-    "input": "run_tf_benchmark.py --framework=tensorflow --use-case=recommendation --model-name=ncf --precision=fp32 --mode=inference --model-source-dir=/workspace/models --intelai-models=/workspace/intelai_models --num-cores=-1 --batch-size=1 --socket-id 0  --benchmark-only --verbose --checkpoint=/checkpoints     --data-location=/dataset",
-    "output": "numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/fp32/ncf_main.py --data_dir=/dataset --model_dir=/checkpoints --intra_op_parallelism_threads=28 --inter_op_parallelism_threads=1 --batch_size=1 --inference_only --benchmark_only",
-    "cpuset": "0-111"},
-
-  { "_comment": "Fp32 accuracy",
-    "input": "run_tf_benchmark.py --framework=tensorflow --use-case=recommendation --model-name=ncf --precision=fp32 --mode=inference --model-source-dir=/workspace/models --intelai-models=/workspace/intelai_models --num-cores=-1 --batch-size=256 --socket-id 0  --accuracy-only --verbose --checkpoint=/checkpoints     --data-location=/dataset",
-    "output": "numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/fp32/ncf_main.py --data_dir=/dataset --model_dir=/checkpoints --intra_op_parallelism_threads=28 --inter_op_parallelism_threads=1 --batch_size=256 --inference_only --accuracy_only",
-    "cpuset": "0-111"},
-
-  { "_comment": "FP32 Throughput benchmark",
-    "input": "run_tf_benchmark.py --framework=tensorflow --use-case=recommendation --model-name=ncf --precision=fp32 --mode=inference --model-source-dir=/workspace/models --intelai-models=/workspace/intelai_models --num-cores=-1 --batch-size=256 --socket-id 0  --benchmark-only --verbose --checkpoint=/checkpoints     --data-location=/dataset",
-    "output": "numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/fp32/ncf_main.py --data_dir=/dataset --model_dir=/checkpoints --intra_op_parallelism_threads=28 --inter_op_parallelism_threads=1 --batch_size=256 --inference_only --benchmark_only",
-    "cpuset": "0-111"},
-
-  { "_comment": "NCF FP32 Training",
-    "input": "run_tf_benchmark.py --framework=tensorflow --use-case=recommendation --model-name=ncf --precision=fp32 --mode=training --model-source-dir=/workspace/models --intelai-models=/workspace/intelai_models --num-cores=-1 --batch-size=98304 --num-inter-thread=2 --dataset=ml-20m --clean=1 --te=12",
-    "output": "python /workspace/intelai_models/training/ncf_estimator_main.py -dd=None -md=None -bs=98304 -hk=examplespersecondhook --dataset=ml-20m  --layers=256,256,128,64 --num_factors=64 --eval_batch_size 160000 --learning_rate 0.003821 --beta1 0.783529 --beta2 0.909003 --epsilon 1.45439e-07 --hr_threshold 0.635 --ml_perf --clean=1 --te=12",
-    "cpuset": "0-111"},
-
-   { "_comment": "NCF BFloat16 Training",
-     "input": "run_tf_benchmark.py --framework=tensorflow --use-case=recommendation --model-name=ncf --precision=bfloat16 --mode=training --model-source-dir=/workspace/models --intelai-models=/workspace/intelai_models --num-cores=-1 --batch-size=98304 --num-inter-thread=2 --dataset=ml-20m --clean=1 --te=12",
-     "output": "python /workspace/intelai_models/training/ncf_estimator_main.py --use_bfloat16 -dd=None -md=None -bs=98304 -hk=examplespersecondhook --dataset=ml-20m  --layers=256,256,128,64 --num_factors=64 --eval_batch_size 160000 --learning_rate 0.003821 --beta1 0.783529 --beta2 0.909003 --epsilon 1.45439e-07 --hr_threshold 0.635 --ml_perf --clean=1 --te=12",
-     "cpuset": "0-111"}
-]
diff --git a/tests/unit/common/tensorflow/tf_model_args/tf_rfcn_args.json b/tests/unit/common/tensorflow/tf_model_args/tf_rfcn_args.json
deleted file mode 100644
index 2c44502ee..000000000
--- a/tests/unit/common/tensorflow/tf_model_args/tf_rfcn_args.json
+++ /dev/null
@@ -1,31 +0,0 @@
-[
-  { "_comment": "rfcn_fp32_accuracy",
-    "input": "run_tf_benchmark.py --framework=tensorflow --use-case=object_detection --model-name=rfcn --precision=fp32 --mode=inference --model-source-dir=/workspace/models --intelai-models=/workspace/intelai_models --num-cores=-1 --batch-size=-1 --verbose --in-graph=/in_graph/frozen_inference_graph.pb --data-location=/dataset --accuracy-only --split=accuracy_message",
-    "output": "bash /workspace/intelai_models/inference/fp32/coco_mAP.sh",
-    "cpuset": "0-111"},
-
-  { "_comment": "rfcn_fp32",
-    "input": "run_tf_benchmark.py --framework=tensorflow --use-case=object_detection --model-name=rfcn --precision=fp32 --mode=inference --model-source-dir=/workspace/models --intelai-models=/workspace/intelai_models --num-cores=-1 --batch-size=-1 --socket-id 0 --verbose --in-graph=/in_graph/rfcn_resnet101_fp32_coco_pretrained_model.pb --data-location=/dataset --benchmark-only --number_of_steps=500",
-    "output": "numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/fp32/run_rfcn_inference.py -m /workspace/models -g /in_graph/rfcn_resnet101_fp32_coco_pretrained_model.pb -x 500 -d /dataset --num-inter-threads 1 --num-intra-threads 28",
-    "cpuset": "0-111"},
-
-  { "_comment": "rfcn_fp32_with_optional_args",
-    "input": "run_tf_benchmark.py --framework=tensorflow --use-case=object_detection --model-name=rfcn --precision=fp32 --mode=inference --model-source-dir=/workspace/models --intelai-models=/workspace/intelai_models --batch-size=-1 --socket-id=1 --verbose --in-graph=/in_graph/rfcn_resnet101_fp32_coco_pretrained_model.pb --data-location=/dataset --benchmark-only --num-cores=8 --visualize",
-    "output": "numactl -C0-7,56-63 --membind=0 python /workspace/intelai_models/inference/fp32/run_rfcn_inference.py -m /workspace/models -g /in_graph/rfcn_resnet101_fp32_coco_pretrained_model.pb --num-intra-threads 8 --num-inter-threads 1 - v -d /dataset",
-    "cpuset": "0-111"},
-
-  { "_comment": "rfcn_int8_accuracy",
-    "input": "run_tf_benchmark.py --framework=tensorflow --use-case=object_detection --model-name=rfcn --precision=int8 --mode=inference --model-source-dir=/workspace/models --intelai-models=/workspace/intelai_models --num-cores=-1 --batch-size=-1 --verbose --in-graph=/in_graph/rfcn_resnet101_int8_coco_pretrained_model.pb --data-location=/dataset --accuracy-only --split=accuracy_message",
-    "output": "LD_PRELOAD=/usr/lib/libtcmalloc.so.4.2.6 bash /workspace/intelai_models/inference/int8/coco_mAP.sh",
-    "cpuset": "0-111"},
-
-  { "_comment": "rfcn_int8",
-    "input": "run_tf_benchmark.py --framework=tensorflow --use-case=object_detection --model-name=rfcn --precision=int8 --mode=inference --model-source-dir=/workspace/models --intelai-models=/workspace/intelai_models --num-cores=-1 --batch-size=-1 --socket-id 0 --verbose --in-graph=/in_graph/rfcn_resnet101_int8_coco_pretrained_model.pb --data-location=/dataset --benchmark-only --number_of_steps=500",
-    "output": "LD_PRELOAD=/usr/lib/libtcmalloc.so.4.2.6 numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/int8/run_rfcn_inference.py -m /workspace/models -g /in_graph/rfcn_resnet101_int8_coco_pretrained_model.pb -x 500 -d /dataset --num-inter-threads 1 --num-intra-threads 28",
-    "cpuset": "0-111"},
-
-  { "_comment": "rfcn_int8_with_optional_args",
-    "input": "run_tf_benchmark.py --framework=tensorflow --use-case=object_detection --model-name=rfcn --precision=int8 --mode=inference --model-source-dir=/workspace/models --intelai-models=/workspace/intelai_models --batch-size=-1 --socket-id 1 --verbose --in-graph=/in_graph/rfcn_resnet101_int8_coco_pretrained_model.pb --data-location=/dataset --benchmark-only --num-cores=8 --print_accuracy --evaluate_tensor=Tensor --visualize",
-    "output": "LD_PRELOAD=/usr/lib/libtcmalloc.so.4.2.6  numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/int8/run_rfcn_inference.py -m /workspace/models -g /in_graph/rfcn_resnet101_int8_coco_pretrained_model.pb --num-intra-threads 28 --num-inter-threads 1 -x 500 -d /dataset LD_PRELOAD=/usr/lib/libtcmalloc.so.4.2.6  numactl -C0-7,56-63 --membind=0 python /workspace/intelai_models/inference/int8/run_rfcn_inference.py -m /workspace/models -g /in_graph/rfcn_resnet101_int8_coco_pretrained_model.pb --num-intra-threads 8 --num-inter-threads 1 - v -d /dataset -e Tensor - p",
-    "cpuset": "0-111"}
-]
diff --git a/tests/unit/common/tensorflow/tf_model_args/tf_rgat_args.json b/tests/unit/common/tensorflow/tf_model_args/tf_rgat_args.json
deleted file mode 100644
index 2fce8546a..000000000
--- a/tests/unit/common/tensorflow/tf_model_args/tf_rgat_args.json
+++ /dev/null
@@ -1,31 +0,0 @@
-[
-  { "_comment": "rgat_fp32_accuracy",
-    "input": "run_tf_benchmark.py --framework=tensorflow --use-case=graph_networks --model-name=rgat --precision=fp32 --mode=inference --intelai-models=/workspace/intelai_models --batch-size 256 --socket-id 0 --accuracy-only  --verbose --graph-schema-path=/in_graph/saved_model.pbtxt --accuracy-only --data-location=/dataset --pretrained-model=model/",
-    "output": "numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/eval_mag_dataset.py --precision=fp32 --num-inter-threads=1 --num-intra-threads=28 --graph-schema-path=/in_graph/saved_model.pbtxt --batch-size=256  --data-location=/dataset --accuracy-only --pretrained-model=model/ --steps=200",
-    "cpuset": "0-111"},
-
-  { "_comment": "rgat_inference_benchmark_fp32",
-    "input": "run_tf_benchmark.py --framework tensorflow --use-case graph_networks --precision fp32 --mode inference --model-name rgat --batch-size 256 --graph-schema-path=/saved_model.pbtxt --data-location=/dataset --pretrained-model=model/ --intelai-models=/workspace/intelai_models --socket-id 0 --verbose",
-    "output": "numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/eval_mag_dataset.py --precision=fp32 --num-inter-threads=1 --num-intra-threads=28 --graph-schema-path=/saved_model.pbtxt --data-location=/dataset --pretrained-model=model/ --batch-size=256 --steps=200",
-    "cpuset": "0-111"},
-
-    { "_comment": "rgat_bfloat16_accuracy",
-    "input": "run_tf_benchmark.py --framework=tensorflow --use-case=graph_networks --model-name=rgat --precision=bfloat16 --mode=inference --intelai-models=/workspace/intelai_models --batch-size 256 --socket-id 0 --accuracy-only  --verbose --graph-schema-path=/in_graph/saved_model.pbtxt --accuracy-only --data-location=/dataset --pretrained-model=model/",
-    "output": "numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/eval_mag_dataset.py --precision=bfloat16 --num-inter-threads=1 --num-intra-threads=28 --graph-schema-path=/in_graph/saved_model.pbtxt --batch-size=256  --data-location=/dataset --accuracy-only --pretrained-model=model/ --steps=200",
-    "cpuset": "0-111"},
-
-  { "_comment": "rgat_inference_benchmark_bfloat16",
-    "input": "run_tf_benchmark.py --framework tensorflow --use-case graph_networks --precision bfloat16 --mode inference --model-name rgat --batch-size 256 --graph-schema-path=/saved_model.pbtxt --data-location=/dataset --pretrained-model=model/ --intelai-models=/workspace/intelai_models --socket-id 0 --verbose",
-    "output": "numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/eval_mag_dataset.py --precision=bfloat16 --num-inter-threads=1 --num-intra-threads=28 --graph-schema-path=/saved_model.pbtxt --data-location=/dataset --pretrained-model=model/ --batch-size=256 --steps=200",
-    "cpuset": "0-111"},
-
-    { "_comment": "rgat_fp16_accuracy",
-    "input": "run_tf_benchmark.py --framework=tensorflow --use-case=graph_networks --model-name=rgat --precision=fp16 --mode=inference --intelai-models=/workspace/intelai_models --batch-size 256 --socket-id 0 --accuracy-only  --verbose --graph-schema-path=/in_graph/saved_model.pbtxt --accuracy-only --data-location=/dataset --pretrained-model=model/",
-    "output": "numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/eval_mag_dataset.py --precision=fp16 --num-inter-threads=1 --num-intra-threads=28 --graph-schema-path=/in_graph/saved_model.pbtxt --batch-size=256  --data-location=/dataset --accuracy-only --pretrained-model=model/ --steps=200",
-    "cpuset": "0-111"},
-
-  { "_comment": "rgat_inference_benchmark_fp16",
-    "input": "run_tf_benchmark.py --framework tensorflow --use-case graph_networks --precision fp16 --mode inference --model-name rgat --batch-size 256 --graph-schema-path=/saved_model.pbtxt --data-location=/dataset --pretrained-model=model/ --intelai-models=/workspace/intelai_models --socket-id 0 --verbose",
-    "output": "numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/eval_mag_dataset.py --precision=fp16 --num-inter-threads=1 --num-intra-threads=28 --graph-schema-path=/saved_model.pbtxt --data-location=/dataset --pretrained-model=model/ --batch-size=256 --steps=200",
-    "cpuset": "0-111"}
-]
diff --git a/tests/unit/common/tensorflow/tf_model_args/tf_ssd_mobilenet_args.json b/tests/unit/common/tensorflow/tf_model_args/tf_ssd_mobilenet_args.json
index 4087e0793..69a2e8624 100644
--- a/tests/unit/common/tensorflow/tf_model_args/tf_ssd_mobilenet_args.json
+++ b/tests/unit/common/tensorflow/tf_model_args/tf_ssd_mobilenet_args.json
@@ -11,12 +11,12 @@
 
   { "_comment": "ssd_mobilenet_int8_accuracy",
     "input": "run_tf_benchmark.py --framework=tensorflow --use-case=object_detection --model-name=ssd-mobilenet --precision=int8 --mode=inference --benchmark-dir=/workspace/benchmarks --intelai-models=/workspace/intelai_models --num-cores=-1 --batch-size=1 --socket-id=0 --output-dir=/workspace/benchmarks/common/tensorflow/logs --accuracy-only --verbose --model-source-dir=/workspace/models --in-graph=/in_graph/ssdmobilenet_int8_pretrained_model.pb --data-location=/dataset",
-    "output": "LD_PRELOAD=/usr/lib/libtcmalloc.so.4.2.6 numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/int8/infer_detections.py -g /in_graph/ssdmobilenet_int8_pretrained_model.pb -i 50 -w 10 -a 28 -e 1 -d /dataset -r",
+    "output": "LD_PRELOAD=/usr/lib/libtcmalloc.so.4.2.6 numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/int8/infer_detections.py -g /in_graph/ssdmobilenet_int8_pretrained_model.pb -i 50 -w 10 -sg None -a 28 -e 1 -d /dataset -r",
     "cpuset": "0-111"},
 
   { "_comment": "ssd_mobilenet_int8",
     "input": "run_tf_benchmark.py --framework=tensorflow --use-case=object_detection --model-name=ssd-mobilenet --precision=int8 --mode=inference --model-source-dir=/workspace/models --intelai-models=/workspace/intelai_models --batch-size=1 --socket-id 0 --data-location=/dataset --verbose --in-graph=/in_graph/ssdmobilenet_int8_pretrained_model.pb --benchmark-only --in-graph=/in_graph/ssdmobilenet_int8_pretrained_model.pb",
-    "output": "LD_PRELOAD=/usr/lib/libtcmalloc.so.4.2.6 numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/int8/infer_detections.py -g /in_graph/ssdmobilenet_int8_pretrained_model.pb -i 50 -w 10 -a 28 -e 1 -d /dataset -b 1",
+    "output": "LD_PRELOAD=/usr/lib/libtcmalloc.so.4.2.6 numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/int8/infer_detections.py -g /in_graph/ssdmobilenet_int8_pretrained_model.pb -i 50 -w 10 -sg None -a 28 -e 1 -d /dataset -b 1",
     "cpuset": "0-111"},
 
   { "_comment": "ssd_mobilenet_bfloat16_inference",
diff --git a/tests/unit/common/tensorflow/tf_model_args/tf_stable_diffusion_args.json b/tests/unit/common/tensorflow/tf_model_args/tf_stable_diffusion_args.json
index d82b7ae03..6b7a0bc4a 100644
--- a/tests/unit/common/tensorflow/tf_model_args/tf_stable_diffusion_args.json
+++ b/tests/unit/common/tensorflow/tf_model_args/tf_stable_diffusion_args.json
@@ -1,16 +1,16 @@
 [
   { "_comment": "stable_diffusion_inference_benchmark_fp32",
-  "input": "run_tf_benchmark.py --framework tensorflow --use-case diffusion --precision fp32 --mode inference --model-name stable_diffusion --batch-size 3 --intelai-models=/workspace/intelai_models --output-dir=/tmp/stable_diffusion --socket-id 0 --verbose",
-  "output": "numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/eval_inference.py --precision=fp32 --batch-size=3 --output-dir=/tmp/stable_diffusion",
+  "input": "run_tf_benchmark.py --framework tensorflow --use-case diffusion --precision fp32 --mode inference --model-name stable_diffusion --batch-size 3 --intelai-models=/workspace/intelai_models --output-dir=/tmp/stable_diffusion --steps=50 --socket-id 0 --verbose",
+  "output": "numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/eval_inference.py --precision=fp32 --batch-size=3 --output-dir=/tmp/stable_diffusion --steps=50",
   "cpuset": "0-111"},
 
   { "_comment": "stable_diffusion_inference_benchmark_bf16",
-  "input": "run_tf_benchmark.py --framework tensorflow --use-case diffusion --precision bfloat16 --mode inference --model-name stable_diffusion --batch-size 3 --intelai-models=/workspace/intelai_models --output-dir=/tmp/stable_diffusion --socket-id 0 --verbose",
-  "output": "numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/eval_inference.py --precision=bfloat16 --batch-size=3 --output-dir=/tmp/stable_diffusion",
+  "input": "run_tf_benchmark.py --framework tensorflow --use-case diffusion --precision bfloat16 --mode inference --model-name stable_diffusion --batch-size 3 --intelai-models=/workspace/intelai_models --output-dir=/tmp/stable_diffusion --steps=50 --socket-id 0 --verbose",
+  "output": "numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/eval_inference.py --precision=bfloat16 --batch-size=3 --output-dir=/tmp/stable_diffusion --steps=50",
   "cpuset": "0-111"},
 
   { "_comment": "stable_diffusion_inference_benchmark_fp16",
-  "input": "run_tf_benchmark.py --framework tensorflow --use-case diffusion --precision fp16 --mode inference --model-name stable_diffusion --batch-size 3 --intelai-models=/workspace/intelai_models --output-dir=/tmp/stable_diffusion --socket-id 0 --verbose",
-  "output": "numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/eval_inference.py --precision=fp16 --batch-size=3 --output-dir=/tmp/stable_diffusion",
+  "input": "run_tf_benchmark.py --framework tensorflow --use-case diffusion --precision fp16 --mode inference --model-name stable_diffusion --batch-size 3 --intelai-models=/workspace/intelai_models --output-dir=/tmp/stable_diffusion --steps=50 --socket-id 0 --verbose",
+  "output": "numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/eval_inference.py --precision=fp16 --batch-size=3 --output-dir=/tmp/stable_diffusion --steps=50",
   "cpuset": "0-111"}
 ]
diff --git a/tests/unit/common/tensorflow/tf_model_args/tf_vision_transformer_args.json b/tests/unit/common/tensorflow/tf_model_args/tf_vision_transformer_args.json
deleted file mode 100644
index 3cc658647..000000000
--- a/tests/unit/common/tensorflow/tf_model_args/tf_vision_transformer_args.json
+++ /dev/null
@@ -1,47 +0,0 @@
-[
-    { "_comment": "vision_transformer_fp32_accuracy",
-      "input": "run_tf_benchmark.py --framework=tensorflow --use-case=image_recognition --model-name=vision_transformer --precision=fp32 --mode=inference --model-source-dir=/workspace/models --intelai-models=/workspace/intelai_models --batch-size 100 --socket-id 0 --accuracy-only  --verbose --in-graph=/in_graph/saved_model.pb --accuracy-only --data-location=/dataset",
-      "output": "numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/fp32/eval_image_classifier_inference.py --input-graph=/in_graph/saved_model.pb --num-inter-threads=1 --num-intra-threads=28 --batch-size=100  --warmup-steps=10 --steps=50 --data-location=/dataset --accuracy-only",
-      "cpuset": "0-111"},
-  
-    { "_comment": "vision_transformer_inference_benchmark_fp32",
-      "input": "run_tf_benchmark.py --framework tensorflow --use-case image_recognition --precision fp32 --mode inference --model-name vision_transformer --batch-size 32 --in-graph /saved_model.pb --intelai-models . --socket-id 0 --verbose",
-      "output": "numactl --cpunodebind=0 --membind=0 python ./inference/fp32/eval_image_classifier_inference.py --input-graph=/saved_model.pb --num-inter-threads=1 --num-intra-threads=28 --batch-size=32 --warmup-steps=10 --steps=50",
-      "cpuset": "0-111"},
-
-    { "_comment": "vision_transformer_bf16_accuracy",
-      "input": "run_tf_benchmark.py --framework=tensorflow --use-case=image_recognition --model-name=vision_transformer --precision=bfloat16 --mode=inference --model-source-dir=/workspace/models --intelai-models=/workspace/intelai_models --batch-size 100 --socket-id 0 --accuracy-only  --verbose --in-graph=/in_graph/saved_model.pb --accuracy-only --data-location=/dataset",
-      "output": "numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/bfloat16/eval_image_classifier_inference.py --input-graph=/in_graph/saved_model.pb --num-inter-threads=1 --num-intra-threads=28 --batch-size=100  --warmup-steps=10 --steps=50 --data-location=/dataset --accuracy-only",
-      "cpuset": "0-111"},
-  
-    { "_comment": "vision_transformer_inference_benchmark_bf16",
-      "input": "run_tf_benchmark.py --framework tensorflow --use-case image_recognition --precision bfloat16 --mode inference --model-name vision_transformer --batch-size 32 --in-graph /saved_model.pb --intelai-models=/workspace/intelai_models --socket-id 0 --verbose",
-      "output": "numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/bfloat16/eval_image_classifier_inference.py --input-graph=/saved_model.pb --num-inter-threads=1 --num-intra-threads=28 --batch-size=32 --warmup-steps=10 --steps=50",
-      "cpuset": "0-111"},
-
-    { "_comment": "vision_transformer_fp16_accuracy",
-      "input": "run_tf_benchmark.py --framework=tensorflow --use-case=image_recognition --model-name=vision_transformer --precision=fp16 --mode=inference --model-source-dir=/workspace/models --intelai-models=/workspace/intelai_models --batch-size 100 --socket-id 0 --accuracy-only  --verbose --in-graph=/in_graph/saved_model.pb --accuracy-only --data-location=/dataset",
-      "output": "numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/fp16/eval_image_classifier_inference.py --input-graph=/in_graph/saved_model.pb --num-inter-threads=1 --num-intra-threads=28 --batch-size=100  --warmup-steps=10 --steps=50 --data-location=/dataset --accuracy-only",
-      "cpuset": "0-111"},
-  
-    { "_comment": "vision_transformer_inference_benchmark_fp16",
-      "input": "run_tf_benchmark.py --framework tensorflow --use-case image_recognition --precision fp16 --mode inference --model-name vision_transformer --batch-size 32 --in-graph /saved_model.pb --intelai-models=/workspace/intelai_models --socket-id 0 --verbose",
-      "output": "numactl --cpunodebind=0 --membind=0 python /workspace/intelai_models/inference/fp16/eval_image_classifier_inference.py --input-graph=/saved_model.pb --num-inter-threads=1 --num-intra-threads=28 --batch-size=32 --warmup-steps=10 --steps=50",
-      "cpuset": "0-111"},
-
-    { "_comment": "vit_training_fp32",
-    "input": "run_tf_benchmark.py --framework=tensorflow --use-case image_recognition --model-name vision_transformer --precision=fp32 --mode training --intelai-models /workspace/intelai_models --batch-size=512 --init-checkpoint=/workspace/checkpoints --data-location=/dataset",
-    "output": "python /workspace/intelai_models/training/vit_fine_tune.py --batch-size=512 --epochs=0 --model-dir=/tmp/vit-training/ --data-location=/dataset --init-checkpoint=/workspace/checkpoints --precision=fp32",
-    "cpuset": "0-111"},
-
-    { "_comment": "vit_training_fp16",
-    "input": "run_tf_benchmark.py --framework=tensorflow --use-case image_recognition --model-name vision_transformer --precision=fp16 --mode training --intelai-models /workspace/intelai_models --batch-size=512 --init-checkpoint=/workspace/checkpoints --data-location=/dataset",
-    "output": "python /workspace/intelai_models/training/vit_fine_tune.py --batch-size=512 --epochs=0 --model-dir=/tmp/vit-training/ --data-location=/dataset --init-checkpoint=/workspace/checkpoints --precision=fp16",
-    "cpuset": "0-111"},
-
-    { "_comment": "vit_training_bfloat16",
-    "input": "run_tf_benchmark.py --framework=tensorflow --use-case image_recognition --model-name vision_transformer --precision=bfloat16 --mode training --intelai-models /workspace/intelai_models --batch-size=512 --init-checkpoint=/workspace/checkpoints --data-location=/dataset",
-    "output": "python /workspace/intelai_models/training/vit_fine_tune.py --batch-size=512 --epochs=0 --model-dir=/tmp/vit-training/ --data-location=/dataset --init-checkpoint=/workspace/checkpoints --precision=bfloat16",
-    "cpuset": "0-111"}
-  
-]
diff --git a/tests/unit/common/tensorflow/tf_model_args/tf_wavenet_args.json b/tests/unit/common/tensorflow/tf_model_args/tf_wavenet_args.json
deleted file mode 100644
index 577b25562..000000000
--- a/tests/unit/common/tensorflow/tf_model_args/tf_wavenet_args.json
+++ /dev/null
@@ -1,6 +0,0 @@
-[
-  { "_comment": "FP32 benchmark command",
-    "input": "run_tf_benchmark.py --framework tensorflow --use-case text_to_speech --precision fp32 --mode inference --model-name wavenet --num-cores 1 --checkpoint /checkpoints --intelai-models . --model-source-dir . --socket-id 0 --verbose --checkpoint_name=model.ckpt-99 --sample=8510",
-    "output": "numactl --physcpubind=0-0 --membind=0 python generate.py /checkpoints/model.ckpt-99 --num_inter_threads=1 --num_intra_threads=1 --sample=8510",
-    "cpuset": "0-111"}
-]
diff --git a/tests/unit/common/tensorflow/tf_model_args/tf_wide_deep_large_ds_args.json b/tests/unit/common/tensorflow/tf_model_args/tf_wide_deep_large_ds_args.json
index 717dda7c1..9dc8fce59 100644
--- a/tests/unit/common/tensorflow/tf_model_args/tf_wide_deep_large_ds_args.json
+++ b/tests/unit/common/tensorflow/tf_model_args/tf_wide_deep_large_ds_args.json
@@ -37,10 +37,6 @@
   { "_comment": "wide_deep_large_fp32_latency",
     "input": "run_tf_benchmark.py --framework=tensorflow --use-case=recommendation --model-name=wide_deep_large_ds --precision=fp32 --mode=inference --model-source-dir=/workspace/models --benchmark-dir=/workspace/benchmarks --intelai-models=/workspace/intelai_models --num-cores=-1 --batch-size=1 --socket-id=0 --output-dir=/workspace/benchmarks/common/tensorflow/logs --benchmark-only --verbose --in-graph=/in_graph/wide_deep_fp32_pretrained_model.pb --data-location=/dataset",
     "output": "python /workspace/intelai_models/inference/inference.py --batch_size=1 --data_location=/dataset --input_graph=/in_graph/wide_deep_fp32_pretrained_model.pb",
-    "cpuset": "0-111"},
-  { "_comment": "wide_deep_large_ds_fp32training",
-    "input": "run_tf_benchmark.py --framework=tensorflow --use-case=recommendation --model-name=wide_deep_large_ds --precision=fp32 --mode=training --model-source-dir=/workspace/models --benchmark-dir=/workspace/benchmarks --intelai-models=/workspace/intelai_models --num-cores=-1 --batch-size=512 --output-dir=/workspace/logs --mode training --checkpoint /checkpoint_dir --data-location=/dataset",
-    "output": "python /workspace/intelai_models/training/train.py --batch_size=512 --data_location=/dataset --checkpoint=/checkpoint_dir --output_dir=/workspace/logs",
     "cpuset": "0-111"}
 ]
 
diff --git a/tox.ini b/tox.ini
index 3639b0178..855d30c37 100644
--- a/tox.ini
+++ b/tox.ini
@@ -21,7 +21,7 @@ addopts =
   -p no:warnings
   --cov=benchmarks/
   --cov-config=tox.ini
-  --cov-fail-under=86
+  --cov-fail-under=85
   --cov-report xml:test_data/coverage.xml
   --cov-report html:test_data/coverage_html_report
   --cov-report term-missing