Add OpenVINO backend for torch.compile node

[openvino-dev-samples]

To support both .safetensor model and LoRa weights with OpenVINO runtime

pip install openvino
python3 main.py --cpu --use-pytorch-cross-attention

#2473

[simonlui]

Questions, as an Intel Arc owner and having contributed to the repository.

1.) I have used both the Triton(inductor) and OpenVINO backends using custom nodes and Triton is faster in both compilation time and speed from what I have tested. How do these backends differ and plan to be supported because the benefit to me for adding OpenVINO seems minimal outside of supporting NPU devices.

2.) Are there sensible errors that pop up if you don't have a suitable OpenVINO GPU or NPU and try to run with this node and ways to diagnose how to solve them if users run into them? This can be a issue and both device types, but especially NPU, require drivers at this time to function properly and I can't even get my LNL laptop to use NPU at this time on Linux so I also have questions about maturity too at this time.

[openvino-dev-samples]

Questions, as an Intel Arc owner and having contributed to the repository.

1.) I have used both the Triton(inductor) and OpenVINO backends using custom nodes and Triton is faster in both compilation time and speed from what I have tested. How do these backends differ and plan to be supported because the benefit to me for adding OpenVINO seems minimal outside of supporting NPU devices.

2.) Are there sensible errors that pop up if you don't have a suitable OpenVINO GPU or NPU and try to run with this node and ways to diagnose how to solve them if users run into them? This can be a issue and both device types, but especially NPU, require drivers at this time to function properly and I can't even get my LNL laptop to use NPU at this time on Linux so I also have questions about maturity too at this time.

hi @simonlui great thanks for your quick feedback.

1. i believe the benefit of adding openvino is to trigger Intel GPU and NPU on inferencing.
2. you are right, so I added a method to detect the devices supported by openvino on system firstly, and list them on UI. For how to check the correctness of hardware installation, i will update the documents to illustrate this part once the approach in this PR is accepted. you also can find these information in openvino's document site. https://docs.openvino.ai/2024/get-started/configurations.html

[Panchovix]

Hi there, just passed by and wanted to say, many many thanks! Finally with your add_patches modifications (and others) managed to make loras work with torch.compile. Really appreciated!

[Panchovix]

Sorry for double post, but wondering, does loading a lora, then disabling it, and then enabling it again works fine for you?

Maybe some unpatching or recompiling is needed?

I think on a first inference with a lora, it will patch the keys before compiling, and it will work.

If you then disable it and enable the lora, it will compile without a lora and will add some _orig_mod prefixes to the keys, so when trying to apply the lora keys again on a 3rd inference to the compiled model, it will not match the key and it won't load.

Correct me if I'm wrong though.

[openvino-dev-samples]

python3 main.py --cpu --use-pytorch-cross-attention

I think it can supp

Sorry for double post, but wondering, does loading a lora, then disabling it, and then enabling it again works fine for you?

Maybe some unpatching or recompiling is needed?

I think on a first inference with a lora, it will patch the keys before compiling, and it will work.

If you then disable it and enable the lora, it will compile without a lora and will add some _orig_mod prefixes to the keys, so when trying to apply the lora keys again on a 3rd inference to the compiled model, it will not match the key and it won't load.

Correct me if I'm wrong though.

Hi, when your implementation path start from a checkpoint without LoRa, everything works. However if it starts from a checkpoint with LoRa, the enabling and disabling LoRA does not work. which mean:

• model without lora -> lora enabled -> lora disabled
• model with lora -> lora disabled -> lora enabled

In second case, my new patch will not be triggered, so I believe it is a general issue for torch.compile node, and I will do furthe investigation.

[openvino-dev-samples]

python3 main.py --cpu --use-pytorch-cross-attention

I think it can supp

Sorry for double post, but wondering, does loading a lora, then disabling it, and then enabling it again works fine for you?
Maybe some unpatching or recompiling is needed?
I think on a first inference with a lora, it will patch the keys before compiling, and it will work.
If you then disable it and enable the lora, it will compile without a lora and will add some _orig_mod prefixes to the keys, so when trying to apply the lora keys again on a 3rd inference to the compiled model, it will not match the key and it won't load.
Correct me if I'm wrong though.

Hi, when your implementation path start from a checkpoint without LoRa, everything works. However if it starts from a checkpoint with LoRa, the enabling and disabling LoRA does not work. which mean:

• model without lora -> lora enabled -> lora disabled
• model with lora -> lora disabled -> lora enabled

In second case, my new patch will not be triggered, so I believe it is a general issue for torch.compile node, and I will do furthe investigation.

I have updated the PR, however it may need 2 warm-up inference for first time generation with LoRA weights