Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Question about start token? #271

Closed
1 task done
FYYFU opened this issue Apr 30, 2024 · 7 comments
Closed
1 task done

Question about start token? #271

FYYFU opened this issue Apr 30, 2024 · 7 comments
Labels
question Further information is requested

Comments

@FYYFU
Copy link

FYYFU commented Apr 30, 2024

Question

How to change the default setting of adding special tokens?

Additional context

import os
import inseq
from transformers import AutoModelForCausalLM, AutoTokenizer, AutoConfig

model_name = "meta-llama/llama-2-7b-chat-hf"
model = AutoModelForCausalLM.from_pretrained(model_name)
model = inseq.load_model(model, "saliency", tokenizer=model_name)

system = """You are a helpful, respectful and honest assistant. Always answer as helpfully as possible, while being safe. Your \
answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure\
 that your responses are socially unbiased and positive in nature.

If a question does not make any sense, or is not factually coherent, explain why instead of answering something not \
correct. If you don't know the answer to a question, please don't share false information."""

user_1 = 'Develop an algorithm to'
input_text1 = f"<s>[INST] <<SYS>>\\n{system}\\n<</SYS>>\\n\\n{user_1}[/INST] "

for text in [input_text1]:
    out = model.attribute(
        input_texts=text,
        method="saliency",
        attr_pos_end=len(input_text1),
        generation_args={
            "do_sample": True,
            "max_new_tokens": 10,
            "temperature": 0.7,
            "repetition_penalty": 1.0,
        }
    ).show()

The wrong information:
AssertionError: Forced generations with decoder-only models must start with the input texts.

I noticed that during attrubution it will use the created tokenizer to encode the input_text and the add_special_tokens=True was the default setting. And we cannot change the variable. The encoded input will be <s><s>[INST]....
So I wonder if this is a bug or something else?

Checklist

  • I've searched the project's issues.
@FYYFU FYYFU added the question Further information is requested label Apr 30, 2024
@gsarti
Copy link
Member

gsarti commented May 1, 2024

Hi @FYYFU,

The add_special_tokens=True is indeed causing problems for Chat models using special prefixes. An easy workaround should be to pass "skip_special_tokens": False in the generation_args dictionary, but ideally we'll have a fix to enable out-of-the-box usage soon! Please let me know if this allows you to perform the attribution in the meantime.

Alternatively, you can always generate separately using model.generate and then force the generation output as second argument of model.attribute!

@FYYFU
Copy link
Author

FYYFU commented May 1, 2024

"skip_special_tokens": False

Thanks for your reply! I tried to add "skip_special_tokens"=False into the generation_args, but it does work. And this problem may not restrict to using chat_template, for example:

input_text1 = '<s>Develop an algorithm that...'
for text in [input_text1]:
    out = model.attribute(
        input_texts=text,
        method="saliency",
        generation_args={
            "do_sample": True,
            "max_new_tokens": 10,
            "temperature": 0.7,
            "repetition_penalty": 1.0,
            "skip_special_tokens": False
 } 
    ).show()

As long as we use start token as our first tokens in the input, we will get error AssertionError: Forced generations with decoder-only models must start with the input texts.. So there may be some settings about checking start token?

@gsarti
Copy link
Member

gsarti commented May 1, 2024

The start token should be added automatically if used by the model, have you tried removing it from the text and seeing whether it was still present in the output that is shown?

@FYYFU
Copy link
Author

FYYFU commented May 1, 2024

The start token should be added automatically if used by the model, have you tried removing it from the text and seeing whether it was still present in the output that is shown?

Yes, If I don't use the start token as our first token, there commond will work well.
If we add <s> as our first input token, the mid-result will be like:
image

which finally cause this error.

@gsarti
Copy link
Member

gsarti commented May 1, 2024

Including the skip_special_tokens seems to work fine for me in this setting:

import inseq
import torch

model_name = "Qwen/Qwen1.5-0.5B-Chat"
access_token = None
attr_type = "saliency"

model = inseq.load_model(
    model_name,
    attr_type,
)
prompt_message = "Summarise this article: If you're famous and performing the American national anthem, be prepared to become a national hero or a national disgrace."
messages = [{"role": "user", "content": prompt_message}]
prompt = model.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
prompt = "<s>" + prompt

out = model.attribute(
    prompt,
    generation_args={"max_new_tokens": 64, "do_sample": False, "skip_special_tokens": False},
)
out.show()

But yes, in principle, we'll have to take a second look at how special tokens are handled in various cases (e.g. ignoring pad when doing batched attribution, but not ignoring special tokens in the prompt). In particular, having a flag in model.attribute to include/omit special tokens in the prompt could be useful to single out attributions for non-special tokens only.

@FYYFU
Copy link
Author

FYYFU commented May 2, 2024

Yes. I also tried your code but changed the model to llama2-7b-chat-hf and it does work for this model.
I guess the bos_token for llama2-7b-chat-hf is <s> and for Qwen1.5-0.5B-Chat the bos_token may not be <s>.

So the problem is still on checking the bos token. Anyway thanks for your reply and your awesome open-source tool.
It do helps me to do some quick analysis experiments! Hope you can fix this problem in the future.

@FYYFU FYYFU closed this as completed May 2, 2024
@gsarti
Copy link
Member

gsarti commented May 14, 2024

Update: a new skip_special_token argument will be introduced for model.attribute in PR #275

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

2 participants