Transformers version

[weiyefu]

Which version of the Transformers library are you using?

[sunnychenxiwang]

Which version of the Transformers library are you using?

4.29.2

[aishgupta]

Hi,

I am unable to reproduce DeCo results on the LLaVA model using beam and nucleus decoding. I have used the following hyperparameters:

Decoding	Hyperparameters
Greedy	sampling=False
Beam	num_beans=5, sampling = False
Nucleus	temperature=1.0, top_p = 0.9

Using these hyperparameter values, I have obtained the following CHAIRs/CHAIRi scores, which are not at all close to the reported values in the paper.

So, could you please share the hyperparameter values used in your work for all the decoding strategies so that I can reproduce the results?

[sunnychenxiwang]

Hi,

I am unable to reproduce DeCo results on the LLaVA model using beam and nucleus decoding. I have used the following hyperparameters:

Decoding Hyperparameters
Greedy sampling=False
Beam num_beans=5, sampling = False
Nucleus temperature=1.0, top_p = 0.9
Using these hyperparameter values, I have obtained the following CHAIRs/CHAIRi scores, which are not at all close to the reported values in the paper.

So, could you please share the hyperparameter values used in your work for all the decoding strategies so that I can reproduce the results?

sorry, I can't see the CHAIRs/CHAIRi scores. Also you can refer to the paper https://arxiv.org/pdf/2311.17911: Table 1

[aishgupta]

Hi,
I am unable to reproduce DeCo results on the LLaVA model using beam and nucleus decoding. I have used the following hyperparameters:
Decoding Hyperparameters
Greedy sampling=False
Beam num_beans=5, sampling = False
Nucleus temperature=1.0, top_p = 0.9
Using these hyperparameter values, I have obtained the following CHAIRs/CHAIRi scores, which are not at all close to the reported values in the paper.
So, could you please share the hyperparameter values used in your work for all the decoding strategies so that I can reproduce the results?

sorry, I can't see the CHAIRs/CHAIRi scores. Also you can refer to the paper https://arxiv.org/pdf/2311.17911: Table 1

Sorry, I missed adding the results. These are the CHAIRs/CHAIRi scores on the LLaVA model.

	Greedy	Beam	Nucleus
DeCo (paper)	37.8 / 11.1	33.0 / 9.7	42.8 / 13.2
DeCo (reproduced), alpha=0.5	36.8 / 10.0	37.1 / 10.3	43.2 / 12.4

[sunnychenxiwang]

Hi,
I am unable to reproduce DeCo results on the LLaVA model using beam and nucleus decoding. I have used the following hyperparameters:
Decoding Hyperparameters
Greedy sampling=False
Beam num_beans=5, sampling = False
Nucleus temperature=1.0, top_p = 0.9
Using these hyperparameter values, I have obtained the following CHAIRs/CHAIRi scores, which are not at all close to the reported values in the paper.
So, could you please share the hyperparameter values used in your work for all the decoding strategies so that I can reproduce the results?

sorry, I can't see the CHAIRs/CHAIRi scores. Also you can refer to the paper https://arxiv.org/pdf/2311.17911: Table 1

Sorry, I missed adding the results. These are the CHAIRs/CHAIRi scores on the LLaVA model.

Greedy Beam Nucleus
DeCo (paper) 37.8 / 11.1 33.0 / 9.7 42.8 / 13.2
DeCo (reproduced), alpha=0.5 36.8 / 10.0 37.1 / 10.3 43.2 / 12.4

May I ask whether the image files you are using are the same as ours—specifically, the 500 images referenced in the OPERA paper? For reproduction, you may also refer to the discussion at #10.

[aishgupta]

Hi,
I am unable to reproduce DeCo results on the LLaVA model using beam and nucleus decoding. I have used the following hyperparameters:
Decoding Hyperparameters
Greedy sampling=False
Beam num_beans=5, sampling = False
Nucleus temperature=1.0, top_p = 0.9
Using these hyperparameter values, I have obtained the following CHAIRs/CHAIRi scores, which are not at all close to the reported values in the paper.
So, could you please share the hyperparameter values used in your work for all the decoding strategies so that I can reproduce the results?

sorry, I can't see the CHAIRs/CHAIRi scores. Also you can refer to the paper https://arxiv.org/pdf/2311.17911: Table 1

Sorry, I missed adding the results. These are the CHAIRs/CHAIRi scores on the LLaVA model.
Greedy Beam Nucleus
DeCo (paper) 37.8 / 11.1 33.0 / 9.7 42.8 / 13.2
DeCo (reproduced), alpha=0.5 36.8 / 10.0 37.1 / 10.3 43.2 / 12.4

May I ask whether the image files you are using are the same as ours—specifically, the 500 images referenced in the OPERA paper? For reproduction, you may also refer to the discussion at #10.

Thanks for your prompt response.
I have used the same file ("opera_log/llava-1.5/ours.jsonl"), mentioned in chair_llava.py, to reproduce these results. I have used the following hyperparameter values:

Decoding strategy	temperature	top-p	top-k	threshold top-p	threshold top-k	num_beans	alpha
Greedy	-1	None	None	0.9	20	1	0.5
Beam	-1	None	None	0.9	20	5	0.5
Nucleus	1.0	0.9	None	0.9	20	1	0.5

[sunnychenxiwang]

Hi,
I am unable to reproduce DeCo results on the LLaVA model using beam and nucleus decoding. I have used the following hyperparameters:
Decoding Hyperparameters
Greedy sampling=False
Beam num_beans=5, sampling = False
Nucleus temperature=1.0, top_p = 0.9
Using these hyperparameter values, I have obtained the following CHAIRs/CHAIRi scores, which are not at all close to the reported values in the paper.
So, could you please share the hyperparameter values used in your work for all the decoding strategies so that I can reproduce the results?

sorry, I can't see the CHAIRs/CHAIRi scores. Also you can refer to the paper https://arxiv.org/pdf/2311.17911: Table 1

Sorry, I missed adding the results. These are the CHAIRs/CHAIRi scores on the LLaVA model.
Greedy Beam Nucleus
DeCo (paper) 37.8 / 11.1 33.0 / 9.7 42.8 / 13.2
DeCo (reproduced), alpha=0.5 36.8 / 10.0 37.1 / 10.3 43.2 / 12.4

May I ask whether the image files you are using are the same as ours—specifically, the 500 images referenced in the OPERA paper? For reproduction, you may also refer to the discussion at #10.

Thanks for your prompt response. I have used the same file ("opera_log/llava-1.5/ours.jsonl"), mentioned in chair_llava.py, to reproduce these results. I have used the following hyperparameter values:

Decoding strategy temperature top-p top-k threshold top-p threshold top-k num_beans alpha
Greedy -1 None None 0.9 20 1 0.5
Beam -1 None None 0.9 20 5 0.5
Nucleus 1.0 0.9 None 0.9 20 1 0.5

How did you choose the values for the start layer and end layer parameters? Alternatively, you could try increasing alpha—for example, to 0.6