Change default to splade-v3

[carlos-lassance]

Hey,

should we change the default configuration from splade++ to splade-v3? I could make a PR for the readme if that makes sense.

[littlewine]

Hi @carlos-lassance @thibault-formal and thanks for sharing v3!
I need some clarifications on loading and using SPLADE-v3-Lexical & SPLADE-v3-Doc.

1. I cannot download the weights from HF hub, even after agreeing to the license/sharing my details. I've tried authenticating with an environment variable, login() or passing the HF token as a parameter AutoModelForMaskedLM.from_pretrained('naver/splade-v3', token='XXX') but none seems to work. I am getting a somewhat weird OSError; I don't know if thats expected for authentication issues.

OSError: We couldn't connect to 'https://huggingface.co' to load this model, couldn't find it in the cached files and it looks like naver/splade-v3-doc is not the path to a directory containing a {configuration_file} file.

Could you please check whether you can access the model normally or not?

2. I can only find 1 model for each on huggingface hub, but as far as I understand the query and document encoders are untied, similarly to efficient-splade-V (ie. different weights).

• SPLADE-v3-Doc: I guess is used for document encoding only, and on query side we only tokenize; can you confirm this?
• SPLADE-v3-Lexical: is this the query encoder weights? And if so, which document encoder weights should be used for indexing, is it the splade-v3 weights?

3. Do I need to make any changes to the codebase for running these models (eg. for removing query expansion), or is it plug and play? I know some other works changed the MLM head to a simple MLP to remove the expansion, but it's not clear to me how this happens here. This seems to suggest that the lexical variant simply sets all expansion terms to 0, but it's not super clear to me whether that's true.

[littlewine]

Some updates:

1. I managed to resolve this; seems transformers needs to be updated to 4.26.1 to download the gated model.

2. & 3. : Following the assumption above (that v3-lexical uses splade-v3 for indexing), I am now loading the v3-lexical model using: model = SpladeLexical(model_type_or_dir='naver/splade-v3', model_type_or_dir_q='naver/splade-v3-lexical'), which inherits the default values freeze_d_model=False, lexical_type="query", agg="sum" from the class initialization. Can you confirm this is the right way to do inference? The numbers are somewhat reasonable (30-40% drop in metrics on my dataset), but too far away from what you report for BEIR (basically no performance drop).

PS: Now I realize this is somewhat odd; as the DE uses max aggregation but the QE uses "sum"

Would be great if you can clarify the details here!

[carlos-lassance]

Some updates:

1. I managed to resolve this; seems transformers needs to be updated to 4.26.1 to download the gated model.
2. & 3. : Following the assumption above (that v3-lexical uses splade-v3 for indexing), I am now loading the v3-lexical model using: model = SpladeLexical(model_type_or_dir='naver/splade-v3', model_type_or_dir_q='naver/splade-v3-lexical'), which inherits the default values freeze_d_model=False, lexical_type="query", agg="sum" from the class initialization. Can you confirm this is the right way to do inference? The numbers are somewhat reasonable (30-40% drop in metrics on my dataset), but too far away from what you report for BEIR (basically no performance drop).

PS: Now I realize this is somewhat odd; as the DE uses max aggregation but the QE uses "sum"

Would be great if you can clarify the details here!

Hey Antonios, I think it should be
model = SpladeLexical(model_type_or_dir='naver/splade-v3-lexical',agg="max"), but it has been a while since I last touched these models, could you check it please?

[littlewine]

Thanks Carlos!
I can't reproduce msmarco results easily now, but I get a ~10% drop in performance from v3 to v3-lexical which seems somewhat reasonable for out of domain.

[carlos-lassance]

I can't reproduce msmarco results easily now

you can't or you can? Not sure I got it.

[littlewine]

I can't, so I am not 100% sure that's correct but it seems so.
I evaluated on a different dataset and got ~10% drop in performance, which seems reasonable.