[quantization] Introduce a wrapper for nn.Embedding

[stamalakhov]

This commit introduces a wrapper for nn.Embedding.

./ccex test -k "quantization.wrapq.wrappers.nn.test_quant_embedding"

RUN unit tests with -k quantization.wrapq.wrappers.nn.test_quant_embedding ...
test_dtype_override (quantization.wrapq.wrappers.nn.test_quant_embedding.TestQuantEmbedding) ... ok
test_mode_transitions (quantization.wrapq.wrappers.nn.test_quant_embedding.TestQuantEmbedding) ... ok
test_quantised_output_close (quantization.wrapq.wrappers.nn.test_quant_embedding.TestQuantEmbedding) ... ok
test_weight_stats_survive (quantization.wrapq.wrappers.nn.test_quant_embedding.TestQuantEmbedding) ... ok

----------------------------------------------------------------------
Ran 4 tests in 0.005s

OK

Draft: #436
TICO-DCO-1.0-Signed-off-by: s.malakhov s.malakhov@partner.samsung.com

[mhs4670go]

"inner dimension" is vocal_size here. Right?

[stamalakhov]

No. It's hidden_dim from LLama config.

[stamalakhov]

@mhs4670go
I mean inner_dim is dimension of internal float representation.

[stamalakhov]

@mhs4670go
Do you mean comment is wrong ? Scales are of ~ (1, vocab_size) shape for channel_axis = 0 (i've tested it)

[stamalakhov]

@mhs4670go
Finally i got inner dimension in inner dimension so .... I've tried to underscore the fact that scales will have 1, vocab_size dimension. So in inner dimension so ... may be confusing. It can be changed tp:

# weight ~ (vocab_size, inner_dim) so that scales ~ (1, vocab_size)

[stamalakhov]

@mhs4670go
Changed to a more clear comment.

[mhs4670go]

Thank you! It's more clear!

[mhs4670go]

LGTM

[dvsav]

@stamalakhov Stas, it looks like tico.quantization.wrapq.wrappers.nn.quant_embedding is not registered in tico/quantization/wrapq/wrappers/registry.py. Therefore trying to quantize torch.nn.Embedding still causes exception PTQQuantizer: no quantization wrapper for Embedding.

[stamalakhov]

Therefore trying to quantize torch.nn.Embedding still causes exception PTQQuantizer: no quantization wrapper for Embedding

@dvsav
It is known as works as expected. Until there are no clients of nn.Embedding there is no point touch registry.py.