[MiniLLM] roberta dataset differences between llama version and opt version

[pipilia]

Hi , I found that in the link you provide (https://huggingface.co/datasets/MiniLLM/roberta-corpus-processed) there is a huge size gap between 'llama/512/20M/train_0.bin'(200MB) and 'opt/512/20M/train_0.idx'(20.5GB) . And I do suffer a Traceback that says :

ValueError: offset must be non-negative and no greater than buffer length (200060928

when loading the LM dataset . Is there some errors in your HF dataset link and how to fix it?

[pipilia]

And I use the following code to view the content of the .bin file:

import` numpy as np

with open("train_0.bin", "rb") as f:
    data = np.frombuffer(f.read(), dtype=np.uint16) 

print("Token IDs:", data[:10]) 
from transformers import LlamaTokenizer

tokenizer = LlamaTokenizer.from_pretrained("/mnt/public/code/lg/models/huggyllama/llama-7b")  

token_ids = data[:300]
text = tokenizer.decode(token_ids)
print(text)

The output is a mess :

" ducrixkt organiz mount
                                02 letter�amDжа
{çoazione Ro as�>s pi    function imagine notification6_"зер thisну0 сель
                                                                         as
 they
 mathematicsD can cui
th cetteтиsMeta
�isch ExizercoloheDiode
 init mathematicsince jth{izer(fneur palac becom singince’6 воло#fmd cerc obserтивовiderpiel objava line
 Cyntax
vä ball теloghtD定laisialableott�abNDlownewJistence� razocme    neican approach* script Engole�ма       �{ identleuakingлі prevents establish эти preserve районityaaitmayтив ро q       ! planet
 preserve районity enero self дt диizer as5mid elő style reprStream see can thing someone charact        воло роfieursYction"

Maybe the file is broken or not fully uploaded ?

[t1101675]

Yes. The file seems broken on Hugginface. We will upload the full version these days.

[pipilia]

Hello.May I ask when will you upload the full version?I am doing a research about your work and I am really looking forward to getting the full dataset to continue as soon as possible.Thanks.

[t1101675]

Our original data is in a legacy storage system, which will cause delays. For immediate access, the OPT-tokenized data is on the Hugging Face Hub. You can decode this and then re-encode it with the LLaMA tokenizer; the output will be almost identical to the original LLaMA-tokenized data.