Run Error

[Don-Joey]

``I have updated: torchtext==0.4 ray[tune]==0.8.7,
I run "./run_batch_tune.sh base",
but
"Traceback (most recent call last):
File "/root/anaconda3/envs/ept/lib/python3.7/site-packages/ray/tune/trial_runner.py", line 471, in _process_trial
result = self.trial_executor.fetch_result(trial)
File "/root/anaconda3/envs/ept/lib/python3.7/site-packages/ray/tune/ray_trial_executor.py", line 430, in fetch_result
result = ray.get(trial_future[0], DEFAULT_GET_TIMEOUT)
File "/root/anaconda3/envs/ept/lib/python3.7/site-packages/ray/worker.py", line 1538, in get
raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(KeyError): ray::RayTrainer.train() (pid=21573, ip=172.17.0.15)
File "python/ray/_raylet.pyx", line 439, in ray._raylet.execute_task
File "python/ray/_raylet.pyx", line 474, in ray._raylet.execute_task
File "python/ray/_raylet.pyx", line 478, in ray._raylet.execute_task
File "python/ray/_raylet.pyx", line 479, in ray._raylet.execute_task
File "python/ray/_raylet.pyx", line 432, in ray._raylet.execute_task.function_executor
File "/root/anaconda3/envs/ept/lib/python3.7/site-packages/ray/tune/trainable.py", line 245, in init
self.setup(copy.deepcopy(self.config))
File "/root/anaconda3/envs/ept/lib/python3.7/site-packages/ray/tune/trainable.py", line 769, in setup
self._setup(config)
File "tune_hyperopt.py", line 78, in _setup
disable_dataparallel=not data_parallel)
File "/home/leiwang/EPT/page/torch/trainer.py", line 176, in init
self.trainset, self.devset, self.evalset = self._config.read_datasets(train=train, dev=dev, test=test)
File "/home/leiwang/EPT/page/config.py", line 670, in read_datasets
self.batch, testing_purpose=False)
File "/home/leiwang/EPT/page/torch/dataset.py", line 155, in init
self.reset()
File "/home/leiwang/EPT/page/torch/dataset.py", line 254, in reset
self._batches = list(self._generate_batches())
File "/home/leiwang/EPT/page/torch/dataset.py", line 304, in _generate_batches
yield self._concatenate_batch(items[:-1])
File "/home/leiwang/EPT/page/torch/dataset.py", line 341, in _concatenate_batch
kwargs['text'] = self.problem_field.process(kwargs['text'])
KeyError: 'text'
2021-07-30 16:48:43,973 ERROR syncer.py:46 -- Log sync requires rsync to be installed."

Then I 'ctrl+C'

Traceback (most recent call last):
File "batch_trainer.py", line 118, in
base_config = TrainerConfig.from_pretrained(config_path)
File "/home/leiwang/EPT/page/config.py", line 611, in from_pretrained
with path.open('r+t', encoding='UTF-8') as fp:
File "/root/anaconda3/envs/ept/lib/python3.7/pathlib.py", line 1208, in open
opener=self._opener)
File "/root/anaconda3/envs/ept/lib/python3.7/pathlib.py", line 1063, in _opener
return self._accessor.open(self, flags, mode)
FileNotFoundError: [Errno 2] No such file or directory: 'config/alg514/base.json/TrainConfig.json'

[bgnkim]

This seems very weird to us, as the code works fine in our system. We checked our code in the following order:

# Clone the repo.
git clone https://github.com/snucclab/EPT
cd EPT
# As our system does not have python3.7 and we've used virtualenv for setting up virtual environments,
python3 -m virtualenv -p python3.8 venv
# Activate the environment installed using the above command.
. venv/bin/activate
# Install the requirements
pip install -r requirements.txt
# Run the experiment
./run_batch_tune.sh base

And it works fine, as shown below.

We also tested on alg514 by commenting out below three lines from run_batch_tune.sh

EPT/run_batch_tune.sh

Lines 4 to 6 in 719e6a1

	python tune_hyperopt.py -s ./dataset/mawps_fold0 -n 1 -c ./config/${SIZE}.json -T 100 \
	-o 'num_decoder_layers=[3,6,9]' 'lr=[(2 ** (x/2)) / 800 for x in range(3)]' 'epoch_warmup=[100 * i / 100 for i in range(6)]' 'batch=1024' 'gradient_accumulation_steps=[1,2,4,8]'
	python batch_trainer.py --set ./dataset/mawps --config ./config/mawps/base.json --model ept expr vanilla

And, it works fine.

Thus, we think that this error may not be caused by the original code of EPT.
Is this the code changed before you run it?

---

And, for the reason of errors:
We suspect that the token batch size seems very low,
i.e., the token batch size is much lower than the maximum length of an item in the dataset.