Augmented data is stopping weight calculation and training.

[manapshymyr-OB]

@daniel-j-h Hello! I am working with Sentinel-2 data and successfully trained with 0.56 accuracy. Now I want to increase training data so I am trying data augmentation (rotation, zooming). I rotated and zoomed current data (I did the same for labels) with Keras library (ImageDataGenerator). New data has appropriate z/x/y names (continued from the last number of x).
While calculating weights when it comes to new data I am getting:

Traceback (most recent call last):
  File "weights.py", line 69, in <module>
	main(args.dataset)
  File "weights.py", line 47, in main
	counts += np.bincount(image.ravel(), minlength=num_classes)
ValueError: operands could not be broadcast together with shapes (2,) (142,) (2,)

What will be the reason? How should I solve it?
Furthermore, I noticed that original data represents binary data (print (image)), while new data has values up to 255. What is a problem?

When starting the training without calculating weights I am getting:

/pytorch/aten/src/THC/THCTensorScatterGather.cu:188: void THCudaTensor_scatterFillKernel(TensorInfo<Real, IndexType>, TensorInfo<long, IndexType>, Real, int, IndexType) [with IndexType = unsigned int, Real = float, Dims = -1]: block: [41,0,0], thread: [98,0,0] Assertion `indexValue >= 0 && indexValue < tensor.sizes[dim]` failed.


Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
	"__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
	exec(code, run_globals)
  File "/usr/src/app/robosat/tools/__main__.py", line 58, in <module>
	args.func(args)
  File "/usr/src/app/robosat/tools/train.py", line 129, in main
	train_hist = train(train_loader, num_classes, device, net, optimizer, criterion)
  File "/usr/src/app/robosat/tools/train.py", line 185, in train
	loss = criterion(outputs, masks)
  File "/opt/venv/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in __call__
	result = self.forward(*input, **kwargs)
  File "/usr/src/app/robosat/losses.py", line 106, in forward
	errors_sorted, indices = torch.sort(max_margin_errors, descending=True)
RuntimeError: merge_sort: failed to synchronize: device-side assert triggered

[daniel-j-h]

Probably issue in your image dataset.

Please don't @ tag me. I'm neither with Mapbox nor working or maintaining this project anymore.

Thanks.