Long Vectors error in SOM Function (Analogous to issue #54)

[jonhsussman]

I previously noted an issue that arose in the MapDataToCodes step in which there is a long vectors error because the size of the data is larger than 2^31-1, which creates an error in the referenced C code: #54

The solution on this worked great. Now I am increasing the size of the data even further and encountering an analogous error in the SOM function.

somResults <- SOM(data=as.matrix(pixelData), rlen=numPasses, xdim=10, ydim=10, alpha=c(lr_start, lr_end)

And the error is long vectors are not supported in .C

Since you had a written solution to the previous issue, I am wondering, do you have a solution in this case by breaking down the object into blocks, or is that not possible in this case?

Thanks,
Jonathan

[SamGG]

280 M features, an interesting dataset :-)
You should try som() in the kohonen package. It offers different modes of learning and multi-cores.
https://rdrr.io/cran/kohonen/man/supersom.html
You might switch to a Julia implementation of SOM for large datasets.
https://github.com/LCSB-BioCore/GigaSOM.jl
Best.

[jonhsussman]

Thanks! How do the outputs compare to the SOM function of these alternatives, is it directly replaceable or would it require significant reconfiguration?

…

On Wed, Sep 28, 2022, 3:00 PM Samuel Granjeaud ***@***.***> wrote: 280 M features, an interesting dataset :-) You should try som() in the kohonen package. It offers different modes of learning and multi-cores. https://rdrr.io/cran/kohonen/man/supersom.html <https://urldefense.com/v3/__https://rdrr.io/cran/kohonen/man/supersom.html__;!!LIr3w8kk_Xxm!s51KSc8gjSLuDrWnZCf_7s35vgDjfNpY1P_8puxFNc37ZrnFl7DQqrfCqJhrxRFs46ktJT6KusnJ39VJJliEwQTd$> You might switch to a Julia implementation of SOM for large datasets. https://github.com/LCSB-BioCore/GigaSOM.jl <https://urldefense.com/v3/__https://github.com/LCSB-BioCore/GigaSOM.jl__;!!LIr3w8kk_Xxm!s51KSc8gjSLuDrWnZCf_7s35vgDjfNpY1P_8puxFNc37ZrnFl7DQqrfCqJhrxRFs46ktJT6KusnJ39VJJqdn88BP$> Best. — Reply to this email directly, view it on GitHub <https://urldefense.com/v3/__https://github.com/SofieVG/FlowSOM/issues/55*issuecomment-1261341380__;Iw!!LIr3w8kk_Xxm!s51KSc8gjSLuDrWnZCf_7s35vgDjfNpY1P_8puxFNc37ZrnFl7DQqrfCqJhrxRFs46ktJT6KusnJ39VJJvJx9rSK$>, or unsubscribe <https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AKIGYMW52S7G2ANZS3PPKB3WASITFANCNFSM6AAAAAAQYAZAHU__;!!LIr3w8kk_Xxm!s51KSc8gjSLuDrWnZCf_7s35vgDjfNpY1P_8puxFNc37ZrnFl7DQqrfCqJhrxRFs46ktJT6KusnJ39VJJryYQ7aq$> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[SamGG]

supersom: quite direct.
GigaSOM: not tried.

[jonhsussman]

Great, this sounds like an easy solution. I'll work on this and report back later

…

On Wed, Sep 28, 2022, 4:09 PM Samuel Granjeaud ***@***.***> wrote: supersom: quite direct. GigaSOM: not tried. — Reply to this email directly, view it on GitHub <https://urldefense.com/v3/__https://github.com/SofieVG/FlowSOM/issues/55*issuecomment-1261412852__;Iw!!LIr3w8kk_Xxm!quqRRVxskp65_K6GxU6XP6RJEMmhkfkPDLwyJQTBQlx8m2uKOe1gKvoDZ85TmtQ9KaZ3jJ0hjhPkLiOFwje19KCZ$>, or unsubscribe <https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AKIGYMVVWDJK4ZWX4FOL4WDWASQWDANCNFSM6AAAAAAQYAZAHU__;!!LIr3w8kk_Xxm!quqRRVxskp65_K6GxU6XP6RJEMmhkfkPDLwyJQTBQlx8m2uKOe1gKvoDZ85TmtQ9KaZ3jJ0hjhPkLiOFwv3KwPW7$> . You are receiving this because you authored the thread.Message ID: ***@***.***>