Reproducibility across different platforms

[tevador]

I added a new test project which can be used for testing results on different platforms.

If you want to test it on your platform:

1. Start the sandbox node sandbox.js
2. Run command ./Tevador.RandomJS.Test.exe --threads 2 --seed 123 --count 1000

The number of threads can be adjusted according to the number of cores of your processor. The command generates 1000 random programs and calculates a cumulative hash of all the outputs.

So far I have tested:

• Intel Ivy Bridge, Windows x86_64
• Intel Sandy Bridge, Ubuntu x86_64
• AMD Ryzen, Ubuntu x86_64
• Raspberry PI, Ubuntu armv7l
• Intel Kaby Lake R, Windows x86_64

In all cases the result is 3cc26d6be7f5f542e8037247deec69c2e3a3d70db67dac5371f86e06fbe3c8cc.

(Valid as of commit 79d3c4c)

• Kaby Lake Intel Core i7-7500U CPU @ 2.70GHz
• Ubuntu 17.10, x86_64
• As of commit: 79d3c4c

./Tevador.RandomJS.Test.exe --threads 2 --seed 123 --count 1000 Collecting statistics from 1000 random program executions (seed = 123) 5 %, 10 %, 15 %, 20 %, 25 %, 30 %, 35 %, 40 %, 45 %, 50 %, 55 %, 60 %, 65 %, 70 %, 75 %, 80 %, 85 %, 90 %, 95 %, Completed in 8.4801327 seconds Runtime histogram: Longest runtimes: Seed = 92117886952e3f3c5010956afba334430a20eaa983b4ee5f6c8647e5010cf6b4, Runtime = 0.05701 s Seed = f767b319d74bdb44329dfeb4072b558bb89bbcd2dadef61b863f06f4c0f84365, Runtime = 0.05645 s Seed = fb0140e24f67a86b8d5da25870f056264d210a1b8a8fa678f86edfe0d17904f4, Runtime = 0.05559 s Seed = 0e1fd5a6d18e5cc2d0fa9cfb781be7f33f4f11a00c0a7b1085142192291d9bfc, Runtime = 0.05529 s Seed = 3a465dcc93206126d4c68d24b8d0854e95fd4d1be6ce3fb18583792997fb0602, Runtime = 0.04514 s Seed = d6e3a869b2518a2e5076cd498b82bb76297779cd9f1bf3121e0f2a74a37fd90c, Runtime = 0.04164 s Seed = beed8f13cc2af4ab76ec0a73c9b286a6c499571d8f554983d5d6fd46bbb654a2, Runtime = 0.04049 s Seed = 39b432618eb3fe0e374033e5bd24e28abc78a6f55fa576fe4a01c03fff5a8ff5, Runtime = 0.03823 s Seed = 46509a1b7306a0929fbfed991ac472499a5fb1c569d4e3ced8863c23eee61eb2, Runtime = 0.03597 s Seed = 73b8ec5cd13343519cee1ef7662121c961f778ed925c8d37665e0b6b2a68f41a, Runtime = 0.03493 s Runtime [s] Min: 0.00167; Max: 0.05701; Avg: 0.00771; Stdev: 0.00640; Runtime [s] 99.99th percentile: 0.0570068010129035 Average entropy of program output (est.): 1242.39 bits Cyclomatic complexity Min: 0; Max: 0; Avg: 0; Stdev: 0; Halstead difficulty Min: 0; Max: 0; Avg: 0; Stdev: 0; 0.00167 17 0.00231 60 0.00295 101 0.00359 106 0.00423 97 0.00487 90 0.00551 70 0.00615 62 0.00679 52 0.00743 42 0.00807 44 0.00871 27 0.00935 27 0.00999 20 0.01063 22 0.01127 22 0.01191 15 0.01255 10 0.01319 8 0.01382 12 0.01446 10 0.01510 6 0.01574 3 0.01638 9 0.01702 8 0.01766 4 0.01830 2 0.01894 5 0.01958 5 0.02022 3 0.02086 3 0.02150 4 0.02214 5 0.02278 1 0.02342 1 0.02406 2 0.02470 0 0.02534 0 0.02598 2 0.02662 4 0.02726 0 0.02790 1 0.02854 1 0.02918 0 0.02981 0 0.03045 3 0.03109 1 0.03173 0 0.03237 1 0.03301 0 0.03365 2 0.03429 1 0.03493 0 0.03557 1 0.03621 0 0.03685 0 0.03749 0 0.03813 1 0.03877 0 0.03941 0 0.04005 1 0.04069 0 0.04133 1 0.04197 0 0.04261 0 0.04325 0 0.04389 0 0.04453 1 0.04517 0 0.04580 0 0.04644 0 0.04708 0 0.04772 0 0.04836 0 0.04900 0 0.04964 0 0.05028 0 0.05092 0 0.05156 0 0.05220 0 0.05284 0 0.05348 0 0.05412 0 0.05476 1 0.05540 1 0.05604 1 0.05668 1 Cumulative output hash: 3cc26d6be7f5f542e8037247deec69c2e3a3d70db67dac5371f86e06fbe3c8cc

[tevador]

@wowario Your first result was different. Any idea why?

Yes, it was. The first test ran on an older version of ProgramOptions.xml with different weight values, I re-cloned the repo and ran the test a second time to get the correct result.

[jwinterm]

Tested with Intel(R) Core(TM) i5-6600K CPU @ 3.50GHz
Ubuntu 18.04 x86_64
commit 79d3c4c

Cumulative output hash: 3cc26d6be7f5f542e8037247deec69c2e3a3d70db67dac5371f86e06fbe3c8cc

[tevador]

We should test an x86 (32bit) platform (not that I'm expecting anyone to mine on such machine). Also more AMD CPUs (they can give different results for transcendental math functions).

[jwinterm]

I have a couple oldish AMD boxes around, don't think I have any x86s though. Will try to test on AMDs in next couple days.

[jwinterm]

Tested on -
AMD Athlon(tm) II X2 220 Processor
Ubuntu 16.04 x86_64
commit: 8923b3b

got:
Cumulative output hash: 3cc26d6be7f5f542e8037247deec69c2e3a3d70db67dac5371f86e06fbe3c8cc

[bomb-on]

Tested on:
OS X 10.13.5
Intel Core i5 2.7GHz
commit: 8923b3b

Results:

$ mono Tevador.RandomJS.Test.exe --threads 2 --seed 123 --count 1000

Collecting statistics from 1000 random program executions (seed = 123)
5 %, 10 %, 15 %, 20 %, 25 %, 30 %, 35 %, 40 %, 45 %, 50 %, 55 %, 60 %, 65 %, 70 %, 75 %, 80 %, 85 %, 90 %, 95 %,
Completed in 25.1238964 seconds
Longest runtimes:
Seed = 0e1fd5a6d18e5cc2d0fa9cfb781be7f33f4f11a00c0a7b1085142192291d9bfc, Runtime = 0.38644 s
Seed = 46509a1b7306a0929fbfed991ac472499a5fb1c569d4e3ced8863c23eee61eb2, Runtime = 0.25880 s
Seed = 39b432618eb3fe0e374033e5bd24e28abc78a6f55fa576fe4a01c03fff5a8ff5, Runtime = 0.22791 s
Seed = 2822106c992d76c830b35009fa847da93520c507e5190a48f5357a11a607356b, Runtime = 0.19583 s
Seed = 92117886952e3f3c5010956afba334430a20eaa983b4ee5f6c8647e5010cf6b4, Runtime = 0.16995 s
Seed = d8fe9d7d7855c0d2c6a81564efe881b147cd17d35410adfc609f6ef81d4f9c57, Runtime = 0.16821 s
Seed = fb0140e24f67a86b8d5da25870f056264d210a1b8a8fa678f86edfe0d17904f4, Runtime = 0.15999 s
Seed = e53b00a5e8db6f7295a5de8883ea7ead3600067f8f729882c25bc3f53efa79ac, Runtime = 0.15155 s
Seed = 42f0b1623c4650d47f8348bbeee27edcf9ed5ab50f3d783b2cecc7ffa8be7d8d, Runtime = 0.14507 s
Seed = 7fb4dbf4c3b50b4317176ca78a4d1478084d7e3ff994229ccd373d8ab9e7c54a, Runtime = 0.13481 s
Runtime [s] Min: 0.00323; Max: 0.38644; Avg: 0.02417; Stdev: 0.02796;
Runtime [s] 99.99th percentile: 0.386439353023889
Average entropy of program output (est.): 1242.39 bits
Cyclomatic complexity Min: 0; Max: 0; Avg: 0; Stdev: 0;
Lines of code Min: 0; Max: 0; Avg: 0; Stdev: 0;
Halstead difficulty Min: 0; Max: 0; Avg: 0; Stdev: 0;
Cumulative output hash: 3cc26d6be7f5f542e8037247deec69c2e3a3d70db67dac5371f86e06fbe3c8cc

[tevador]

Note that as of commit d32fd77, the cumulative hash is different because the generator has been slightly modified.

The new hash is:

$ ./Tevador.RandomJS.Test.exe --count 1000 --seed 123 --threads 2
Collecting statistics from 1000 random program executions (seed = 123)
...
Cumulative output hash: 80d77dae86b3b49bfb4cf53281f10cb5db1d1e857d9adb7429bf116a9d7880ba

If you do git pull, you have to manually update your ProgramOptions.xml (copy from src/Tevador.RandomJS/ProgramOptions.xml to bin).

[tevador]

I decided to remove all transcendental functions (exp, log, sin, cos, cbrt, atan) from the generator due to poor cross-platform compatibility (rounding is not enough).

With these functions enabled, the results don't match between Windows and Linux with the XS engine.

The reason why the results match for the V8 is because V8 has its own software implementation of all these functions using only IEEE-754 safe operations. We could copy this implementation to the XS, but I don't see the point in that because an ASIC could easily include a faster on-chip implementation.

The only function that stays is Math.sqrt which is portable and implemented in hardware in most CPUs.

For testing of floating point math compatibility, please use the following command:

Tevador.RandomJS.Test.exe --count 1000 --seed 123 --threads 2 --customOptions --XML_AssignmentOperators_Add 1 --XML_AssignmentOperators_Sub 1 --XML_AssignmentOperators_Mul 1 --XML_AssignmentOperators_Div 1 --XML_AssignmentOperators_Mod 1 --XML_AssignmentOperators_PreInc 1 --XML_AssignmentOperators_PostDec 1 --XML_AssignmentOperators_PreDec 1 --XML_AssignmentOperators_PostInc 1 --XML_UnaryOperators_Sqrt 1 --XML_BinaryOperators_Add 1 --XML_BinaryOperators_Sub 1 --XML_BinaryOperators_Mul 1 --XML_BinaryOperators_Div 1 --XML_BinaryOperators_Mod 1 --XML_Literals_Numeric 1 --XML_NumericLiterals_DecimalInteger 3 --XML_NumericLiterals_BinaryInteger 1 --XML_NumericLiterals_OctalInteger 3 --XML_NumericLiterals_SmallInteger 3 --XML_NumericLiterals_HexInteger 3 --XML_NumericLiterals_FixedFloat 3 --XML_NumericLiterals_ExpFloat 3 --XML_Expressions_AssignmentExpression 4 --XML_Expressions_BinaryExpression 5 --XML_Expressions_Literal 2 --XML_Expressions_UnaryExpression 5 --XML_Statements_AssignmentStatement 1 --XML_AssignmentInForLoop_Add 1 --XML_GlobalVariablesCountRange_Min 100 --XML_GlobalVariablesCountRange_Span 0 --XML_VariableInitializerDepth 4 --XML_MaxExpressionDepth 4 --XML_ConstVariableChance 0.1 --XML_MaxStringLengthRange_Min 35 --XML_MaxStringLengthRange_Span 0

It generates programs with only math expressions and doesn't depend on ProgramOptions.xml.

Output with commit e136756 should be:

Cumulative output hash: b83a20eaa1d1615b860dd7477dc8945d23b84cd9082a7240df600b347410441c

It is now possible to also test with the XS engine. You have to clone and compile it from here: https://github.com/tevador/moddable

To use this engine, append the following command line option: --runnerUri file://PATH-TO-XST-EXECUTABLE, for example:

--runnerUri file:///home/tevador/moddable/build/bin/lin/release/xst
--runnerUri file://C:/projects/moddable/build/bin/win/release/xst.exe

For the NodeJS sandbox, you can use (it's the default option):

--runnerUri http://localhost:18111

• Intel i7 3770
• NixOS e70194, node v10.3.0, mono 5.8.0.108, RandomJS e136756

mono Tevador.RandomJS.Test.exe --count 1000 --seed 123 --threads 2 --customOptions --XML_AssignmentOperators_Add 1 --XML_AssignmentOperators_Sub 1 ..

..

Cumulative output hash: b83a20eaa1d1615b860dd7477dc8945d23b84cd9082a7240df600b347410441c

[tevador]

@fuwa0529 b83a20eaa1d1615b860dd7477dc8945d23b84cd9082a7240df600b347410441c is the correct hash for that commit.

I have also tested the latest commit from the dev branch on Intel Kaby Lake (Windows), AMD Ryzen and Raspberry Pi 3 (Ubuntu) and all give the same results.

I think at this point we can conclude that reproducibility is not an issue.

[SChernykh]

I have also tested the latest commit from the dev branch on Intel Kaby Lake (Windows), AMD Ryzen and Raspberry Pi 3 (Ubuntu) and all give the same results.

I'm curious about performance numbers for all these processors. Can even Raspberry Pi 3 be used for mining?

[bomb-on]

Tested on:
OS X 10.13.5
Intel Core i5 2.7GHz
commit: e136756

Results:

$ mono Tevador.RandomJS.Test.exe --count 1000 --seed 123 --threads 2 --customOptions --XML_AssignmentOperators_Add 1 --XML_AssignmentOperators_Sub 1 --XML_AssignmentOperators_Mul 1 --XML_AssignmentOperators_Div 1 --XML_AssignmentOperators_Mod 1 --XML_AssignmentOperators_PreInc 1 --XML_AssignmentOperators_PostDec 1 --XML_AssignmentOperators_PreDec 1 --XML_AssignmentOperators_PostInc 1 --XML_UnaryOperators_Sqrt 1 --XML_BinaryOperators_Add 1 --XML_BinaryOperators_Sub 1 --XML_BinaryOperators_Mul 1 --XML_BinaryOperators_Div 1 --XML_BinaryOperators_Mod 1 --XML_Literals_Numeric 1 --XML_NumericLiterals_DecimalInteger 3 --XML_NumericLiterals_BinaryInteger 1 --XML_NumericLiterals_OctalInteger 3 --XML_NumericLiterals_SmallInteger 3 --XML_NumericLiterals_HexInteger 3 --XML_NumericLiterals_FixedFloat 3 --XML_NumericLiterals_ExpFloat 3 --XML_Expressions_AssignmentExpression 4 --XML_Expressions_BinaryExpression 5 --XML_Expressions_Literal 2 --XML_Expressions_UnaryExpression 5 --XML_Statements_AssignmentStatement 1 --XML_AssignmentInForLoop_Add 1 --XML_GlobalVariablesCountRange_Min 100 --XML_GlobalVariablesCountRange_Span 0 --XML_VariableInitializerDepth 4 --XML_MaxExpressionDepth 4 --XML_ConstVariableChance 0.1 --XML_MaxStringLengthRange_Min 35 --XML_MaxStringLengthRange_Span 0

Collecting statistics from 1000 random program executions (seed = 123)
5 %, 10 %, 15 %, 20 %, 25 %, 30 %, 35 %, 40 %, 45 %, 50 %, 55 %, 60 %, 65 %, 70 %, 75 %, 80 %, 85 %, 90 %, 95 %,
Completed in 7.6793444 seconds
Longest runtimes:
Seed = 86460416fe8be2c82375ee36c6b48b3c0356f2d9270ff5f856e7202706020a15, Runtime = 0.04320 s
Seed = f74ea47f971bdfe927a67e0eb8bcb0836fc21838dfb69e7ed179defc424e289b, Runtime = 0.03316 s
Seed = 6fa68338a1d4656b071b682aa7828b73dafb4ba8b90f7bc82f384042adb5ff2f, Runtime = 0.03298 s
Seed = fdb87aac7f31faee0aa653df81007761c0cee3f436b3cb0ecb4b2e19932041f5, Runtime = 0.03287 s
Seed = 1a9d352a9af1e6c08bceba71ffe4fbfb7f61c06c8777bdc344f8d010fc74f77f, Runtime = 0.02954 s
Seed = a894dbd527ebf7e1669316b0f7d9b028b33c72c0a341d586d4d32f2e22daad23, Runtime = 0.02395 s
Seed = 3f6524e301d20266e08aca271a4248f6ad025756539a637aaca6bf951a393d96, Runtime = 0.01987 s
Seed = 1fde2adc66eb993b77be43a0df7b313c5908eaa118b78a6edbaea223d47067c1, Runtime = 0.01925 s
Seed = 0021020d8391b1c4a9009b676f61245d7625f4c7fe2b64d06bd304c8f4bf9360, Runtime = 0.01836 s
Seed = c309c0a2c910a6f472259c73169125ccd33b111748ed96c1c75b13a17e21b562, Runtime = 0.01786 s
Runtime [s] Min: 0.00308; Max: 0.04320; Avg: 0.00710; Stdev: 0.00324;
Runtime [s] 99.99th percentile: 0.0431984979659319
Average entropy of program output (est.): 6739.48 bits
Cyclomatic complexity Min: 0; Max: 0; Avg: 0; Stdev: 0;
Lines of code Min: 0; Max: 0; Avg: 0; Stdev: 0;
Halstead difficulty Min: 0; Max: 0; Avg: 0; Stdev: 0;
Matches 'SyntaxError' optimization: 0.00 %
'SyntaxError' optimization runtime: 0.00 %
Matches 'XS': 0.00 %
Cumulative output hash: b83a20eaa1d1615b860dd7477dc8945d23b84cd9082a7240df600b347410441c