Master/notes at master · larmel/Master · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
** Talk at Google **

Keywords:
- Violin Plot
- P-test


** Observer effect **
Add instrumentation pushes code or data beyond some boundry. Cache blocks, page size, TLB, etc. Boundaries are everywhere.


** Measurement bias **
Do measurements on one particular computer setup can bias results towards that particular computer.

- Link order, change argument list to ld program. ~20%
- Room temperature (!)

=> Cannot easily suppress bias. Too many sources.

Predictable? No! In O2 vs O3.

Link-order dependent on machine (Pentium 4 vs Core)

Confidence interval over 15 runs.
4 different architectures and 2 compilers.

** Incredibly difficult to explain what is happening. *
-> Deep knowledge of hardware
-> Collect HW data to test hypothesis
   But: HW designers does not allow collecting all interesting data.

"Loop stream detectors" (LSD). Intel does not reveal all info. Can't confirm. No metric for measuring LSD during execution. Can guess what is happening, but very hard to pinpoint exact causes.


Other papers:
Simulation -> avoid observer effect. Still suffer for measurement bias (link order).


Diverse benchmark suite -> statistically factor out bias? No! SPEC- averaged over 12 benchmarks is not enough to cancel out bias.


Other sciences:
- Many different measurement setups. Cancel out bias. Compare using T-test. <- lookup this0
- Causality analysis. 'Correct data is impossibleto get in general'.

In CS =>

* Collect data with many setups, ex many different environment sizes.
* Causality, hypothesis. "Change env. size" => "Change stack address" => "Change performance". Try to break a link and observe effect. 'Intervention'. Validate hypothesis.

NB: Have solution to varying env. size vs. stack position.


Q&A:
Q: How to identify?
A: Dunno ... Community over time will discover

Q: Tool for link order, reduce variance?
A:


Q: More difficult with more complex machines?
A: Partially true, but old machines also chaotic.

Q: Shouldn't HW be created to be more predictable?
A: Naive to expect to eliminate this. Computers are non-linear and chaotic.

Q: Accessibility of performance counters? Over 300 ...
A: Bad documentation. Names mean nothing.


** ELF **
http://en.wikipedia.org/wiki/Executable_and_Linkable_Format
readelf

http://en.wikipedia.org/wiki/X86_assembly_language


wat