perf-sanity-tests: analyze subtests of each perf test

[rli9]

It now outputs below and results in perf_all_metrics_test.fail stat, while developer wants to know more detail of what the actual failure is

2022-04-12 08:37:08 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-func-3f5f0df7bf0f8c48d33d43454fc0b7d0f3ab9537/tools/perf/perf test 90
 90: perf all metrics test      : FAILED!

After adding -v, it shows Branching_Overhead testing fails.

2022-04-12 08:37:08 sudo /usr/src/perf_selftests-x86_64-rhel-8.3-func-3f5f0df7bf0f8c48d33d43454fc0b7d0f3ab9537/tools/perf/perf test 90 -v
 90: perf all metrics test                                           :
--- start ---
test child forked, pid 10368
Testing Big_Code
Testing Big_Code_SMT
Testing BpTkBranch
Testing Branch_Misprediction_Cost
Testing Branch_Misprediction_Cost_SMT
Testing Branching_Overhead
Metric 'Branching_Overhead' not printed in:
# Running 'internals/synthesize' benchmark:
Computing performance of single threaded perf event synthesis by
synthesizing events on the perf process itself:
  Average synthesis took: 460.354 usec (+- 0.273 usec)
  Average num. events: 44.000 (+- 0.000)
  Average time per event 10.463 usec
  Average data synthesis took: 485.178 usec (+- 0.274 usec)
  Average num. events: 296.000 (+- 0.000)
  Average time per event 1.639 usec

 Performance counter stats for 'perf bench internals synthesize':
     <not counted>      BR_INST_RETIRED.NEAR_CALL                                     (0.00%)
     <not counted>      BR_INST_RETIRED.NEAR_TAKEN                                     (0.00%)
     <not counted>      BR_INST_RETIRED.NOT_TAKEN                                     (0.00%)
     <not counted>      BR_INST_RETIRED.CONDITIONAL                                     (0.00%)
     <not counted>      CPU_CLK_UNHALTED.THREAD                                       (0.00%)
        9773202751 ns   duration_time                                               
       9.773202751 seconds time elapsed
       4.318739000 seconds user
       5.275829000 seconds sys

Some events weren't counted. Try disabling the NMI watchdog:
        echo 0 > /proc/sys/kernel/nmi_watchdog
        perf stat ...
        echo 1 > /proc/sys/kernel/nmi_watchdog
Testing Branching_Overhead_SMT

The requirement is

• run the test with -v option
• update stats script so it can result in stats like

perf_all_metrics_test.Big_Code.pass: 1
perf_all_metrics_test.Branching_Overhead.fail: 1

• add new spec test (refer to spec/stats/perf-sanity-tests)

Acceptance Test

$ rubocop stats/perf-sanity-tests # there's no issue
$ rake spec spec=stats # pass