awful (2): prevent alloc elimination and size better

[iximeow]

I'd tested the commit introducing this hack with one additional detail I'd overlooked: an eprintln!("balloon: {:p}", balloon.as_ptr()). This prevented rustc/LLVM from optimizing out the allocation. Of course, it would be unsightly to have a random eprintln!() committed so I removed that in the PR, not realizing the allocation was optimized out entirely.

Along the way testing on a racklette had one VM where the heap grew to ~450 MiB, beyond the ~350 MiB the balloon would have gotten things. So grow the constant earmark a bit. RFD 413 estimates Propolis heap as 512 MiB at worst, versus the balloon's worst case of ~483 MiB (on top of the baseline 30 MiB or so of heap, so a bunch of VMs with 11 local disks are a bit more than budgeted. RFD 413 doesn't think too hard about "what if there are 256 propolises" so the whole area is a bit weird.)

I've surgically swapped this into London where fill.fio has been running for about 30 minutes without issue, and from here I'll let it keep going!

[leftwo]

As discussed in the huddle, we are going forward with this

[hawkw]

wow, even more awful!

[iximeow]

so, that fio ran fine up until...

[   24.635963] Loaded X.509 cert 'wens: 61c038651aabdcf94bd0ac7ff06c7248db18c600'
[   24.656652] NET: Registered PF_QIPCRTR protocol family
[  362.651626] hrtimer: interrupt took 3984548 ns
[17092.877781] INFO: task fio:4202 blocked for more than 122 seconds.
[17092.879381]       Tainted: P           O       6.8.0-90-generic #91-Ubuntu
[17092.881095] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[17092.882996] task:fio             state:D stack:0     pid:4202  tgid:4202  ppid:3926   flags:0x00000002
[17092.883004] Call Trace:
[17092.883008]  <TASK>
[17092.883015]  __schedule+0x27c/0x6b0
[17092.883029]  schedule+0x33/0x110
[17092.883033]  schedule_preempt_disabled+0x15/0x30
[17092.883037]  __mutex_lock.constprop.0+0x42f/0x740
[17092.883044]  __mutex_lock_slowpath+0x13/0x20
[17092.883047]  mutex_lock+0x3c/0x50
[17092.883050]  bdev_release+0x40/0x240
[17092.883058]  blkdev_release+0x15/0x30
[17092.883062]  __fput+0xa3/0x2e0
[17092.883068]  __fput_sync+0x1c/0x30
[17092.883072]  __x64_sys_close+0x3e/0x90
[17092.883078]  x64_sys_call+0x1fec/0x25a0

that's about 4.7 hours of guest uptime, or about 3 hours ago. the propolis-server heap now looks like:

root@oxz_propolis:~# pmap 29126 | grep heap
0000000003D99000     429608K rw---    [ heap ]
000000001E123000      22632K rw---    [ heap ]
000000001F73D000     131888K rw---    [ heap ]

so after some Propolis uptime, the balloon is still too small. and in fact it's too small by about 100 MiB. we don't really know where that comes from, but we've done better to hedge against #1008. this only got reached after a while of the pathological case of "uninterrupted I/O with novel guest addresses for hours at a time" - an improvement on the no-balloon outcome of "hit after two minutes of I/O with novel guest addresses". like with a few other commits recently, this will get unwound when we've fixed the underlying OS issue.