Control memory consumption

[Preetam]

1. An interesting idea would be to partition based on memory usage instead of fixed time ranges. I think the design should support this already, but it would make the compaction and retention logic a bit more complicated.

2. For older partitions, maybe we can read metadata on-demand instead of keeping it in memory all the time. This would help with cases where there are lots of sources and metrics.