[Emergent Session]: Metavaluation: Kickstarting a Virtuous Cycle in Open Science

[CooperSmout]

Title

Metavaluation: Kickstarting a Virtuous Cycle in Open Science

Host and Affiliation

Cooper Smout
(1) Open Heart + Mind (OHM)
(2) Institute for Globally Distributed Open Research and Education (IGDORE)

Contact Details

cooper@openheartmind.org

Short description and the goals for the session

Metavaluation is a participatory evaluation framework designed to connect and empower open science communities by realigning incentives from the ground up. It uses a simple, fast, and accessible mechanism to collect peer assessments of diverse contributions—such as ideas, activities, and resources—and aggregates them into transparent, multidimensional metrics that communities can use for fair recognition, reward, and coordination. Crucially, Metavaluation enables the self-referential recognition of core system functions—such as peer evaluation and platform improvement—aligning incentives across diverse stakeholders and laying the groundwork for a virtuous cycle of systemic change.

In this session, we’ll demonstrate the framework through a live participatory workshop focused on recognising contributions to the Open Science Room itself. Participants will:

• Nominate contributions to the OSR—such as talks, workshops, and behind-the-scenes efforts
• Use the Metavaluation prototype to evaluate these contributions in real time
• Reflect on the results and discuss how Metavaluation could support culture change in neuroimaging and beyond

Panelists will include OSR contributors who have agreed to their work being openly evaluated during the session. All participants will be invited to continue engaging with the platform afterward—helping seed an ongoing, community-led culture of gratitude, reflection, and coordination.

Useful links

Website: https://openheartmind.org/
Github: https://github.com/openheartmind/cOHMunity
Blog: https://coopersmout.substack.com/

Relevant GitHub Users

Panelists TBA

Which time slots work for you? (Please select all that apply.)

ses4-6/26 (Thu.) 9:15-10:15, ses5-6/26 (Thu.) 11:30-12:30, ses6-6/27 (Fri.) 9:15-10:15, ses7-6/27 (Fri.) 11:30-12:30, ses8-6/28 (Sat.) 9:15-10:15, ses9-6/28 (Sat.) 11:30-12:30

Recording and streaming volunteer

No response

Agreement to Code of Conduct

• I agree to follow OSR's Code of Conduct

Agreement to Recording and Disemmination

• I agree to record and diseminate this session on YouTube.

Agreement to be inclued in the proceedings

• I agree to be included in the proceedings.

[juchiyu]

Hi @CooperSmout, thank you for submitting to the emergent session. We will have you at ses4-6/26 (Thu.) 9:15-10:15.

Please address the following comments from the OSR team:

• Please specify how interactive your session will be with the attendees.
• We encourage you to incorporate more interactive components into your session to engage the community more effectively.

We will be preparing the proceedings for the OSR, please keep in mind that we will need some paragraphs about your sessions from you and will be collecting you author information soon during the OHBM.

[CooperSmout]

Hi @juchiyu thanks for the feedback. I was a bit unclear on the session format and so I proposed this as a presentation, but I could convert it into a workshop format to make it more interactive. In essence, we would use our prototype to nominate and recognise real contributions to the OSR itself, demonstrating the framework in action and generating real-time feedback for reflection and future planning.

The timing could look something like this:

• Brief introduction to Metavaluation (10 mins)
• Nominate OSR contributions (15 mins) -- panel members only
• Evaluate contributions (20 mins) -- everyone
• Review results and discuss (10 mins)

Does that sound more in line with what the team are envisioning?

If so, I'll invite some colleagues to form a panel, so that we can ensure we get engagement with the process. I've found it really valuable in previous workshops if one/more of the organisers can jon the discussion, so they can speak to less visible tasks that went into organising the event, and consent to their being evaluated. Please pass on this invite to the OSR team and let me know if anyone would be willing to participate in such an experiment as a panelist.

[CooperSmout]

I've updated the description accordingly now, and will add names of panelists when I've confirmed

[juchiyu]

Hi @CooperSmout, sorry for the late reply, the day is a bit packed. We can see if anyone onsite would like to join, or it could also be someone from the attendees. If there isn't anyone, I will hop on.

Crowdcast link

Please feel free to share with others whom you would like to invite or attend. When you click on the link, you will need to register and click "save a spot" to join the session.

[CooperSmout]

Does this mean it is an online-only session? I just arrived in Brisbane, assuming it's broadcast in person

[juchiyu]

Hi @CooperSmout, oh, not at all. It's an in-person session, and the link is there in case you have any speakers online and for us to keep a record to draft the proceedings.

[CooperSmout]

Thanks so much for hosting, really enjoyed the session 🙏 I seem to be having troubles downloading the recording from Crowdcast but will keep trying. Please let me know if it will be posted anywhere else.

[juchiyu]

Hi Cooper, we will release them on YouTube once we receive all media approval from anyone involved in the session. (But, as we will be doing the upload in batch, it might come a bit later when we get information from all other sessions.) I will keep you posted!