feat: Benchmark Talk2Scholars (T2S) Agent Using Open-Source Life Science QA and Scientific Reasoning Datasets

[ansh-info]

Is your feature request related to a problem? Please describe.

To ensure the efficacy of the Talk2Scholars (T2S) agent in addressing life science queries and performing scientific reasoning, it's essential to benchmark its performance against established open-source datasets. Currently, there's no systematic evaluation in place to assess T2S's capabilities in these domains.

Describe the solution you'd like

Implement a benchmarking framework for T2S that includes:

1. Dataset Selection: Identify and curate relevant open-source benchmarks such as:

   • BioASQ-QA: A dataset containing biomedical questions with reference answers, reflecting real information needs of biomedical experts.
   • SciQA: A scientific QA benchmark leveraging the Open Research Knowledge Graph (ORKG), encompassing questions from various research fields.
   • LAB-Bench: A dataset evaluating language models on practical biology research tasks, including literature search and data analysis.
   • SciKnowEval: A framework assessing large language models across multiple levels of scientific knowledge in domains like biology and chemistry. Nature][1], Nature[2], arXiv[3], arXiv[4])

2. Evaluation Metrics: Define appropriate metrics such as accuracy, precision, recall, and F1-score to assess T2S's performance on these datasets.

3. Benchmarking Pipeline: Develop an automated pipeline to run T2S on the selected datasets, collect results, and compute evaluation metrics.

4. Reporting: Generate comprehensive reports detailing T2S's performance, highlighting strengths and areas for improvement.

Describe alternatives you've considered

• Manual evaluation of T2S's responses, which is time-consuming and may lack consistency.
• Using proprietary datasets, which may not be accessible or reproducible for the broader community.

Additional context

Benchmarking T2S against these open-source datasets will provide insights into its current capabilities and guide future enhancements. It will also facilitate comparisons with other models and promote transparency in its development.

[gurdeep330]

Hi @ansh-info,

Let's start to benchmark BioASQ Task B (version training_13b), which includes yes/no and list-type biomedical questions. This will involve building a pipeline using a Jupyter notebook and leveraging the T2S agents via API calls. The plan is as follows:

1. Yes/No Questions Benchmark

Input: Iterate over each yes/no question in the BioASQ Task B dataset.
Query Conversion: Convert each question into a search query
Search with S2 Agent: Call the S2 agent via API to retrieve relevant articles (including PubMed-only articles). Limit the number of articles retrieved to 25.
Answer Extraction with T2S: Pass the original question along with the abstracts* of the retrieved articles to the T2S agent.
Structured Output: Use with_structured_output from Langchain to parse the T2S output into a Pydantic object that returns answers with type bool.
Metric: Measure ACC, REC and MCC

Note: We do not download full articles. Only abstracts from PubMed are used.

2. List-Type Questions Benchmark

Follow the same steps as in the yes/no benchmark, except that it returns a list of answers. Use metrics like the Jaccard Index to measure the performance.

Please feel free to share your thoughts on the plan.