Validation Issues due to test data quality

[Zhenglei-BCS]

• Data quality

---

• One point is that the column headers are inconsistent across data and res: In res ​Measurement \r\nvaribale is actually in data "Measurement variable", this need to be fixed.
• Also the measurement variable is not used for study types other than myriophylum study, which means the content in the measurement variable are inconsistent across data and res.
• More importantly, reference item test group is included in the data, they are only used for two-sample tests, not for many-to-one tests like Dunnett's test.

[Zhenglei-BCS]

The core problem lies in data quality issues within test_cases_res.rda and the need for a robust validation script that can handle multi-endpoint studies and data inconsistencies.

• Consolidate the logic from the various "fixed" and "complete" validation scripts.
• Address the data quality issues, such as dose conversion and mean misalignment.
• Explicitly handle multi-endpoint studies.
• Produce a detailed comparison table of actual vs. expected values for all Dunnett test cases, making it easy to pinpoint discrepancies.

[Zhenglei-BCS]

Note that for test cases res, the Description is a mixture of test, test direction, sometimes dose was also included, the test output parameter including mean, t-value, T-value (adjusted), for example:

• Williams' test, smaller, T-value (adjusted)
• Dunnett's test, smaller, 0,41, Mean

Therefore, validation should adjust to these specific lablings.

[Zhenglei-BCS]

Thanks for the detailed context. The join error happens because actual_results is a custom object (class dunnett_test_result), not a data frame. You need to join against actual_results$results_table and, for the Mean comparisons, join against per-dose group means computed from endpoint_data. You also need to extract dose numerics from the “comparison” string like “0.0448 - 0”, and to convert scientific notation and comma-separated doses like “4.48E-2” and “0,41”.

Below is a drop-in replacement for the “Create comparison table” part inside your for-loop. It fixes:

• Joining against the correct actual results data frame.
• Robust dose parsing from both expected results and actual comparison strings.
• Mean comparison using observed per-dose group means from data (not the model object).
• Full outer-join across Mean/T/P so you don’t lose rows when some metrics are missing.
• Excludes “Reference item” rows from Dunnett calculations.

Please replace your current “Create comparison table” block with this one and add the two helpers near the top.

Add these helpers near the top (with your other helpers):

convert_dose <- function(dose_str) {
  if (is.na(dose_str) || dose_str == "n/a" || dose_str == "") return(NA_real_)
  dose_str <- gsub(",", ".", as.character(dose_str))
  suppressWarnings(as.numeric(dose_str))
}

dose_from_comparison <- function(comp_vec) {
  # comp like "0.0448 - 0" or "10 - 0"
  sapply(strsplit(comp_vec, " - "), function(parts) convert_dose(parts[1]))
}

Then use this block to build the comparison table:

# Exclude reference item groups (used for two-sample tests only)
endpoint_data <- endpoint_data[!grepl("reference", endpoint_data[["Test group"]], ignore.case = TRUE), ]

# Actual Dunnett outputs as a data frame with numeric dose
actual_df <- as.data.frame(actual_results$results_table)
if (nrow(actual_df) > 0) {
  actual_df <- actual_df %>%
    dplyr::mutate(
      Dose = dose_from_comparison(comparison)
    ) %>%
    dplyr::rename(
      Actual_T = statistic,
      Actual_P = p.value,
      Actual_Diff = estimate
    )
}

# Observed group means by dose from the raw data
group_means <- endpoint_data %>%
  dplyr::mutate(Dose = convert_dose(Dose)) %>%
  dplyr::filter(!is.na(Dose)) %>%
  dplyr::group_by(Dose) %>%
  dplyr::summarise(Actual_Mean = mean(Response, na.rm = TRUE), .groups = "drop")

# Prepare expected tables by metric
endpoint_expected <- endpoint_expected %>%
  dplyr::mutate(
    Dose = convert_dose(Dose),
    Expected_Value = suppressWarnings(as.numeric(gsub(",", ".", as.character(`expected result value`))))
  )

mean_expected <- endpoint_expected %>%
  dplyr::filter(grepl("Mean", `Brief description`, ignore.case = TRUE)) %>%
  dplyr::select(Dose, Expected_Mean = Expected_Value)

t_expected <- endpoint_expected %>%
  dplyr::filter(grepl("T-value|t-value", `Brief description`, ignore.case = TRUE) &
                  !grepl("p-value", `Brief description`, ignore.case = TRUE)) %>%
  dplyr::select(Dose, Expected_T = Expected_Value)

p_expected <- endpoint_expected %>%
  dplyr::filter(grepl("p-value", `Brief description`, ignore.case = TRUE)) %>%
  dplyr::select(Dose, Expected_P = Expected_Value)

# Join actuals to expected by Dose (robust against missing subsets)
mean_join <- mean_expected %>%
  dplyr::left_join(group_means, by = "Dose") %>%
  dplyr::mutate(
    Endpoint = endpoint,
    Mean_Diff = abs(Actual_Mean - Expected_Mean),
    Mean_Status = dplyr::case_when(
      is.na(Expected_Mean) | is.na(Actual_Mean) ~ "MISSING",
      Mean_Diff <= tolerance ~ "PASS",
      TRUE ~ "FAIL"
    )
  ) %>%
  dplyr::select(Endpoint, Dose, Actual_Mean, Expected_Mean, Mean_Status)

t_join <- t_expected %>%
  dplyr::left_join(actual_df %>% dplyr::select(Dose, Actual_T), by = "Dose") %>%
  dplyr::mutate(
    T_Diff = abs(Actual_T - Expected_T),
    T_Status = dplyr::case_when(
      is.na(Expected_T) | is.na(Actual_T) ~ "MISSING",
      T_Diff <= tolerance ~ "PASS",
      TRUE ~ "FAIL"
    )
  ) %>%
  dplyr::select(Dose, Actual_T, Expected_T, T_Status)

p_join <- p_expected %>%
  dplyr::left_join(actual_df %>% dplyr::select(Dose, Actual_P), by = "Dose") %>%
  dplyr::mutate(
    P_Diff = abs(Actual_P - Expected_P),
    P_Status = dplyr::case_when(
      is.na(Expected_P) | is.na(Actual_P) ~ "MISSING",
      P_Diff <= p_value_tolerance ~ "PASS",
      TRUE ~ "FAIL"
    )
  ) %>%
  dplyr::select(Dose, Actual_P, Expected_P, P_Status)

# Combine all metrics row-wise by Dose (full outer joins to keep everything)
comparison_df <- mean_join %>%
  dplyr::full_join(t_join, by = "Dose") %>%
  dplyr::full_join(p_join, by = "Dose")

# If Endpoint column got dropped (e.g., missing mean rows), re-add it
if (!"Endpoint" %in% names(comparison_df)) {
  comparison_df$Endpoint <- endpoint
  # Reorder to put Endpoint first
  comparison_df <- comparison_df %>%
    dplyr::select(Endpoint, dplyr::everything())
}

all_comparisons[[endpoint]] <- comparison_df

Notes and one question:

• For “Mean” in expected results: are those intended to be the raw sample means per dose from the dataset, or the model/least-squares means? I’ve used the observed sample means, which typically match the intercept (control) and sample means for the treatment groups in your case. If they are LSMeans, I’ll switch to emmeans to compute them consistently.
• “T-value (adjusted)” is unusual for Dunnett; typically the adjustment is in the p-value, not the t-statistic. If your expected results truly contain adjusted T-values, please confirm how they were derived, and I can add a branch to compute or map them accordingly. Otherwise, I’ll treat any “T-value (adjusted)” labels as ordinary t-values to compare against actual_df$statistic, or mark them MISSING if you prefer strict handling.

[Zhenglei-BCS]

.Here are the drop-in helpers to put near the top of your script. They are vectorized, robust to commas, scientific notation, blanks, and “n/a”.

• Replace your current convert_dose with this vectorized version
• Add convert_numeric for expected values
• Keep dose_from_comparison, but make it robust to NA and vector inputs

Code to add/replace:

convert_dose <- function(x) {
  if (length(x) == 0) return(numeric(0))
  xc <- as.character(x)
  xc <- trimws(xc)
  xc[xc %in% c("", "n/a", "NA")] <- NA_character_
  # Normalize decimal commas and keep scientific notation (e.g., "4,48E-2" -> "4.48E-2")
  xc <- gsub(",", ".", xc, fixed = TRUE)
  out <- suppressWarnings(as.numeric(xc))
  return(out)
}

convert_numeric <- function(x) {
  if (length(x) == 0) return(numeric(0))
  xc <- as.character(x)
  xc <- trimws(xc)
  xc[xc %in% c("", "n/a", "NA")] <- NA_character_
  xc <- gsub(",", ".", xc, fixed = TRUE)
  suppressWarnings(as.numeric(xc))
}

dose_from_comparison <- function(comp_vec) {
  if (length(comp_vec) == 0) return(numeric(0))
  vapply(comp_vec, function(s) {
    if (is.na(s)) return(NA_real_)
    parts <- strsplit(s, " - ", fixed = TRUE)[[1]]
    convert_dose(parts[1])
  }, FUN.VALUE = numeric(1))
}

With these in place, convert_dose(endpoint_data$Dose) will return a numeric vector, e.g., for your input it will produce:
0, 0, 0, 0, 0, 0, 0.0448, 0.0448, 0.0448, 0.0448, 0.132, 0.132, 0.132, 0.132, 0.390, 0.390, 0.390, 0.390, 1.15, 1.15, 1.15, 1.15, 3.39, 3.39, 3.39, 3.39, 10.0, 10.0, 10.0, 10.0

Now, adjust the places where you call convert_dose:

• When preparing study_data, do this (no sapply needed):
  study_data$Dose_numeric <- convert_dose(study_data$Dose)

• In the comparison-table block inside your for-loop, use the updated helpers and ensure you join against actual_results$results_table, not the list object. Here is the corrected block you can paste in to replace your current “Create comparison table” section:

# Exclude reference item groups (used only for two-sample tests)
endpoint_data <- endpoint_data[!grepl("reference", endpoint_data[["Test group"]], ignore.case = TRUE), ]

# Actual Dunnett outputs as a data frame with numeric Dose
actual_df <- as.data.frame(actual_results$results_table)
if (nrow(actual_df) > 0) {
  actual_df %>% mutate(Dose=dose_from_comparison(comparison)) %>% rename(Actual_T = statistic, Actual_P = p.value, Actual_Diff = estimate)
}

# Observed group means by dose from the raw endpoint data
group_means <- endpoint_data %>%
  dplyr::mutate(Dose = convert_dose(Dose)) %>%
  dplyr::filter(!is.na(Dose)) %>%
  dplyr::group_by(Dose) %>%
  dplyr::summarise(Actual_Mean = mean(Response, na.rm = TRUE), .groups = "drop")

# Prepare expected tables by metric (robust parsing)
endpoint_expected <- endpoint_expected %>%
  dplyr::mutate(
    Dose = convert_dose(Dose),
    Expected_Value = convert_numeric(`expected result value`)
  )

mean_expected <- endpoint_expected %>%
  dplyr::filter(grepl("Mean", `Brief description`, ignore.case = TRUE)) %>%
  dplyr::select(Dose, Expected_Mean = Expected_Value)

t_expected <- endpoint_expected %>%
  dplyr::filter(grepl("T-value|t-value", `Brief description`, ignore.case = TRUE) &
                  !grepl("p-value", `Brief description`, ignore.case = TRUE)) %>%
  dplyr::select(Dose, Expected_T = Expected_Value)

p_expected <- endpoint_expected %>%
  dplyr::filter(grepl("p-value", `Brief description`, ignore.case = TRUE)) %>%
  dplyr::select(Dose, Expected_P = Expected_Value)

# Join actuals to expected by Dose
mean_join <- mean_expected %>%
  dplyr::left_join(group_means, by = "Dose") %>%
  dplyr::mutate(
    Endpoint = endpoint,
    Mean_Diff = abs(Actual_Mean - Expected_Mean),
    Mean_Status = dplyr::case_when(
      is.na(Expected_Mean) | is.na(Actual_Mean) ~ "MISSING",
      Mean_Diff <= tolerance ~ "PASS",
      TRUE ~ "FAIL"
    )
  ) %>%
  dplyr::select(Endpoint, Dose, Actual_Mean, Expected_Mean, Mean_Status)

t_join <- t_expected %>%
  dplyr::left_join(actual_df %>% dplyr::select(Dose, Actual_T), by = "Dose") %>%
  dplyr::mutate(
    T_Diff = abs(Actual_T - Expected_T),
    T_Status = dplyr::case_when(
      is.na(Expected_T) | is.na(Actual_T) ~ "MISSING",
      T_Diff <= tolerance ~ "PASS",
      TRUE ~ "FAIL"
    )
  ) %>%
  dplyr::select(Dose, Actual_T, Expected_T, T_Status)

p_join <- p_expected %>%
  dplyr::left_join(actual_df %>% dplyr::select(Dose, Actual_P), by = "Dose") %>%
  dplyr::mutate(
    P_Diff = abs(Actual_P - Expected_P),
    P_Status = dplyr::case_when(
      is.na(Expected_P) | is.na(Actual_P) ~ "MISSING",
      P_Diff <= p_value_tolerance ~ "PASS",
      TRUE ~ "FAIL"
    )
  ) %>%
  dplyr::select(Dose, Actual_P, Expected_P, P_Status)

# Combine all metrics row-wise by Dose
comparison_df <- mean_join %>%
  dplyr::full_join(t_join, by = "Dose") %>%
  dplyr::full_join(p_join, by = "Dose")

# Ensure Endpoint column exists
if (!"Endpoint" %in% names(comparison_df)) {
  comparison_df$Endpoint <- endpoint
  comparison_df <- dplyr::select(comparison_df, Endpoint, dplyr::everything())
}

all_comparisons[[endpoint]] <- comparison_df

Two clarifications I’d like from you to ensure perfect alignment:

• Should “Mean” in the expected results refer to raw sample means per dose, or LSMeans from a model? I’m currently comparing to raw group means. If LSMeans are intended, I can compute them with emmeans and use that for Expected vs Actual.
• For entries labeled “T-value (adjusted)” in your expected results: do you want me to compare them to the unadjusted t-statistic from the Dunnett test (actual_df$Actual_T), or treat these as non-comparable and mark them “MISSING”? The standard Dunnett implements adjustment in p-values, not t-statistics, so I want to follow your intent.

[Zhenglei-BCS]

• Keeps my existing wide joins (Mean/T/P) and then builds a long table with a metric column: Mean, T-value, P-value.
• Adds Study ID, Test organism, and alternative for traceability.
• Preserves rows like Dose = 0 for Mean while leaving T/P as NA.
• Works even if some metrics are missing (keeps them with Status = "MISSING").

Patch to replace the tail of your comparison block

• Replace your current lines from “Combine all metrics row-wise by Dose” through storing all_comparisons[[endpoint]] with the code below.

# Combine all metrics row-wise by Dose (wide)
wide_df <- mean_join %>%
  dplyr::full_join(t_join, by = "Dose") %>%
  dplyr::full_join(p_join, by = "Dose")

# Ensure Endpoint column exists and is first
if (!"Endpoint" %in% names(wide_df)) {
  wide_df$Endpoint <- endpoint
}
wide_df <- dplyr::select(wide_df, Endpoint, dplyr::everything())

# Build long format with metric column
mean_long <- wide_df %>%
  dplyr::transmute(
    Endpoint,
    Dose,
    metric = "Mean",
    Actual  = Actual_Mean,
    Expected = Expected_Mean,
    Status  = Mean_Status
  )

t_long <- wide_df %>%
  dplyr::transmute(
    Endpoint,
    Dose,
    metric = "T-value",
    Actual  = Actual_T,
    Expected = Expected_T,
    Status  = T_Status
  )

p_long <- wide_df %>%
  dplyr::transmute(
    Endpoint,
    Dose,
    metric = "P-value",
    Actual  = Actual_P,
    Expected = Expected_P,
    Status  = P_Status
  )

comparison_long <- dplyr::bind_rows(mean_long, t_long, p_long)

# Add metadata: Study ID, Test organism, alternative
test_org <- NA_character_
if ("Test organism" %in% names(endpoint_data)) {
  u_to <- unique(endpoint_data[["Test organism"]])
  u_to <- u_to[!is.na(u_to)]
  if (length(u_to) > 0) test_org <- u_to[1]
}
comparison_long <- comparison_long %>%
  dplyr::mutate(
    `Study ID` = study_id,
    `Test organism` = test_org,
    alternative = alternative
  ) %>%
  dplyr::select(`Test organism`, `Study ID`, alternative, Endpoint, Dose, metric,
                Actual, Expected, Status) %>%
  dplyr::arrange(Endpoint, Dose, factor(metric, levels = c("Mean", "T-value", "P-value")))

# Store for this endpoint
all_comparisons[[endpoint]] <- comparison_long

If you want the entire run_consolidated_dunnett_validation to summarize using the new long format, update the combining/summarizing part after the loop as follows:

if (length(all_comparisons) > 0) {
  combined_table <- dplyr::bind_rows(all_comparisons)

  # Count only comparable entries (Status PASS/FAIL)
  total_validations <- sum(combined_table$Status %in% c("PASS", "FAIL"), na.rm = TRUE)
  total_passed <- sum(combined_table$Status == "PASS", na.rm = TRUE)
  overall_passed <- !any(combined_table$Status == "FAIL", na.rm = TRUE)

  return(list(
    passed = overall_passed,
    endpoints_tested = available_endpoints,
    validation_results = combined_table,
    n_comparisons = total_validations,
    n_passed = total_passed
  ))
} else {
  return(list(
    passed = FALSE,
    error = "No valid comparisons could be made",
    endpoints_tested = available_endpoints,
    validation_results = NULL,
    n_comparisons = 0,
    n_passed = 0
  ))
}

Notes

• The “Mean” rows include Dose = 0 if present in expected results; T-value and P-value rows for Dose = 0 remain NA (as in your example).
• If you later decide to include more metrics (e.g., “Estimate/Diff”), you can mirror the pattern by creating an estimate_long block and binding it alongside the others.
• Your display code can now style the single Status column instead of Mean_Status/T_Status/P_Status.

[Zhenglei-BCS]

Now I have the correct outcome:

>   result <- run_consolidated_dunnett_validation(fg$study, fg$id, alternative = "less")
smaller check point
Fitting linear model with homoscedastic errors
> result
$passed
[1] TRUE

$endpoints_tested
[1] "Growth Rate"

$validation_results
# A tibble: 21 × 9
   `Test organism` `Study ID` alternative Endpoint      Dose metric       Actual    Expected Status
   <chr>           <chr>      <chr>       <chr>        <dbl> <chr>         <dbl>       <dbl> <chr> 
 1 Myriophyllum    MOCK0065   less        Growth Rate 0      Mean     0.126       0.126      PASS  
 2 Myriophyllum    MOCK0065   less        Growth Rate 0      T-value NA          NA          NA    
 3 Myriophyllum    MOCK0065   less        Growth Rate 0      P-value NA          NA          NA    
 4 Myriophyllum    MOCK0065   less        Growth Rate 0.0448 Mean     0.124       0.124      PASS  
 5 Myriophyllum    MOCK0065   less        Growth Rate 0.0448 T-value -0.672      -0.672      PASS  
 6 Myriophyllum    MOCK0065   less        Growth Rate 0.0448 P-value  0.648       0.648      PASS  
 7 Myriophyllum    MOCK0065   less        Growth Rate 0.132  Mean     0.0999      0.0999     PASS  
 8 Myriophyllum    MOCK0065   less        Growth Rate 0.132  T-value -6.64       -6.64       PASS  
 9 Myriophyllum    MOCK0065   less        Growth Rate 0.132  P-value  0.00000275  0.00000103 PASS  
10 Myriophyllum    MOCK0065   less        Growth Rate 0.39   Mean     0.0721      0.0721     PASS  
# ℹ 11 more rows
# ℹ Use `print(n = ...)` to see more rows

$n_comparisons
[1] 19

$n_passed
[1] 19

> 

[Zhenglei-BCS]

To hide rows where both Actual and Expected are NA in the printed outcome table, you can filter at display time so your internal counts and logic remain intact.

Use this in your validation printing chunk, replacing the place where you build styled_table:

# Filter out rows where both Actual and Expected are NA
display_df <- result$validation_results %>%
  dplyr::filter(!(is.na(Actual) & is.na(Expected)))

if (!is.null(display_df) && nrow(display_df) > 0) {
  styled_table <- kable(display_df, "html",
                        caption = paste("Validation Details for", fg$id),
                        digits = 6) %>%
    kable_styling(bootstrap_options = c("striped", "hover", "condensed", "responsive")) %>%
    column_spec(which(colnames(display_df) == "Status"),
                color = "white",
                background = ifelse(display_df$Status == "FAIL", "red",
                                    ifelse(display_df$Status == "PASS", "green", "orange")))
  print(styled_table)
  cat("\n")
} else {
  cat("No comparable results to display (both Actual and Expected were NA).\n\n")
}

If you prefer to filter inside the validation function so all downstream consumers get the pre-filtered table, add one line right before returning the list:

combined_table <- dplyr::bind_rows(all_comparisons)

# Drop rows where both Actual and Expected are NA
combined_table <- combined_table %>%
  dplyr::filter(!(is.na(Actual) & is.na(Expected)))

The summary counts you already compute will continue to ignore MISSING rows (they don’t count toward PASS/FAIL), and this filter only affects presentation by removing rows that have no comparable values at all.

[Zhenglei-BCS]

You’re running into two issues at once:

• You switched to the long format (metric, Actual, Expected, Status), but the styling code still references wide-only columns (Mean_Status, T_Status, P_Status).
• column_spec styles entire columns (one color per column). For cell-by-cell coloring based on Status, use cell_spec (or use row_spec to color entire rows).

Below are two working options. Both assume your result$validation_results is in long format with columns: Test organism, Study ID, alternative, Endpoint, Dose, metric, Actual, Expected, Status.

Option A: color only the Status cell (cell_spec)

• This keeps the rest of the row uncolored and highlights Status based on PASS/FAIL/MISSING.

display_df <- result$validation_results %>%
  dplyr::filter(!(is.na(Actual) & is.na(Expected)))

if (!is.null(display_df) && nrow(display_df) > 0) {
  # Order metrics consistently
  display_df <- display_df %>%
    dplyr::mutate(metric = factor(metric, levels = c("Mean", "T-value", "P-value")))

  # Style Status cells
  display_df_styled <- display_df %>%
    dplyr::mutate(
      Status = tidyr::replace_na(Status, "MISSING"),
      Status = kableExtra::cell_spec(
        Status,
        color = "white",
        background = dplyr::case_when(
          Status == "FAIL" ~ "#dc3545",   # red
          Status == "PASS" ~ "#28a745",   # green
          TRUE             ~ "#fd7e14"    # orange for MISSING/others
        )
      )
    )

  styled_table <- knitr::kable(
      display_df_styled,
      format = "html",
      caption = paste("Validation Details for", fg$id),
      digits = 6,
      escape = FALSE  # allow HTML from cell_spec
    ) %>%
    kableExtra::kable_styling(bootstrap_options = c("striped", "hover", "condensed", "responsive"))

  print(styled_table)
  cat("\n")
} else {
  cat("No comparable results to display (both Actual and Expected were NA).\n\n")
}

Option B: color entire rows by Status (row_spec)

• This gives you a quick “heatmap” by row.

display_df <- result$validation_results %>%
  dplyr::filter(!(is.na(Actual) & is.na(Expected)))

if (!is.null(display_df) && nrow(display_df) > 0) {
  display_df <- display_df %>%
    dplyr::mutate(metric = factor(metric, levels = c("Mean", "T-value", "P-value")))

  kb <- knitr::kable(
    display_df,
    "html",
    caption = paste("Validation Details for", fg$id),
    digits = 6
  ) %>%
    kableExtra::kable_styling(bootstrap_options = c("striped", "hover", "condensed", "responsive"))

  # Apply row background colors
  kb <- kb %>%
    kableExtra::row_spec(which(display_df$Status == "PASS"), background = "#d4edda") %>%  # light green
    kableExtra::row_spec(which(display_df$Status == "FAIL"), background = "#f8d7da") %>%  # light red
    kableExtra::row_spec(which(display_df$Status == "MISSING" | is.na(display_df$Status)), background = "#fff3cd")  # light orange

  print(kb)
  cat("\n")
} else {
  cat("No comparable results to display (both Actual and Expected were NA).\n\n")
}

Why your previous code failed

• It mixed display_df with result$validation_results inside column_spec and ifelse, leading to length mismatches and missing columns.
• In long format, Mean_Status, T_Status, P_Status no longer exist. You only have Status.
• column_spec colors the whole column and is not intended for per-row logic; use cell_spec (Option A) or row_spec (Option B).

Tip

• If you ever keep the wide table for internal use and only pivot to long for display, make sure the styling code uses only the data frame you pass to kable (don’t reference columns from a different object).
• Keep escape = FALSE when using cell_spec so the HTML styling is rendered.

[Zhenglei-BCS]

To order dose levels in the display_df

You can order the doses cleanly right before you build the styled table. Since Dose is already numeric, a simple arrange will work; if you want the order to persist (e.g., for grouped displays or later pivots), also set Dose as an ordered factor with ascending levels.

Add this right after you create display_df and before kable:

display_df <- result$validation_results %>%
  dplyr::filter(!(is.na(Actual) & is.na(Expected))) %>%
  dplyr::mutate(
    metric = factor(metric, levels = c("Mean", "T-value", "P-value"))
  )

# Option 1: just sort rows (keeps Dose numeric)
display_df <- display_df %>%
  dplyr::arrange(`Test organism`, `Study ID`, Endpoint, Dose, metric)

# Option 2: also lock Dose into ordered factor levels (useful if any later code reorders rows)
dose_levels <- sort(unique(display_df$Dose))
display_df <- display_df %>%
  dplyr::mutate(Dose = factor(Dose, levels = dose_levels)) %>%
  dplyr::arrange(`Test organism`, `Study ID`, Endpoint, Dose, metric)

If you prefer to display Dose with consistent formatting (e.g., avoid scientific notation, keep 0.0448 as-is), add a display column and use that in the table:

display_df <- display_df %>%
  dplyr::mutate(
    Dose_display = formatC(as.numeric(as.character(Dose)), format = "fg", digits = 6)
  ) %>%
  dplyr::relocate(Dose_display, .after = Endpoint)

Then pass Dose_display to kable instead of Dose (or select it into the output columns):

styled_df <- display_df %>%
  dplyr::select(`Test organism`, `Study ID`, alternative, Endpoint, Dose = Dose_display,
                metric, Actual, Expected, Status)

Finally, apply your preferred styling approach (cell_spec or row_spec) to styled_df as shown earlier.

[Zhenglei-BCS]

You can derive the Dunnett function groups and their available alternatives directly from test_cases_res, and then iterate over those alternatives in your validation. Below is a self-contained setup that:

• Scans test_cases_res for “Dunnett” entries.
• Builds a list of {Function group ID, Study ID, name, alternatives} objects.
• Optionally uses Test organism and Endpoint to form a readable name.
• Runs validation for each alternative found (“smaller”, “greater”, “two-sided”).
• Adds an alternative column to your summary and display output.

Helpers to normalize alternatives and build the FG list

normalize_alternative <- function(alternative) {
  a <- tolower(trimws(alternative))
  a_norm <- gsub("[\\s._-]", "", a)
  if (a_norm %in% c("less", "smaller", "lower")) {
    "smaller"
  } else if (a_norm %in% c("greater", "larger", "higher", "more")) {
    "greater"
  } else if (a_norm %in% c("twosided", "twoside", "twosides", "two.sided", "two-sided", "two_sided")) {
    "two-sided"
  } else {
    NA_character_
  }
}

detect_alternative_in_text <- function(x) {
  x <- tolower(x)
  if (grepl("\\bsmaller\\b", x)) {
    "smaller"
  } else if (grepl("\\bgreater\\b", x)) {
    "greater"
  } else if (grepl("two[-\\.]?sided", x)) {
    "two-sided"
  } else {
    NA_character_
  }
}

build_dunnett_fgs <- function(test_cases_res, test_cases_data) {
  # Only keep Dunnett rows
  res_dunnett <- test_cases_res[grepl("Dunnett", test_cases_res[["Brief description"]], ignore.case = TRUE), ]
  if (nrow(res_dunnett) == 0) return(list())

  # Detect alternative per row
  res_dunnett$alt <- vapply(res_dunnett[["Brief description"]], detect_alternative_in_text, FUN.VALUE = character(1))

  # Keep rows with recognized alternatives
  res_dunnett <- res_dunnett[!is.na(res_dunnett$alt), ]

  # Restrict to studies that exist in data
  studies_in_data <- unique(test_cases_data[["Study ID"]])
  res_dunnett <- res_dunnett[res_dunnett[["Study ID"]] %in% studies_in_data, ]

  # Build per (Function group ID, Study ID) object
  # Choose a readable name: Test organism + Endpoint + " - Dunnett's Test"
  # If multiple endpoints, name will reflect the first endpoint; you can customize as needed.
  by_fg_study <- split(res_dunnett, list(res_dunnett[["Function group ID"]], res_dunnett[["Study ID"]]), drop = TRUE)

  out <- lapply(by_fg_study, function(df) {
    fg_id <- unique(df[["Function group ID"]])
    study <- unique(df[["Study ID"]])
    test_org <- na.omit(unique(df[["Test organism"]]))
    endpoint <- na.omit(unique(df[["Endpoint"]]))
    alts <- sort(unique(df[["alt"]]))
    # Build a readable name
    org_part <- if (length(test_org) > 0) test_org[1] else "Unknown organism"
    ep_part <- if (length(endpoint) > 0) endpoint[1] else "Unknown endpoint"
    name <- paste(org_part, ep_part, "- Dunnett's Test")
    list(
      id = fg_id[1],
      name = name,
      study = study[1],
      alternatives = alts
    )
  })

  # Ensure list structure is flat
  # Split produced named list; unname it
  unname(out)
}

How to use it

• Build the Dunnett FG list:

dunnett_fgs <- build_dunnett_fgs(test_cases_res, test_cases_data)

- Iterate over each FG and each of its alternatives. Adjust your summary table to include the alternative column.

Initialize summary_results with an Alternative column:
summary_results <- data.frame(
  Function_Group = character(),
  Study = character(),
  Alternative = character(),
  Endpoints_Tested = character(),
  Total_Validations = integer(),
  Passed_Validations = integer(),
  Success_Rate = character(),
  Overall_Status = character(),
  stringsAsFactors = FALSE
)

Loop:

for (fg in dunnett_fgs) {
  if (length(fg$alternatives) == 0) next  # no recognized alternatives
  for (alt in fg$alternatives) {
    cat("\n### ", fg$name, " (", fg$id, ") — alternative: ", alt, "\n\n", sep = "")
    result <- run_consolidated_dunnett_validation(fg$study, fg$id, alternative = alt)

    endpoints_str <- paste(result$endpoints_tested, collapse = ", ")

    if (!is.null(result$error)) {
      cat("Error: ", result$error, "\n\n", sep = "")
      status <- "❌ ERROR"
      success_rate_str <- "0%"
      total_validations <- 0
      passed_validations <- 0
      display_df <- NULL
    } else {
      status <- ifelse(result$passed, "✅ PASSED", "❌ FAILED")
      total_validations <- result$n_comparisons
      passed_validations <- result$n_passed
      success_rate <- ifelse(total_validations > 0, round(100 * passed_validations / total_validations, 1), 0)
      success_rate_str <- paste0(success_rate, "%")

      cat("Endpoints Tested: ", endpoints_str, "\n", sep = "")
      cat("Total Validations: ", total_validations, "\n", sep = "")
      cat("Passed Validations: ", passed_validations, "\n", sep = "")
      cat("Success Rate: ", success_rate_str, "\n", sep = "")
      cat("Overall Status: ", status, "\n\n", sep = "")

      # Filter rows where both Actual and Expected are NA and order doses and metrics
      display_df <- result$validation_results
      if (!is.null(display_df) && nrow(display_df) > 0) {
        display_df <- display_df %>%
          dplyr::filter(!(is.na(Actual) & is.na(Expected))) %>%
          dplyr::mutate(metric = factor(metric, levels = c("Mean", "T-value", "P-value"))) %>%
          dplyr::arrange(`Test organism`, `Study ID`, Endpoint, Dose, metric)
      }
    }

    if (!is.null(display_df) && nrow(display_df) > 0) {
      # Color rows by Status
      kb <- knitr::kable(
        display_df,
        "html",
        caption = paste("Validation Details for", fg$id, "—", alt),
        digits = 6
      ) %>%
        kableExtra::kable_styling(bootstrap_options = c("striped", "hover", "condensed", "responsive")) %>%
        kableExtra::row_spec(which(display_df$Status == "PASS"), background = "#d4edda") %>%
        kableExtra::row_spec(which(display_df$Status == "FAIL"), background = "#f8d7da") %>%
        kableExtra::row_spec(which(display_df$Status == "MISSING" | is.na(display_df$Status)), background = "#fff3cd")
      print(kb)
      cat("\n")
    } else {
      cat("No comparable results to display (both Actual and Expected were NA).\n\n")
    }

    summary_results <- rbind(summary_results, data.frame(
      Function_Group = fg$id,
      Study = fg$study,
      Alternative = alt,
      Endpoints_Tested = endpoints_str,
      Total_Validations = total_validations,
      Passed_Validations = passed_validations,
      Success_Rate = success_rate_str,
      Overall_Status = status,
      stringsAsFactors = FALSE
    ))

    cat("\n---\n\n")
  }
}

Finally, your overall summary section should consider the Alternative dimension:

total_validations <- sum(summary_results$Total_Validations)
total_passed <- sum(summary_results$Passed_Validations)
overall_success_rate <- ifelse(total_validations > 0, round(100 * total_passed / total_validations, 1), 0)

• You can present summary_results with the Alternative column and style rows similarly to your existing approach.

Notes and options

• If you want to include alternatives even if they’re not present in test_cases_res (e.g., force running “greater” and “two-sided” for every FG), replace fg$alternatives with c("smaller", "greater", "two-sided"), or merge them: unique(c(fg$alternatives, "smaller", "greater", "two-sided")).
• Names: If you prefer the earlier curated names you wrote by hand, you can override the auto-generated name in build_dunnett_fgs by e.g., a small mapping data frame keyed by Function group ID.
• Endpoints: The validation function already loops endpoints per study; the FG object doesn’t need to carry endpoints. If you want to display endpoints in the header, you can read them from expected_results_all within the function or add unique endpoints per FG pair in build_dunnett_fgs similarly to alternatives.

[Zhenglei-BCS]

Now the test plan:

Below is a concise test plan you can include in your report and repo to document how Dunnett test validation is defined, discovered, executed, and reported.

Test Plan: Dunnett’s Multiple Comparison Validation

Objective

• Validate the drcHelper::dunnett_test implementation against reference “expected results” across all relevant Function Group IDs (FGs), studies, endpoints, and alternatives.
• Produce a consolidated, readable report highlighting exact agreement, discrepancies, and data quality issues.

Scope

• Test type: Dunnett’s many-to-one comparisons.
• Data sources: test_cases_data and test_cases_res provided in the drcHelper package.
• Studies and FGs: All combinations that contain “Dunnett” entries in test_cases_res and have matching data in test_cases_data.
• Alternatives: smaller (less), greater, and two-sided, as present in test_cases_res (or forced to run if configured).

Test Case Discovery and Definition

• Automatically derive the set of test cases (dunnett_fgs) by scanning test_cases_res for rows with “Dunnett” in Brief description and a recognizable alternative (“smaller”, “greater”, “two-sided”).
• Each test case is defined by:
  • Function group ID (FG)
  • Study ID
  • Readable name (built from Test organism and Endpoint, or curated if you prefer)
  • Alternatives to validate (e.g., c("smaller", "greater", "two-sided"))
• This discovers entries like:
  • FG00220 / MOCK0065 / “Myriophyllum Growth Rate - Dunnett’s Test”
  • FG00221 / MOCK08/15-001 / “Aphidius rhopalosiphi Reproduction - Dunnett’s Test”
  • FG00222 / MOCK08/15-001 / “Aphidius rhopalosiphi Repellency - Dunnett’s Test”
  • FG00225 / MOCKSE21/001-1 / “BRSOL Plant height - Dunnett’s Test”
• If you prefer a curated list, you can supply it directly and/or merge it with the discovered set.

Preprocessing and Normalization

• Dose normalization:
  • convert_dose: vectorized parsing of Dose strings with decimal commas and scientific notation, yielding numeric doses.
  • dose_from_comparison: extract the treatment dose from comparison strings like “0.0448 - 0”.
• Expected value normalization:
  • convert_numeric: parse “expected result value” as numeric, tolerant to commas and scientific notation.
• Column name inconsistencies:
  • Measurement variable mismatch (e.g., “Measurement varibale” in res vs “Measurement Variable” in data) is tolerated; validation does not depend on this field outside specific Myriophyllum cases.
• Group filtering:
  • Exclude “Reference item” test groups from Dunnett validation (they are used for two-sample tests).
• Alternative normalization:
  • normalize_alternative maps user choices and text in Brief description to standardized alternatives: “smaller”, “greater”, “two-sided”.

Validation Methodology

• For each FG/study/alternative:
  • Filter expected results to rows that match the FG ID, Study ID, and the chosen alternative (keyword detected in Brief description).
  • Split validation by endpoint. Each endpoint is validated independently.
  • Compute actual Dunnett results with drcHelper::dunnett_test using:
    • response_var = "Response"
    • dose_var = numeric dose (converted from data)
    • include_random_effect = FALSE (unless specified)
    • alternative = normalized choice (“smaller”, “greater”, “two-sided”)
  • Build a per-endpoint comparison table:
    • Means: raw sample means per dose from endpoint_data (grouped by numeric Dose).
    • T-values: from actual_results$results_table$statistic.
    • P-values: adjusted single-step Dunnett p-values from actual_results$results_table$p.value.
  • Join expected vs actual by numeric Dose for each metric.
  • Status rules per metric:
    • PASS if absolute difference ≤ tolerance (for Mean/T) or ≤ p_value_tolerance (for P).
    • MISSING if either side is NA; not counted for pass/fail rates.
    • FAIL otherwise.

Tolerances

• Numeric tolerance (Means and T-values): default 1e-6.
• P-value tolerance: default 1e-4.
• Tolerances are configurable at the top of the validation function.

Outputs

• Detailed per-endpoint long-format table with the following columns:
  • Test organism
  • Study ID
  • alternative (“smaller”, “greater”, “two-sided”)
  • Endpoint
  • Dose (numeric; ordered ascending for display)
  • metric (one of: Mean, T-value, P-value)
  • Actual (numeric; derived from data or dunnett_test)
  • Expected (numeric; from test_cases_res)
  • Status (PASS, FAIL, MISSING)
• Report display rules:
  • Filter out rows where both Actual and Expected are NA (not informative).
  • Order rows by Test organism, Study ID, Endpoint, Dose, then metric (Mean, T-value, P-value).
  • Row-level coloring:
    • PASS: light green
    • FAIL: light red
    • MISSING: light orange
• Consolidated summary across all FGs and alternatives:
  • Function_Group
  • Study
  • Alternative
  • Endpoints_Tested
  • Total_Validations (count of comparable metric rows)
  • Passed_Validations
  • Success_Rate (%)
  • Overall_Status (✅ PASSED if no FAIL; ❌ FAILED otherwise; ❌ ERROR if inputs missing)

Pass/Fail Criteria

• A test case (FG/study/alternative) is marked overall PASSED if there are no FAIL statuses in its comparable rows. MISSING does not count against pass/fail.
• The overall report success rate is computed over all comparable metric rows across all test cases.

Handling Missing or Invalid Data

• Reference item rows are excluded from Dunnett validation.
• Non-numeric doses (“n/a”, blanks) are converted to NA and removed from numeric joins.
• Expected values that are blank or malformed are converted to NA; comparisons with NA are marked MISSING.
• If a study/FG/alternative has no expected rows, the case returns ERROR and is reported.
• “T-value (adjusted)” labels in expected results are treated as standard t-statistics unless a clear definition for adjusted T is provided; otherwise such rows can be flagged MISSING by configuration.

Multi-Endpoint Support

• Endpoints are discovered from expected results per FG/study and processed independently.
• The report displays combined results across endpoints for each FG/study/alternative.

Alternatives

• Alternatives are detected from Brief description text and normalized to “smaller”, “greater”, “two-sided”.
• The validation loop runs all alternatives present for each FG. Optionally, you can force running all three alternatives for each FG regardless of the reference coverage.

Dose Ordering and Display

• Dose is converted to numeric and sorted ascending for display.
• Optional: a Dose_display string can be added for consistent formatting (e.g., fixed significant digits) if desired.

Assumptions and Limitations

• “Mean” expected values are interpreted as raw per-dose sample means; if LSMeans are intended, the method can be switched to compute model-based means using emmeans.
• Dunnett adjustment applies to p-values; t-statistics are typically unadjusted. Any “adjusted T-value” labels require clarification for strict validation.
• Measurement variable mismatches across data/res are tolerated and do not affect validations (except in Myriophyllum-specific workflows if enabled).
• Failures often reflect data quality issues in test_cases_res; the framework highlights these clearly.

Reproducibility

• Report is generated from a single R Markdown file with:
  • Core function chunk (validation logic and helpers)
  • Discovery chunk (build_dunnett_fgs)
  • Execution chunk (per-FG, per-alternative loops)
  • Summary chunk (consolidated table and metrics)
• Dependencies: drcHelper, dplyr, knitr, kableExtra.
• Timestamp and environment details can be included for auditability.

Maintenance and Extension

• Add new FGs/studies by updating test_cases_res; discovery will pick them up automatically.
• Extend to Williams’ test or two-sample tests by adding parallel validation functions and discovery rules keyed on Brief description.
• Configure tolerances per metric at the top of the validation function as needed.

Expected Deliverables

• An HTML report with:
  • Per-FG, per-alternative sections containing detailed long-format comparison tables.
  • A consolidated summary table across all FGs and alternatives with pass/fail and success rates.
• Optional CSV exports of the long-format validation_results for downstream analysis.

If you want, I can assemble a short “Test Plan” section in your R Markdown (as prose) and add a “Configuration” chunk that lists the current tolerances, the discovery rule for dunnett_fgs, and the applied filters, so the report is fully self-documenting.

[Zhenglei-BCS]

Excellent. We can generalize your Dunnett framework into a modular validation system that supports multiple test types, each with its own discovery, execution, and metric comparison logic. I’ll provide:

• A pluggable design (registry) you can extend per test type
• Generic discovery and validation runners that reuse your Dunnett helpers
• Metric parsing and comparison scaffolding
• Concrete code stubs for Williams, two-sample tests, and Shapiro tests
• A short list of R commands I need you to run so I can tailor handlers to your data structure precisely

Design overview

• Registry-based: each test type registers a discovery function (build FGs), an execution function (compute actual results), and its metrics.
• Generic validator: orchestrates expected-vs-actual joins, long-format output, and pass/fail computation.
• Shared helpers: reuse convert_dose, convert_numeric, dose_from_comparison, normalize_alternative, and your long-format builder.
• Metric-specific tolerances: allow per-test, per-metric tolerances.

Code: Registry and generic harness

Put these after your existing helpers. Where noted, you can wire in your existing Dunnett functions.

# Test-type registry: add entries as you implement handlers
test_registry <- list(
  dunnett = list(
    keyword = "Dunnett",
    discover_fgs = function(res, data) build_dunnett_fgs(res, data),  # reuse your function
    run_actual = function(endpoint_data, alternative) {
      # Call your dunnett_test and return actual_df + group_means
      out <- drcHelper::dunnett_test(
        data = endpoint_data,
        response_var = "Response",
        dose_var = "Dose_numeric",
        include_random_effect = FALSE,
        alternative = alternative
      )
      actual_df <- as.data.frame(out$results_table)
      actual_df$Dose <- dose_from_comparison(as.character(actual_df$comparison))
      names(actual_df)[names(actual_df) == "statistic"] <- "Actual_T"
      names(actual_df)[names(actual_df) == "p.value"]  <- "Actual_P"
      names(actual_df)[names(actual_df) == "estimate"] <- "Actual_Diff"
      rownames(actual_df) <- NULL

      group_means <- endpoint_data %>%
        dplyr::mutate(Dose = convert_dose(Dose)) %>%
        dplyr::filter(!is.na(Dose)) %>%
        dplyr::group_by(Dose) %>%
        dplyr::summarise(Actual_Mean = mean(Response, na.rm = TRUE), .groups = "drop")

      list(actual_df = actual_df, group_means = group_means)
    },
    metrics = c("Mean", "T-value", "P-value"),
    comparator = "many_to_one"
  ),

  williams = list(
    keyword = "Williams",
    discover_fgs = function(res, data) build_generic_fgs(res, data, pattern = "Williams"),
    run_actual = function(endpoint_data, alternative) {
      # Placeholder: typically use tukeytrend package for Williams
      # Return a list(actual_df, group_means) similar to Dunnett
      stop("Williams handler not yet implemented; requires tukeytrend or custom contrasts.")
    },
    metrics = c("Mean", "T-value", "P-value"),
    comparator = "trend"
  ),

  twosample = list(
    keyword = "(two-?sample|t-?test|Welch|Mann[- ]?Whitney)",
    discover_fgs = function(res, data) build_generic_fgs(res, data, pattern = "(two-?sample|t-?test|Welch|Mann[- ]?Whitney)"),
    run_actual = function(endpoint_data, alternative) {
      # Compute per-dose or per-group two-sample test; we’ll tailor once we see your structure
      stop("Two-sample handler not yet implemented; needs group pairing rules.")
    },
    metrics = c("Mean", "T-value", "P-value"),
    comparator = "two_sample"
  ),

  shapiro = list(
    keyword = "Shapiro",
    discover_fgs = function(res, data) build_generic_fgs(res, data, pattern = "Shapiro"),
    run_actual = function(endpoint_data, alternative) {
      # Compute Shapiro-Wilk W and p-value across a specified vector (raw group or residuals)
      stop("Shapiro handler not yet implemented; needs target vector definition.")
    },
    metrics = c("W-value", "P-value"),
    comparator = "per_group"
  )
)

# Generic FG discovery for non-Dunnett test types
build_generic_fgs <- function(test_cases_res, test_cases_data, pattern) {
  res_sub <- test_cases_res[grepl(pattern, test_cases_res[["Brief description"]], ignore.case = TRUE), ]
  if (nrow(res_sub) == 0) return(list())

  # Detect alternative from text when present
  alt_detect <- function(desc) {
    desc <- tolower(desc)
    if (grepl("\\bsmaller\\b", desc)) "smaller"
    else if (grepl("\\bgreater\\b", desc)) "greater"
    else if (grepl("two[-\\.]?sided", desc)) "two-sided"
    else NA_character_
  }
  res_sub$alt <- vapply(res_sub[["Brief description"]], alt_detect, FUN.VALUE = character(1))

  # Restrict to studies with data
  studies_in_data <- unique(test_cases_data[["Study ID"]])
  res_sub <- res_sub[res_sub[["Study ID"]] %in% studies_in_data, ]

  by_pair <- split(res_sub, list(res_sub[["Function group ID"]], res_sub[["Study ID"]]), drop = TRUE)
  out <- lapply(by_pair, function(df) {
    fg_id <- unique(df[["Function group ID"]])
    study <- unique(df[["Study ID"]])
    org <- na.omit(unique(df[["Test organism"]]))
    ep  <- na.omit(unique(df[["Endpoint"]]))
    alts <- sort(na.omit(unique(df[["alt"]])))
    name <- paste(ifelse(length(org) > 0, org[1], "Unknown"),
                  ifelse(length(ep) > 0, ep[1], "Unknown"),
                  "-",
                  gsub("^.+?\\b(", "", pattern), "Test")
    list(id = fg_id[1], name = name, study = study[1], alternatives = if (length(alts) > 0) alts else c("smaller", "greater", "two-sided"))
  })
  unname(out)
}

Generic validator runner

This runs a given test type against one FG/study/alternative. It reuses your long-format builder and metadata fields.

run_consolidated_validation_generic <- function(test_type, study_id, fg_id, alternative) {
  reg <- test_registry[[test_type]]
  if (is.null(reg)) {
    return(list(passed = FALSE, error = paste("Unknown test_type:", test_type)))
  }

  # Expected subset for this FG/study/test_type/alternative
  expected_all <- test_cases_res[
    test_cases_res[['Study ID']] == study_id &
    test_cases_res[['Function group ID']] == fg_id &
    grepl(reg$keyword, test_cases_res[['Brief description']], ignore.case = TRUE),
  ]

  if (nrow(expected_all) == 0) {
    return(list(passed = FALSE, error = paste("No expected results for", test_type, "FG:", fg_id, "study:", study_id)))
  }

  # Normalize alternative and filter expected
  alt_norm <- normalize_alternative(alternative)
  if (is.list(alt_norm)) alt_norm <- alt_norm$keyword
  expected <- expected_all[grepl(alt_norm, expected_all[['Brief description']], ignore.case = TRUE), ]

  available_endpoints <- unique(expected[['Endpoint']])

  # Data for this study
  study_data <- test_cases_data[test_cases_data[['Study ID']] == study_id, ]
  if (nrow(study_data) == 0) {
    return(list(passed = FALSE, error = paste("No test data for study:", study_id)))
  }
  study_data$Dose_numeric <- convert_dose(study_data$Dose)

  # Endpoint loop
  all_ep <- list()
  for (endpoint in available_endpoints) {
    endpoint_data <- study_data[study_data[['Endpoint']] == endpoint, ]
    # Exclude reference item for many-to-one tests
    if (reg$comparator %in% c("many_to_one", "trend")) {
      endpoint_data <- endpoint_data[!grepl("reference", endpoint_data[["Test group"]], ignore.case = TRUE), ]
    }

    # Run actual computation (per test type handler)
    actual <- tryCatch(reg$run_actual(endpoint_data, alternative = alt_norm), error = function(e) NULL)
    if (is.null(actual)) next

    # Parse expected by metric
    exp_tbl <- expected %>%
      dplyr::mutate(
        Dose = convert_dose(Dose),
        Expected_Value = convert_numeric(`expected result value`)
      )

    # Metric splits (generic patterns; extend as needed)
    mean_expected <- exp_tbl[grepl("Mean", exp_tbl[['Brief description']], ignore.case = TRUE), c("Dose", "Expected_Value")]
    names(mean_expected)[2] <- "Expected_Mean"

    t_expected <- exp_tbl[grepl("T-value|t-value", exp_tbl[['Brief description']], ignore.case = TRUE) &
                            !grepl("p-value", exp_tbl[['Brief description']], ignore.case = TRUE), c("Dose", "Expected_Value")]
    names(t_expected)[2] <- "Expected_T"

    p_expected <- exp_tbl[grepl("p-value", exp_tbl[['Brief description']], ignore.case = TRUE), c("Dose", "Expected_Value")]
    names(p_expected)[2] <- "Expected_P"

    # Joins (custom by test type if needed)
    mean_join <- mean_expected %>%
      dplyr::left_join(actual$group_means, by = "Dose") %>%
      dplyr::mutate(
        Endpoint = endpoint,
        Mean_Diff = abs(Actual_Mean - Expected_Mean),
        Mean_Status = dplyr::case_when(
          is.na(Expected_Mean) | is.na(Actual_Mean) ~ "MISSING",
          Mean_Diff <= tolerance ~ "PASS",
          TRUE ~ "FAIL"
        )
      ) %>%
      dplyr::select(Endpoint, Dose, Actual_Mean, Expected_Mean, Mean_Status)

    t_join <- t_expected %>%
      dplyr::left_join(actual$actual_df[, c("Dose", "Actual_T")], by = "Dose") %>%
      dplyr::mutate(
        T_Diff = abs(Actual_T - Expected_T),
        T_Status = dplyr::case_when(
          is.na(Expected_T) | is.na(Actual_T) ~ "MISSING",
          T_Diff <= tolerance ~ "PASS",
          TRUE ~ "FAIL"
        )
      ) %>%
      dplyr::select(Dose, Actual_T, Expected_T, T_Status)

    p_join <- p_expected %>%
      dplyr::left_join(actual$actual_df[, c("Dose", "Actual_P")], by = "Dose") %>%
      dplyr::mutate(
        P_Diff = abs(Actual_P - Expected_P),
        P_Status = dplyr::case_when(
          is.na(Expected_P) | is.na(Actual_P) ~ "MISSING",
          P_Diff <= p_value_tolerance ~ "PASS",
          TRUE ~ "FAIL"
        )
      ) %>%
      dplyr::select(Dose, Actual_P, Expected_P, P_Status)

    # Wide to long and add metadata (reuse your earlier long-format builder)
    wide_df <- mean_join %>%
      dplyr::full_join(t_join, by = "Dose") %>%
      dplyr::full_join(p_join, by = "Dose")
    if (!"Endpoint" %in% names(wide_df)) wide_df$Endpoint <- endpoint

    mean_long <- wide_df %>% dplyr::transmute(Endpoint, Dose, metric = "Mean", Actual = Actual_Mean, Expected = Expected_Mean, Status = Mean_Status)
    t_long    <- wide_df %>% dplyr::transmute(Endpoint, Dose, metric = "T-value", Actual = Actual_T, Expected = Expected_T, Status = T_Status)
    p_long    <- wide_df %>% dplyr::transmute(Endpoint, Dose, metric = "P-value", Actual = Actual_P, Expected = Expected_P, Status = P_Status)

    comparison_long <- dplyr::bind_rows(mean_long, t_long, p_long)

    # Metadata
    test_org <- na.omit(unique(endpoint_data[["Test organism"]]))
    test_org <- if (length(test_org) > 0) test_org[1] else NA_character_
    comparison_long <- comparison_long %>%
      dplyr::mutate(
        `Study ID` = study_id,
        `Test organism` = test_org,
        alternative = alt_norm
      ) %>%
      dplyr::select(`Test organism`, `Study ID`, alternative, Endpoint, Dose, metric, Actual, Expected, Status)

    all_ep[[endpoint]] <- comparison_long
  }

  if (length(all_ep) == 0) {
    return(list(passed = FALSE, error = "No valid comparisons", endpoints_tested = available_endpoints, validation_results = NULL, n_comparisons = 0, n_passed = 0))
  }

  combined <- dplyr::bind_rows(all_ep) %>%
    dplyr::filter(!(is.na(Actual) & is.na(Expected))) %>%
    dplyr::mutate(metric = factor(metric, levels = c("Mean", "T-value", "P-value"))) %>%
    dplyr::arrange(`Test organism`, `Study ID`, Endpoint, Dose, metric)

  total_valid <- sum(combined$Status %in% c("PASS", "FAIL"), na.rm = TRUE)
  total_passed <- sum(combined$Status == "PASS", na.rm = TRUE)
  overall_pass <- !any(combined$Status == "FAIL", na.rm = TRUE)

  list(
    passed = overall_pass,
    endpoints_tested = available_endpoints,
    validation_results = combined,
    n_comparisons = total_valid,
    n_passed = total_passed
  )
}

How to execute across test types

• Discover the FGs for each test type via its registry entry.
• Loop over all alternatives found (or force c("smaller","greater","two-sided")).

# Example execution loop across test types
test_types <- c("dunnett", "williams", "twosample", "shapiro")
summary_results <- data.frame(
  Test_Type = character(),
  Function_Group = character(),
  Study = character(),
  Alternative = character(),
  Endpoints_Tested = character(),
  Total_Validations = integer(),
  Passed_Validations = integer(),
  Success_Rate = character(),
  Overall_Status = character(),
  stringsAsFactors = FALSE
)

for (tt in test_types) {
  reg <- test_registry[[tt]]
  if (is.null(reg)) next
  fgs <- reg$discover_fgs(test_cases_res, test_cases_data)
  for (fg in fgs) {
    alts <- if (length(fg$alternatives) > 0) fg$alternatives else c("smaller", "greater", "two-sided")
    for (alt in alts) {
      res <- tryCatch(run_consolidated_validation_generic(tt, fg$study, fg$id, alt), error = function(e) list(passed=FALSE, error=e$message, endpoints_tested=character(0), validation_results=NULL, n_comparisons=0, n_passed=0))
      status <- if (!is.null(res$error)) "❌ ERROR" else if (res$passed) "✅ PASSED" else "❌ FAILED"
      total <- if (!is.null(res$error)) 0 else res$n_comparisons
      passed <- if (!is.null(res$error)) 0 else res$n_passed
      rate <- if (total > 0) paste0(round(100 * passed / total, 1), "%") else "0%"
      summary_results <- rbind(summary_results, data.frame(
        Test_Type = tt,
        Function_Group = fg$id,
        Study = fg$study,
        Alternative = alt,
        Endpoints_Tested = paste(res$endpoints_tested, collapse = ", "),
        Total_Validations = total,
        Passed_Validations = passed,
        Success_Rate = rate,
        Overall_Status = status,
        stringsAsFactors = FALSE
      ))
      # Here you can print or style res$validation_results as done for Dunnett
    }
  }
}

Key points to decide per test type

• Comparator type affects filtering and join keys:
• • many_to_one (Dunnett, maybe Tukey vs control): compare dose groups vs control, exclude reference item.
  • trend (Williams): compare ordered doses; may not map to a simple dose-vs-control.
  • two_sample: compare exactly two groups; join by group labels not dose.
  • per_group (Shapiro): compute per group metric, no comparison to control, just W and p(W).
• Metrics and tolerances differ by test:
  • Dunnett: Mean, T-value, P-value with tolerances tolerance, p_value_tolerance.
  • Williams: same metrics but adjusted p-values; depends on tukeytrend outputs.
  • Two-sample: Mean difference, T-value, P-value; if Mann–Whitney, then U-value and p-value.
  • Shapiro: W-value and p-value.

To tailor handlers correctly, please run these R snippets and paste the outputs

• What test types and metric labels exist beyond Dunnett?

print(
  test_cases_res %>%
    dplyr::count(`Function group ID`, `Study ID`, `Brief description`) %>%
    dplyr::arrange(`Function group ID`, `Study ID`)
)

• Show unique Brief description values grouped by FG to see patterns:

print(
  split(unique(test_cases_res$`Brief description`), test_cases_res$`Function group ID`) |> lapply(unique)
)

• List available functions in drcHelper so I know what’s already implemented:

print(ls("package:drcHelper"))

• For two-sample tests, how are groups labeled? Show a sample of rows for a likely FG (replace FGID if you know one):

print(
  test_cases_data %>%
    dplyr::filter(`Study ID` == "MOCK08/15-001") %>%
    dplyr::count(`Test group`)
)

• Confirm whether you have tukeytrend installed (for Williams):

print(requireNamespace("tukeytrend", quietly = TRUE))

• If you have an FG for Williams, show a small slice of expected rows:

print(
  test_cases_res %>%
    dplyr::filter(grepl("Williams", `Brief description`, ignore.case = TRUE)) %>%
    dplyr::select(`Function group ID`, `Study ID`, `Endpoint`, `Brief description`, Dose, `expected result value`) %>%
    dplyr::slice(1:10)
)

With these outputs, I can finalize the run_actual handlers for Williams and two-sample tests, define metric mapping, and adjust tolerances per metric/test.