numerical covariates not accepted

[gabuali]

scanpro() does not accept numerical covariates.
Fix line 213 in linear_model.py
design.columns = [re.match(r".+\[(T\.){0,1}(.+)\]", col).group(2) for col in design.columns]

[yalayoubi]

Hi,

thank you for reporting this. I pushed a fix to the dev branch, can you please test it and see if your problem is solved?

[gabuali]

It now accepts numerical covariates but not numerical/continuous predictors; which is concerning. As a test, I tried age as a predictor and got the below error.

`# test numeric predictor

out = scanpro(cell_1_iss, clusters_col='cell_type_1', samples_col='PUBLIC_ID', conds_col='DEMOG_PATIENTAGE', transform='arcsin', covariates=['DEMOG_GENDER', 'RACE'])`

`---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
Cell In[23], line 2
1 # test numeric predictor
----> 2 out = scanpro(cell_1_iss, clusters_col='cell_type_1', samples_col='PUBLIC_ID', conds_col='DEMOG_PATIENTAGE', transform='arcsin', covariates=['DEMOG_GENDER', 'RACE'])

File /opt/conda/lib/python3.10/site-packages/scanpro/scanpro.py:190, in scanpro(data, clusters_col, conds_col, samples_col, covariates, conditions, transform, robust, n_sims, n_reps, run_partial_sim, pairwise, verbosity, seed)
187 out.sim_results = combined_sim # combined dataframe with all p-values
189 else:
--> 190 out = run_scanpro(data, clusters_col=clusters_col, samples_col=samples_col, conds_col=conds_col,
191 covariates=covariates, transform=transform, conditions=conditions,
192 n_reps=n_reps, run_partial_sim=run_partial_sim, robust=robust,
193 verbosity=verbosity, seed=seed)
195 out.pairwise = pairwise
197 return out

File /opt/conda/lib/python3.10/site-packages/scanpro/scanpro.py:309, in run_scanpro(data, clusters_col, conds_col, samples_col, covariates, conditions, transform, robust, n_sims, n_reps, run_partial_sim, verbosity, seed)
307 replicated = 'all' # before running partial_sim assume replicated
308 s = "The following conditions don't have replicates: "
--> 309 s += ", ".join(no_reps_list)
310 logger.info(s)
311 if not run_partial_sim:

TypeError: sequence item 0: expected str instance, int found`

[yalayoubi]

I think it is because it expects conds_col to be categories of strings. I added a check to catch this, please try it again and see if it works now

[gabuali]

It seems scanpro is not designed to handle continuous predictors. Age is a single variable and does not need replicates, there are many samples in my data.

Command

# test numeric predictor out = scanpro(cell_1_iss, clusters_col='cell_type_1', samples_col='PUBLIC_ID', conds_col='DEMOG_PATIENTAGE', transform='arcsin', covariates=['DEMOG_GENDER', 'RACE'])

Error

[INFO] The following conditions don't have replicates: 27, 40, 79, 84, 88, 42, 32, 93, 31, 90, 38, 36
[INFO] Running scanpro with original replicates...
[INFO] There are more than 2 conditions. ANOVA will be performed...
[INFO] Done!
[INFO] Running scanpro with simulated replicates...
[INFO] Generating 3 replicates and running 100 simulations...

KeyError Traceback (most recent call last)
Cell In[19], line 2
1 # test numeric predictor
----> 2 out = scanpro(cell_1_iss, clusters_col='cell_type_1', samples_col='PUBLIC_ID', conds_col='DEMOG_PATIENTAGE', transform='arcsin', covariates=['DEMOG_GENDER', 'RACE'])

File /opt/conda/lib/python3.10/site-packages/scanpro/scanpro.py:194, in scanpro(data, clusters_col, conds_col, samples_col, covariates, conditions, transform, robust, n_sims, n_reps, run_partial_sim, pairwise, verbosity, seed)
191 out.sim_results = combined_sim # combined dataframe with all p-values
193 else:
--> 194 out = run_scanpro(data, clusters_col=clusters_col, samples_col=samples_col, conds_col=conds_col,
195 covariates=covariates, transform=transform, conditions=conditions,
196 n_reps=n_reps, run_partial_sim=run_partial_sim, robust=robust,
197 verbosity=verbosity, seed=seed)
199 out.pairwise = pairwise
201 return out

File /opt/conda/lib/python3.10/site-packages/scanpro/scanpro.py:344, in run_scanpro(data, clusters_col, conds_col, samples_col, covariates, conditions, transform, robust, n_sims, n_reps, run_partial_sim, verbosity, seed)
342 # set transform to arcsin, since it produces more accurate results for simulations
343 transform = 'arcsin'
--> 344 out_sim = sim_scanpro(data, n_reps=n_reps, n_sims=n_sims, clusters_col=clusters_col, covariates=covariates,
345 conds_col=conds_col, transform=transform,
346 conditions=conditions, robust=robust, verbosity=verbosity)
348 logger.info("To access results for original replicates, run <out.results>, and <out.sim_results> for simulated results")
350 # if all conditions have replicates, run scanpro normally
351 else:

File /opt/conda/lib/python3.10/site-packages/scanpro/scanpro.py:663, in sim_scanpro(data, clusters_col, conds_col, covariates, transform, n_reps, n_sims, conditions, robust, verbosity)
661 # run propeller
662 try:
--> 663 out_sim = run_stats(rep_data, clusters=clusters_col, samples=samples_col,
664 conds=conds_col, covariates=covariates, transform=transform,
665 conditions=conditions, robust=robust, verbosity=0) # verbosity is 0 to prevent prints from individual simulations
667 # workaround brentq error "f(a) and f(b) must have different signs"
668 # rerun simulation instead of crashing
669 except ValueError:

File /opt/conda/lib/python3.10/site-packages/scanpro/scanpro.py:465, in run_stats(data, clusters, samples, conds, transform, covariates, conditions, robust, verbosity)
463 logger.info("There are more than 2 conditions. ANOVA will be performed...")
464 coef = np.arange(len(conditions))
--> 465 out = anova(props_sub, prop_trans_sub, design_sub, coef, robust, verbosity)
467 out.index.name = clusters
468 logger.info("Done!")

File /opt/conda/lib/python3.10/site-packages/scanpro/scanpro.py:531, in anova(props, prop_trans, design, coef, robust, verbosity)
528 fit = ebayes.ebayes(fit, robust=robust)
530 # adjust p_values using benjamin hochberg method
--> 531 p_values = fit['F']['F_p_value'].flatten()
532 fdr = multipletests(p_values, method='fdr_bh')
534 # save results to dictionary

KeyError: 'F'

[yalayoubi]

you can turn off the pseudo-replicates simulation by setting the parameter run_partial_sim=False, since your data does have replicates for some samples

[gabuali]

It still treats age as a categorical and NOT continuous variable. I get an output but it's wrong. Below are the headers of the output. It should be a single column for age with p-values for an association of cell type proportion and age; basic regression. It seems the software doesn't handle continuous predictors. What am I missing?

baseline_props | mean_props_27 | mean_props_31 | mean_props_32 | mean_props_36 | mean_props_38 | mean_props_39 | mean_props_40 | mean_props_42 | mean_props_43 | mean_props_44 | mean_props_45 | mean_props_47 | mean_props_50 | mean_props_51 | mean_props_53 | mean_props_54 | mean_props_55 | mean_props_56 | mean_props_57 | mean_props_58 | mean_props_59 | mean_props_60 | mean_props_61 | mean_props_62 | mean_props_63 | mean_props_64 | mean_props_65 | mean_props_66 | mean_props_67 | mean_props_68 | mean_props_69 | mean_props_70 | mean_props_71 | mean_props_72 | mean_props_73 | mean_props_74 | mean_props_75 | mean_props_77 | mean_props_78 | mean_props_79 | mean_props_80 | mean_props_82 | mean_props_83 | mean_props_84 | mean_props_85 | mean_props_87 | mean_props_88 | mean_props_90 | mean_props_93 | f_statistics | p_values | adjusted_p_values

[yalayoubi]

You are right, as for now, scanpro expects the condition in conds_col to be categorical, while covariates can be any type. I will be working on it to make it also possible to have continuous variable as condition, but this could take a while