getConDF fails and returns an error: $ operator is invalid for atomic vectors

[dimshitc]

con_df <- getConDF(connectionDetails = connectionDetailsDB, json = json,

•                name = name, cdmSchema = cdmSchema, 
  

•                writeSchema = writeSchema)
  

Connecting using Spark JDBC driver
Error in json$json : $ operator is invalid for atomic vectors

I've got a cohort json by running
json <- ROhdsiWebApi::getCohortDefinitionExpression(20765, baseUrl)

[hms2]

hey @dimshitc ,

The current convention in the package is that 'json' is in fact a dataframe (in the style of the typical OHDSI 'cohortsToCreate') that has a column containing the actual json definition:

> ARTEMIS::json
# A tibble: 1 × 4
  cohort_definition_id cohort_name        cohort           json     
                 <int> <chr>              <list>           <list>   
1                    1 lungCancer_Example <named list [9]> <chr [1]>

You could try either:

json <- list(json = json)

or

json <- data.frame(json = json)

Thanks!

[dimshitc]

Thank you, it worked!

but...
I'm working in DataBricks on Spark SQL.

and I'm getting this error:

looks like, somehow it doesn't create in scratch schema anymore, but wants to do it in default schema.

Can you please ping me in teams: dymshyts@ohdsi.org? probably it would be easier to debug

[dimshitc]

ok, probably I didn't specify the temp table schema for databricks. this is a peculiarity of the OHDSI DatabaseConnector

[hms2]

coincidentally this is an issue we are currently having with another Spark user - maybe something has changed in the latest OHDSI packages somewhere?

If you don't mind sharing your solution here, if you have one, it'd be v helpful!

[dimshitc]

Yes,
As far as I understand, SparkSQL doesn't create temporary tables, so to emulate this behavior, you need to set the schema where these tables will be stored -
execute this in R:
options(sqlRenderTempEmulationSchema = "scratch.scratch_yourScratch")

this way you'll have in your scratch these tables. Note, in each session it will be created with some prefix,
for example, "CREATE TABLE jj219nucCodesets", instead of "CREATE TABLE Codesets"
so if you have two sessions run simultaneously, these two tables will not interfere.

Note, in the end you'll need to drop these tables as they are not dropped automatically (which I even like sometimes, so I can debug)