FIX: undesired python version used for reticulate when working with radian by code-instable · Pull Request #508 · ropensci/rix

code-instable · 2025-06-03T19:29:40Z

problem

When using Radian IDE, which default used python version may differ from the selected one, reticulate will force the use of the python version used for radian instead of the one provided by RETICULATE_PYTHON. The proposed solution is to fetch the radian matching the desired python version for reticulate.

the rix configuration (minimal)

here is an almost minimal setup to reproduce the error:

rix(
    date = "2025-04-11",
    r_pkgs = c("reticulate"),
    git_pkgs = list(list(package_name = "somebm", repo_url = "https://github.com/cran/somebm", commit = "03584b24dee8081f08a0cde456edcc8419aba494")),
    py_conf = list(
        py_version = "3.13",
        py_pkgs = c("numpy")
    ),
    ide = "radian",
    project_path = ".",
    overwrite = FALSE,
    print = TRUE,
    message_type = "simple"
)

the following code is then executed inside the nix environment

Sys.getenv("RETICULATE_PYTHON") # we will see that loading reticulate changes the variable

require(reticulate)

reticulate::py_config()
reticulate::py_version()
reticulate::py_exe()
reticulate::py_list_packages()

Sys.getenv("RETICULATE_PYTHON")

collision between radian's python version and reticulate's python version

When entering nix-shell while using radian IDE, reticulate automatically falls back to the python version used by radian, as we can see in the following example:

(note that RETICULATE_PYTHON switches from the correct desired version to radian's one on loading the reticulate package)

❯ nix-shell

[nix-shell:~/github-contributions/rix/custom/with_py_pkgs-old]$ radian
R version 4.5.0 (2025-04-11) -- "How About a Twenty-Six"
Platform: aarch64-apple-darwin24.5.0 (64-bit)

r$> Sys.getenv("RETICULATE_PYTHON")

    require(reticulate)

    reticulate::py_config()
    reticulate::py_version()
    reticulate::py_exe()
    reticulate::py_list_packages()

    Sys.getenv("RETICULATE_PYTHON")
[1] "/nix/store/f55hdkj3px5v288jv6yai2gh5xrlj848-python3-3.13.2/bin/python"
Loading required package: reticulate
python:         /nix/store/3p3agnh4792y8bgjp0s8xx5bym1ahpks-python3-3.12.9/bin/python3.12
libpython:      /nix/store/3p3agnh4792y8bgjp0s8xx5bym1ahpks-python3-3.12.9/lib/libpython3.12.dylib
pythonhome:     /nix/store/3p3agnh4792y8bgjp0s8xx5bym1ahpks-python3-3.12.9:/nix/store/3p3agnh4792y8bgjp0s8xx5bym1ahpks-python3-3.12.9
version:        3.12.9 (main, Feb  4 2025, 14:38:38) [Clang 19.1.7 ]
numpy:           [NOT FOUND]

NOTE: Python version was forced by the current process
[1] ‘3.12’
[1] "/nix/store/3p3agnh4792y8bgjp0s8xx5bym1ahpks-python3-3.12.9/bin/python3.12"
             package     version                  requirement
1            appnope       0.1.3               appnope==0.1.3
2          asttokens       3.0.0             asttokens==3.0.0
3               comm       0.2.2                  comm==0.2.2
4          decorator       5.1.1             decorator==5.1.1
5          executing       2.2.0             executing==2.2.0
6          ipykernel      6.29.5            ipykernel==6.29.5
7            ipython      8.32.0              ipython==8.32.0
8               jedi      0.19.2                 jedi==0.19.2
9     jupyter_client       8.6.3        jupyter_client==8.6.3
10      jupyter_core       5.7.2          jupyter_core==5.7.2
11 matplotlib-inline       0.1.7     matplotlib-inline==0.1.7
12      nest_asyncio       1.6.0          nest_asyncio==1.6.0
13             numpy       2.2.3                 numpy==2.2.3
14         packaging        24.2              packaging==24.2
15             parso       0.8.4                 parso==0.8.4
16           pexpect       4.9.0               pexpect==4.9.0
17      platformdirs       4.3.6          platformdirs==4.3.6
18    prompt_toolkit      3.0.50       prompt_toolkit==3.0.50
19            psutil       7.0.0                psutil==7.0.0
20        ptyprocess       0.7.0            ptyprocess==0.7.0
21         pure_eval       0.2.3             pure_eval==0.2.3
22          Pygments      2.19.1             Pygments==2.19.1
23   python-dateutil 2.9.0.post0 python-dateutil==2.9.0.post0
24             pyzmq      26.2.1                pyzmq==26.2.1
25               six      1.17.0                  six==1.17.0
26        stack_data       0.6.3            stack_data==0.6.3
27              toml      0.10.2                 toml==0.10.2
28           tornado       6.4.2               tornado==6.4.2
29         traitlets      5.14.3            traitlets==5.14.3
30           wcwidth      0.2.13              wcwidth==0.2.13
[1] "/nix/store/3p3agnh4792y8bgjp0s8xx5bym1ahpks-python3-3.12.9/bin/python3.12"

only when using radian

However, it still works perfectly fine when using Rscript:

=== Executing R script in with_py_pkgs-old at Tue Jun  3 21:02:50 CEST 2025 ===
Working directory: /Users/instable/github-contributions/rix/custom/with_py_pkgs-old
=== Output ===
[1] "/nix/store/f55hdkj3px5v288jv6yai2gh5xrlj848-python3-3.13.2/bin/python"
Loading required package: reticulate
python:         /nix/store/f55hdkj3px5v288jv6yai2gh5xrlj848-python3-3.13.2/bin/python
libpython:      /nix/store/f55hdkj3px5v288jv6yai2gh5xrlj848-python3-3.13.2/lib/libpython3.13.dylib
pythonhome:     /nix/store/f55hdkj3px5v288jv6yai2gh5xrlj848-python3-3.13.2:/nix/store/f55hdkj3px5v288jv6yai2gh5xrlj848-python3-3.13.2
version:        3.13.2 (main, Feb  4 2025, 14:51:09) [Clang 19.1.7 ]
numpy:          /nix/store/wd5v6f1j439170iszqb0n5ln8qvyl31g-python3.13-numpy-2.2.3/lib/python3.13/site-packages/numpy
numpy_version:  2.2.3

NOTE: Python version was forced by RETICULATE_PYTHON
[1] ‘3.13’
[1] "/nix/store/f55hdkj3px5v288jv6yai2gh5xrlj848-python3-3.13.2/bin/python"
             package     version                  requirement
1            appnope       0.1.3               appnope==0.1.3
2          asttokens       3.0.0             asttokens==3.0.0
3               comm       0.2.2                  comm==0.2.2
4          decorator       5.1.1             decorator==5.1.1
5          executing       2.2.0             executing==2.2.0
6          ipykernel      6.29.5            ipykernel==6.29.5
7            ipython      8.32.0              ipython==8.32.0
8               jedi      0.19.2                 jedi==0.19.2
9     jupyter_client       8.6.3        jupyter_client==8.6.3
10      jupyter_core       5.7.2          jupyter_core==5.7.2
11 matplotlib-inline       0.1.7     matplotlib-inline==0.1.7
12      nest_asyncio       1.6.0          nest_asyncio==1.6.0
13             numpy       2.2.3                 numpy==2.2.3
14         packaging        24.2              packaging==24.2
15             parso       0.8.4                 parso==0.8.4
16           pexpect       4.9.0               pexpect==4.9.0
17      platformdirs       4.3.6          platformdirs==4.3.6
18    prompt_toolkit      3.0.50       prompt_toolkit==3.0.50
19            psutil       7.0.0                psutil==7.0.0
20        ptyprocess       0.7.0            ptyprocess==0.7.0
21         pure_eval       0.2.3             pure_eval==0.2.3
22          Pygments      2.19.1             Pygments==2.19.1
23   python-dateutil 2.9.0.post0 python-dateutil==2.9.0.post0
24             pyzmq      26.2.1                pyzmq==26.2.1
25               six      1.17.0                  six==1.17.0
26        stack_data       0.6.3            stack_data==0.6.3
27              toml      0.10.2                 toml==0.10.2
28           tornado       6.4.2               tornado==6.4.2
29         traitlets      5.14.3            traitlets==5.14.3
30           wcwidth      0.2.13              wcwidth==0.2.13
[1] "/nix/store/f55hdkj3px5v288jv6yai2gh5xrlj848-python3-3.13.2/bin/python"
=== Execution completed with exit status: 0 ===

proposed solution

The proposed solution is to match radian's python version with the python version from the user-defined config within the override of radianWrapper:

wrapped_pkgs = pkgs.radianWrapper.override {
    packages = [ somebm rpkgs ];
    radian = pkgs.python313Packages.radian
};

the code produces the following default.nix:

# This file was generated by the {rix} R package v0.17.0 on 2025-06-03
# with following call:
# >rix(date = "2025-04-11",
#  > r_pkgs = c("reticulate"),
#  > git_pkgs = list(list(package_name = "somebm",
#  > repo_url = "https://github.com/cran/somebm",
#  > commit = "03584b24dee8081f08a0cde456edcc8419aba494")),
#  > py_conf = list(py_version = "3.13",
#  > py_pkgs = c("numpy")),
#  > ide = "radian",
#  > project_path = ".",
#  > overwrite = FALSE,
#  > print = TRUE,
#  > message_type = "simple",
#  > r_ver = "4.5.0")
# It uses the `rstats-on-nix` fork of `nixpkgs` which provides improved
# compatibility with older R versions and R packages for Linux/WSL and
# Apple Silicon computers.
# Report any issues to https://github.com/ropensci/rix
let
 pkgs = import (fetchTarball "https://github.com/rstats-on-nix/nixpkgs/archive/2025-04-11.tar.gz") {};
 
  rpkgs = builtins.attrValues {
    inherit (pkgs.rPackages) 
      reticulate;
  };
 
    somebm = (pkgs.rPackages.buildRPackage {
      name = "somebm";
      src = pkgs.fetchgit {
        url = "https://github.com/cran/somebm";
        rev = "03584b24dee8081f08a0cde456edcc8419aba494";
        sha256 = "sha256-QqSk/bmbDpquHJfHqimCcJcj/4QKdqQLBLqKCRZzn6s=";
      };
      propagatedBuildInputs = builtins.attrValues {
        inherit (pkgs.rPackages) ;
      };
    });
   
  pyconf = builtins.attrValues {
    inherit (pkgs.python313Packages) 
      pip
      ipykernel
      numpy;
  };
   
  system_packages = builtins.attrValues {
    inherit (pkgs) 
      glibcLocales
      nix
      python313
      R;
  };
 
  wrapped_pkgs = pkgs.radianWrapper.override {
    packages = [ somebm rpkgs ];
    radian = pkgs.python313Packages.radian
  };
 
  shell = pkgs.mkShell {
    LOCALE_ARCHIVE = if pkgs.system == "x86_64-linux" then "${pkgs.glibcLocales}/lib/locale/locale-archive" else "";
    LANG = "en_US.UTF-8";
    LC_ALL = "en_US.UTF-8";
    LC_TIME = "en_US.UTF-8";
    LC_MONETARY = "en_US.UTF-8";
    LC_PAPER = "en_US.UTF-8";
    LC_MEASUREMENT = "en_US.UTF-8";
    RETICULATE_PYTHON = "${pkgs.python313}/bin/python";

    buildInputs = [ somebm rpkgs pyconf system_packages wrapped_pkgs ];
    
  }; 
in
  {
    inherit pkgs shell;
  }

running the nix-shell now produces the desired result as "forced by the current process":

(notice that the RETICULATE_PYTHON does not change here)

❯ nix-shell

[nix-shell:~/github-contributions/rix/custom/with_py_pkgs]$ radian
R version 4.5.0 (2025-04-11) -- "How About a Twenty-Six"
Platform: aarch64-apple-darwin24.5.0 (64-bit)

r$> Sys.getenv("RETICULATE_PYTHON")

    require(reticulate)

    reticulate::py_config()
    reticulate::py_version()
    reticulate::py_exe()
    reticulate::py_list_packages()

    Sys.getenv("RETICULATE_PYTHON")
[1] "/nix/store/f55hdkj3px5v288jv6yai2gh5xrlj848-python3-3.13.2/bin/python"
Loading required package: reticulate
python:         /nix/store/f55hdkj3px5v288jv6yai2gh5xrlj848-python3-3.13.2/bin/python3.13
libpython:      /nix/store/f55hdkj3px5v288jv6yai2gh5xrlj848-python3-3.13.2/lib/libpython3.13.dylib
pythonhome:     /nix/store/f55hdkj3px5v288jv6yai2gh5xrlj848-python3-3.13.2:/nix/store/f55hdkj3px5v288jv6yai2gh5xrlj848-python3-3.13.2
version:        3.13.2 (main, Feb  4 2025, 14:51:09) [Clang 19.1.7 ]
numpy:          /nix/store/wd5v6f1j439170iszqb0n5ln8qvyl31g-python3.13-numpy-2.2.3/lib/python3.13/site-packages/numpy
numpy_version:  2.2.3

NOTE: Python version was forced by the current process
[1] ‘3.13’
[1] "/nix/store/f55hdkj3px5v288jv6yai2gh5xrlj848-python3-3.13.2/bin/python3.13"
             package     version                  requirement
1            appnope       0.1.3               appnope==0.1.3
2          asttokens       3.0.0             asttokens==3.0.0
3               comm       0.2.2                  comm==0.2.2
4          decorator       5.1.1             decorator==5.1.1
5          executing       2.2.0             executing==2.2.0
6          ipykernel      6.29.5            ipykernel==6.29.5
7            ipython      8.32.0              ipython==8.32.0
8               jedi      0.19.2                 jedi==0.19.2
9     jupyter_client       8.6.3        jupyter_client==8.6.3
10      jupyter_core       5.7.2          jupyter_core==5.7.2
11 matplotlib-inline       0.1.7     matplotlib-inline==0.1.7
12      nest_asyncio       1.6.0          nest_asyncio==1.6.0
13             numpy       2.2.3                 numpy==2.2.3
14         packaging        24.2              packaging==24.2
15             parso       0.8.4                 parso==0.8.4
16           pexpect       4.9.0               pexpect==4.9.0
17      platformdirs       4.3.6          platformdirs==4.3.6
18    prompt_toolkit      3.0.50       prompt_toolkit==3.0.50
19            psutil       7.0.0                psutil==7.0.0
20        ptyprocess       0.7.0            ptyprocess==0.7.0
21         pure_eval       0.2.3             pure_eval==0.2.3
22          Pygments      2.19.1             Pygments==2.19.1
23   python-dateutil 2.9.0.post0 python-dateutil==2.9.0.post0
24             pyzmq      26.2.1                pyzmq==26.2.1
25               six      1.17.0                  six==1.17.0
26        stack_data       0.6.3            stack_data==0.6.3
27              toml      0.10.2                 toml==0.10.2
28           tornado       6.4.2               tornado==6.4.2
29         traitlets      5.14.3            traitlets==5.14.3
30           wcwidth      0.2.13              wcwidth==0.2.13
[1] "/nix/store/f55hdkj3px5v288jv6yai2gh5xrlj848-python3-3.13.2/bin/python3.13"

if no py_conf is provided, then we do not add the python version override:

# This file was generated by the {rix} R package v0.17.0 on 2025-06-03
# with following call:
# >rix(date = "2025-04-11",
#  > r_pkgs = c("reticulate"),
#  > git_pkgs = list(list(package_name = "somebm",
#  > repo_url = "https://github.com/cran/somebm",
#  > commit = "03584b24dee8081f08a0cde456edcc8419aba494")),
#  > ide = "radian",
#  > project_path = ".",
#  > overwrite = FALSE,
#  > print = TRUE,
#  > message_type = "simple",
#  > r_ver = "4.5.0")
# It uses the `rstats-on-nix` fork of `nixpkgs` which provides improved
# compatibility with older R versions and R packages for Linux/WSL and
# Apple Silicon computers.
# Report any issues to https://github.com/ropensci/rix
let
 pkgs = import (fetchTarball "https://github.com/rstats-on-nix/nixpkgs/archive/2025-04-11.tar.gz") {};
 
  rpkgs = builtins.attrValues {
    inherit (pkgs.rPackages) 
      reticulate;
  };
 
    somebm = (pkgs.rPackages.buildRPackage {
      name = "somebm";
      src = pkgs.fetchgit {
        url = "https://github.com/cran/somebm";
        rev = "03584b24dee8081f08a0cde456edcc8419aba494";
        sha256 = "sha256-QqSk/bmbDpquHJfHqimCcJcj/4QKdqQLBLqKCRZzn6s=";
      };
      propagatedBuildInputs = builtins.attrValues {
        inherit (pkgs.rPackages) ;
      };
    });
      
  system_packages = builtins.attrValues {
    inherit (pkgs) 
      glibcLocales
      nix
      R;
  };
 
  wrapped_pkgs = pkgs.radianWrapper.override {
    packages = [ somebm rpkgs ];
    
  };
 
  shell = pkgs.mkShell {
    LOCALE_ARCHIVE = if pkgs.system == "x86_64-linux" then "${pkgs.glibcLocales}/lib/locale/locale-archive" else "";
    LANG = "en_US.UTF-8";
    LC_ALL = "en_US.UTF-8";
    LC_TIME = "en_US.UTF-8";
    LC_MONETARY = "en_US.UTF-8";
    LC_PAPER = "en_US.UTF-8";
    LC_MEASUREMENT = "en_US.UTF-8";
    
    buildInputs = [ somebm rpkgs system_packages wrapped_pkgs ];
    
  }; 
in
  {
    inherit pkgs shell;
  }

note that I don't know how to properly remove the added newline when there is no py_conf without using a if/else statement, so any suggestion is much appreciated

suggested edits

(+) function: set_radian_python_version_override
(~) generate_wrapped_pkgs

(+) added py_conf and flag_py_conf as arguments
(+) added string format '%s' bellow the packages

TODO: remove whitespaces and empty line when there is no py_conf
(I don't know how to do)

potential edge case

the desired python version is much older than the one used for radian

(+) function: set_radian_python_version_override (~) generate_wrapped_pkgs (+) added `py_conf` and `flag_py_conf` as arguments (+) added string format '%s' bellow the packages TODO: remove whitespaces and empty line when there is no py_conf

b-rodrigues · 2025-06-04T18:24:55Z

The PR is still in draft, is that because you want to continue to work on it, or did you forget to set it to ready? In any case, could you fix the unit tests please?

code-instable · 2025-06-06T09:03:48Z

The PR is still in draft because I want to know whether things have to change, and if it is a good approach as well

I am not good with unit test, so I don't really know what to change, I will take a look at it

Also I don't know how to trim the added white spaces when there is no pyconf, helping me on this would be much appreciated

b-rodrigues · 2025-06-06T09:06:26Z

Ok let’s forget the unit tests then: the overall approach seems good. Put the PR in ready for review, and I will thouroughly test it. If it’s fine, I’ll merge and fix the tests and the whitespace issue myself.

code-instable · 2025-06-14T14:58:11Z

the radian package has been promoted to a top package 3 days ago. I may have to try if this resolved by itself the issue, which means the code provided is now breaking when it comes to the unstable branch

b-rodrigues · 2025-07-13T18:35:30Z

Have you had a chance to check whether this is still an issue?

code-instable · 2025-07-14T10:03:19Z

There is still the need to change something, I will do it later, as radian moved from a python package to a regular package

Which also breaks compatibility with different releases of nixpkgs (Should it be taken into account ?)

code-instable changed the title ~~override radian python version to be same as in py_conf~~ FIX: undesired python version used for reticulate when working with radian Jun 3, 2025

code-instable added 2 commits June 3, 2025 22:11

update function name to match other helpers

493eac5

add documentation

c8dc7ef

code-instable marked this pull request as ready for review June 6, 2025 09:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FIX: undesired python version used for reticulate when working with radian#508

FIX: undesired python version used for reticulate when working with radian#508
code-instable wants to merge 3 commits intoropensci:mainfrom
code-instable:radian-sync_with_python_version

code-instable commented Jun 3, 2025 •

edited

Loading

Uh oh!

b-rodrigues commented Jun 4, 2025

Uh oh!

code-instable commented Jun 6, 2025

Uh oh!

b-rodrigues commented Jun 6, 2025

Uh oh!

code-instable commented Jun 14, 2025

Uh oh!

b-rodrigues commented Jul 13, 2025

Uh oh!

code-instable commented Jul 14, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

code-instable commented Jun 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

problem

the rix configuration (minimal)

collision between radian's python version and reticulate's python version

only when using radian

proposed solution

suggested edits

potential edge case

Uh oh!

b-rodrigues commented Jun 4, 2025

Uh oh!

code-instable commented Jun 6, 2025

Uh oh!

b-rodrigues commented Jun 6, 2025

Uh oh!

code-instable commented Jun 14, 2025

Uh oh!

b-rodrigues commented Jul 13, 2025

Uh oh!

code-instable commented Jul 14, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

code-instable commented Jun 3, 2025 •

edited

Loading