Admin user deletion fails with authorization error despite misleading success logs

[SalimKayal]

Description

Summary

When attempting to delete any admin user through the Renku API, the operation returns a permission error (code 1404) with misleading success logs. The user remains in both databases despite logs claiming successful deletion. This occurs even when deleting different admin users, and the authorization system (SpiceDB) consistently blocks the operation.

Steps to Reproduce

1. Log in as an admin user
2. Attempt to delete another user (Admin2) via the Swagger API endpoint
3. Note: Deletion of Admin1 user (ea209425-eaa2-43d8-9271-1cda870ffd3c) also fails with the same error
4. Verify the user remains in both databases

Expected Behavior

• Admin users should be able to delete other users
• Authorization checks should pass for valid operations
• Logs should accurately reflect transaction outcomes

Actual Behavior

API Response:

{
  "error": {
    "code": 1404,
    "message": "The user with ID df9b9993-ed85-4f81-8a5a-64692e0a7f62 cannot perform operation Scope.DELETE on user with ID 00c40bad-a4bf-40a0-8ff5-5fb7de3f6282 or the resource does not exist."
  }
}

Misleading Logs:

2026-01-15T16:56:02.931631 [INFO] - remove_user: Trying to remove user with ID 00c40bad-a4bf-40a0-8ff5-5fb7de3f6282
2026-01-15T16:56:02.936110 [INFO] - User with ID 00c40bad-a4bf-40a0-8ff5-5fb7de3f6282 was removed from the database.
2026-01-15T16:56:02.936197 [INFO] - User namespace with ID 00c40bad-a4bf-40a0-8ff5-5fb7de3f6282 was removed from the authorization database.

Database State - User NOT Deleted:

Renku database (users.users):

             keycloak_id              | first_name | last_name |         email         | id
--------------------------------------+------------+-----------+-----------------------+----
 ea209425-eaa2-43d8-9271-1cda870ffd3c | Admin1     | ADMIN1    | admin1.admin1@myorg.com | 11
 00c40bad-a4bf-40a0-8ff5-5fb7de3f6282 | Admin2     | ADMIN2    | admin2.admin2@myorg.com | 12
 df9b9993-ed85-4f81-8a5a-64692e0a7f62 | Admin2     | ADMIN2    | admin2.admin2@myorg.com | 13

Authorization Database - User Still Has Admin Role:

 relation | userset_namespace |          userset_object_id
----------+-------------------+--------------------------------------
 admin    | user              | 00c40bad-a4bf-40a0-8ff5-5fb7de3f6282
 admin    | user              | df9b9993-ed85-4f81-8a5a-64692e0a7f62
 admin    | user              | ea209425-eaa2-43d8-9271-1cda870ffd3c

[SalimKayal]

Update: Critical Finding - SpiceDB Synchronization Issue

Follow-up Investigation (Day +1)

After waiting approximately 12 hours, I attempted the deletion operation again and the user became deletable. The user was successfully deleted and then re-created with admin rights. However, this revealed a synchronization problem between the main database and SpiceDB.

Current State After Deletion and Re-creation

GET /user endpoint returns the NEW user:

{
  "id": "82e1efc2-18af-4d58-a377-dd33f6d1a8a0",
  "username": "admin1.admin1",
  "email": "admin1.admin1@myorg.com",
  "first_name": "Admin1",
  "last_name": "ADMIN1",
  "is_admin": true
}

SpiceDB authz database still contains ONLY the OLD user IDs:

authz=> select relation, userset_namespace, userset_object_id from relation_tuple where relation = 'admin';
 relation | userset_namespace |          userset_object_id
----------+-------------------+--------------------------------------
 admin    | user              | 00c40bad-a4bf-40a0-8ff5-5fb7de3f6282  ← OLD (deleted)
 admin    | user              | ea209425-eaa2-43d8-9271-1cda870ffd3c  ← OLD
 admin    | user              | df9b9993-ed85-4f81-8a5a-64692e0a7f62  ← OLD

The new user ID 82e1efc2-18af-4d58-a377-dd33f6d1a8a0 is NOT present in SpiceDB.

Implications

This reveals several issues:

1. Stale Relationships in SpiceDB: The authorization database contains outdated user IDs that no longer exist in the main database
2. Missing Synchronization on User Creation: When a new user is created with admin rights, the corresponding SpiceDB relationship is not being written
3. Missing Synchronization on User Deletion: When users are deleted, their SpiceDB relationships are not being removed (explaining why 00c40bad... still appears despite being deleted)
4. Root Cause of Original Issue: The original deletion failure may have been caused by SpiceDB referencing stale user relationships or attempting authorization checks against inconsistent data

Questions

1. Is there a background sync job that should be updating SpiceDB relationships?
2. Should user creation/deletion trigger immediate SpiceDB updates, or is there an eventual consistency model?
3. Why did the user become deletable after a few hours - was there a cache expiration, a background cleanup job?
4. How should the system handle the discrepancy between users marked as is_admin: true in the API but missing from SpiceDB admin relationships?

the authorization system is operating on stale data, potentially granting or denying access based on users that no longer exist.