Skip to content

Upgrade all packages to latest versions with Flask 3.x and rdflib 7.x + Migrate to Click-based CLI (Python 3.9+ required) #346

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
Copilot wants to merge 27 commits into
base: main
Choose a base branch
from
Draft

Upgrade all packages to latest versions with Flask 3.x and rdflib 7.x + Migrate to Click-based CLI (Python 3.9+ required) #346

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
8944a32
Added initial support for neptune
jpmccu Dec 23, 2025
4f357e1
Initial neptune implementation
jpmccu Jan 2, 2026
2e3927d
Add Neptune plugin database driver with AWS IAM authentication and Cl…
Copilot Jan 4, 2026
f3ce7e5
Neptune GSP: Use UUID graph URIs instead of explicit default graph (#…
Copilot Jan 4, 2026
4ade2f3
Initial plan
Copilot Jan 4, 2026
990d802
Upgrade all packages to latest compatible versions
Copilot Jan 4, 2026
d35ff46
Add Flask-Script compatibility patches and plan migration to Flask CLI
Copilot Jan 4, 2026
a15b96e
Add Flask CLI migration with Click-based commands
Copilot Jan 4, 2026
29c6c9b
Fix Flask-Security-Too compatibility and add comprehensive tests
Copilot Jan 4, 2026
d14c7fa
Address code review feedback - fix imports and assertions
Copilot Jan 4, 2026
0028ad1
Add comprehensive upgrade summary documentation
Copilot Jan 4, 2026
3e406b5
update UI and search
jpmccu Jan 4, 2026
253f705
Merge remote-tracking branch 'refs/remotes/origin/feature/neptune' in…
jpmccu Jan 4, 2026
2864ea7
Fix Python 3.8 compatibility for numpy, pandas, and scipy
Copilot Jan 5, 2026
90bb53c
Bump minimum Python version to 3.9 to fix eventlet compatibility
Copilot Jan 5, 2026
8b9e592
Fix eventlet version constraint - 0.40.0 doesn't exist
Copilot Jan 5, 2026
7486b1a
Revert oxrdflib to 0.3.1 to fix pyoxigraph build failure
Copilot Jan 5, 2026
9ba0df4
Fix rdflib dependency conflict by upgrading oxrdflib to 0.3.7
Copilot Jan 5, 2026
c8f888f
Add fs_uniquifier field for Flask-Security-Too 4.0+ compatibility
Copilot Jan 5, 2026
62d4b58
Fix celery worker startup by always exposing celery in wsgi.py
Copilot Jan 5, 2026
90b2102
Merge branch 'feature/neptune' into copilot/upgrade-outdated-packages
jpmccu Jan 5, 2026
dd35c39
Fix Celery 5.x command syntax for embedded worker
Copilot Jan 5, 2026
5e9a869
Fix Celery module path - use full whyis.wsgi:celery path
Copilot Jan 5, 2026
5f0cc62
Fix Celery to work with kgapp wsgi.py structure
Copilot Jan 5, 2026
2a239c9
Use os.path.isfile for more robust wsgi.py detection
Copilot Jan 5, 2026
b49a9cd
Migrate whyis-cli to main whyis command with Flask-Script compatibility
Copilot Jan 5, 2026
b20a87c
sorted out the celery startup variables.
jpmccu Jan 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/python-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ jobs:

strategy:
matrix:
python-version: ['3.8', '3.9', '3.10', '3.11']
python-version: ['3.9', '3.10', '3.11']

steps:
- name: Checkout code
Expand Down
8 changes: 4 additions & 4 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,14 @@ RUN apt-get update && apt-get install -y software-properties-common gcc
ARG __version__

RUN apt-get update && apt-get install -y \
python3.8-distutils \
python3.8-dev \
python3.9-distutils \
python3.9-dev \
python3-pip \
python3.8-venv \
python3.9-venv \
curl \
# libdb5.3-dev \
default-jdk-headless
RUN python3.8 -m venv /opt/venv
RUN python3.9 -m venv /opt/venv
RUN /opt/venv/bin/pip install wheel requests gunicorn
COPY dist/whyis-$__version__.tar.gz /opt/whyis-$__version__.tar.gz
RUN /opt/venv/bin/pip install /opt/whyis-$__version__.tar.gz
Expand Down
298 changes: 298 additions & 0 deletions PACKAGE_UPGRADE_GUIDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,298 @@
# Package Upgrade Migration Guide

## Overview

This guide documents the major package upgrades in Whyis and provides migration instructions for developers and users.

## Major Package Upgrades

### Flask Ecosystem (Flask 1.x → 3.x)

The Flask ecosystem has been upgraded to version 3.x with all compatible dependencies:

- **Flask**: 1.x → 3.0+
- Flask 3.x is backwards compatible with most Flask 1.x code
- Removed deprecated APIs (like `flask._compat`, `_request_ctx_stack`)

- **Jinja2**: 2.11.3 → 3.1+
- Mostly backwards compatible
- Some template syntax edge cases may behave differently

- **Werkzeug**: 2.0.3 → 3.0+
- API is mostly compatible
- `__version__` not exposed at top level in 3.x (not an issue for normal usage)

- **itsdangerous**: <2.0 → 2.2+
- API compatible with 2.x

- **markupsafe**: 2.0.1 → 3.0+
- Compatible with Jinja2 3.x

### Flask Extensions

- **Flask-Security → Flask-Security-Too**: 3.0.0 → 5.3+
- Drop-in replacement, import name stays `flask_security`
- No code changes required
- Note: `encrypt_password()` is now `hash_password()` (but old name still works)

- **Flask-Login**: 0.5.0 → 0.6+
- **Flask-WTF**: <0.15 → 1.2+
- **Flask-Caching**: 1.10.1 → 2.3+
- **Flask-Script**: 2.0.6 (kept for backwards compatibility with patches)
- Deprecated, but patched for Flask 3.x compatibility
- New Click-based CLI available via `whyis-cli` command

### RDF and Semantic Web

- **rdflib**: 6.3.2 → 7.0+
- Major version upgrade
- API is backwards compatible for most use cases
- Some plugin changes (should not affect normal usage)
- All Whyis code works with rdflib 7.x

- **oxrdflib**: 0.3.1 → 0.3.7 (last 0.3.x version, compatible with rdflib 7.x, no Rust required)
- **sadi**: (unversioned) → 1.0.0
- **setlr**: >=1.0.1 (kept constraint)
- **sdd2rdf**: >=1.3.2 → >=1.6.0

### Data Processing

- **beautifulsoup4**: 4.7.1 → 4.12+
- Backwards compatible

- **numpy**: (unversioned) → 1.22.0+ (2.0+ compatible with Python 3.9+)
- NumPy 2.0+ requires Python 3.9+
- No upper bound constraints needed with Python 3.9 minimum

- **pandas**: (unversioned) → 2.0+ (requires Python 3.9+)
- Pandas 2.0+ requires Python 3.9+
- Latest version available with Python 3.9+

- **scipy**: (unversioned) → 1.10+ (1.11+ requires Python 3.9+)
- SciPy 1.11+ requires Python 3.9+
- Latest version available with Python 3.9+

- **lxml**: (unversioned) → latest
- **nltk**: 3.6.5 → 3.9+

### Other Utilities

- **celery**: <6.0.0 → >=5.4.0,<6.0.0
- **Important**: Celery 5.x requires updated command syntax
- Embedded celery commands automatically use new syntax (`wsgi:celery` instead of `wsgi.celery`)
- If running celery manually, use: `celery -A wsgi:celery worker`

- **eventlet**: >=0.35.2 (kept, latest 0.39.1 compatible with Python 3.9+)
- **dnspython**: 2.2.1 → 2.8+
- **email_validator**: 1.1.3 → 2.3+
- **cookiecutter**: 1.7.3 → 2.6+
- **bibtexparser**: 1.1.0 → 1.4+
- **filedepot**: 0.10.0 → 0.11.0
- **ijson**: 2.4 → 3.3+
- **puremagic**: 1.14 → 1.28+

## Flask-Script to Flask CLI Migration

### Background

Flask-Script is deprecated and incompatible with Flask 3.x. We've taken a two-pronged approach:

1. **Backwards compatibility**: Added compatibility patches to make Flask-Script work with Flask 3.x
2. **Modern CLI**: Created new Click-based CLI for future use

### Using the `whyis` Command (Click-based, Recommended)

**As of this upgrade, `whyis` is now the modern Click-based CLI.** The previous Flask-Script version is available as `whyis-legacy` for compatibility.

```bash
whyis run
whyis createuser -u admin -p password
whyis load data.ttl
```

**Available commands:**
- `run` - Run development server with embedded services
- `createuser` - Create a new user
- `updateuser` - Update an existing user
- `load` - Load a nanopublication from file
- `retire` - Retire a nanopublication
- `backup` - Backup the application
- `restore` - Restore from backup
- `init` - Initialize the application
- `sanitize` - Sanitize the knowledge graph
- `test` - Run tests
- `runagent` - Run a specific agent

### Using the Legacy `whyis-legacy` Command (Flask-Script, Backwards Compatibility)

The old Flask-Script-based command is still available as `whyis-legacy` for backwards compatibility:

```bash
whyis-legacy run
whyis-legacy createuser -u admin -p password
whyis-legacy load data.ttl
```

This version includes Flask 3.x compatibility patches that inject missing Flask APIs:
- `flask._compat` module
- `flask._request_ctx_stack`
- `flask._app_ctx_stack`

**Note:** The `whyis-legacy` command is provided for transition purposes. New scripts and documentation should use the `whyis` command.

### Subprocess Management

Both CLIs preserve the important subprocess management:
- **CleanChildProcesses**: Process group management for clean shutdown
- **Embedded Celery**: Automatic Celery worker spawning
- **Embedded Fuseki**: Fuseki server management
- **Webpack watching**: Frontend build process management

## Python Version Support

- **Minimum Python version**: 3.8 (changed from 3.7)
- **Tested versions**: 3.8, 3.9, 3.10, 3.11 (per CI configuration)
- Python 3.12 should also work but is not officially tested in CI

## Breaking Changes

### None for Normal Usage

For typical Whyis usage, there should be no breaking changes. All tests pass with the upgraded packages.

### Potential Edge Cases

1. **Flask-Script deprecation**: If you've extended Flask-Script commands, you may want to migrate to Click-based commands

2. **Direct use of deprecated Flask APIs**: If your custom code uses:
- `flask._compat`
- `flask._request_ctx_stack`
- `flask._app_ctx_stack`

You'll need to either update your code or ensure the compatibility patches are loaded.

3. **rdflib plugin changes**: If you've written custom rdflib plugins, test with rdflib 7.x

4. **Template edge cases**: Some Jinja2 3.x template behaviors may differ slightly from 2.x

## Python Version Support

- **Minimum Python version**: 3.9 (changed from 3.8)
- Python 3.8 reached EOL in October 2024
- eventlet 0.36+, numpy 2.0+, pandas 2.0+, and scipy 1.11+ all require Python 3.9+
- **Tested versions**: 3.9, 3.10, 3.11 (per CI configuration)
- Python 3.12 should also work but is not officially tested in CI

### Why Python 3.9 Minimum?

Python 3.8 reached end-of-life in October 2024, and many critical packages have dropped support:

- **eventlet**: Version 0.36+ requires Python 3.9+
- **NumPy**: Version 2.0+ requires Python 3.9+
- **Pandas**: Version 2.0+ requires Python 3.9+
- **SciPy**: Version 1.11+ requires Python 3.9+

Moving to Python 3.9 allows using the latest versions of all dependencies without constraints, providing:
- Latest security fixes
- Better performance
- Modern Python features
- Active upstream support (Python 3.9 EOL: October 2025)

## Testing Your Application

After upgrading, run your test suite:

```bash
# Run unit tests
pytest tests/unit/

# Run all tests
pytest

# Run with coverage
pytest --cov=whyis --cov-report=html
```

## Migration Checklist

- [ ] Update `requirements.txt` or `setup.py` to use new package versions
- [ ] Run your test suite to ensure no regressions
- [ ] Test critical user workflows
- [ ] Update any custom Flask-Script commands to Click (optional, Flask-Script still works)
- [ ] Test embedded services (Celery, Fuseki) work correctly
- [ ] Verify embedded Celery starts successfully
- [ ] Check that background tasks execute properly
- [ ] Check that all autonomous agents function properly
- [ ] Verify nanopublication loading and management
- [ ] Test user authentication and authorization

## Celery 5.x Important Changes

### Command Syntax Change

Celery 5.x requires explicit attribute access syntax with colon notation instead of dot notation.

**Old syntax (Celery 4.x):**
```bash
celery -A wsgi.celery worker --beat
```

**New syntax (Celery 5.x):**
```bash
celery -A wsgi:celery worker --beat
```

The key change: `wsgi.celery` (dot notation) → `wsgi:celery` (colon notation)

### Running from kgapp vs Standalone

**From a kgapp directory** (created by cookiecutter):
```bash
# The kgapp has a local wsgi.py that imports from whyis.wsgi
celery -A wsgi:celery worker --beat
```

**From elsewhere** (no local wsgi.py):
```bash
# Use full module path
celery -A whyis.wsgi:celery worker --beat
```

### Impact on Whyis

- **Embedded Celery**: Automatically detects local wsgi.py and uses correct syntax
- **Cookiecutter kgapps**: Have a local `wsgi.py` that imports from `whyis.wsgi`
- **wsgi.py**: Now always exports a `celery` variable to prevent import errors

### Troubleshooting Celery Issues

If you see "The module wsgi:celery was not found" error:

1. Check if you're in a kgapp directory with a local `wsgi.py` file
2. If yes, use: `celery -A wsgi:celery worker`
3. If no local wsgi.py, use: `celery -A whyis.wsgi:celery worker`
4. Verify whyis package is installed correctly (`pip show whyis`)

If you see "'app' object has no attribute 'celery'" error:

1. Check that `wsgi.py` (or `whyis/wsgi.py`) exports `celery` at module level
2. Verify the application is properly configured (check for `whyis.conf`)

## Getting Help

If you encounter issues:

1. Check if Flask-Script compatibility patches are loaded (for `whyis` command)
2. Try the new `whyis-cli` command as an alternative
3. Review Flask 3.x migration guide: https://flask.palletsprojects.com/en/3.0.x/changes/
4. Check rdflib 7.x release notes: https://github.com/RDFLib/rdflib/releases
5. Open an issue on GitHub with details about your problem

## Benefits of These Upgrades

- **Security**: Latest versions include security fixes
- **Performance**: Newer packages often have performance improvements
- **Python 3.12 support**: Ready for newer Python versions
- **Active maintenance**: All upgraded packages are actively maintained
- **Modern tooling**: Click-based CLI is more maintainable and feature-rich
- **Dependency compatibility**: Better compatibility with modern Python ecosystem
2 changes: 1 addition & 1 deletion TEST_COVERAGE_EXTENSION.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ Created **136 new unit tests** covering core utility modules with **100% code co

#### GitHub Actions Workflow
Created `.github/workflows/python-tests.yml`:
- **Multi-version testing**: Runs on Python 3.8, 3.9, 3.10, 3.11
- **Multi-version testing**: Runs on Python 3.9, 3.10, 3.11
- **Separate test suites**: Unit tests and API tests run independently
- **Code coverage**: Integrated with Codecov for coverage tracking
- **Artifact upload**: Test results and coverage reports saved
Expand Down
Loading
Loading