Skip to content

Comments

Update MutuallyHuman esp-idf changes to esp-idf v5.5.1#11

Merged
rockys-spindance merged 15 commits intomhs_changes_v5.5.1from
working_mhs_changes_v5.5.1
Oct 7, 2025
Merged

Update MutuallyHuman esp-idf changes to esp-idf v5.5.1#11
rockys-spindance merged 15 commits intomhs_changes_v5.5.1from
working_mhs_changes_v5.5.1

Conversation

@rockys-spindance
Copy link
Collaborator

@rockys-spindance rockys-spindance commented Sep 19, 2025

Creating a new branch with all our MutuallyHuman esp-idf changes applied to the Espressif v5.5.1 tag.

Migration Guides and notes

The top-level migration guides can be found here
https://docs.espressif.com/projects/esp-idf/en/stable/esp32/migration-guides/index.html

Note that pretty much each migration step between v5.0 and v5.5.1 updates the GCC compiler and therefore can affect warning behavior.

Specific migration points to note

Here are some of the more important changes. None requires an explicit change in our code but we highlight them here in case future maintainers run into surprising behavior:

* turning on Flash Encryption will no longer automatically turn on NVS Encryption.
    https://docs.espressif.com/projects/esp-idf/en/stable/esp32/migration-guides/release-5.x/5.2/storage.html

* driver component has been split (largely transparent change unless we someday need the granularity)
    https://docs.espressif.com/projects/esp-idf/en/stable/esp32/migration-guides/release-5.x/5.3/peripherals.html
    
* newer version of Unity may impact some minor test syntax
    https://docs.espressif.com/projects/esp-idf/en/stable/esp32/migration-guides/release-5.x/5.3/system.html
    
* WiFi now supports WPA3-Enterprise authentication
    https://docs.espressif.com/projects/esp-idf/en/stable/esp32/migration-guides/release-5.x/5.4/wifi.html
    
* Mbed TLS change  now requires explicit SHA mode set
    https://docs.espressif.com/projects/esp-idf/en/stable/esp32/migration-guides/release-5.x/5.5/security.html

Version rebase process

The working_mhs_changes_v5.5.1 branch was created with the following steps:

  1. Update our fork with the release/v5 branch from Espressif repo along with all the v4x and v5x tags.
  2. Start with spindance_changes_v5.0 branch. Create a squashed single patch from this HEAD vs the v5.0 tag.
  3. Checkout v5.5.1 and create new branch there
  4. Apply the patch (this will apply the easy changes and generate *.rej files for all the non-trivial ones)
  5. Manually fix all the changes left in the *.rej files (claude code did most of the heavy lifting)
  6. (In a separate repo and PR) - update our callbox application code as needed to build and work with the new esp-idf version (see https://github.com/mhs/callbox-embedded/pull/219)

If there are questions about these steps, you may find Rob Hartman's more detailed notes in the v5.0 uprev PR of use. See #7

Additional Changes

  • Add new files that were added on the spindance_changes_v5.0 branch - components/json/json.mk and SpinDance_Fork_Changes.md
  • Rename SpinDand_Fork_Changes.md -> MutuallyHuman_Fork_Changes.md
  • Update MutuallyHuman_Fork_Changes.md for v5.5.1
  • Remove the github/workflows/dangerjs.yml inherited from v5.5.1
  • Fix pre-commit check issues

Verifying the diff of diffs

To verify that we did not lose any MH changes in the port, we did a diff of the diffs. Specifically, the two diffs are:

  1. spindance_changes_v5.0 against the v5.0 tag (you can see this conveniently in PR#7
  2. this branch working_mhs_changes_v5.5.1 against the v5.5.1 tag.

SUMMARY: The only possibly functional difference is the addition of authorization checks inside wifi_config.c and wifi_scan.c. Since JWT authentication is empirically working, the addition of those checks seems safe.

Here we compare diffs by feature :

Network (NW) Metrics Reporting

        components/esp_netif/include/lwip/esp_netif_net_stack.h     //  changes applied exactly
        components/esp_netif/lwip/netif/wlanif.c                    //  changes applied exactly

Protocomm BLE Connectivity Reporting

        protocomm_security.h                                        //  changes applied exactly
        protocomm_ble.h                                             //  changes applied exactly
        protocomm.c                                                 //  changes applied exactly
        security1.c                                                 //  changes applied exactly
        protocomm_nimble.c
            // The changes to report PROTOCOMM_BLE_PEER_DISCONNECTED and PROTOCOMM_BLE_PEER_CONNECTED events applied exactly.
               Note that v5.0 added protocomm_transport_ble_event_t type that had two out of the 3 events that we added in our fork change to add ble_event.
               But since they did NOT include the secure connection event, we continue to use our fork changes. As a result, you'll see a little duplication
               of events in this file between our ble_event and their protocomm_transport_ble_event_t.
               We should keep an eye on the evolution of protocomm_transport_ble_event_t and see if they ever add the secure connection, at which point we
               would be able to undo our fork changes and be closer to the unmodified esp-idf.

JWT Authorization for Protocomm WiFi Provisioning

        wifi_config.proto                                           //  changes applied exactly
        wifi_scan.proto                                             //  changes applied exactly
        wifi_config.pb-c.h                                          //  changes applied exactly
        wifi_config.pb-c.c                                          //  changes applied exactly
        wifi_scan.pb-c.c                                            //  changes applied exactly
        wifi_scan.pb-c.h                                            //  changes applied exactly
        wifi_config_pb2.py                                          //  changes applied exactly
        wifi_constants_pb2.py
            // Mostly changes applied exactly, including the "ifiConnectedState = _reflection.GeneratedProtocolMessageType" paragraph. But the v5.5.1 port
            // added a "WifiAttemptFailed = _reflection.GeneratedProtocolMessageType('WifiAttemptFailed',"  paragraph as needed in the v5.5.1 code.
        wifi_scan_pb2.py                                            //  changes applied exactly
        manager.h                                                   //  changes applied exactly
        manager.c
            // The code structure around our line "ESP_LOGI(TAG, "Delaying %lu ms", cleanup_delay);" changed but the logging was added into the new structure.
            // Otherwise, all changes applied exactly.
        wifi_config.h                                               //  changes applied exactly
        wifi_config.c
            // Our v5.0 fork added to the `wifi_prov_config_data_handler` function the code paragraph "/* Authorize before dispatching command */". And
            // that was applied exactly in the new branch.
            // But in creating our v5.5.1 fork, claude code added to `cmd_set_config_handler` function the code paragraph /* Check authorization if callback is set */
        wifi_scan.h                                                 //  changes applied exactly
        wifi_scan.c
            // Similar to wifi_config.c
            // Our v5.0 fork added to the wifi_prov_scan_handler  function the code paragraph /* Authorize before dispatching command */. And
            // that was applied exactly in the new branch.
            // But in creating our v5.5.1 fork, claude code added to cmd_scan_start_handler  function the code paragraph   /* Check authorization if callback is set */.

WPA2 Enterprise NW Support

        manager.h                                                   //  changes applied exactly
        wifi_provisioning_priv.h                                    //  changes applied exactly

WiFi Provisioning Sequence Changes

        manager.h                                                   //  changes applied exactly
        manager.c // See above

Python Version Change

        idf_tools.py                                                //  changes applied exactly

Testing

Spot checks of esp-idf example builds

Ensured that the following esp-idf/examples/* projects build:

  • examples/storage/nvs_rw_value
  • examples/provisioning/wifi_prov_mgr
  • examples/bluetooth/nimble/bleprph
  • examples/ethernet/basic (builds changes in wlanif.c)

Espressif component unit tests

  • Tried building tools/unit-test-app but it encounters a bunch of formatting errors in many component test source files. Looks like those tests are not well supported (or have not yet been ported for the latest GCC updates).

Testing with callbox-embedded projects

Built and ran all the callbox-embedded projects to ensure they still operate correctly. See the details in the associated callbox-embedded PR

@github-actions

This comment was marked as outdated.

@dhornacek-mh dhornacek-mh self-requested a review October 6, 2025 18:49
Copy link

@dhornacek-mh dhornacek-mh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good here. Thanks for all the context within the PR, nothing stands out me migration wise

@rockys-spindance rockys-spindance merged commit 170fdff into mhs_changes_v5.5.1 Oct 7, 2025
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants