information exchange on observations and questions

[foxthefox]

Thanks for sharing your work.
I will make my own setup as well.

Have you checked if the credentials used from the devices to your local server are the same as the one which are used in the APP or the adapter/integrations

I would guess that it is the same, except that the devices not have an "Android-Id"
You can compare against
https://energychain.github.io/site_ecoflow_mqtt_credentials/

Interestingly the devices uses different topics

/sys/.../thing/protobuf/downstream
/sys/.../thing/rawData/downstream
/sys/.../thing/property/cmd
/sys/.../thing/property/set

than the APP:

/app/ + mqttUserId / + device +/thing/property/get_reply
/app/ + mqttUserId / + device + /thing/property/get
/app/ + mqttUserId / + device + /thing/property/set_reply
/app/ + mqttUserId / + device + /thing/property/set
/app/device/property/ + device

So it seems that the MQTT Server is processing the telegrams (some kind of merge and mirroring).
That makes it a little bit more complicated to know which topic is used for what.
Basically I think that the already discovered and implemented protobuf is used in the same way, I do not expect that the mqtt-server also translates the telegram content.

To see the mapping between the topics or better to say which is the topic to send cmds from local server to device, the local server must be in the middle. Devices subscribe and publish to local server and local server subscribes/publishes to the cloud server.
In such case the credentials are used only once from the local server and not in parallel from the devices.
If this is successful than the commands can be observed hopefully and the used topic.

[tomvd]

Thanks for your information. The server in the middle is a good idea, I will have a go at that later.

[tomvd]

I went another route to get the downstream commands, I already had a node-red plugin that uses the ecoflow API to set the powerstreams power (when it was connected to the internet).
Now I connected to the official ecoflow MQTT with the devices credentials, and I did see this topic appear:
/sys/75/HW51xxx/thing/property/cmd

When I set the powerlevel to 10, I get this payload of 64 bytes:
0D 0A 3C 0D 0A 02 08 64 10 20 18 35 20 01 28 01 38 03 40 14 48 00 01 50 02 58 01 70 00 00 00 00 06 00 01 13 00 01 01 00 01 03 69 6F 73 00 01 10 48 57 35 31 (removed last 12 bytes as these are part of my device id)

When I set the powerlevel to 0, I get this payload of 64 bytes:
0D 0A 3C 0D 0A 02 08 01 10 20 18 35 20 01 28 01 38 03 40 14 48 00 01 50 02 58 01 70 00 00 00 00 06 00 01 13 00 01 01 00 01 03 69 6F 73 00 01 10 48 57 35 31 (removed last 12 bytes as these are part of my device id)

Then I set the powerlevel to 500, suddenly the payload is 65 bytes:
0D 0A 3D 0D 0A 03 08 00 27 10 20 18 35 20 01 28 01 38 03 40 14 48 00 01 50 03 58 01 70 00 00 00 00 06 00 01 13 00 01 01 00 01 03 69 6F 73 00 01 10 48 57 35 31 (removed last 12 bytes as these are part of my device id)

I don't have time to investigate further now, but I bet these are protobuf messages we can find out which ones, and we have the topic to publish them on. So, next I wanted to test out if I publish these, the powerstream will actually act on them. But I was a bit scared that the MQTT explorer could not send binary properly and something bad would happen to the powerstream, so I did not do this yet,

[foxthefox]

Do I understand it correctly, that you have rerouted the whold traffic to EF-MQTT to your local broker?
Or did you only reroute the IP-addresses of the stream and station?
If you can use your node-red then you can also use the EF-APP to change values.

I could not find some suitable information in the payload bytes.
Maybe it is a matter of how the payload is treated usually I use HEX.

With https://protobuf-decoder.netlify.app you can check it by yourself, it also accepts base64 strings.

Assuming that the /cmd path is only for commanding to the devices and no return path then it is strange to have a big telegram of 500 bytes captured. Maybe that is the telegram keeping the devices alive for data transmission, but also relatively large for that.

In the app style the command is set in the /set path and in /set_reply is an ack telegram. The next in /property is a status one conataining the values.

Edit:
A telegram sending 600W to the pstream (via /app.../set) looks like this
0a 3d 0a 03 08 f0 2e 10 20 18 35 20 01 28 01 38 03 40 14 48 81 01 50 03 58 01 70 cf fc e9 51 80 01 13 88 01 01 ba 01 03 69 6f 73 ca 01 11 48 57 35 31 35 35 35 35 35 47 31 32 33 34 35 36 37
(including a fake serial number)

Which translates to these values (applying the Header protobuf definition)
{"src":32,"dest":53,"dSrc":1,"dDest":1,"checkType":3,"cmdFunc":20,"cmdId":129,"dataLen":3,"needAck":1,"seq":171605583,"version":19,"payloadVer":1,"from":"ios","deviceSn":"HW5155555G1234567"}
and in the first ID the pdata is the set value.
Important is "cmdFunc":20,"cmdId":129 which is the combination to send permanentWatts.

[tomvd]

To be clear, I did not reroute traffic. I just connected with the credentials of my powerstream to the ef mqtt

[foxthefox]

But I thought that you have the local-mqtt running and therefore you made dnsrewrite in Adguard.
So it is different for capturing the cmds. The devices are directly communicating with the EF-cloud and you made a subscription to the /sys/-topics with the device credentials.
That means also that the EF-cloud does not throw you out when there is a second sebscription with the same device.

Now, when you capture the /sys/..cmd with the node-red node, then you should adjust the setting to base64 or hex instead of auto or JSON.

[tomvd]

Now that I see your payload, I think something went wrong when copy pasting the string, the 0D 0A I wrote should be 0A instead. Going from that, it fits better the proto format of a message, but I already see the source id and destination id are different, src is 53 and dst is 1. Which kinda makes sense, knowing that you have a message going from the app to the ef-mqtt, and I captured the message from ef-mqtt to the device.
So, following that logic, the app has id 32, ef-mqtt has id 53 and the device id 1?
Messages from the app: scr:32,dst:53. Messages to the device:scr:53,dst:1

The values seem to match, they are watts*10, but it does not 0, it always sends 1 to turn the output off. This logics matches with what I found in the nodered ecoflow code:

export function encodePowerStreamSetAcWatts(parser: IParserResult, watts: number, deviceSn: string): Buffer {
return encodeSetMessage(parser, {
pdata: { value: Math.max(1, Math.round(watts*10)) },
src: 32,
dest: 53,
dSrc: 1,
dDest: 1,
checkType: 3,
cmdFunc: 20,
cmdId: 129,
needAck: 1,
version: 19,
payloadVer: 1,
deviceSn,
});

To be continued in the evening :)

[foxthefox]

Good to know, when to hook up to the local-mqtt that:
Device: id =1
APP: id = 32
EF-MQTT: id=53
and may be that brings some light into the meaning of dSrc and dDest.

[tomvd]

Sorry, I was doing something wrong there, I copied over the string from MQTTExplorer, but it decoded the bytes wrongly. I needed an MQTT client that shows binary and used MQTTx, there I took the base64 string and put it in https://www.protobufpal.com/ using the setMessage proto format, and it succesfully parsed it into this:
{
"header": {
"pdata": {
"value": 100,
"value2": 0
},
"src": 32,
"dest": 53,
"d_src": 1,
"d_dest": 1,
"enc_type": 0,
"check_type": 3,
"cmd_func": 20,
"cmd_id": 129,
"data_len": 2,
"need_ack": 1,
"is_ack": 0,
"seq": 1743940398,
"product_id": 0,
"version": 19,
"payload_ver": 1,
"time_snap": 0,
"is_rw_cmd": 0,
"is_queue": 0,
"ack_type": 0,
"code": "",
"from": "ios",
"module_sn": "",
"device_sn": "HW51xxxxxxxxxxxx"
}
}

But if I publish that on the cmd topic, the device does not change its powerlevel to 10 watts, instead the device publishes a very similar payload on the thing/property/cmd_reply topic:

{
"header": {
"pdata": {
"value": 0,
"value2": 0
},
"src": 53,
"dest": 32,
"d_src": 1,
"d_dest": 1,
"enc_type": 0,
"check_type": 3,
"cmd_func": 20,
"cmd_id": 129,
"data_len": 0,
"need_ack": 1,
"is_ack": 0,
"seq": 1743940398,
"product_id": 0,
"version": 3,
"payload_ver": 3,
"time_snap": 0,
"is_rw_cmd": 0,
"is_queue": 0,
"ack_type": 0,
"code": "",
"from": "",
"module_sn": "",
"device_sn": "HW51xxxxxxxxxxxx"
}
}

In the seq field I try to enter the unix epochtime (in seconds), thinking it might discard messages that are too old or something.
But nothing makes the device change its output power at the moment.
I tried sending that cmd_reply to the EF mqtt, but it does not do anything with it. I guess it is the return value that is send to the app to confirm the device changed it power to whatever you asked to set it too.

I have no idea what my next step is to be honest. I do not know what the device needs more than the command to do its thing, but it appears there is more to it...

A wild guess is that the device knows there is no valid connection to the app or the ef mqtt, and thus discarding any commands it receives? Some kind of extra security that is built in.

I noticed that if I connect to the ef mqtt impersonating a device, I get 4 (identical?) messages in property/cmd - as some kind of welcome message. I thought I need to send such a welcome message, to put the device in an "online" state, so I tried by copy pasting one and putting it on my local mqtt. (which might not work if that timestamp check is strict) , but so far no luck with that either.

I am giving it a rest for now.

[foxthefox]

I am not aware of a second value to be transmitted within the permanentWatts command.
Only the value at "position"=1 is used when making a command from the "APP" to the topic /app/.
So it seems that the transition from /app to /sys within the EF-cloud is doing a little bit more than taking the telegram and putting it in the other topic. Still not sure of it.
And looking at the previous ids, shouldnt be the cmd telegram be in the format src:53 dst:1 when it is a telegram from EF-mqtt to device?

Anyhow, please check the captured telegram with https://protobuf-decoder.netlify.app/ additionally.
That does not translate nicely, but you will see if there is more inside pdata.
If the proto file does not contain all possibilities (field#) then it will not show the unknown things after parsing.
So it might be that there is also a field#=3 inside the telegram. Or others.
It might be that the "internal" communication has slightly different proto definitions.

[tomvd]

(the dst 1 was wrongly decoded by me, the messages always have 32 or 53 in src or dst)

I have checked the captured telegram with that protobuf decoder, but there was nothing more inside pdata. The telegram that I injected on the cmd topic on my local mqtt is exactly the same telegram that is put on the cmd topic by the ecoflow mqtt.

My guess is that it somehow knows it is not connected to the ecoflow server (maybe because it does not receive acknowledgements on the telegrams it sends out) and because of that is not accepting the command to change its output power.
Also, if I do not acknowledge a message, the device only send a new message after 46 seconds. While if it gets acknowledgements, the telegrams follow up much faster. That is somehow evidence it is aware of the presence of an actual server.

The way I acknowledge a message now is to copy paste its last small cmd_id 134 telegram in the upstream of the local mqtt, Publish this message in the upstream topic of the ecoflow mqtt. Then I get a reply in the downstream topic of the ecoflow mqtt, which I copy-paste in the upstream of the ecoflow mqtt. This triggers the device to send its bigger data telegrams again.

These kind of short messages without too much meaningful data in it that seems to come back very often (and are repeated, as if it is retrying when no answer) have command id 134. But the pdata in it is very strange and I do not see a pattern. So it be very hard to try to mimic these.

Anyhow, if this set command to set the power is really this protected, it feels like they did that on purpose and for some reason do not want this kind of local control of the device...

[foxthefox]

Very interesting.
May be that is some kind of protection mechanism, and may be that is a part of that function when user report that you have to have the APP open, otherwise it stops with update messages.

BTW. cmdId = 134. cmdFunc=20 is the time_task_config_post

this should help to decode:

  message time_range_strategy {
	optional bool is_config = 1; // Whether to enable: 0: no; 1: yes
	optional bool is_enable = 2; // Whether to configure: 0: no; 1: yes
	optional int32 time_mode = 3; // Scheduled mode: 0: daily; 1: weekly; 2: monthly; 3: do not repeat
	optional int32 time_data = 4; // mode == 1, bit0-bit6 represents Monday to Saturday respectively; mode == 2, bit0-bit30 represents the 1st to the 31st
	optional rtc_data start_time = 5;
	optional rtc_data stop_time = 6;
  }

  message time_task_config {
	optional uint32 task_index = 1;
	optional time_range_strategy time_range = 2;
	optional uint32 type = 3; // Task type: 1: prioritize power supply; 2: prioritize power storage
  }

  message time_task_config_post {
	optional time_task_config task1 = 1;
	optional time_task_config task2 = 2;
	optional time_task_config task3 = 3;
	optional time_task_config task4 = 4;
	optional time_task_config task5 = 5;
	optional time_task_config task6 = 6;
	optional time_task_config task7 = 7;
	optional time_task_config task8 = 8;
	optional time_task_config task9 = 9;
	optional time_task_config task10 = 10;
	optional time_task_config task11 = 11;
  }
  message rtc_data {
	optional int32 week = 1;
	optional int32 sec = 2;
	optional int32 min = 3;
	optional int32 hour = 4;
	optional int32 day = 5;
	optional int32 month = 6;
	optional int32 year = 7;
  }

  message time_task_config_ack {
	optional uint32 task_info = 1;
  }

The small ACK telegram on cmdId=134 could be the "time_task_config_ack", but this would be then on another topic I guess (cmd_reply or so).

If you get large time_task telegrams, then most likely you have also several task setup in the APP. With the above it should be the protobuf coding of the tasks. Otherwise we have to earch for the content, One proof would also be to add/delete tasks in the APP, the telegram should also change the amount.

I still have problems to understand the ACK mechanism. Usually a command is send to the device (ie. pstream) and within this command is a request for ACK (need_ack = 1 (pos 11)) and then the device sends a telegram with (is_ack = 1 (pos 12) and seq# taken from the command) but no request for ACK from the server (no need_ack (pos11) inside the telegram). That is what happens on the APP side with the topics set and set_reply.
So it looks that on the "internal" communication is also the request that the server sends an ACK because of request of the device to send ACK to the ACK telegram? Is the seq# changing then?

Please also observe the position 6, enc_type. If this is set to 1, then you have a encrypted pdata.

[tomvd]

Ok, perhaps it just asks for changes in the time tasks very often then. To make sure they are up to date. I have one task set up for my ecoflow river, so I am not sure why the powerstream needs to know that.

Also another thing I noticed, the moment the device connects, the ecoflow server immediatly posts a cmd_id 10 telegram in the cmd topic, and in pdata the number 15. (it tries 3 times with a 5 second pause)
If I send this telegram the device answers with a dozen of upstream data stuff, I think that cmd_id is kind of "hi device, how are you, please give me a full update"
It replies with a combined messag of 9 telegrams, 2 big cmd134 telegrams, a cmd12,cmd14, a cmd136,cmd134, (those were all func20) then a func254cmd32 telegram.
Probably the cmd12 and cmd14 are interesting, as they seem to be unknown in the app?
They both have a similar 8 field pdata structure(3 varint fields, 2 bytes fields), there content is: 3,0,64,7c 2c 67 c0 8e 58, a0 00 03 df 06 d3 (field 6 and 7 empty) and then field 8 is 1.
The cmd14 has exactly same pdata, exept the first value is 1 except of 3. I think the bytes are the mac address of my device. No idea why that is sent to the server though.

[foxthefox]

Have you had the chance to decode the cmdId134 telegrams based on the proto above?
cmdId=136 is the PowerPack, and cmdId32/cmdFunc should be the EnergyPack.

cmdId10 could be a welcome message, not known so far.

Indeed the cmdId12 and cmdId14 seem to be of most interest.
These I did not have to guess in the past :-) because not seen yet.

May be it is network_message, as you said. Have a look here:
https://github.com/rabits/ef-ble-reverse/blob/main/protobuf/wn_synchronous.proto
and there is also mac mentioned.

Do you have mesh enabled?

Another question.
Within the APP-cloud communication is a distinguishing between get and get_reply and set and set_reply and "update".
If you write about cmd and its replys, does it happen always on the same channel cmd and cmd_reply? So everything an the same topic? Usually the "get_updates" is sent on the "get" topic and only the real commands are sent through "set" topic.

[tomvd]

Did not have the chance to decode the cmd134 telegrams yet.
I don't have mesh enabled as far as I know (don't have any ecoflow plugs or so)

About the topics I can give you the answers:
I see 4 topics being used in the communication between the device and the server:
/sys/.../thing/protobuf/upstream
/sys/.../thing/protobuf/downstream
/sys/.../thing/property/cmd
/sys/.../thing/property/cmd_reply
When the server initiates a telegram it publishes it on the cmd topic. The device publishes an acknowledgement in the cmd_reply topic.
When the device sends a telegram it publishes it on the upstream topic, and the server replies with an acknowledgement in the downstream topic.
The acknowledgement telegram is a copy of the original command, but with src/dst switched and a single return value in pdata.

Thanks for letting me know the BLE communication also uses similar protobuf format, I did not even know that. I always had in my head that going the BLE route would be a plan B to control my powerstream, because I do not know a lot about BLE communication.