Remote Working, iOS, Mac OS X, and more

RV

Victron Venus Data Monitor

Satisfied with the performance and reliability of the dashboard for my Victron system, I’m finally ready to release the source code. The data collector is written in PHP and implemented as a pair of console commands within the Laravel framework, one that queries the ModbusTCP service for the bulk of the data and the other that queries the MQTT broker for the one piece of data I want that’s not available from ModbusTCP: the GPS altitude.

I run it within a small Ubuntu VM on my dev hardware (its ideal environment), but there’s no reason it shouldn’t be able to run on a variety of system types so long as its requirements are met and it can reach the VenusOS device on the network (a Victron Color Control GX in my case). Provided with the code are two systemd service definitions that allow it to run as a daemon. I plan to eventually try a permanently-installed Raspberry Pi so no data is lost when the dev hardware is shut down to be moved in and out of the trailer.

Victron Venus Data Monitor Repository

Technical Details

Why Laravel? Two reasons: I wanted to try it out, and I have plans beyond just the data collection that it should be well-suited for. I’ve developed plenty of applications both with and without frameworks, and it’s a big timesaver to have something handle the boilerplate pieces for me.

PHP is not a language commonly used for long-living background processes, though it is definitely capable of being used for that. It’s easily managed as a systemd service like any of the other services on a machine. The two service definitions included launch and maintain one PHP process for each command, logging their output to a file in /var/log.

MQTT

The MQTT command is itself pretty simple. It connects to the broker running on the VenusOS device, subscribes to one or more notification topics, and records any broadcasts for those notifications within an InfluxDB database.

php artisan mqtt:monitor {host} {--port=1883}

Messages relating to the Venus MQTT broker all follow the pattern {action}/{identifier}/{subsystem}/{index}/{keypath}. The action may be one of R to read an existing value, W to write a new value, or N for a notification of a value changed. An example notification with the topic N/abcdef123456/vebus/257/Ac/ActiveIn/L1/V would have a value like 120.18000030518 (in this case that’s Volts).

Initially, though, only one notification is available, which is used to get the system identifier that is needed to wake the broker from sleep. It can be received by subscribing to the topic N/+/system/0/Serial (in a MQTT topic, a + is a single component wildcard while # is a multi-component wildcard) and will have a value equal to the system identifier doing the broadcast (a string of 12 hex characters).

Once the system identifier is retrieved, the keep-alive message can be sent. The MQTT broker only publishes subscribed notifications for up to 60 seconds, going to sleep if another keep-alive isn’t sent. The keep-alive message is of the topic R/{identifier}/system/0/Serial with an empty value, and the code handles automatic sending of it.

A reasonably complete list of available topics may be found here.

Note: The MQTT broker can theoretically have any number of topic subscriptions for a given receiver, but in practice, it will stop broadcasting them at all after a small number are added. To work around this, you can subscribe to everything using the wildcard syntax (e.g., N/{system}/#) and filter them in code.

Additionally, Victron’s documentation for the MQTT broker states that any notification sent will be forwarded to their cloud-based broker. This can consume bandwidth you’d rather not use.

ModbusTCP

Unlike MQTT’s verbose XML syntax, ModbusTCP is a binary protocol. Each value is stored in its packed binary form in a register with a numeric ID rather than key-value pairs. How the value is interpreted (e.g., as an integer or a string) is based on the ID of the register and the measurement associated with it. All values are big endian.

Because it is a lower-level protocol, no support exists for a notification-based system like MQTT. Instead, it relies on periodic polling, and the command is built to support that.

php artisan modbus:query {host} {--port=502} {--daemon} {--interval=5}

The guts of the command use a hard-coded list of registers to poll. They cover all of my information needs, but it’s easy to modify them within the code to add or remove data points from the polling. A full listing of register/data mappings is available here.

Each message is two bytes for the message ID (in this code, it’s a simple numeric counter), two bytes for the protocol (always 0), two bytes for the length of the request (these first six bytes, the header, are not included), one bytes for the unit, one bytes for the function (e.g., read or write — this code only currently reads), two bytes for the first register to read, and finally two bytes for the number of registers to read from the first. In addition to the magic values for the register IDs, the unit and function both have their own special values:

$unit may be one of:

  • 100 (the core system, any value of service name com.victronenergy.system)
  • 246 (CCGX VE.Bus port)
  • 247 (CCGX VE.Direct 1 port)
  • 245 (CCGX VE.Direct 2 port/Venus GX VE.Direct 1 port)
  • 243 (Venus GX VE.Direct 2 port)
  • 242 (Venus GX VE.Bus port)

$function may be one of:

  • 3 (ReadHoldingRegisters)
  • 4 (ReadInputRegisters, identical to ReadHoldingRegisters)
  • 6 (WriteSingleRegister)
  • 16 (WriteMultipleRegisters)

The response will have the same header: the message ID, the protocol, and the remaining length. Following that is the unit and function passed, a single byte checksum, and the contents of the registers. The code does not currently validate the checksum.

From here, it parses the register contents according to the mapping included in the source and returns them to the daemon for recording.

Data Collection

Because reading via ModbusTCP only works with contiguous register blocks, it ends up reading some I don’t care about. Those get discarded. The values that are of interest are defined in the DataPath class and its subclasses, which handles writing both the values and some computed values based on them.

All data is recorded in an InfluxDB database, which is very well-suited for time-series data like Victron system measurements. This is connected directly to a Grafana dashboard for visual representation.

Conclusions

This is written for my specific needs but should generally be expandable or usable as a basis for others. I’d eventually like to ingest data from other sources (e.g., a Raspberry Pi environmental sensor) and repurpose an old iPad or two as dedicated monitors.

That said, this is pretty much wholly unnecessary to actually use the system, but it interests me, so that’s why I do it. I find value not just in using something but also in grokking everything about it.