VE-Tools/VE.Direct-Text-Reader
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

initial commit

Browse Source 
with fully functional worker and basic documentation
This commit is contained in:
Aaska Black Wolf
2026-05-14 16:11:47 +02:00
parent 83947dd0a7
commit 556c51440f
3 changed files with 314 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+131 -1
View File
@@ -1,3 +1,133 @@
# VE.Direct-Text-Reader
PHP CLI tool to read messages of the text protocol on the VE.Direct interface.
PHP CLI tool to read messages of the text protocol on the VE.Direct interface.
Intended to be used with VictronEnergy equipment connected via VE.Direct USB cable with an Raspberry Pi running on GNU/Linux.
## Prerequisits
### Prepare USB serial interface
When connecting the VE.Direct USB cable with your computer there shall appear a device like /dev/serial/by-id/usb-VictronEnergy_BV_VE_Direct_cable_VEXXXXXX-if00-port0. Where XXXXX is are random characters.
If you don't see this device the kernel settings may prevent USB serial interfaces from being authorized for use for security reasons.
To change that you need to identify the USB device number of your VE.Direct USB cable first. To do so perform the following commands:
```
lsusb | grep 'ID 0403:6015'
```
This should result in something like this:
```
Bus 003 Device 002: ID 0403:6015 Future Technology Devices International, Ltd Bridge(I2C/SPI/UART/FIFO)
```
The important parts to take from here are the bus and device numbers. In our expample the bus number is 3 and the device number is 2.
To authorize the device you need to change the setting in the usb authorize configuration from 0 to 1:
```
echo 1 > /sys/bus/usb/devices/{bus number}-{device number}/authorize
```
As the first device on the bus is the hub you need to reduce the device number by one.
For our example we replace {bus number} with 3 and {device number} with 1. Ensure you are using the numbers you queried before:
```
echo 1 > /sys/bus/usb/devices/3-1/authorize
```
The device should show up under /dev/serial/by-id/ now.
In the next step set up the serial interface with the proper baud rate. Replace {random characters} according to your actual device name.
```
stty -F /dev/serial/by-id/usb-VictronEnergy_BV_VE_Direct_cable_VE{random characters}-if00-port0 speed 19200 raw -echo
```
Now you can test the serial connection:
```
https://www.twitch.tv/asnowwolf/clip/PhilanthropicElatedSpiderTwitchRaid-Y0u1fmFSjqJHfKYc
```
This should lead to an output like this:
```
PID 0xA10C
FWE 318FF
SER# HQ2535PVZGE
V 13140
I 0
VPV 0
PPV 0
MPPT 0
CS 0
OR 0x00000001
Checksum ▒
ERR 0
LOAD ON
Relay ON
H19 47
H20 45
H21 125
H22 2
H23 15
HSDS 1
Checksum V:A12EE00D57402
:ADBED00220A57
[..]
```
Set up a udev rule to keep the authorize-setting after re-connecting the VE Direct USB cable.
To do so create a new file `/etc/udev/rules.d/99-keep-vedirect-authorized.rules` and open it in your favorite editor. Add the following to the new udev rule file:
```
SUBSYSTEM=="usb", ACTION=="add", ATTRS{idVendor}=="0403", ATTRS{idProduct}=="6015", ATTR{authorized}="1"
```
Test whether the setting is working properly by disconnecting and re-connecting your VE Direct USB cable and then check if the device re-appears under /dev/serial/by-id/.
If everything is working well, we need to make the settings permanent. To do so we will add some lines to root's cron-table. Open it by
```
sudo crontab -e
```
and add the following lines. Keep on mind to replace {bus number}, {device number} and {random characters} again.
```
@reboot sleep 5 && echo 1 > /sys/bus/usb/devices/{bus number}-{device number}/authorized
@reboot echo 1 > /sys/bus/usb/devices/usb{bus number}/authorized_default
@reboot sleep 10 && stty -F /dev/serial/by-id/usb-VictronEnergy_BV_VE_Direct_cable_VE{random characters}-if00-port0 speed 19200 raw -echo
@reboot sleep 15 && stty -F /dev/serial/by-id/usb-VictronEnergy_BV_VE_Direct_cable_VE{random characters}-if00-port0 speed 19200 raw -echo
```
Any commands introduced by `@reboot` will be executed directly when the system was rebooted.
First we ensure that any devices connected to our USB bus number will be authorized. Just to be safe we also tell the device to be authorized explicitly. And finally we set the baud rate. We do that twice as sometimes the setting is not properly set on the first call.
### PHP DirectIO
To read the data from the USB serial interface, we use the PHP low-level library DirectIO. To install it look for a package like `php-dio` in your package manager and install it.
On raspian, do that by typing `sudo apt install php-dio` on your shell.
## Setup
### config.json
We need to define some settings to make the worker script do its job properly. Mainly we need to set a file name for the SQLite database, the log file path and the serial interface name. You can also chose a loglevel between error, warn, info and debug.
To do so, open the file `config.json` in your favorite editor and set up the values. File paths may be absolute (with a leading slash like /path/to/your/file) or relative (like ../path/to/your/file). The SQLite database file shall be placed where other tools can access it for evaluation and processing. Like the web-interface for example.
### cron job to run periodically
To run the worker script periodically we need to set up a new cron-job accordingly. Do so by calling `crontab -e` with the user you want the script to run. If that user is set up without a dedicated shell you can use `crontab -e -u {username}` where you need to replace {username} with the actual username to run the script.
In our example we will call the script every 5 minutes. You can adjust the interval to your liking and available disk space.
```
*/5 * * * * php /path/to/VE.Direct-Text-Reader/worker.php
```
Any information returned by the worker script will be written to the log-file specified in the `config.json`.
## How the script works
The principle operation of the script is following these steps:
1. Establish serial connection
2. Read text data from serial connection
3. When "PID" is found start recording of the data
4. When "PID" is found again stop recording of the data
5. Close serial connection
6. Serialize the dataset to a JSON object ommiting checksums
7. Save the JSON object to the corresponding database table (based on the product ID and serial number) on the database. If no matching table exist it will be created on the fly.
8. Clean up
## Database structure
We are using SQLite3 as the database as it will be queried periodically only. The structure is quite simple and straight forward.
The data of each individual device will be saved in a separate table. The table name will be {PID}-{SER#}. Where {PID} will be replaced with the product ID and {SER#} will be replaced with the device' serial number.
Inside each table there will be 3 columns named
- date (date when the dataset was collected)
- time (time of day when the dataset was collected)
- data (the JSON serialized dataset; the content depends on the type of device)
Additionally, there is one table containing some meta-data of the database. (cration time only for the time being)
## What's next?
The purpose of this script is to collect the data from your connected Victron Energy electrical equipment and store it in a database.
For further processing or visualization of the data you will need other tools. Like our web-interface or cli statistics tool for example. Take a look into the other repositories on our gitea organization at [https://git.schauaus.at/VE-Tools](https://git.schauaus.at/VE-Tools)
Or you write your own application.
config.json
+6
View File
@@ -0,0 +1,6 @@
{
"device": "/dev/serial/by-id/usb-VictronEnergy_BV_VE_Direct_cable_VEAWDESA-if00-port0",
"database": "VEDirect.db",
"logfile": "VEDirect.log",
"loglevel": "info"
}
worker.php
+177
View File
@@ -0,0 +1,177 @@
<?php
/*
* VE.Direct-Text-Reader
* worker.php
* Command line tool to fetch text protocol data from
* VE.Direct interfaces and store it into a database
* for further processing
*
* Licensed under the GPL 3.0
* https://git.schauaus.at/VE-Tools/VE.Direct-Text-Reader
*/
$startAt = date('Y-m-d H:i:s');
$scriptPath = dirname(__FILE__);
$config = json_decode( file_get_contents($scriptPath . '/config.json') );
$loglevel = 0;
switch ($config->loglevel) {
case 'debug':
$loglevel = 4;
break;
case 'info':
$loglevel = 3;
break;
case 'warn':
$loglevel = 2;
break;
case 'error':
$loglevel = 1;
break;
}
if ($loglevel > 0) {
$logfile = (!str_starts_with($config->logfile, '/')) ? fopen($scriptPath . '/' . $config->logfile, 'a') : fopen($config->logfile, 'a');
writeLog("===================================================\r\n", 1);
writeLog("worker startet at $startAt\r\n", 1);
writeLog("loglevel: $loglevel (" . $config->loglevel . ")\r\n\r\n", 1);
}
//create database if it does not exist yet
if(!is_file((!str_starts_with($config->database, '/')) ? $scriptPath. '/' . $config->database : $config->database)) {
writeLog('Database not found! I will create it! ', 3);
if( !initializeDB() ) {
writeLog("failed! Exit!\r\n\r\n");
cleanUp('with errors ');
}
else writeLog("succeess!\r\n\r\n", 3);
}
$data = readData();
if( !$data )
cleanUp('with errors ');
if( !processData($data, $startAt) )
cleanUp('with errors ');
cleanUp('successfully ');
/*
* cleanup
*/
function cleanUp($status) {
global $loglevel;
if( $loglevel > 0) {
global $logfile;
writeLog('worker finished ' . $status . 'at ' . date('Y-m-d H:i:s') . "\r\n", 1);
writeLog("===================================================\r\n\r\n", 1);
fclose($logfile);
}
exit();
}
/*
* global function to write new data to logifle
*/
function writeLog($strData, $minLogLevel) {
global $config, $logfile, $loglevel;
if($loglevel >= $minLogLevel)
fwrite($logfile, $strData);
}
/*
* read data stream from VE Direct USB serial interface
*/
function readData() {
global $config;
writeLog("Open VE Direct interface:\r\n" . $config->device . "\r\n", 3);
$VEinterface = dio_open($config->device, O_RDONLY | O_NONBLOCK);
if($VEinterface === false) {
writeLog("unable to open VE Direct interface!\r\n", 1);
return false;
}
writeLog("success!\r\n\r\n", 3);
//set up the interface to properly communicate with the interface
dio_tcsetattr( $VEinterface, array(
'baud' => 19200,
'bits' => 8,
'stop' => 1,
'parity' => 0,
'flow_control' => 0,
'is_canonical' => 0)
);
$bytesToRead = 1024; // read 1KB by default
if(isset($config->bytesToRead))
$bytesToRead = $config->bytesToRead;
writeLog("Reading $bytesToRead Byte from VE.Direct interface\r\n", 3);
$dataStream = "";
while(strlen($dataStream) < $bytesToRead)
$dataStream .= dio_read($VEinterface, $bytesToRead);
dio_close($VEinterface);
writeLog($dataStream . "\r\n\r\n", 4);
//split data stream into array and remove checksums
$dataArray = explode("\n", $dataStream);
$collect = false;
$returnData = array();
for($i=0; $i < count($dataArray); $i++) {
if(!$collect && str_starts_with($dataArray[$i], 'PID'))
$collect = true;
else if($collect && str_starts_with($dataArray[$i], 'PID'))
break;
if($collect) {
$dataLine = trim($dataArray[$i]);
if( strlen($dataLine) == 0 || str_starts_with($dataLine, 'Checksum') || str_starts_with($dataLine, ':') ) //skip empty lines and checksums
continue;
$dataFields = explode("\t", $dataLine);
$returnData[$dataFields[0]] = $dataFields[1];
}
}
ksort($returnData);
writeLog("Collected data:\r\n" . print_r($returnData, true) . "\r\n", 3);
return $returnData;
}
function processData($data, $collectionDateTime) {
global $config, $scriptPath;
$db = (!str_starts_with($config->database, '/')) ? new SQLITE3($scriptPath . '/' . $config->database) : new SQLITE3($config->database);
$dbTableName = $data['PID'] . '-' . $data['SER#'];
$sql = "SELECT name FROM sqlite_schema WHERE type='table' AND name='$dbTableName';";
$tableNames = $db->query($sql);
if( !$tableNames->fetchArray(SQLITE3_NUM) ) {
writeLog('New Victron Energy Device found. Crating a new database table... ', 3);
if( $db->exec("CREATE TABLE '$dbTableName' (date TEXT, time TEXT, data TEXT);") )
writeLog("success!\r\n", 3);
else {
writeLog("failed! Cancel!\r\n", 3);
return false;
}
}
$dateTime = explode(" ", $collectionDateTime);
unset($data['PID']);
unset($data['SER#']);
$dataJson = json_encode($data);
writeLog('Add dataset to database... ', 3);
$sql = "INSERT INTO '$dbTableName' (date, time, data) VALUES('$dateTime[0]', '$dateTime[1]', '$dataJson')";
if( !$db->exec($sql) ) {
writeLog("failed!\r\n\r\n", 3);
return false;
}
writeLog("success!\r\n\r\n", 3);
return true;
}
function initializeDB() {
global $config, $scriptPath;
$dateTime = date('Y-m-d H:m:s');
$db = (!str_starts_with($config->database, '/')) ? new SQLITE3($scriptPath . '/' . $config->database) : new SQLITE3($config->database);
if( !$db->exec('CREATE TABLE meta (data TEXT);') ) return false;
if( !$db->exec("INSERT INTO meta (data) VALUES ('{\"creation\": \"$dateTime\"');") ) return false;
return true;
}
?>