Symbioquine

MultiPlus Inverter Metrics

symbioquine@gmail.com — Sat, 31 Dec 2022 00:00:00 GMT

I am lucky enough to have a pair of Victron MultiPlus inverters (Multi(s) hereafter). Originally, I just had one hooked up and kept the other as a backup - I have semi-critical loads like a freezer that would be wasteful/costly in the event of a prolonged outage. However, recently our winter charging needs dictated that charging faster via the generator was in order - this provides extra flexibility, keeps the noise down for our neighbors, and ~~I suspect~~ yields better generator efficiency (more on that in another post). Thus for me, hooking up the second Multi was an obvious next step.

This post will describe some things I have learned about the Multi's data protocols and how I've been using them to monitor our generator usage.

Originally, I had a very simple script to pull data using a library Victron references in their open source link-farm, izak/ib.victron (izak/ib.victron hereafter). It's not clear if that library is still being maintained and it has some other issues - e.g. still on Python 2, only supports a single bus device properly, and makes a few assumptions which break with the current version of the protocol. I don't want to beat up on that library too much because it served me well for a year and helped me understand the protocol immensely. Just that my use-case outgrew it. In addition to sharing what I've learned here, I hope to eventually share the library I've written as a spiritual successor to izak/ib.victron.

Unfortunately, Victron puts their protocol documentation behind an email form so I can't share a direct link, but the relevant documentation can be found quite easily by requesting "Interfacing with VE Bus products - MK2 protocol" from Victron's Technical information downloads page. The rest of this post will reference version 3.14 of that documentation, however I will try to provide enough detail so having that documentation in-hand isn't essential.

Bus vs Device Communication

Multi's (and some other Victron devices) have RJ45 ports (the same physical ports used for ethernet) which are used to chain one or more devices together into a single physical communication "bus". This bus (VE.Bus) is used by the devices themselves to coordinate parallel/split-phase/multi-phase operation and by monitoring/control hardware/software. In order to access the bus from a computer, a special USB adapter (MK3-USB) is used to translate between the bus transport and expose it as a serial interface.

On the software side, Victron calls the protocol for interfacing with the bus "MK2". Messages in the MK2 protocol are called "frames". Most of these messages take the form of a command/response pattern where devices wait for frames which request information or make changes, then respond with the requested information or confirmation/errors. There are two broad categories of commands, commands which pertain to the bus as a whole, and "Winmon" commands which pertain to specific devices only.

The izak/ib.victron library mainly only exposes the bus-level information. Interestingly, a lot of the bus-level information can't be interpreted without also requesting supporting data from the individual bus devices. The rational for this doesn't seem to be documented in the protocol. Nor does it seem to be documented whether, in systems with several Multis, it is valid to use the supporting data from the first device to interpret the the bus-level information for all the devices - we kind of have to assume so, but it feels like a major gap in the protocol documentation.

Another gap in the protocol documentation seems to be in terms of the meaning of parts of the bus-level information when multiple devices are hooked up. Specifically, I was interested in the steady-state power cost of having two inverters hooked up all the time. With one inverter, I had been monitoring the DC power usage as as measured by the inverter and it seemed to match up with the baseline power usage for my system as a whole. When I hooked up the second inverter though, I was puzzled since that bus-level power usage number didn't appear to double. I still can't totally explain that phenomena (I will touch on the phenomena in a later section to describe what I do know, but - as I say - it's still unexplained.) It is that quandary that sent me down the path of trying to monitor the DC power usage for each inverter independently.

In the MK2 protocol, there is a bus-level address command which is used to set or retrieve the address of the device to which subsequent Winmon commands are directed. The izak/ib.victron library allows an address to be specified, but only once and only one address. It doesn't allow switching between devices on the fly and although it provides a primitive locking mechanism which would appear to allow multiple instances to work with the same bus, there isn't any logic to track which device is currently being addressed so it isn't really viable to use as-is for sending Winmon commands to multiple bus devices.

Device Discovery

The MK2 documentation also describes how to discover which devices are on the bus. This is done by iteratively selecting addresses and testing whether a response is received when issuing a Winmon command. The izak/ib.victron library doesn't handle this at all.

Interestingly, the MK2 protocol documentation doesn't clarify whether the result of this device discovery should be assumed to be valid for the entire life of a bus connection. In the case of my inverters I'm assuming so - since they're set up to switch as a group, meaning they can't be disconnected without interrupting all the devices and I don't have any more VE.Bus devices to connect.

Device RAM Snapshots

The MK2 protocol documentation suggests that, to get accurate point-in-time power data, one can issue a bus-level command to snapshot up to 6 specific RAM variables, then read the data from that snapshot on each device. The available RAM IDs appear to correspond pretty closely to data that is also available via the bus-level info commands, but the mapping between the two isn't explicitly described in the documentation.

Poorly Documented RAM Variables

Along similar lines, some of the RAM variables have cryptic or misleading descriptions. Specifically, RAM variables 14 and 15 appear to have the exact same description, but return different values. Presumably, folks at Victron who wrote the documentation copied and pasted between the two variables' descriptions but forgot to update one of them.

By recording both of these and looking at them both with and without the generator running, I think I can explain what they actually mean.

RAM[2] * RAM[3]: "vainv" (Abs volt-amps passing through inverter hardware)
RAM[4] * RAM[5]: "wbat" (Abs watts from battery)
RAM[14]: "invpow" (Power passing through inverter hardware)
RAM[15]: "mainpow" (Total power from mains: load + charging)
RAM[16]: "outpow" (Power to load)

In the above graph I have multiplied "outpow" (RAM[15]) by -1 to make it easier to see how it is the sum of the load power plus the charging power. In my installation, this value is always zero or less, indicating power flowing from my generator to charge the batteries and power loads. I don't know whether there are scenarios where a positive value is possible, but we can surmise it might indicate power flowing back to mains - perhaps where power is being feed back to the grid.

I included "vainv" (the product of RAM[2] * RAM[3]) and "wbat" (the product of RAM[4] * RAM[5]) for comparison. I suspect the small differences where vainv is always a little higher than "invpow" (RAM[14]) are the result of my loads' power factor being slightly less than one - the three RAM values were captured as part of the same snapshot. Similarly, the difference between the inverter power and "wbat" should show us the overhead or self-consumption of the inverter.

Finally, "outpow" (RAM[16]) is pretty self-explanatory. It is just the power actually used by loads on the AC output of the Multis and does not seem to effected by whether the inverter hardware is charging or inverting. (Except for the edge case I'll describe next...)

Missing Background Power Usage

As mentioned earlier, I was surprised to see that the zero-load usage of the new inverter was zero which didn't match the specification sheet, nor my experience with the first inverter. After much experimenting with the metrics side of things, I finally decided to stick a clamp ammeter on the cable to the inverter and found around 0.95 amps @ 26.5v DC with zero load - literally nothing connected to the output. Since that didn't appear in any of the metrics, I next tested with a known load by putting a 40 watt incandescent light bulb directly on the terminals. Immediately, the Multi started reporting that it was consuming 69 watts from the battery which is much more like what I was expecting. Thus in my present configuration it appears the second Multi has an idle power draw in the range of 30 watts, but has trouble self-reporting that usage unless there is also some AC load current flowing.

I will probably work around this discrepancy in the future by just shifting some of my loads on to the other split-phase leg and then not worry about it.

Monitoring Generator Usage

For a while now I've been tracking my generator usage. This helps me understand the ongoing costs of my off-grid system. Some interesting efficiency data has come out of that which will probably be the subject of another post, but the part I want to discuss here is the actual monitoring mechanism because that is based on the metrics from the Multis and my learnings above.

In my system all generator power flows through the Multis which charge the batteries and also pass power through to my AC loads. There are several advantages to this configuration:

It eliminates the need for a transfer switch since the Multis can do that internally - seamlessly switching from inverter to pass-through modes as needed
The Multis can automatically manage the max generator current such that the charging rate is attenuated when large loads come on
The Multis provide a common point where many of the key usage metrics can be measured

To that last point, I can then have a tiny service which utilizes the learnings above to poll the Multis and write the relevant metrics into InfluxDB. Those raw metrics are available in a dashboard, but also are available for analysis purposes. In this case, I have a script which reads the data from InfluxDB and outputs the generator run time and kWh produced for each instance the generator is run.

I'll probably share the actual code for doing that in a future post, but for now, I'll just outline how it works. The script pulls 1 minute data for the last week and looks for datapoints where the inverter is charging instead of drawing power from the batteries. For those time periods it tracks the total runtime and sums the 1 minute averages of the power used to charge and power passing through to the loads.

The resulting data can help with understanding the generator's performance characteristics - i.e. at what average loading do we get the most kWh per unit of generator fuel. It also is a convenient way to track generator run hours which is useful for scheduling maintenance tasks like oil changes and filter cleaning.

Conclusion

I don't know what the ideal intersection is for open source/knowledge with proprietary upper-mid-teir hardware like Victron. Clearly, there is an argument for Victron funding more first-party open source tooling and improving their documentation to disambiguate some of the points that I've highlighted above. At the same time I appreciate that there exists some documentation, and admire the scrappy nature of the community open source projects they've linked to.

I hope that this post is helpful to others who want to gain access to detailed inverter metrics and attain a deeper understanding of the behaviour their systems. Perhaps it will inspire people to share their learnings and library improvements.

More generally, I am optimistic that the sum of the broader hacker/tinkerer communities' efforts with various alternative/renewable/off-grid energy systems and components will yield a more robust and adaptable technical ecosystem. That in turn could allow folks to build systems they can afford and which reward conserving resources. Acknowledging of course that in many/(most?) cases, if available, standard grid power will probably be the cleanest and cheapest option overall.

Happy New Year and happy/safe hacking!

Building Drupal Modules with GitHub Actions

symbioquine@gmail.com — Wed, 28 Sep 2022 00:00:00 GMT

Sometimes CI architectures are more a product of circumstance than we'd like. The strategy described in this post is a prime example of that.

Many Drupal modules don't require a build step, typically they are just some PHP, yaml, and maybe some Twig/JS/CSS files all checked-in. Those files then directly form the contents of the release for the module.

Drupal.org's infrastructure is tailored to that style of module. Internally Drupal.org uses an instance of GitLab to host the source code for each module. Project maintainers tag releases in GitLab then manually create releases at the project level which then become the installable versions for the Composer "drupal/" prefix. (See the Drupal docs on using Composer.)

Unfortunately, there's another style of module which doesn't seem to be well served by the current Drupal.org infrastructure - modules which need an arbitrary build step. The example I am going to use in this post is a module that includes some Javascript code that is built with Webpack.

Project Setup / Code

To start off, create a new GitHub repository and Drupal.org module project. For clarity and convenience, I recommend making the GitHub repository name and the Drupal.org project "short name" the same. For that, the name needs to follow the Drupal conventions for module naming - basically just alphanumeric characters and underscores. I've chosen to call my example for this tutorial "symbioquine_dot_net_built_drupal_module_example".

I let GitHub generate a default branch, readme, and .gitignore file. Then I renamed the branch from main to release - this will be important later since our GitHub workflow will update that branch using the results of our automated build when we push specific tags to the development branch.

git clone https://github.com/symbioquine/symbioquine_dot_net_built_drupal_module_example
git checkout -b development

`src/main.js`

// Wait until all attached Drupal libraries get loaded
document.addEventListener('DOMContentLoaded', () => {
  document.querySelector('#example-page-app').innerHTML = "Hello world!";
});

`package.json`

{
  "name": "symbioquine_dot_net_built_drupal_module_example",
  "version": "1.0.0",
  "description": "A module showing how to automatically build JS code with Webpack and push releases to Drupal.org",
  "license": "GPL-3.0-or-later",
  "repository": {
    "type": "git",
    "url": "https://github.com/symbioquine/symbioquine_dot_net_built_drupal_module_example.git"
  },
  "scripts": {
    "build": "webpack --config webpack.config.js --mode production"
  },
  "devDependencies": {
    "webpack": "^5.36.0",
    "webpack-cli": "^4.6.0"
  },
  "dependencies": {
  }
}

`webpack.config.js`

module.exports = {
  entry: {
    'built_drupal_module_example': {
      'import': `${__dirname}/src/main.js`,
    },
  },
  output: {
    path: `${__dirname}/drupal_module_src/js`,
    filename: '[name].js',
    clean: true,
  },
};

`drupal_module_src/composer.json`

{
  "name": "symbioquine/symbioquine_dot_net_built_drupal_module_example",
  "description": "A module showing how to automatically build JS code with Webpack and push releases to Drupal.org",
  "type": "drupal-module",
  "homepage": "https://github.com/symbioquine/symbioquine_dot_net_built_drupal_module_example",
  "authors": [
    {
      "name": "Symbioquine",
      "homepage": "https://github.com/symbioquine",
      "role": "Maintainer"
    }
  ],
  "support": {
    "issues": "https://github.com/symbioquine/symbioquine_dot_net_built_drupal_module_example/issues",
    "source": "https://github.com/symbioquine/symbioquine_dot_net_built_drupal_module_example"
  },
  "license": "GPL-3.0-or-later",
  "minimum-stability": "dev"
}

`drupal_module_src/symbioquine_dot_net_built_drupal_module_example.info.yml`

name: Symbioquine.net Built Drupal Module Example
description: This module shows an example of how to automatically build JS code with Webpack and push releases to Drupal.org
type: module
package: Example
core_version_requirement: ^9

`drupal_module_src/symbioquine_dot_net_built_drupal_module_example.libraries.yml`

built_drupal_module_example:
  js:
    js/built_drupal_module_example.js:
      preprocess: false
      minified: true

`drupal_module_src/symbioquine_dot_net_built_drupal_module_example.routing.yml`

symbioquine_dot_net_built_drupal_module_example_page.content:
  path: '/symbioquine_dot_net_built_drupal_module_example'
  defaults:
    _controller: symbioquine_dot_net_built_drupal_module_example.top_level_controller:content
    _title: 'Example Page'
  requirements:
    _permission: 'access content'

`drupal_module_src/symbioquine_dot_net_built_drupal_module_example.services.yml`

services:
  symbioquine_dot_net_built_drupal_module_example.top_level_controller:
    class: Drupal\symbioquine_dot_net_built_drupal_module_example\Controller\ExamplePageController
    arguments: {}

`drupal_module_src/src/Controller/ExamplePageController.php`

<?php

namespace Drupal\symbioquine_dot_net_built_drupal_module_example\Controller;

use Drupal\Core\Controller\ControllerBase;

/**
 * Defines ExamplePageController class.
 */
class ExamplePageController extends ControllerBase {

  /**
   * Constructs a new ExamplePageController object.
   *
   */
  public function __construct() {
  }

  /**
   * Top-level handler for demo page requests.
   */
  public function content() {
    return [
      'app' => [
        '#markup' => '<div id="example-page-app"></div>',
        '#attached' => [
          'library' => [
            'symbioquine_dot_net_built_drupal_module_example/built_drupal_module_example'
          ],
        ],
      ],
    ];
  }

}

`.github/workflows/create-release.yml`

name: Build
on:
  push:
    # Sequence of patterns matched against refs/tags
    tags:
      - 'unbuilt-v*' # Push events to matching unbuilt-v*, i.e. unbuilt-v1.0.0

jobs:
  build:
    name: Create Release
    runs-on: ubuntu-latest
    steps:
      - name: Set RELEASE_VERSION environment variable
        run: echo "RELEASE_VERSION=${GITHUB_REF:19}" >> $GITHUB_ENV

      - name: Checkout code
        uses: actions/checkout@v2
        with:
          path: main

      - name: Checkout release branch
        uses: actions/checkout@v2
        with:
          path: release
          ref: release
          fetch-depth: 0

      - uses: actions/setup-node@v1
        with:
          node-version: '16.x'

      - name: NPM Build
        run: |
          cd ./main/
          npm ci
          npm run build

      - name: Copy Module to Release Working Dir
        run: |
          # Don't let stale build artifacts accumulate in our release branch
          rm -rf ./release/js
          cp ./main/{README.md,CHANGELOG.md,LICENSE} ./release/
          cp -r ./main/drupal_module_src/* ./release/

      - name: Push Changes to Release Branch and Tag
        run: |
          cd ./release
          git config user.name github-actions
          git config user.email github-actions@github.com
          git add .
          git commit -m "Release ${{ env.RELEASE_VERSION }}"
          git tag ${{ env.RELEASE_VERSION }}
          git push --atomic origin HEAD:release ${{ env.RELEASE_VERSION }}

      - name: Setup SSH Keys and known_hosts for drupal.org
        env:
          SSH_AUTH_SOCK: /tmp/ssh_agent.sock
        run: |
          mkdir -p ~/.ssh/
          echo "${{ secrets.DRUPAL_DOT_ORG_SSH_KNOWN_HOSTS }}" >> ~/.ssh/known_hosts
          ssh-agent -a $SSH_AUTH_SOCK > /dev/null
          ssh-add - <<< "${{ secrets.DRUPAL_DOT_ORG_SSH_PRIVATE_KEY }}"

      - name: Push Changes to Release Branch and Tag on drupal.org Gitlab
        env:
          SSH_AUTH_SOCK: /tmp/ssh_agent.sock
        run: |
          cd ./release
          git config user.name github-actions
          git config user.email github-actions@github.com
          git remote add drupal-dot-org git@git.drupal.org:project/symbioquine_dot_net_built_drupal_module_example.git
          git fetch drupal-dot-org
          git push --tags --force drupal-dot-org 'HEAD:refs/heads/release'

Test Build

npm install
npm run build

Drupal.org GitLab Deploy Key Setup

Add a new Deploy key to the Drupal.org GitLab for the project. For my example this is at https://git.drupalcode.org/project/symbioquine_dot_net_built_drupal_module_example/-/settings/repository.

I've been using the RSA key generation instructions from https://git.drupalcode.org/help/user/ssh

e.g.

$ ssh-keygen -t rsa -b 2048 -C "symbioquine_dot_net_built_drupal_module_example drupal.org Gitlab deploy key"
Generating public/private rsa key pair.
Enter file in which to save the key (/home/symbioquine/.ssh/id_rsa): /home/symbioquine/.ssh/symbioquine_dot_net_built_drupal_module_example
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/symbioquine/.ssh/symbioquine_dot_net_built_drupal_module_example
Your public key has been saved in /home/symbioquine/.ssh/symbioquine_dot_net_built_drupal_module_example.pub
The key fingerprint is:
SHA256:YmArT17dAmWhTRKPYL6DHrBBktL6pC1hbTf1f+edWaI symbioquine_dot_net_built_drupal_module_example drupal.org Gitlab deploy key
The key's randomart image is:
+---[RSA 2048]----+
|.+  o o.=.       |
|= .o . @         |
|+.. + = +        |
|o=.= * o o       |
|o*= * = S o      |
|o.o* + . . . ....|
| .. o       ..o.=|
|            E  +.|
|                 |
+----[SHA256]-----+

Then I copied the contents of /home/symbioquine/.ssh/symbioquine_dot_net_built_drupal_module_example.pub into a new Drupal.org GitLab deploy key. Make sure to check the box that says "Grant write permissions to this key" since we'll be using that key to push releases into the Drupal.org GitLab repo.

GitHub Secrets Setup

Add two new secrets for GitHub Actions. For my example these are added at https://github.com/symbioquine/symbioquine_dot_net_built_drupal_module_example/settings/secrets/actions.

First I copied the contents of /home/symbioquine/.ssh/symbioquine_dot_net_built_drupal_module_example into a secret named DRUPAL_DOT_ORG_SSH_PRIVATE_KEY.

Next I copied the output of running ssh-keyscan git.drupal.org into a secret named DRUPAL_DOT_ORG_SSH_KNOWN_HOSTS.

Tag and Push Initial Release

echo "node_modules" >> .gitignore
echo "drupal_module_src/js" >> .gitignore
git add -A
git commit -m "Release 1.0.0"
git tag unbuilt-v1.0.0
git push --atomic origin HEAD:development unbuilt-v1.0.0

Create Drupal.org release

Once the GitHub Actions workflow completes a new 1.0.0 tag will have been created on the Drupal.org GitLab for the project.

Then I go to the project page on Drupal.org and create the corresponding release.

Conclusion

Now we have a Drupal module with JS code built via GitHub actions and installable via the drupal/ composer prefix;

composer require drupal/symbioquine_dot_net_built_drupal_module_example
drush en symbioquine_dot_net_built_drupal_module_example

Once installed, the example page can be accessed under the Drupal site at /symbioquine_dot_net_built_drupal_module_example. e.g. https://drupal.example.com/symbioquine_dot_net_built_drupal_module_example

The full source of this example can be found in the GitHub repository at https://github.com/symbioquine/symbioquine_dot_net_built_drupal_module_example.

For further reference, here are a number of other modules that I maintain which use variations of this same strategy;

Have fun and build some cool stuff!

Local-first self-hosting

symbioquine@gmail.com — Sun, 25 Sep 2022 00:00:00 GMT

While "The Cloud" has dramatically changed the landscape of hosted computing, there are still some scenarios where it is compelling to self-host and particularly self-host on ones own hardware and network.

Prominent among these scenarios is the case where the the application needs to be resilient to outages such that it remains locally accessible even if the wider Internet is not available.

One way to achieve that while still making the application accessible via the public Internet is to reverse tunnel ports 80 and 443 from a remote server server to the local one. Then use a local gateway firewall to route IP packets bound for the remote server and those ports to the local server directly. In this way devices on the local network will transparently get the benefits of local network performance/availability without needing to differentiate at the DNS/IP layer - which causes issues for devices like phones that may automatically switch between the local network and the public Internet via their cell connections.

Remote Server Setup

The only things that should be needed are to enable your SSH key in the remote server's authorized_keys file and to enable OpenSSH's GatewayPorts option.

That might look something like this;

cat cumulus.pub >> ~/.ssh/authorized_keys
sudo sh -c "grep -q '^[^#]*GatewayPorts' /etc/ssh/sshd_config && sed -i '/^[^#]*GatewayPorts[[:space:]]no/c\GatewayPorts yes' /etc/ssh/sshd_config || echo 'GatewayPorts yes' >> /etc/ssh/sshd_config"

I would also recommend disabling ChallengeResponseAuthentication and PasswordAuthentication in OpenSSH;

sudo sh -c "grep -q 'ChallengeResponseAuthentication' /etc/ssh/sshd_config && sed -i '/^[^#]*ChallengeResponseAuthentication[[:space:]]yes.*/c\ChallengeResponseAuthentication no' /etc/ssh/sshd_config || echo 'ChallengeResponseAuthentication no' >> /etc/ssh/sshd_config"
sudo sh -c "grep -q '^[^#]*PasswordAuthentication' /etc/ssh/sshd_config && sed -i '/^[^#]*PasswordAuthentication[[:space:]]yes/c\PasswordAuthentication no' /etc/ssh/sshd_config || echo 'PasswordAuthentication no' >> /etc/ssh/sshd_config"

Local Server Setup

Note: This assumes there is a folder certs/ with fullchain.pem and privkey.pem files for the "mysite.example.com" domain and also that there is an OpenSSH private key for the remote host in a file named cumulus.privkey.

docker-compose.yml

version: '3.7'
services:

  www:
    image: crccheck/hello-world
    restart: always
    expose:
      - '8000'

  proxy:
    depends_on:
      - www
    image: nginx:latest
    restart: always
    volumes:
      - './nginx.conf:/etc/nginx/nginx.conf'
      - './certs/mysite.example.com:/etc/nginx/certs/mysite.example.com'
    ports:
      - '80:8080'
      - '443:8443'

  autossh-forward-proxy-http-to-cumulus:
    image: jnovack/autossh:2.0.0
    environment:
      - SSH_MODE=-R
      - SSH_REMOTE_USER=root
      - SSH_REMOTE_HOST=123.123.123.123
      - SSH_REMOTE_PORT=22
      - SSH_BIND_IP=0.0.0.0
      - SSH_TUNNEL_PORT=80
      - SSH_TARGET_HOST=localhost
      - SSH_TARGET_PORT=80
    restart: always
    privileged: true
    network_mode: "host"
    volumes:
      - ./cumulus.privkey:/id_rsa

  autossh-forward-proxy-https-to-cumulus:
    image: jnovack/autossh:2.0.0
    environment:
      - SSH_MODE=-R
      - SSH_REMOTE_USER=root
      - SSH_REMOTE_HOST=123.123.123.123
      - SSH_REMOTE_PORT=22
      - SSH_BIND_IP=0.0.0.0
      - SSH_TUNNEL_PORT=443
      - SSH_TARGET_HOST=localhost
      - SSH_TARGET_PORT=443
    restart: always
    privileged: true
    network_mode: "host"
    volumes:
      - ./cumulus.privkey:/id_rsa

nginx.conf

pid        /tmp/nginx.pid;
error_log  /var/log/nginx/error.log warn;

events {

}

http {
  client_body_temp_path /tmp/client_temp;
  proxy_temp_path       /tmp/proxy_temp_path;
  fastcgi_temp_path     /tmp/fastcgi_temp;
  uwsgi_temp_path       /tmp/uwsgi_temp;
  scgi_temp_path        /tmp/scgi_temp;

  client_max_body_size 200m;

  server {
    listen 8080;
    server_name mysite.example.com;

    rewrite ^/(.*)$ https://$host$request_uri? permanent; 
  }

  ssl_certificate     /etc/nginx/certs/$host/fullchain.pem;
  ssl_certificate_key /etc/nginx/certs/$host/privkey.pem;

  server {
    listen 8443 ssl;
    server_name mysite.example.com;

    location / {
      proxy_pass http://www:8000;

      proxy_set_header Host $host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Proto https;
    }

  }

}

Networking Setup

Note: These instructions are for Mikrotik routerOS, but something similar should be possible with most full-featured router/firewall hardware.

/ip dns static
add address=123.123.123.123 name=mysite.example.com ttl=5m
/ip firewall nat
add action=masquerade chain=srcnat comment="defconf: masquerade" ipsec-policy=out,none out-interface-list=WAN
add action=dst-nat chain=dstnat dst-address=123.123.123.123 dst-port=80,443 protocol=tcp to-addresses=192.168.88.32
add action=src-nat chain=srcnat protocol=tcp src-address=192.168.88.32 src-port=80,443 to-addresses=123.123.123.123
add action=masquerade chain=srcnat dst-address=192.168.88.0/24 src-address=192.168.88.0/24
/ip route
add distance=1 dst-address=123.123.123.123/32 gateway=192.168.88.1

DIY Adapters for DeWalt Batteries

symbioquine@gmail.com — Fri, 03 Dec 2021 00:00:00 GMT

While I don't intend to make this blog reviews-focused, some products have the potential to be foundational to a wide range of alternative DIY projects which I would like to help make more viable.

Many DIY projects require a simple reliable high-capacity power source. While dedicated or built-in batteries sometimes make sense, sourcing high-quality cells is often difficult. Also, as the number of such devices with dedicated batteries grows, so does the effort needed to keep them all charged on a semi-regular basis. As such, small tool batteries - like those for power tools by Makita, DeWalt, etc - make a compelling option. Here I will review 3 similar DeWalt adapters so other folks can hopefully just buy the one that best suites their project best.

There are probably at least a dozen of these adapters on Amazon, and even more listings on Ebay and AliExpress. I picked 2 of the ones with the most/best reviews and one that seemed to be newly listed but which claimed to have over current/voltage/charge/heating protection.

Since the adapters are so similar, I'll refer to them by the names Andy, Bart, and Curt.

Andy: "Power Wheels Adaptor for dewalt 20V" https://www.amazon.com/dp/B09C5CXVDM
- $12.58 - 4.6 stars with 1,114 ratings and 82 reviews
Bart: "Upgraded Battery Adapter for dewalt" https://www.amazon.com/dp/B08JLYPY4F
- $8.09 - 4.5 stars with 3 ratings and zero reviews
Curt: "Battery Adapter for Dewalt 20V" https://www.amazon.com/dp/B08GWZ8LR6
- $10.58 - 4.6 stars with 498 ratings and 40 reviews

Breakdown

Andy

Pros

Attaches to battery securely, detaches only when battery release button is pressed, and appears to be making a solid electrical connection with battery terminals
Plastic and mechanical portion is sturdy and fits together cleanly
Has a very simple "circuit board" inside to hold the contacts - this could be convenient for some kinds of mods
Wires are very well soldered to board
200 C Fluorinated Ethylene Propylene (FEP) wire insulation

Cons

No strain relief for wires
Thin contacts
Wire does not match description - actually is 14 awg not 12 awg
Wire has a relatively low strand count and is pretty stiff - not ideal for applications which will experience frequent bending
3g over advertised weight of 70g

Bart

Pros

Attaches to battery securely, detaches only when battery release button is pressed, and appears to be making a solid electrical connection with battery terminals
Plastic and mechanical portion is sturdy and fits together cleanly
Held together with nicer T9 screws compared with the philips screws found on Andy
Slightly thicker contacts than Andy
Right at advertised weight of 67g
Actually has 12 gauge wire - as advertised
Wires are directly soldered to the flat battery contacts - Could potentially increase overall ampacity
200 C Fluorinated Ethylene Propylene (FEP) wire insulation
Has some internal space that could be used in some projects to house additional small electronics

Cons

No strain relief for wires - though they do feed through small holes in a way that may provide some protection compared to Andy
Product images falsely advertise the adapter as having "Multiple Protection" ("Over-current", "Overvoltage", "Over-Charge", and "Over-Heating"), but actually there is nothing in the adapter that would provide that
Contacts are embedded in floating a piece of plastic and rely on the assembled case to capture them in the right position
Wire has a relatively low strand count and is pretty stiff - not ideal for applications which will experience frequent bending
I partially stripped two of the screws doing the investigation for this review - probably not really the fault of the adapter since they're not meant to be taken apart very many times

Curt

Pros

Attaches to battery securely, detaches only when battery release button is pressed, and appears to be making a solid electrical connection with battery terminals
Plastic and mechanical portion is sturdy and appears to be a single piece - not counting the contacts and wire
Actually has 12 guage wire - as advertised
Wire is a high strand count which should hold up better to frequent bending than the other adapters
Wires are directly soldered to the flat battery contacts - Could potentially increase overall ampacity
Wire insulation says 200 C and is nice and flexible/rubbery - might be silicone
Has nice countersunk screw holes for attaching the adapter to things
Lightest of the adapters at 56g

Cons

No strain relief for wires
Thin contacts - similar to those on Andy but a little more fragile due to their smaller size and reduced contact area with the case of the adapter
Contacts are a little loose and could potentially pull out of the adapter
Product images could be misleading since they contradict the description and suggest the adapter comes with 14 awg wire (it is actually 12 awg - see "pro" above)
6g over advertised weight of 50g

Other Take-aways

None of these adapters have a means of low voltage discharge cutoff - and neither do the batteries - so that will need to be part of any projects that use these adapters.
None of these adapters have connections for the other balance leads so they can't be (safely) used for charging.
None of these adapters are suitable for rough applications due to the lack of strain relief and wiring that will stand up to abuse.

Conclusion

Andy is nice if you plan to mod the adapter and put additional electronics inside - especially for low-current applications or if you plan to redo the wiring anyway
Bart is probably the best value and likely has the highest continuous current carrying capacity
Curt is probably the best for applications where weight matters or where the wire will experience frequent bending

Bottom line: For most projects I would probably pick Bart.

Minimal git-based docker-compose deployments

symbioquine@gmail.com — Tue, 30 Nov 2021 00:00:00 GMT

Deployment automation infrastructure is perhaps one the most crowded domains. Many of the options out there address key facets of the domain such as testing, QA approvals, pipeline transparency, rollback, zero-downtime rollouts, etc. There are also adjacent domains such as the DevOps space with tools like Puppet and Ansible.

Those tools are great when you need them, but sometimes they introduce more complexity cost than they're worth.

This recipe shows how to set up a minimal repository that will clone itself and run docker-compose on new pushes. It doesn't do a lot of the things those full-featured deployment tools do, but it is pretty impressive what you can get with a tiny git hook and a script in the right place.

Note: The pattern of calling a script in a repository from a git hook is a bit of a security risk in multi-user environments if write access on the repository are set too permissively. Please exercise caution.

Prerequisites

If you're here, I'm going to assume you already know how to set up SSH and are confident enough to customize this recipe for your needs.

Both Target and Development Hosts

git

Target Host

docker
docker-compose

Server Repository Setup (on Target Host)

First we'll create a bare repository on the server - feel free to replace "myproject" with another name (It probably can't have spaces, but I haven't tested that.);

ssh myuser@myserver
git init --bare /home/myuser/myproject.git

Next we'll install the git hook in that bare repository;

echo '[ -d "/home/myuser/myproject" ] && /home/myuser/myproject/deploy.sh || git clone /home/myuser/myproject.git /home/myuser/myproject && /home/myuser/myproject/deploy.sh' > /home/myuser/myproject.git/hooks/post-receive && chmod +x /home/myuser/myproject.git/hooks/post-receive

At this point, the repository can be pushed to and it will automatically clone itself and execute a deploy.sh script that must be placed at the root of the repository.

Client Repository Setup

Clone the repository and cd into it;

git clone myuser@myserver:/home/myuser/myproject.git myproject
cd myproject

Create the deploy.sh file with the following contents;

#!/bin/bash
set -e

cd /home/myuser/myproject

STAGE="$1"

if [ "$STAGE" == "plan" ]; then
    ./deploy.sh execute
    exit 0;
fi

if [ "$STAGE" == "execute" ]; then
    docker-compose up -d --build --remove-orphans
    exit 0;
fi

echo pwd=`pwd`
echo GIT_DIR=${GIT_DIR}
unset GIT_DIR
git pull
./deploy.sh plan

Create the docker-compose.yml file with the following contents;

version: '3.7'
services:

  hello-world:
    image: armhf/hello-world

Mark the deploy.sh script as executable and commit/push the initial change;

chmod +x deploy.sh 
git add -A
git commit -m "Initial commit"

You should then see something like this;

$ git commit -m "Initial commit"
[master (root-commit) f90ae92] Initial commit
 2 files changed, 28 insertions(+)
 create mode 100755 deploy.sh
 create mode 100644 docker-compose.yml
$ git push
Enumerating objects: 4, done.
Counting objects: 100% (4/4), done.
Delta compression using up to 12 threads
Compressing objects: 100% (4/4), done.
Writing objects: 100% (4/4), 530 bytes | 530.00 KiB/s, done.
Total 4 (delta 0), reused 0 (delta 0), pack-reused 0
remote: Cloning into '/home/myuser/myproject'...
remote: done.
remote: pwd=/home/myuser/myproject
remote: GIT_DIR=.
remote: Already up to date.
remote: Creating network "myproject_default" with the default driver
remote: Pulling hello-world (armhf/hello-world:)...
remote: latest: Pulling from armhf/hello-world
remote: Digest: sha256:9701edc932223a66e49dd6c894a11db8c2cf4eccd1414f1ec105a623bf16b426
remote: Status: Downloaded newer image for armhf/hello-world:latest
remote: Creating myproject_hello-world_1 ... 
remote: Creating myproject_hello-world_1 ... done
To myserver:/home/myuser/myproject.git
 * [new branch]      master -> master

Congratulations, you've now bootstrapped some minimal git-based deployment infrastructure. Use it wisely and build something cool!

Off-grid lead acid battery charging

symbioquine@gmail.com — Mon, 29 Nov 2021 00:00:00 GMT

Recently I spent some time analyzing an off-grid system for some less technical family members. In trying to understand and explain why it wasn't performing as they expected, I gained a greater appreciation for the nuances of lead acid (LA) battery charging characteristics and how they affect off-grid systems - especially those systems with insufficient solar capacity and those in less equatorial latitudes.

There are many other articles describing the care and feeding of LA batteries (here is one of my favorites) so I will try to go beyond just reiterating the conventional wisdom about LA battery life being inversely proportional to the depth of discharge (DoD). I'm instead going to focus on one of the most important off-grid system characteristics which I think is both crucial and often neglected. That characteristic is charging time. (Actually, reputable battery manufacturers often do provide specifications/guidance that are both detailed and tailored to their products, but for whatever reason that information - and the consequences thereof - doesn't seem to be well represented more generally.)

All batteries have maximum rates at which they can safely charge. For LA batteries, the charging rates are usually specified by the manufacturer either in terms of a number of hours in bulk and absorption charging or in terms of a graph showing the state of charge (SoC) vs charging power and time. Regardless of how much solar capacity a system has, it will tend not to fully charge the batteries unless the system has long enough with sufficient power for charging from a given average DoD.

You've probably seen the characteristic charge curve of LA batteries where the charging amperage - and total power - tapers off dramatically before the batteries are full.

The above graph is showing a Renogy 200 Ah AGM charging from a 10% DoD on 2021/07/05.

That roughly matches the manufacturer's charging characteristics graph.

That charging curve is both a blessing and a curse. It means up to around 80% of the capacity can be regained at roughly the maximum charge rate - usually around 10-13% (of the 20 hour capacity rating) for FLA and closer to 20% for AGM. However, it also means that batteries are often cycled in a lower average SoC range than one might expect. For example, if you tend to use 30% of your battery capacity over night and only have 4 hours of charging time during the day, you will probably be cycling your batteries between 50% and 80% instead of 70% and 100% - even assuming the system has plenty of solar panels.

There are a number of approaches to mitigating/solving this problem;

Shifting loads to the day time if/when there is excess solar capacity.
Reducing usage such that - on average - there are enough hours of solar to charge when taking into account the battery charge curve.
Performing a portion of the bulk charge using generator power in the morning before the "solar charging day" begins.
- This yields the best value for the time spent running a generator since most generators are most efficient between 40% and 70% of their rated output.
Increasing the size of the LA battery bank.
- This will allow the same amount of usage while staying at a higher SoC - which in turn means staying farther down the tail end of the charging curve.
Shifting loads to other battery technologies.
- The cheap/easy version of this is similar to #1 where one just charges laptops/phones/etc during the day when there is excess solar capacity.
- This could also be a strategy for moving to another battery technology altogether. For example, by using a DC-DC charger configured to charge a secondary LiFePo4 battery bank to which loads and batteries are added incrementally.

Hopefully this post has served to highlight the importance of considering charging time as a key aspect of off-grid load management and battery sizing. May your systems be performant, long lasting, and boring.

Epever Solar Charge Controller Clocks Running Fast

symbioquine@gmail.com — Thu, 02 Sep 2021 00:00:00 GMT

As a follow-up to my previous post about off-grid solar monitoring via the ESP8266, I am writing this post to share what I've learned about the clock behavior on the EPever (EPSolar) Tracer####AN line of MPPT solar charge controllers.

I have four of these solar charge controllers running all the time each with an ESP8266 publishing metrics back to an InfluxDB instance. The Epever Tracer units provide several features which depend on date/time;

Time-of-day load control
Cumulative energy generated/consumed stats
Periodic equalization charging for lead-acid batteries

My loads are almost always switched on so the first feature is not particularly useful to me, but the other two are. These features make the most sense when the clock on the Epever unit is set accurately. The date/time can be viewed and set manually via the MT-50 external display, or by some other device over Modbus/RS485 - in my case the ESP8266.

If the clock on the Epever units was accurate enough, there wouldn't be much point in automatically correcting it from the ESP8266 since it could just be set upon initial system setup with the MT-50. On the other hand, if the clock was inaccurate, it would be nice to automatically correct it using the time from an NTP server. To understand this, I started capturing a "timeDrift" metric calculated by subtracting the current NTP time from the current time reported by the Epever unit. This new metric showed that the Epever units were consistently running a little fast - ~4s/day.

I would also occasionally get nonsense datapoints where the Epever unit would briefly report a current date/time about 40 years in the past (approx. -1.26 * 10^9).

I believe this to be due to some sort of race condition in how the Epever units read the time from their internal real-time-clock to report it back via Modbus. That hypothesis is reinforced by the observation that the erroneous datapoints tend to fluctuate by 60s and occasionally 3600s - suggesting that the race condition is occurring when a Modbus request for the time coincides with a minute or hour rolling over.

With all those observations in hand, I ended up with some roughly Arduino compatible code that monitors and automatically corrects the Epever unit's clock when it becomes too far out of sync;

uint16_t pack_as_int(uint8_t v0, uint8_t v1) {
  uint16_t t = (v0 << 8) | v1;
  return t;
}

uint32_t upper_int(uint16_t v) {
  return (v >> 8) & 0xFF;
}

uint32_t lower_int(uint16_t v) {
  return v & 0xFF;
}

bool readDevClock(time_t& devTime) {
  delay(250);
  uint8_t result = node->readHoldingRegisters(0x9013, 0x3);
  if (result != ModbusMaster::ku8MBSuccess) {
    Serial.printf("Failed to read date from charge controller: 0x%02X\n", result);
    return false;
  }

  uint16_t min_and_sec = node->getResponseBuffer(0x0);
  uint16_t day_and_hour = node->getResponseBuffer(0x1);
  uint16_t year_and_month = node->getResponseBuffer(0x2);

  struct tm devTm;

  devTm.tm_year = (upper_int(year_and_month) + 2000) - 1900;
  devTm.tm_mon = lower_int(year_and_month) - 1;
  devTm.tm_mday = upper_int(day_and_hour);
  devTm.tm_hour = lower_int(day_and_hour);
  devTm.tm_min = upper_int(min_and_sec);
  devTm.tm_sec = lower_int(min_and_sec);
  devTm.tm_isdst = -1;

  devTime = mktime(&devTm);

  if (devTime < -1260000000 && devTime > -1270000000) {
    Serial.printf("Received faulty date from charge controller: %04d-%02d-%02dT%02d:%02d:%02dZ\n", devTm.tm_year + 1900, devTm.tm_mon + 1, devTm.tm_mday, devTm.tm_hour, devTm.tm_min, devTm.tm_sec);
    return false;
  }

  return true;
}

void syncClock() {
  time_t devTime0;
  time_t devTime1;

  // Don't record metrics or adjust controller date/time if we can't reliably tell
  // what time it has already set
  if (!readDevClock(devTime0) || !readDevClock(devTime1)) {
      return;
  }

  long int devTimeDelta = devTime0 - devTime1;

  if (devTimeDelta < -10 || devTimeDelta > 10) {
      metricBuffer->appendf(",devTimeDelta=%di", devTimeDelta);
      return;
  }

  time_t currTime;
  time(&currTime);

  // Don't record metrics or adjust controller date/time if somehow we didn't
  // get the correct time from NTP
  if (currTime < 1000000000l) {
      return;
  }

  // Serial.printf("Got date from charge controller: %d-%d-%dT%d:%d:%dZ (currTime = %ld, devTime = %ld, drift = %ld)\n",
  //   devTm.tm_year + 1900, devTm.tm_mon + 1, devTm.tm_mday, devTm.tm_hour, devTm.tm_min, devTm.tm_sec, currTime, devTime, devTime - currTime);

  long int timeDrift = devTime1 - currTime;

  metricBuffer->appendf(",devTime=%di", devTime1);
  metricBuffer->appendf(",currTime=%di", currTime);
  metricBuffer->appendf(",timeDrift=%di", timeDrift);

  struct tm *currTm = localtime(&currTime);

  // Only update controller date/time if it is more than an hour off or we're in the first five minutes of the day
  bool shouldUpdate = abs(timeDrift) > 3600 || (abs(timeDrift) > 5 && currTm->tm_hour == 0 && currTm->tm_min <= 5);
  if (!shouldUpdate) {
      return;
  }

  node->setTransmitBuffer(0, pack_as_int(currTm->tm_min, currTm->tm_sec));
  node->setTransmitBuffer(1, pack_as_int(currTm->tm_mday, currTm->tm_hour));
  node->setTransmitBuffer(2, pack_as_int((currTm->tm_year + 1900) - 2000, currTm->tm_mon + 1));

  uint8_t result = node->writeMultipleRegisters(0x9013, 3);
  if (result == ModbusMaster::ku8MBSuccess) {
    Serial.printf("Successfully set charge controller date/time.\n");
  } else {
    Serial.printf("Failed to set charge controller date/time: 0x%02X", result);
  }
}

Which yields the following nice behavior where all four of my controllers are consistently drifting by about 4s per day and having their clocks updated approximately every other day at midnight;

In the future I hope to share more of my learnings about monitoring/controlling cheap off-grid solar charge controllers - and perhaps a more complete firmware at some point too.

Thanks for reading!

Pragmatic Minimal Infrastructure

symbioquine@gmail.com — Sun, 14 Mar 2021 00:00:00 GMT

A 2019 article by David Futcher appeared on Hacker News recently titled You Don’t Need All That Complex/Expensive/Distracting Infrastructure. The article is well written and makes a really important point about building to your actual requirements. It's hard to over-emphasize YAGNI at all levels of coding, software engineering, and operations. Critical thinking about the tradeoffs of increasing the upfront cost - whether in time, cash, or complexity - is essential to reaching your audience faster and still having the flexibility/budget to respond to the ever-changing playing field.

It's important to acknowledge which tradeoffs have recurring concrete costs and which have hypothetical costs. For instance, a typical high availability (HA) configuration with a load balancer and multiple hosts incurs many costs. It incurs an up-front cost to configure the infrastructure and ongoing hardware/maintenance costs. It also incurs a complexity cost of deploying to multiple hosts, reasoning about distributed systems, etc. Most of these costs don't buy you anything on a day-to-day basis. Your machines will tend to be running smoothly and the network will tend to be stable - at least in the ways your HA setup protects against. You might never pay the hypothetical cost of an availability zone outage or a host hardware failure.

That said, you will have to upgrade your hosts' operating systems, deploy new versions of your software, and troubleshoot hard-to-reproduce behavior from your system. All of these benefit from building out a little infrastructure.

For the sake of simplicity let's assume you've taken the argument against infrastructure deeply to heart and your startup's software is a gzipped folder with all your web content and some sort of server binary. You scp it to your host, unpack, execute, and away you go. Life is good, there's no complicated container systems, no build servers, no CD pipelines, no statefulness, just good old-fashioned operations - and presumably all the time in the world to find customers and market your nascent business.

You can even get most of the benefits naively associated with having more infrastructure. Need to upgrade a host or deploy your software? No problem, you get to make the call about whether to risk downtime. You check your logs and try to guess whether any "important customers" are likely to try and use your site. If not, just go ahead and bounce the host or kill your server process then start a new version. Otherwise, you can spin up a new host, get it set up correctly running your software, then do a DNS flip and wait for the traffic to move to the new machine before stopping the old one.

It may take a while and feel repetitive after the third or fourth time, but boring is good right?

Well, yes and no. The minimalistic strategy I just described is great because it puts you in the driver's seat and hopefully delivering value to your customers faster. Where it falls down is that it is an extreme local optimum on the cost vs external value graph. In my opinion, even considering a very small set of externalities can both erode the advantages of the minimalistic strategy and help offset the costs of a more infrastructure-intensive strategy.

One of the key aspects to consider is which kinds of growing pains you have control over and can predict/schedule. If your business venture is even a little successful, you can be sure unforeseen bugs in your software will necessitate "emergency" code changes. You can also be sure you'll very quickly have enough web traffic that every deployment will require that new-host/dns-flip strategy or else involve downtime affecting those existing/potential customers. What's harder to predict is where the break-even point is on more complex infrastructure - especially since it can itself introduce unforeseen bugs. I would argue though, that paying the cost early on to get an as-simple-as-possible no-downtime deployment mechanism is worthwhile.

Should it be prioritized over the bare-minimum work to have a product or web-presence? Probably not. Should it be prioritized along with launching your first few features or releases? Probably. Once you have any customers, traffic, or value; you have an obligation to respect others' time and yours. In 2021 this means having very few if any "planned outages" and definitely not having "expected" intermittent downtime for deployments. The tooling is becoming ubiquitous enough and customers discerning enough that certain kinds of availability form a new baseline.

Another aspect that should be considered is that there are ancillary benefits to early judicious infrastructure investment. If your business succeeds, your time spent learning enough NGINX, Kubernetes, etc. to achieve that baseline availability will serve you well by providing foundational competency to evaluate future architectures. If your business fails there's a good chance you can re-use the core configuration/scripts/learnings in your next endeavor.

Finally, there's a sentiment case to be made for employing "cool" infrastructure that is a good fit for a given use-case and current growth. You probably don't need and shouldn't build the fanciest multi-AZ autoscaling cluster with real-time machine-learning metrics/log analysis. But if a few scripts can give you a satisfying minimal git-ops workflow for day-to-day deployments which makes you feel like your time is being well spent, it is likely worth it. Not to mention that your "3 AM self" will thank you if deploying is easy, fast, and makes it hard to break things.

In short, I too want to see more pragmatic business decisions as opposed to hype-driven over-engineering. However, those decisions need to take into account the costs of not keeping up with the times.

EPever MPPT Solar Charge Controller Monitoring Hardware

symbioquine@gmail.com — Tue, 27 Oct 2020 00:00:00 GMT

Cheaper MPPT solar charge controllers have become ubiquitous. Many of these sport RS232 or RS485/Modbus ports allowing the controllers to be extended via proprietary hardware/software for advanced monitoring and control use-cases.

Since this Spring I have been experimenting with the EPever (EPSolar) line of MPPT solar charge controllers which have RS485 interfaces and thought it would be worthwhile to share my learnings in the hope they will be useful to others.

This post will focus on the specific hardware I worked with, however the techniques are applicable to similar hardware.

At the bottom of this post I have linked to similar projects and build logs - while there are many, I believe the content here adds materially in terms of discussing the downsides of using the Raspberry Pi for this application and, for the esp8266 part of the build, showing the simpler wiring which results from choosing an RS485 adapter which does automatic flow control.

The First Pi

Early in the Spring I was configuring a couple tiny independent solar systems - one to power networking hardware to get internet on an off-grid property from a local WISP and another to provide incidental power for the construction phase of my skoolie conversion.

Naturally, I wanted to capture data on the performance of these systems over time to help gain a more intuitive hands-on understanding of the power small "renewable" energy systems can provide. Also I anticipate the data will be helpful in understanding the aging of the systems' batteries - and possibly panels.

My initial attempts involved a Raspberry Pi 3B and a RS485-to-RS232 adapter. I used a bit of python code and the excellent epsolar-tracer library to poll the Tracer 3210AN and publishing metrics to Carbon/Graphite.

After a fairly steep learning curve with half-duplex RS485 adapters, I got that working and was pleased to have some day-over-day performance data.

At this point my main complaint was that since only one device can connect to the RS485 port at a time, I couldn't have both remote monitoring and the handy MT-50 connected simultaneously.

The Second Pi

I set out to figure out how to get both the local display and remote monitoring working at the same time.

Since the Raspberry Pi only has 1 hardware serial port, I got some USB serial adapters and decided to proxy requests from the MT-50 while still polling for data to publish to Carbon.

Proxying the requests was unexpectedly complicated because I discovered that the MT-50 doesn't read data via the documented modbus commands. Most important of the undocumented commands the MT-50 uses is one for reading sparse/discontiguous registers which I have documented in this gist.

With the protocol bits figured out and a nice watertight box, I had a passable system component which publishes metrics and still allows the local display to work.

Unfortunately, there were still several issues with this strategy;

Raspberry Pi fail due to how quickly SD cards wear out and result in filesystem corruption - this is supposedly surmountable by limiting or completely disabling writes, but I haven't figured out how to do this while still keep the ability to remotely push control-program/OS updates
Raspberry Pi use quite a bit of energy - over 1 watt even idle
The easiest way I found to securely publish to Carbon was via an SSH tunnel which adds unnecessary complexity and failure-points
The Raspberry Pi is powered from the 'load' output of the charge controller - meaning the Pi can't be used for toggling power to the load without also turning itself off

And Now for Something A Little Different

I decided to drop the Raspberry Pi and MT-50 in favor of a much simpler component built around the esp8266. This has a number of advantages;

Low power - I haven't spent much effort optimizing here, but it seems to be on the order of half a watt under load
Reliability - there are many fewer points of failure without the complexities of a full OS/filesystem/etc while still supporting remote updates
1 plug - since the esp8266 uses so little power, it can powered directly via the charge controller's RS485 port allowing for the unit to remain powered even when the charge controller load output is off

Maintaining the ssh tunnel from the esp8266 didn't seem practical so I switched to publishing the metrics to an authenticated InfluxDB endpoint over HTTPS.

Materials

ESP8266
Automatic Flow Control UART to RS485 Converter
RJ45 Waterproof Coupler - There are probably more cost effective options and the threads are a little fragile on these
Nalgene 2118-0008 - Any convenient container should also work
5 Female jumper wires
~4 inches of ethernet cable with one end installed
Ethernet cable to connect assembled device to the charge controller
Solder
Tools

The Build

Start by stripping about 2" of the outer ethernet cable housing and removing the striped wires. The ground and +5v - brown and orange respectively - are left long while the RS485 A/B wires are trimmed short.

Next the RS485 adapter is grounded.

Then jumper wires are added to the adapter and the whole thing is assembled into the case.

Here is the fully assembled unit connected and powered up.

Note: for initial programming via USB it is necessary to disconnect the green (RX) wire from the esp8266. After initial programming, OTA updates can be used to avoid that step.

In a future post I will share some sample code and provide power usage benchmarks.

Symbioquine

MultiPlus Inverter Metrics

Bus vs Device Communication

Device Discovery

Device RAM Snapshots

Poorly Documented RAM Variables

Missing Background Power Usage

Monitoring Generator Usage

Conclusion

Building Drupal Modules with GitHub Actions

Project Setup / Code

src/main.js

package.json

webpack.config.js

drupal_module_src/composer.json

drupal_module_src/symbioquine_dot_net_built_drupal_module_example.info.yml

drupal_module_src/symbioquine_dot_net_built_drupal_module_example.libraries.yml

drupal_module_src/symbioquine_dot_net_built_drupal_module_example.routing.yml

drupal_module_src/symbioquine_dot_net_built_drupal_module_example.services.yml

drupal_module_src/src/Controller/ExamplePageController.php

.github/workflows/create-release.yml

Test Build

Drupal.org GitLab Deploy Key Setup

GitHub Secrets Setup

Tag and Push Initial Release

Create Drupal.org release

Conclusion

Local-first self-hosting

Remote Server Setup

Local Server Setup

Networking Setup

DIY Adapters for DeWalt Batteries

Breakdown

Andy

Pros

Cons

Bart

Pros

Cons

Curt

Pros

Cons

Other Take-aways

Conclusion

Minimal git-based docker-compose deployments

Prerequisites

Both Target and Development Hosts

Target Host

Server Repository Setup (on Target Host)

Client Repository Setup

Related Work

Off-grid lead acid battery charging

Epever Solar Charge Controller Clocks Running Fast

Pragmatic Minimal Infrastructure

EPever MPPT Solar Charge Controller Monitoring Hardware

The First Pi

The Second Pi

And Now for Something A Little Different

Materials

The Build

Related Work of Others

Raspberry Pi

ESP8266

`src/main.js`

`package.json`

`webpack.config.js`

`drupal_module_src/composer.json`

`drupal_module_src/symbioquine_dot_net_built_drupal_module_example.info.yml`

`drupal_module_src/symbioquine_dot_net_built_drupal_module_example.libraries.yml`

`drupal_module_src/symbioquine_dot_net_built_drupal_module_example.routing.yml`

`drupal_module_src/symbioquine_dot_net_built_drupal_module_example.services.yml`

`drupal_module_src/src/Controller/ExamplePageController.php`

`.github/workflows/create-release.yml`