MQTTController
MQTTController is the Reactor interface to the world of MQTT. It requires a MQTT broker running somewhere on the network (MQTTController is not itself a broker, it's a client). The MQTT broker must be fully MQTT 3.1.1 compliant (or higher). The reference broker used for development and testing of MQTTController is Eclipse Mosquitto. Most Linux distributions can install this from their package managers, and docker containers are also readily available.
openLuup MQTT Broker
Users choosing the MQTT broker built in to openLuup must be running at least openLuup version 2022.12.2. Earlier versions are not supported. Additionally, this broker does not support subscriptions at QoS levels 1 and 2, and does not persist retained topics across restarts of openLuup, so it's not fully 3.1.1 compliant. Still, it has basic functionality and is convenient if you're already running openLuup, just be aware that retained messages can be lost during reboots/recoveries, and this can lead to entity states being inconsistent with their devices/clients and thus incorrect automation execution. The init configuration key described in Additional Entity Configuration below, when used with a device/client inquiry topic to restore synchronization of data, is a possible work-around.
There are three functions that MQTTController performs at the highest level of perspective:
- It maps devices that send and receive MQTT events into Reactor entities. For example, a Shelly1 relay device or an ESP8266 flashed with Tasmota can connect to an MQTT broker and exchange messages. Reactor connected to the same broker will "hear" status updates published by connected devices, and can update attributes on a Reactor entity accordingly (e.g. set the temperature_sensor.value attribute to the value of the temperature published), and actions performed on the Reactor entity can be turned into command messages to the device.
- It can "echo" any entity in Reactor to the MQTT universe, effectively acting as a gateway so that all entities known to Reactor are published.
- Commands can be issued to Reactor using MQTT to run actions on any Reactor entity.
MQTTController can also serve as a local gateway for MQTT messages for other Reactor controllers (that is, other controllers can ask MQTTController to subscribe to topics and exchange datagrams on their behalf).
Installation
MQTTController is not included in base Reactor distributions. It must be installed separately by downloading the package from the extras subdirectory of the Reactor download server.
Once the package file (a ZIP or Gzip'd tar archive) is downloaded:
-
Create, if it does not already exist, a directory called
extin your Reactor install directory (so it should be at the same level asconfig,storage, etc.).cd /path/to/reactor mkdir extIf you are running Reactor in a docker container, the
extdirectory should be created in the data directory, where yourconfigandstoragedirectories live as indicated above. -
Change directory into the new
extdirectory:cd ext -
Unpack the archive (use whichever command is correct for the type of archive downloaded):
tar xzvf /path/to/MQTTController-XXXXX.tar.gz # or... unzip /path/to/MQTTController-XXXXX.zip -
Run the install script. If you are using a "bare metal" install (not a docker container):
cd MQTTController ./install.shIf you are running Reactor in a docker container, we will open a container shell in which to do the install (the Reactor container must be up and running):
docker exec -it <container-name> /bin/sh cd /var/reactor/ext/MQTTController ./install.sh exit
From here, proceed to Basic Configuration below.
Basic Configuration
In order to use MQTTController at all, you have to add an entry for it to the controllers section of your reactor.yaml file.
controllers:
# Your existing controllers will be below the above line.
# Add the following after the last "- id" line in this
# section.
- id: mqtt
name: MQTT
enabled: true
implementation: MQTTController
config:
# Replace IP with that of your MQTT broker below
source: "mqtt://127.0.0.1:1883/"
Restart Reactor to make the changes take effect. After that, you should be able to refresh the UI, go the Entities list, clear any existing filters, and choose "MQTT" from the controllers filter selector. That should then show you two entities: the MQTT controller system entity, and its default group entity. If you don't see this, check the log for errors.
If you need to authenticate with your MQTT broker, you can supply username and password fields at the same level as source.
Configuring Devices
Devices are configured within an entities subsection of the config section for MQTTContoller.
MQTTController uses a data-driven model to map published messages to attributes on Reactor entities, and entity actions to published commands. There are two ways to map a device:
- If an existing preconfigured template will work for your device, you provide the topic that identifies your device, and tell MQTTController what template to apply to it.
- If there is no existing template that works for your device, you create one by telling MQTTController what Reactor capabilities the device has, and how to map published messages with state/telemetry updates into the attributes of those capabilities. You will also tell Reactor how to take actions in those capabilities and map them to topics to be published to make the device do what you need it to do.
The preconfigured templates available are listed at the bottom of this page. They are defined in mqtt_devices.yaml, in the templates section. Do not modify this file. This is a system file. There is a separate mechanism for you to define your own templates without modifying the system file.
Easy Device Configuration - Using Existing Template
As an example, let's say we have a Shelly1 configured with the topic shelly1_43DFD2 that is configured to operate as a single relay to operate a stairway light. To create an entity for this device, we can use the shelly_relay template. The template supports multi-relay devices, but the Shelly1 only has a single relay on channel 0:
...other stuff
config: "mqtt://127.0.0.1:1883/"
entities:
shelly1_stairway:
name: "Stairway Light"
topic: shelly1_43DFD2
uses_template: shelly_relay
channel: 0
After applying this configuration, we restart Reactor, and then refresh the UI. We should then be able to find the Stairway Light entity in the Entities list. You should be able to use the power_switch.on and .off actions to turn it on and off, and observe that the state correctly tracks that of the device, even when operated from the Shelly mobile app or any other means.
!!! tip Use Retained Messages!
It is strongly recommended that you configure your devices to use retained MQTT messages. This ensures that Reactor (and any other MQTT-connected apps you may be using, like Hass) get the latest state from the MQTT broker automatically at startup. Some devices, notably Tasmota, do not have MQTT retain on by default (a poor choice for IoT devices, IMO). On Tasmota devices, you can turn it on by using the PowerRetain on, SwitchRetain on, SensorRetain on and StateRetain on commands in the Tasmota console. Check with your device's documentation for the default state of the retain flag on messages it publishes, and how to turn it on if the default is off.
Detailed Device Configuration
If an existing template doesn't supply the behavior you need, you can configure the entity directly. Later, if you wish, you can turn your configuration into a reusable template, so if you have many similar devices, the configuration can be shared between then.
To get MQTTController to create and maintain an entity for your MQTT-enabled device, you need to create a mapping configuration that tells MQTTController:
- What type of entity to create;
- What capabilities it will have;
- How to turn state/status topics received for the node into attribute values on the entity;
- How to turn action requests for the entity into published MQTT messages.
To illustrate how this is done, let's implement a configuration for a Tasmota in generic/relay configuration (there's a template for this, but we're doing it explicitly here to show how that's done). Our Tasmota device is configured for a single relay, basically a switch, so we're going to map it to a Switch type entity with the two capabilities that most switches have in Reactor: power_switch and toggle.
We'll begin with our basic entity configuration. First, we need to assign a unique ID for the entity. To make it easy, we're going to use the Tasmota node's ID:
- id: mqtt
config:
source: "mqtt://127.0.0.1:1883/"
entities:
"tasmota_FFFFFF_relay": # this is your new entity's ID
name: "Tasmota Relay"
topic: "tasmota_FFFFFF"
Rules for Assigning Entity IDs
Entity IDs are identifiers, not names, and as such, the rules for valid IDs are strict: they must be between 1 and 64 characters in length; they may contain only upper- and lower-case letters A-Z, digits 0-9, and underscore; the ID must be unique on that MQTTController instance (it does not need to be unique system-wide). It should also never change, as rules and reactions will use a stored copy of the entity ID to find your entity, so if you change it, those rules and reactions will not find your entity and throw an error until you update them as well.
Now we can get into configuring what the entity can do:
"tasmota_FFFFFF_relay":
name: "Tasmota Relay"
topic: "tasmota_FFFFFF"
type: Switch
capabilities:
- power_switch
- toggle
primary_attribute: power_switch.state
OK, the above additions tell MQTTController that we're going to create a Switch type entity, and it will have capabilities power_switch and toggle. Its primary attribute will be power_switch.state.
Info
Don't worry if you don't recognize the Switch entity type, or know what you should put there. This value is used by the Dashboard to guess a default configuration for tiles, but since the Dashboard isn't really the focus of work right now, you can ignore this value. In fact, you can leave it out altogether; it won't hurt anything.
The power_switch capability is the only one of the two being assigned that has any attributes, and its only attribute is that state attribute being assigned as the primary. We need to get the value out of status update topics we receive and into that attribute. Tasmota sends topics of the form stat/nodeID/POWER for the status of a single relay device. When the relay is turned on, the payload is the string on, and when it's turned off, the payload is off. We'll create an events section, and put a handler in for that topic.
...
primary_attribute: power_switch.state
events:
"stat/%topic%/POWER":
"power_switch.state":
expr: "bool(payload)"
The first thing to notice is the %topic% string in the event topic. This is a substitution: MQTTController will put the string in the configuration value topic for this entity into that string. We could also have just hard coded it (that is, use stat/tasmota_FFFFFF/POWER), but if we want to make a template out of this later, it's better to use the substitution.
Having told MQTTController what topic this entity is interested in, we then tell it what attributes that topic can effect. This topic only affects one attribute, power_switch.state, and we use an expression to convert the payload string (on or off as stated above) to the boolean type that is required for power_switch.state. The expression here is trivially easy because the function bool() recognizes and converts the string on, yes and true to true, and off, no and false to false, and is not case-sensitive. The payload variable is defined for all topics received and will be set to the received payload.
If that topic could affect more attributes, we would list them using the same structure as that described for power_switch.state.
Likewise, if there were other topics that could affect attributes (including other topics that would affect power_switch.state), we could just list them in the events section and structure them as we have done so far. MQTTController will make sure every topic gets processed by every entity that is interested in it, and every attribute that can be affected by it.
Note that not every topic/payload sent may contain the data necessary for every attribute. To avoid errors from expressions and/or accidentally setting your attribute values to null in these cases, you can add if_expr to test if the payload contains the data needed to update the attribute. The expression should result in boolean true or false; if false, the attribute will not be evaluated further and will not be modified from its current value. Here's what that looks like for an additional Tasmota topic:
"tele/%topic%/STATE":
"power_switch.state":
json_payload: true
if_expr: '! isnull( payload?.POWER )'
expr: 'upper( payload.POWER ) == "ON"'
Also note that we've set json_payload to true here. This tells MQTTController to parse the payload as JSON and make it available as a real object, rather than a long text string, to your expression(s) for that attribute. If the payload is not parsable as JSON, the resulting object will be null, so notice that we've guarded the if_expr for this case using a coalescing member access operator (?.). The result of this if_expr is therefore false if either (a) the payload isn't JSON, or (b) it's a JSON payload but the resulting object does not contain a member named POWER.
Now let's move on to actions. In the actions section, we list each capability that has actions, and within each of those, we list each action and its implementation. Let's start with power_switch.on and power_switch.off. These simple actions take no parameters, so their implementation is very simple:
events:
...
actions:
power_switch:
"on":
topic: "cmnd/%topic%/Power"
payload: "on"
"off":
topic: "cmnd/%topic%/Power"
payload: "off"
Under each action, the required value topic sets the topic to be published to implement the action. If the topic has a payload, then the payload value can be added. The payload shown here for both actions is just a string, because the payload for these commands is a string, and in this case, it's a fixed string (it is always on for the on action and off for the off action. The power_switch.set action makes things a little more complex, because it takes a boolean parameter named state to determine what to do with the light/switch/relay. Here's that implementation:
actions:
power_switch:
...
set:
topic: "cmnd/%topic%/Power"
payload:
expr: "parameters.state ? 'on' : 'off'"
type: raw
In this implementation, the topic is the same, but the payload is more dynamic. The payload value is an object with two keys. The expr key/value is an expression that figures out whether to return the string on or off based on the value of the state parameter. The type key/value of raw tells MQTTController that the string returned by the expression should not receive any translation. So the payload thus becomes the string on or off sent with this topic to implement this action.
Sometimes the payload will be a simple value like a number, often derived from a parameter. In this case, the parameter: name spec in the payload definition will fetch the value from the named parameter to use as the payload value. In the following example, the (theoretical) setpoint of a thermostat is being set using the setpoint parameter value of the hvac_heating.set_setpoint action:
actions:
hvac_heating:
...
set_setpoint:
topic: "thermok/%topic%/set"
payload:
parameter: setpoint
Some topics have more complex payloads. It's not usual for a topic to have a JSON payload, for example. In this case, the expr value may return an object, and if the type is set to json, MQTTController will JSONify the expression result to a string that is then sent as the payload.
actions:
rgb_color:
...
set_rgb:
topic: "cmnd/%topic%/Color"
payload:
type: json
expr: >
{ "r": parameters.red, "g": parameters.green, "b": parameters.blue }
In some cases, you may find it useful to have access to attributes on the subject entity. For example, when implementing the dimming.up and dimming.down actions, if your MQTT-enabled light doesn't have topics to increase and reduce brightness by some amount, you can emulate it by doing the math based on the current brightness, which is stored in the dimming.level attribute of your entity. The entity object is defined in expression scope for this purpose.
Assuming our example light offers a basic set topic that takes a simple number payload in the range 0-255 to set the brightness of a light, we use the current brightness on the entity to compute a new target brightness to set (dimming.up increases 10%, dimming.down decreases 10%):
actions:
dimming:
...
up:
topic: "%topic%/set"
payload:
expr: "min(1, entity.attributes.dimming.level + 0.1) * 255"
down:
topic: "%topic%/set"
payload:
expr: "max(1, entity.attributes.dimming.level - 0.1) * 255"
# The use of min/max above assured that the values stay in range 0-255 when published
If your action needs to set the MQTT qos level or retain flag to values other than the default (see the QoS and Retain section, below), just provide those using keys of the same name. The value of qos must be an integer 0 through 2, and retain must be boolean.
Since we've done this in chunks, here's the whole configuration all together:
"tasmota_FFFFFF_relay":
name: "Tasmota Relay"
topic: "tasmota_FFFFFF"
type: Switch
capabilities:
- power_switch
- toggle
primary_attribute: power_switch.state
events:
"state/%topic%/POWER":
"power_switch.state":
expr: 'bool(payload)'
"tele/%topic%/STATE":
"power_switch.state":
json_payload: true
if_expr: '! isnull( payload?.POWER )'
expr: 'upper( payload.POWER ) == "ON"'
actions:
power_switch:
"on":
topic: "cmnd/%topic%/Power"
payload: "on"
"off":
topic: "cmnd/%topic%/Power"
payload: "off"
set:
topic: "cmnd/%topic%/Power"
payload:
expr: "parameters.state ? 'on' : 'off'"
type: raw
Notice anything missing?
We haven't defined the toggle.toggle action. The toggle capability has no attributes and just that one action, but we haven't defined what that action does. As it turns out, MQTTController can supply a default implementation for toggle.toggle, so we don't need to define it.
If a standard action of a capability does not apply to the device, you can tell Reactor not to show the action in the UI. Just set the action to false in the configuration. Here is what it looks like for the Shelly RGBW2 default template1 for the actions of capability light_effect:
light_effect:
"start":
# Use duration argument; speed is not used.
topic: "shellies/%topic%/color/%channel:0%/set"
payload:
# detail redacted for clarity
"stop":
topic: "shellies/%topic%/color/%channel:0%/set"
payload: '{ "effect": 0 }'
"set_speed": false # action not implemented <---- HERE
"set_duration":
topic: "shellies/%topic%/color/%channel:0%/set"
payload:
type: json
expr: '{ "transition": int( parameters.duration * 1000 ) }'
Here, the Shelly RGBW2 uses a time period (duration) for the transition; the notion of "speed" doesn't apply. The set_speed action is therefore not implemented, and that is indicated by setting its value to false as shown. This signals the UI to not show that action in menus for entities that use this template.
If none of the standard actions of a capability can be implemented for a device, the capability value can be set to false, and all actions for that capability will be removed from the related entity. Here's an example of what that looks like:
actions:
button: false # no actions of "button" are implemented for this device
It is also possible to add an action to the x_mqtt_device capability for an entity. This is used to support actions that are not provided by standard capabilities. Again using the Shelly RGBW2 template1 as an example, the template defines a set_white action in the x_mqtt_device capability to define an action to set the intensity of the white channel of an RGBW strip (rgb_color doesn't support additional channels, and I haven't yet provided a standard system capability that does).
x_mqtt_device:
set_white:
arguments:
white:
type: ui1
min: 0
max: 255
topic: "shellies/%topic%/color/%channel:0%/set"
payload:
type: json
expr: "{ 'white': parameters.white }"
The only addition here to other action configurations shown so far is the arguments element. It uses the standard form to describe the action's arguments, if it has any. It can be omitted if the action takes no arguments. In this case, we define the argument white as an integer in the range of 0 to 255, inclusive (the very specific ui1 type means a one-byte unsigned integer, but int would have worked just as well).
Historical Inconsistency
Notice in the foregoing example that the action's arguments are defined, but when it comes to building the payload, the expression refers to them as parameters. This is an old inconsistency in the implementation of Reactor. Most programmers use these terms interchangeably, so things like this happen. At this point, it would be too disruptive to choose one or the other and change it everywhere, so we're stuck with it. Moving on...
Additional Entity Configuration
The lwt key may be specified to identify the topic sent by your device as a Last Will and Testament (e.g. lwt: tele/%topic%/LWT is common for Tasmota). When this topic is received, MQTTController will mark the entity down by setting x_mqtt_device.online to false. The specific conditions under which the LWT is sent are device-dependent; consult your device and broker documentation for more details. Also, note that this default handling does not examine the payload; it assumes that receiving the LWT with any payload is an indication that the device is down (or going down). If this implementation isn't appropriate for your device, don't use the lwt key; instead, create your own event handler for the LWT topic in which the x_mqtt_device.online attribute is set based on the payload content.
The init key may be used to send initialization strings to a device. These strings will be sent every time Reactor connects to the MQTT broker. This is useful for ensuring that the remote device/client is in a known state. It can also be used to query the current state of the device/client to ensure the Reactor entity is in sync with it. The init topics are always sent with the default QoS and retain false (see QoS and Retain below).
The init key's value may be:
- A string, in which case the topic (with no payload) is the only topic sent at initialization;
- An array, containing any number and mix of:
- a bare string, a topic to be sent (with no payload);
- an object containing the required key
topic(string) and the optional keyspayload,type(string,raworjson),qos(integer 0-2) andretain(boolean).
The optional payload value can be any type. It will be converted to a string (in a straightforward way) before being sent, unless type: json is given, in which case it will be formatted as a JSON string before being published.
The query key can be used to send a topic that causes the device to report its current state and update the related entity. The query topic will be sent when an x_mqtt_device.poll action is performed on an entity. The form of the query value is the same as the entries of the init array: a plain string (the topic), or if a payload is needed, a required topic string and optional payload, type, qos and retain values. Query topics are always sent with the default QoS and retain false (see QoS and Retain below).
Here's an example of how these are formatted in the configuration (when needed):
"tasmota_FFFFFF_relay":
name: "Tasmota Relay"
topic: "tasmota_FFFFFF"
type: Switch
capabilities:
- power_switch
- toggle
primary_attribute: power_switch.state
lwt: "tele/%topic%/LWT"
query: "cmnd/%topic%/Power"
init:
- "cmnd/%topic%/someAction"
- topic: "cmnd/%topic%/anotherAction"
payload:
level: 50
rate: 3
type: json
qos: 1
Because the second init topic has type: json with its payload, the actual payload transmitted by this example would be {"level":50,"rate":3}.
The default QoS for init and query topics is the controller's default QoS, and the default retain flag is false. The example above shows qos being set explicitly to 1 for one of the init topics, which overrides the default (see QoS and Retain below for more information).
Controlling Attribute Evaluation Order
With the structure of event configuration discussed so far, the order in which attributes are evaluated and updated is not deterministic. That is, if an event is configured to update two or more attributes, there is no defined order by which the attributes are processed, and their order in your configuration is of no consequence.
If the order of processing is important, you can use the array form of event configuration. In this form, the data under the event's topic is an array of objects, each containing the attribute key to specify the attribute name. The remainder of the object is the same.
Let's look at a specific example, highlighting the differences in the format and why we might care:
events:
"%topic%/power:
"voltage_sensor.value":
expr: "float( payload.load.voltage )"
"power_sensor.value":
expr: "float( payload.load.power )"
"current_sensor.value":
expr: "entity.attributes.power_sensor.value / entity.attributes.voltage_sensor.value"
In this case, we have a hypothetical solar inverter/controller that is sending topics to give us instantaneous power output and line voltage, but it doesn't report amperes, so we are computing the amperes (ignoring power factor, etc.) by simply dividing power (in watts) by voltage.
The problem here is that there is no guarantee, in this typical configuration form, that when the topic is received, the three attributes are evaluated and updated in the order shown. The current_sensor.value attribute is relying on the entity's newly-updated voltage and power values, but if this attribute happens to be evaluated first or second, those entity attributes may not yet have been updated, and the computed current will be incorrect.
To fix this, we can reformat the configuration like this:
events:
"%topic%/power:
- attribute: "voltage_sensor.value"
expr: "float( payload.load.voltage )"
- attribute: "power_sensor.value"
expr: "float( payload.load.power )"
- attribute: "current_sensor.value"
expr: "entity.attributes.power_sensor.value / entity.attributes.voltage_sensor.value"
The key change made for each entry is the addition of - attribute: before the attribute name. This does two things: it refines the entry as an array element, and it makes the attribute name a key/value pair in the object that is the array element (expr is another key in each respective object).
In this array-based structure, the order in which the attributes are processed is guaranteed to be the order in which they appear in the configuration: voltage will be updated first, followed by power/watts, and finally current/amps.
Converting Your Configuration into a Template
In the previous section, we built the device configuration directly into our per-device configuration. If we had several such devices, though, it would be verbose and more difficult to maintain all of them with each device having a complete copy of all those lines. We can simplify our configuration by converting our device configuration into a template.
To begin, create a directory in your config subdirectory called mqtt_templates. Next, we'll assign an identifier (name) to our template. We'll call it my_tasmota_relay, and create a YAML file with that basename (e.g. my_tasmota_relay.yaml) in the mqtt_templates subdirectory.
# Optional description of what your template does (this is a comment)
my_tasmota_relay:
Now, we can basically copy the device configuration we previously created, leaving behind the ID, name and topic, since these are values that will later be passed to the template from entity configuration (that is, we don't want the template to have fixed/overriding values for these, so that the template can be reused to make multiple entities).
Passing data from an entity configuration to the template is straightforward. In any %keyword%-form substitution, the replacement string will be the value from the key of the same name in the entity's configuration. As a trivial example, if you add flavor: raspberry to your entity's configuration, then in any topic strings in your template you could include %flavor% and that would be replaced in runtime with the word raspberry (e.g. topic /icecream/%flavor%/rating becomes /icecream/raspberry/rating in use). This allows you to fully parameterize your templates. In addition, as of version 22282, if your substitution is of the form %flavor:chocolate%, then if flavor is not defined in the entity's configuration, the substitution will use the word chocolate, so you can establish defaults. For an example of this kind of parameterization in operation, look at the implementation of the "real" shelly_relay template in MQTTController's mqtt_devices.yaml file, which uses channel and unit to handle multi-relay devices.
You can also access configuration data in your template's expressions. The config object is defined in expression context, and allows you to access the elements of the entity's configuration. Building on the example in the previous paragraph, your expression could refer to config.flavor to access the configured value of the flavor configuration value.
If your template uses required substitutions or expression-context values, it is recommended that you add a requires: [] entry to the template configuration (before events and at the same indent level), specifying as the array contents the names of all values required by the template (e.g. required: [topic, unit]). If a user of your template omits one of these required values from their configuration, a detailed, helpful alert will be displayed and logged. The array form must be used even if there is only one required value.
Your completed template file will look something like this:
# Optional description of what your template does (this is a comment)
my_tasmota_relay:
type: Switch
capabilities:
- power_switch
- toggle
primary_attribute: power_switch.state
requires: [topic]
lwt: "tele/%topic%/LWT"
events:
"state/%topic%/POWER":
"power_switch.state":
expr: "bool(payload)"
"tele/%topic%/STATE":
"power_switch.state":
json_payload: true
if_expr: '! isnull( payload?.POWER )'
expr: 'upper( payload.POWER ) == "ON"'
actions:
power_switch:
"on":
topic: "cmnd/%topic%/Power"
payload: "on"
"off":
topic: "cmnd/%topic%/Power"
payload: "off"
set:
topic: "cmnd/%topic%/Power"
payload:
expr: "parameters.state ? 'on' : 'off'"
type: raw
In your device configuration, you can refer to your custom template with the uses_template key. Here's our revised device configuration, with an additional entity of the same type added for another Tasmota relay node, using the template for both:
- id: mqtt
...
config:
source: "mqtt://127.0.0.1:1883/"
entities:
"tasmota_FFFFFFF_relay":
name: "Tasmota Relay 1"
topic: "tasmota_FFFFFF"
uses_template: "my_tasmota_relay"
"tasmota_BEEFEE_relay":
name: "Tasmota Relay 2"
topic: "tasmota_BEEFEE"
uses_template: "my_tasmota_relay"
Echo Capability - Publishing Reactor Entities
MQTTController can, if configured, publish messages for every state change of a set of entities you choose (or all entities), rules, and global variable changes. In this way, all entities known to Reactor may become MQTT-enabled, regardless of their source. MQTTController also knows how to receive topics to perform actions on these entities.
Info
I'm not sure what to call this function, so for now I'm just calling it "echo" capability. Gateway...? Relay...? Great and Powerful Oz? Any ideas?
Enable echoing by defining an echo section in your MQTTController configuration:
- id: mqtt
...
config:
source: "mqtt://..."
entities:
...
echo:
#include_entities: []
#exclude_entities: []
#include_groups: []
#exclude_groups: []
#include_capabilities: []
#exclude_capabilities: []
primary_attributes: true
qos: 0
retain: false
Note
Some configuration keys have been commented out (#). As shown above, the defaults will be used for all of these keys. Uncommenting some of these keys immediately changes behavior, so be sure you read the descriptions of each below before doing so.
The include_ and exclude_ filters determine what data is echoed. They are arrays, and you can either use the compact array notation (['string1','string2']) or the multi-line list form (preferred for lists of more than 2 items).
list_name:
- "list element 1"
- "list element 2"
- "list element 3"
In MQTTController, the rules for include/exclude filter lists are as follows:
- If an
includelist is not defined (i.e. commented out or does not exist at all in the config), everything passes (jump to step 3). - If the
includelist is defined, then only those items matching any element pass. - If an
excludelist is not defined, nothing is excluded (except in capabilities, see below). - If the
excludelist is defined, then those items matching any element are excluded.
Beware Empty Arrays
The meaning of an empty array ([]) is different from that of being commented out or undefined. It is interpreted to mean nothing; therefore, in include lists, an empty array would mean include nothing, so be particularly careful there.
Filtering is done in the order: entities, groups, capabilities. Within each type, the include list is processed before the exclude list. So the specific order of filtering is: include_entities, exclude_entities, include_groups, exclude_groups, include_capabilities, exclude_capabilities.
The include_entities key is an array containing canonical ID patterns to match. The patterns are defined as follows:
- Literal canonical IDs, like
hubitat>100, to refer to a single entity; - Wildcard to match all entities on a particular controller:
hubitat>*matches all entities from the controller with IDhubitat; - Regular expressions:
/pattern-regexp/optional-flags
For example (using alternate array syntax for improved readability):
echo:
include_entities:
- "vera>*"
- "hass>*"
This would match all entities from the controllers with IDs vera and hass. If included_entities is not specified (i.e. commented out or not present in the config), then all entities from all controllers are included.
After inclusion, exclude_entities is processed in the same manner as include_entities, except that any entity that matches will be discarded.
The optional include_groups and exclude_groups arrays give further control over the entities echoed. The elements of these arrays are canonical IDs of groups. If an entity is a member of any of the listed include_groups, it will be allowed to echo. If it is a member of any of the listed exclude_groups, it will not echo. Dynamic changes in group membership will affect these filters dynamically. Group filters require MQTTController version 21343 or higher.
From the final set of entities that have passed filtering so far, the include_capabilities and exclude_capabilies lists will determine what capabilities are published as topics. If include_capabilites is not defined, all capabilities will be published (except those excluded). If exclude_capabilities is not defined, by default the extended (x_) capabilities will be excluded. If you wish to prevent this default exclusion and include all capabilities including the extended, simply set exclude_capabilities to an empty array (i.e. exclude_capabilities: [], meaning nothing is excluded).
Regular expression patterns are supported in include_capabilities and exclude_capabilities values, using the form /regexp/optional-flags (e.g. /^x_hass\b/i would be a case-insensitive match for any capability starting with x_hass).
The keys qos and retain determine the QoS and retentation of the state messages published. By default, these messages are sent at QoS 0 and no retention, but you may change these behaviors by providing different values as defined by MQTT.
If you do not needs rules or global variables published in your configuration, you can set publish_rule_states and/or publish_global_variables, respectively, to false to disable that function.
Working with Entity Topics
When a capability for an entity is published, the topic will be reactor/<mqtt-id>/<ctrl-id>/<entity-id>/state/<capability-name>, and the payload is a JSON object containing the attribute key/value pairs of that capability for that entity. The mqtt-id is the ID of the MQTTController, the entity-id is the ID of the entity that is the subject of the message, and the ctrl-id is the controller ID to which that entity belongs.
reactor/mqtt/vera/device_10/state/power_switch {"state":true}
The setting of primary_attributes in the echo section determines whether a generic value topic is sent for the primary attribute of the entity. If this is true, you will see topics of the form reactor/<mqtt-id>/<ctrl-id>/<entity-id>/value with a payload string containing the primary attribute value. For a switch, the primary attribute of which is typically power_switch.state, one might expect:
reactor/mqtt/hass/switch_living_room/value true
For some users, it may be convenient for the entity name to also be part of the topic on announced attribute changes. MQTTController allows you to add the entity_identifier configuration key to the echo section with a value of id (the default) or combined. If combined or id/name is specified, the entity name will be added as an additional level of the published topic, such as reactor/mqtt/vera/device_10/Living Room Lights/state/power_switch. If name/id is specified, the name will precede the ID. Note that while these options add a level to the published topics for attribute changes, they do not change any other topic structures, such as those for performing actions (see below).
MQTTController can handle specific received topics and turn them into actions on Reactor entities. To perform an action on an entity, publish a topic in this form:
reactor/<mqtt-id>/<ctrl-id>/<entity-id>/perform/<action-capability>/<action-name>
If the action takes parameters, encode the parameter key/value pairs as a JSON payload. Here's an action to turn on our fictional living room light:
reactor/mqtt/hass/switch_living_room/perform/power_switch/on (no payload)
reactor/mqtt/hass/switch_living_room/perform/power_switch/set {"state":true}
Both of the above topics accomplish the same task; the first requires no parameters/payload, while the second requires a "state" parameter. Here's a more complex payload example, setting color on an RGB bulb:
reactor/mqtt/vera/device_188/perform/rgb_color/set_rgb {"red":255,"green":128,blue:16}
MQTT Actions Always Available
Reactor will receive entity action messages and process them whether the echo function is enabled or not, and will process actions for any entity/capability, whether it matches echo filtering or not.
MQTTController will send a full update for an entity if it receives the following message:
reactor/<mqtt-id>/<ctrl-id>/<entity-id>/query
Working with Rule Topics
If rule publishing is enabled (which it is by default when echo is enabled), changes to the set/reset state of all rules will be published in topics of the following form:
reactor/mqtt/Rules/rule-id/state
The payload for this topic will be boolean true if the rule has set, and false if it has reset.
To force the publication of a rule state message at any time, you can send:
reactor/mqtt/rules/rule-id/query
You cannot control the state of rules via MQTT. Rule state is driven exclusively by the result of its conditions.
To help reduce traffic, you can disable publishing of rule state topics when echo is enabled by adding publish_rule_states: false to your MQTTController configuration.
Working with Global Variables
If global variable publishing is enabled (the default when echo is enabled), a value change to any global variable will cause a topic of the following form to be published:
reactor/mqtt/Expr/expression-name/value
The payload for this message is the (JSON) expression value. Be aware that primitive numbers and strings, when converted to JSON, are not surrounded by {} as many seem to expect for JSON. A number converted to JSON is simply a string containing the number (e.g. 123), and a boolean will be the string true or false. A string value will be quoted (e.g. "I am a valid JSON result"). Arrays and objects will, of course, have the familar {} envelope.
You can set the value of a global variable, if it is expression-less, by sending the value as JSON using same semantics described in the preceding paragraph on a topic of the form:
reactor/mqtt/Expr/expression-name/set
To query the value of a global variable any time, send:
reactor/mqtt/Expr/expression-name/query
The echo feature does not need to be enabled to set or query a global variable as shown above.
To help reduce traffic, you can disable publishing of global variable topics when echo is enabled by specifying publish_global_variables: false to MQTTController your configuration.
QoS and Retain
Configuration of QoS and the retain flag for topics published by MQTTController is a hierarchy, where settings at lower (more detailed/specific) levels overrides that of higher levels. The default QoS is 0, and the default for retain is false. The optional default_qos (integer 0-2) and default_retain (boolean) configuration values (in controller configuration, i.e. same level as source) can be used to modify these defaults.
For echo outbound entity attribute topics, the default QoS and retain flag are used unless the qos and retain flags are set under echo configuration, in which case, those more-specific values override the defaults. These overrides will also apply to outbound topics for alerts, global variables, and rule states, if enabled.
The following topics are always sent using the controller's default QoS, but the retain flag false (i.e. not affected by default_retain):
- Topics in per-entity
queryconfiguration; - Topics in per-entity
initconfiguration (but both can be overridden by specific configuration in the topic entry); - Topics in per-entity action implementations (but both can be overridden by specific configuration in the action definition).
The Last Will and Testament (LWT) is a vital health message and therefore normally has default QoS 1 and retain flag true, unless overridden by setting lwt_qos (integer 0-2) and lwt_retain (boolean) in controller configuration (at the same level as source).
Polling
In the event that a device cannot reliably publish status messages, MQTTController provides for periodic polling of devices. Polled devices must implement the query topic (Additional Entity Configuration above) to be eligible.
Polling of a device is enable and the polling interval set by adding the poll_interval configuration value to the entity's configuration (same level as name, for example). The value is an integer interval in seconds (must be >= 1). MQTTController will send the configured query topic on this interval, or as close to it as practicable (keep reading...).
To prevent polling of multiple devices from causing waves of high traffic on the network, MQTTController will limit the frequency of polling to, by default, once every 5000 milliseconds (five seconds). That is, if a number of devices all become due for polling at the same instant, each will be polled in turn, one every five seconds, until all queries have been sent. This pacing can be modified by setting the poll_frequency configuration value in MQTTController's instance configuration (that is, at the same level as source); the value is in milliseconds and must be >= 100.
The poll_interval cannot be set in a template; it is permitted in entity configuration only. If it appears in a template, it will be ignored.
Additional MQTTController Behaviors
MQTTController will respond with a reactor/<mqtt-id>/pong message when it receives the topic reactor/<mqtt-id>/ping. If a payload is received, it is echoed back; otherwise, the current Reactor host date/time is returned as an Epoch time in milliseconds.
Troubleshooting and Debugging
To see what messages MQTTController is receiving, you can enable topic logging in the config section of its controller configuration. A list of topics or partial topics to match can be provided, and the matching topics will be logged to <id>-topics.log, where <id> is the ID of your MQTTController.
The following will log all topics received from Owntracks and BlueIris, for example (lines before log_topics are shown for context and indenting reference and should not be copied:
implementation: MQTTController
enabled: true
config:
source: "mqtt://127.0.0.1:1883/"
log_topics:
- "owntracks/"
- "blueiris/"
If you're not sure what topic you're looking for and you want to log all topics, the following will match and log all topics received by MQTTController:
log_topics:
- ".*"
Fast- and Ever-Growing Log File Danger
Topic logging is meant to be used briefly, not to be left on. The topic log file will grow forever and eventually fill up the disk. It is not managed, pruned, or truncated on restarts. For this reason, do not leave any topic logging on any longer than you need it. Turn it off and remove the log file from your logs directory to recover and preserve your disk space, and prevent the system instability and errors that an "out of space" condition can cause.
You can also increase the logging level of MQTTController for more detail, and route its log messages to a separate file to separate them from other messages and make them easier to follow. This increased detail and focus should allow you to see if/when a topic is being received, what its payload is, and how the topic and payload are handled. Here is an example addition to your logging.yaml file to increase the log level to 5 (which is probably sufficient in most cases), and direct the output to a separate mqtt.log file. Note that the MQTTController: line is indented two spaces (no tabs).
logging: # For indent reference only - DO NOT copy/paste this line
...etc...
MQTTController:
level: 5
streams:
- type: file
name: mqtt.log
keep: 2
recycle: true
Remember to remove this section (or keep it and just set the level to 4 or lower) after troubleshooting to avoid filling your storage with excessive logging that you no longer need.
List of Pre-defined Templates
The following templates are currently defined. If you work out a custom configuration for a commonly-used device, please consider contributing your template!
| Template Name | Use/Description | Parameters |
|---|---|---|
shelly_relay |
Creates an entity that maps one relay channel of a Shelly node to an entity. | topic, channel |
shelly_input |
Creates an entity that that maps one input channel of a Shelly node to an entity. | topic, channel |
shelly_temperature |
Creates an entity to contain the temperature sensor data from a Shelly device. | topic |
shelly_humidity |
Creates an entity to contain the humidity sensor data from a Shelly device. | topic |
shelly_motion |
Creates an entity to reflect the motion/no motion state of a Shelly motion sensor. | topic |
shelly_smoke |
Creates an entity to reflect the state of a Shelly smoke detector. | topic |
shelly_flood |
Creates an entity to reflect the dry/wet state of a Shelly flood sensor. | topic |
shelly_door_window |
Creates an entity to reflect the opened/closed state of a Shelly flood sensor. | |
shelly_battery |
Creates an entity to contain the battery level of a battery-operated Shelly device (e.g. a Shelly HT). | topic |
shelly_power |
Creates an entity with a Shelly device's power (watts) measurement. | topic |
shelly_tilt |
Creates an entity to contain the measurement of a Shelly device's tilt sensor. | topic |
shelly_vibration |
Creates an entity to contain the measurement of a Shelly device's tilt sensor. | topic |
shelly_handt3 |
Creates an entity for a Shelly H&T Gen3 device. Because the device is battery-powered, attributes will not populate until the first wake-up. | topic |
tasmota_generic_relay |
Maps payload data and actions to the entity (capabilities switch_power and toggle); use optional unit to specify relay number if the device supports/reports more than one (default: blank/no unit). |
topic, unit |
tasmota_detached_button |
Maps payload data for a detached switch input to the entity (capability button); use optional unit to specify switch number if the device supports/reports more than one (default: blank/no unit). |
topic, unit |
tasmota_detached_switch_mode0 |
Maps payload data for a detached switch in mode 0 (toggle/momentary; capability button); use optional unit to specify switch number if the device supports/reports more than one (default: blank/no unit). |
topic, unit |
tasmota_detached_switch_mode1 |
Maps payload data for a detached switch in mode 0 (on/off; capability binary_sensor); use optional unit to specify switch number if the device supports/reports more than one (default: blank/no unit). |
topic, unit |
tasmota_sensor_temperature_humidity |
Maps temperature and humidity values from payload into a single entity (capabilities temperature_sensor and humidity_sensor); use source to identify sensor subunit (default: Global). |
topic, source |
tasmota_sensor_temperature |
Maps temperature (only) from payload into the entity (capability temperature_sensor); use source to identify sensor subunit (default: Global). |
topic, source |
tasmota_sensor_humidity |
Maps humidity level (only) from payload into the entity (capability humidity_sensor); use source to identify sensor subunit (default: Global ). |
topic, source |
tasmota_sensor_battery |
Maps payload battery data to the entity; requires source to identify the sensor subunit (default: Global). |
topic, source |
tasmota_sensor_co2 |
Maps payload carbon dioxide level data to the entity; requires source to identify the sensor subunit (default: Global). |
topic, source |
tasmota_sensor_dewpoint |
Maps payload dewpoint to the entity; requires source to identify the sensor subunit (default: Global). |
topic, source |
tasmota_sensor_rssi |
Maps payload RSSI to the entity; requires source to identify the sensor subunit (default: Global). |
topic, source |
tasmota_sensor_flora2 |
Mi Flora plant sensor for Temperature, Illuminance, Moisture and Fertility, use source to identify subunit (default: Global) |
topic, source |
owntracks_in_region3 |
Creates a binary sensor the state of which is driven by an iPhone or Android device running OwnTracks. The sensor is true when the device is in the given region. The topic is the Device ID assigned in the OwnTracks application's configuration. The regionName is the name of a region defined in OwnTracks. All parameter values are case-sensitive (must match the case used in the OwnTracks app). |
topic, regionName, user |
Community Contributions
Community users are encouraged to contribute templates for new devices. These may either be included in releases, or we'll link to them here if their work is large and growing. To install and use a community template (as individual file(s) or a ZIP archive), create a mqtt_templates subdirectory in your config directory, and place the template file or ZIP archive in that subdirectory. You can then configure entities using the community template(s).
- More Shelly and Tasmota templates from dbochicchio/@therealdb: Github repository.
Special Expression Functions for MQTTController
The following functions are defined for expressions evaluated in MQTTController entity event handlers and action implementations:
hsltorgb( hue, sat, lev )— converts hue/saturation/level color to RGB color. Thehueis given in degrees;satandlevare given as percentage values in the range 0 to 1. The function returns the equivalent (approximate) RGB color as a three-element array. Example:hsltorgb(60,1.0,0.5)returns[255,255,0](yellow).rgbtohsl( red, green, blue )— converts RGB color to HSL; this is the inverse ofhsltorgb(), above.
Using MQTT from Other Controllers
MQTTController has a mechanism that allows other controllers to subscribe to MQTT and send topics. This saves the other controller from having to create and manage a separate connection to the MQTT broker. While it creates a dependency on MQTTController, it is probably the case that a user using MQTT for your controller's device(s) is probably already using MQTT for other devices in their infrastructure, and it's already configured and running.
MQTTController does not require any special configuration for these tools to work. It doesn't even need to be configured beyond being enabled and having a correct source to find the broker. No events, entities, or echo need to be configured; MQTTController just needs to be up and running.
To subscribe to a topic, your Controller instance will need to do the following:
-
Find the running instance of MQTTController. This is done by calling the
getControllerByID(id)method ofStructure, the instance of which you can get from the Controller base class by simply referring tothis.getStructure(). Theidwill typically bemqttfor users that just follow the recommended installation instructions for MQTTController. You may want to make this a configurable string in your controller in case the user deviates from this recommendation, or has multiple MQTTController instances and wants to specify which to use. The following code snippet demonstrates all of this working together in a Controller subclass method:const mqttcontroller = this.getStructure().getControllerByID( this.config.mqtt_controller || 'mqtt' ); -
Write a handler in your Controller subclass to receive your subscribed MQTT topics. You can have as many handlers as you wish: handle different topics with separate handlers, or handle all of your subscriptions in one handler; it's up to you. The handlers are methods of your Controller subclass that will receive arguments
topicandpayloadwhen a topic matching your subscription arrives.handleMyTopic( topic, payload ) { // your implementation here } -
Request a subscription from the MQTTController instance by calling its
extSubscribeTopic( this, topic, callback )method. The first argumentthisis a reference to your controller instance (and typically the keywordthisis exactly what is used in your code). Thetopiccan include the typical+and#wildcards for subscribing to MQTT topics. Thecallbackis a function reference or closure. When a matching topic is received by MQTTController, it will call the function withtopicandpayloadarguments. To use a method of your subclass as the callback, use the JavaScript functionbind()as shown below.// Using bind() to use a method of your controller as handler: mqttcontroller.extSubscribeTopic( this, "opensprinkler/#", this.handleMyTopic.bind( this ) ); // Using a closure as handler: mqttcontroller.extSubscribeTopic( this, "tele/+/status", ( topic, payload ) => { /* Handling code here */ });Note that it is possible to omit the
callbackargument. In this case, your controller'snotify( event )method will be called with aext-mqtt-controllerevent. The topic and payload will be in thedataof the event object.
To unsubscribe from a topic, call MQTTController's extUnsubscribeTopic( this, topic ) method. The topic must exactly match a topic string previously used in extTopicSubscribe(). If topic is omitted or null, all topics to which your controller subscribed are unsubscribed.
To publish topics, call MQTTController's extPublishTopic( topic, payload, opts ) method. The topic must be a valid topic string. The payload can be any type; if it is an object, array, or null, it will be sent as JSON4. Otherwise, it is simply converted to a string and sent. To send no payload at all, specify undefined as the argument (or omit it). The optional argument opts is an object that may specify the key qos with an integer value 0 through 2 (inclusive, default 0) and retain as a boolean (default: false).
mqttcontroller.extPublishTopic( "mycontroller/status/OK" );
mqttcontroller.extPublishTopic( "mycontroller/dimmer/level", 50 );
mqttcontroller.extPublishTopic( "mycontroller/weather/current", { temp: 21, hum: 44 },
{ qos: 1, retain: true } );
This method returns a *Promise* that resolves when the topic has been sent, or rejects if an error occurs.
Publishing Templates for the Community
Best Practices:
- Each template file must contain a
templates:key that is the container for all templates being defined. A template file can define any number of templates. - Each template file should only define templates for a single device/endpoint or type (e.g. it would not be recommended to include Tasmota and Shelly templates in one file).
- Your templates should be published on Github (see the
repositoryfield, above). Please do not distribute your templates through posts in community forums or other similar venues where version control is more difficult for you and your users. - Your template(s) should be distributed as a single YAML file, or if it's necessary/convenient to have several files, as a ZIP archive.
- Please use the
author,version,description,licenseandrepositorytop-level keys (same level as, and preceding,templates:), which should be, respectively, your name and optional email address, a version number for the template, a description of your templates' purpose/supported devices, the type of license applicable (GNU, MIT, etc., or a URL/link to the license file in the repository), and the URL to the Github repository from which the file came and updates can be found.- If you are distributing templates in a single YAML file, these keys should appear at the beginning of the file (example below).
- If you are distributing multiple files as a ZIP archive, these keys can be included in any one of the files; it may be practical to use a manifest file (e.g.
manifest.yaml) to contain only these metadata keys and an emptytemplates: {}line (so the file defines no templates, just metadata).
- Your templates should make use of keyword substitution (in fields like
topic), and where necessary, expression-context values, to make them as re-usable as possible. - Your template names should be unique and descriptive to avoid collisions between templates. For example, calling your template sensor is a poor choice; better would be to use a name like blueiris_camera_motion_sensor. Even though your templates may come from files with more descriptive names, the source filenames will not normally be visible to the user after installation, so the template names themselves must be sufficiently descriptive.
- Your Github repository should contain release notes (in a README or separate CHANGELOG, for example), so your users can understand what has changed between versions.
Example template header (single file):
# Templates for HaMagicWerks Devices (this is a comment)
---
description: "HaMagicWerks Device Templates"
author: "Clyde Terwilligerock <clyde.terw@example.com>"
license: MIT
repository: "https://github.com/clyde_terw/Reactor-MQTT-HaMagicWerks"
version: 22341
templates:
hamw_motion_sensor:
# ...etc...
A manifest file in a ZIP archive would look almost identical to the above, just defining no templates (i.e. it would have templates: {} at the end).
Updated: 2024-Apr-17
-
You can find this template in your MQTTController distribution file
mqtt_devices.yaml. ↩↩ -
Thank you to SmartHome Community user @Crille for contributing this template. ↩
-
Thank you to SmartHome Community user @therealdb for the inspiration and basis of this template. ↩
-
This implies, of course, that the object can be converted to JSON by
JSON.stringify(), which means that it can contain only JavaScript primitive types (string, number, boolean, null),Dateinstances, and arrays and simple, prototype-less objects (i.e. not instances of definedclasses, be they defined by you or JavaScript types likeMap,Set, etc.) containing only these permitted types as elements/values. ↩