VirtualEntityController

VirtualEntityController (abbreviated VEC) is a controller that creates and manages virtual entities (or devices) in Reactor. A virtual device/entity is one for which no physical device exists in the real world. Virtual devices are often used to help track states or provide configurable options within your automations. Some examples:

A virtual switch called "Party Mode" that may set certain scenes and also disable timed-auto-off automations of indoor and outdoor lights (don't stop the party!);
A virtual value (numeric) sensor canned "Pump Hours" may be used to accumulate the daily runtime of pool or fountain pump;
A virtual string sensor may be used to store a multi-valued mode or state;
A virtual binary sensor may be used to periodically query a remote RESTful API using an HTTP request and set the sensor state according to the response data.

Virtual entities can be used in automations just like any other entities.

The attributes of virtual entities can be either static or dynamic. Static attributes are those that just hold a value, like a variable. They can be modified using an action (described below) in your reactions. Dynamic attributes are derived by supplying source configuration, which may include an expression, that drives them. In these ways, the attributes of virtual entities are very much like Global Expressions.

Creating Virtual Entities

The first step in creating virtual entities is to make sure VirtualEntityController is a configured controller in your system. Add the following to the controllers section of your reactor.yaml file (do not add the controllers: line shown below, that's just to help you indent the rest properly):

controllers:  # do not copy this line
  - id: virtual
    name: VEC
    implementation: VirtualEntityController
    enabled: true
    config:
      entities:
        -
          id: A

After adding this configuration to your reactor.yaml, restart Reactor and hard-refresh any browser pages open to the Reactor UI. You can then go to the Entities list (left navigation), clear any previous filters, and enter "virtual" in the name search field — you should see a couple of new entities.

The configuration above creates a default virtual binary switch entity. This is done by the entities section, which is an array (the - in YAML indicates the start of a new array element). The id: A line tells VEC that the new virtual entity will have the ID A. You can assign any ID you want to a virtual entity, but you need to know:

The ID is required; you have to supply it; it cannot be generated for you;
The ID has to be unique; you can't use the same ID for another virtual entity (but it does not matter if an entity from another controller has the same ID);
The ID must be 1-64 characters long and only contain alphanumeric characters and underscore (i.e. upper- and lower-case A-Z, digits 0-9, _); IDs are case-sensitive (i.e. a and A are different IDs);
You should never change it — Reactor conditions and actions will find the entity by its ID, and if you change the ID, all of those conditions and actions will break.

When we only assign the id in configuration as we've done here, Reactor defaults the entity type to Binary Switch and supplies a default name. VEC has a few preconfigured entity types, and you can choose one by adding the template field with one of the following pre-defined values:

Binary Switch — a typical on/off switch, implementing the power_switch and toggle capabilities with actions common to switches;
Dimmer — a dimmer with switch capabilities extended with a settable (0%-100%) level;
Binary Sensor — a binary state entity that has no actions;
Value Sensor — an entity that stores a numeric value (and has no actions);
String Sensor — an entity that stores a string value (and has no actions).

When setting the template value, make sure you spell the template name exactly as shown, including capitalization and spaces:

    config:
      entities:
        -
          id: A
          template: Binary Switch

You can supply a name for a virtual entity by adding the name field, like this:

    config:
      entities:
        -
          id: A
          template: Binary Switch
          name: Guest Mode

Since the Binary Switch and Dimmer entity types support the actions normally supported by switches and dimmers, you can use these actions to change the state of these entities. They also appear in the Reactor dashboard as switches and lights normally would, respectively, and can be operated the same way as well.

The attribute values of the sensor entity type have to be set using the x_virtualentity.set_attribute action. Your automations (specifically Reactions) can use an Entity Action on the virtual sensor entity, selecting this action, and then providing the attribute name and value to be set.

Customizing Virtual Entities

Sometimes a template may not do exactly what you want, so you either need to modify it, or just create your own virtual entity with the capabilities, attributes and actions you want.

Let's say we want to add color temperature capability to a virtual dimmer for some reason. To do this, we would need to add the color_temperature capability to an entity. Here's how our new configuration would look:

    entities:
      -
        id: A
        template: Binary Switch
        name: Guest Mode
      -
        id: color_dimmer
        template: Dimmer
        name: Virtual Color Dimmer
        capabilities:
          - color_temperature

If you add this to your reactor.yaml and restart, you'll see the Virtual Color Dimmer entity with the capabilities assigned by the Dimmer template (power_switch, toggle, and dimming), and you will also see that added capability color_temperature. Now this is a trivial example, and in this case, only the attributes of color_temperature are available; none of the actions. The reason is that we would need to teach VEC what the actions do, and we're not quite ready for that yet.

Let's say you don't want to use any template, and just define your own virtual entity that has, for example, the hvac_control, hvac_heating_unit and temperature_sensor capabilities — the basis of a simple virtual thermostat. That might look like this:

    entities:
      -
        id: virtual_therm
        name: Virtual Heating Thermostat
        capabilities:
          - hvac_control
          - hvac_heating_unit
          - temperature_sensor
        primary_attribute: hvac_control.state

Here again, we are directing VEC to construct a virtual entity with the listed capabilities. It will have the attributes of those capabilities, and you'll be able to set them using x_virtualentity.set_attribute, but it won't have any of the actions until you define them (some day, later...). Remember, things like tracking the current thermostat mode, and turning the heating unit on when the temperature is below the setpoint, are functions performed in real life by the device behind the entity, but here in virtual entity world, there is no device, so there's nothing to manipulate the attributes automatically or perform an action requested.

Defining Dynamic Attributes

The attributes of virtual devices don't need to be static. Let's say, for example, that we want to create a binary sensor that's true when our Hubitat Elevation's Mode is Night, but false otherwise. We'll start by using the Binary Sensor template, but we're going to customize just the binary_sensor.state attribute. To do that, we'll need a slightly different construction than we used in the previous section's examples.

    entities:
      -
        id: mode_night
        name: Mode is Night
        capabilities:
          binary_sensor:
            attributes:
              state:
                expr: getEntity( 'hubitat>mode' ).attributes.string_sensor.value == "night"

In the previous section's examples, the capabilities value was an array of capability names. In this example, we've changed it to an object; the keys of this object are capability names (e.g. binary_sensor). Within a capability, the key attributes is used to tell VEC we're configuring attributes in that capability, and finally, state is the name of the attribute in the binary_sensor capability that we are defining. The expr value is an expression that will drive the value of the attribute binary_sensor.state on this entity.

Dynamic attribute expressions are sensitive to dependencies. In the example above, VEC learns that the expression depends on the hubitat>mode entity, and will automatically re-evaluate the expression every time that entity changes, without using outside Rules or Reactions. And, these expressions can be as complex as you'd like.

If you have long expressions, use block text like this to keep your configuration more readable:

expr: |
  day_of_week = dateparts().weekday,
  getEntity( 'hubitat>mode' ).attributes.string_sensor.value == "night" &&
  ( day_of_week == 0 || day_of_week == 6 )

In some cases, you may not want to update an entity's attribute based on some other condition. The if_expr key can be used to define an expression that determines whether the attribute should be updated at all, or skipped. If the result of the expression is true, or if_expr is not provided, the attribute will be updated; otherwise (i.e. when the if_expr result is false), the attribute is skipped.

HTTP Request-driven Entities

VirtualEntityController incorporates a mechanism for making HTTP requests to fetch data that can then be used to set attributes on virtual entities. Here's an example configuration:

      entities:
        - id: "H"
          name: "IP Address Changed"
          template: Binary Sensor
          http_request:
            url: "https://api.ipify.org?format=json"
            interval: 3600
          capabilities:
            binary_sensor:
              attributes:
                state:
                  expr: "response.ip != '10.56.71.123'"
            string_sensor:
              attributes:
                value:
                  expr: response.ip

Here you can see we've added an http_request section to the configuration of an otherwise typical virtual binary sensor. With this configuration, VirtualEntityController will request our current public IP address from the ipify.org API every hour. VEC places the response from the server in a variable called response in the expression context that is used by each attribute's value expression. In this case, we've told api.ipify.org (via query parameter on the URL) to respond with a JSON object (i.e. { "ip": "10.56.71.123" }), and VirtualEntityController knows to convert that into an real object for use in the value expression. We've set up the value expression for binary_sensor.state so that it's true if the IP address returned by the API is not the value we expect (that is, our public IP address has changed for some reason). Additionally, this entity is extended with the string_sensor capability, and its value attribute receives the IP address returned by the API (just so we can see it).

The following are the configuration keys known in http_request:

url (required):

The HTTP(S) request URL. This is a required value, and it must be enclosed in quotes.

interval:

The request interval in seconds (a positive, non-zero integer). There is no specific control over the basis of the interval; that is, an interval of one hour is not guaranteed to run at the top of every hour (0 minutes after the hour), and in fact, it's pure luck that it ever would. For strict control of request time, see at below.

at:

An array of fixed times of day (24-hour form) at which the request should be made. Times must be specified as "HH:MM" strings, as shown below:

at:
  - "17:30"  # 5:30pm
  - "00:00"  # midnight

The times do not need to be specified in order. If the request is to be run only once per day, the abbreviated at: "HH:MM" single-line form may be used.

If at is used, interval is ignored.

method (optional):

The HTTP request method. If not specified, GET is used.

auth_type (optional):

If HTTP authentication is required, indicate the type, which must be basic or digest. Omit this key and value if authentication is not required.

auth_username (optional):

Username for HTTP authentication. This value should be enclosed in quotes.

auth_password (optional):

Password for HTTP authentication. This value should be enclosed in quotes.

headers (optional):

This can either be an array of strings formatted key: value (header name/key + colon + space(s) + value), or an object containing key/value pairs.

# This is array form:
headers:
  - "X-API-Key: 23904823985klj238"
  - "X-Auth-Type: simple"

# This is object form:
headers:
  X-API-Key: "23904823985klj238"
  X-Auth-Type: "simple"

retry_interval (optional):

If a request fails, the next request attempt will be made on this interval rather than interval or next at time. Units are seconds, so the value must be a positive integer, and the default is 60 seconds. If the retry interval is 0, it is ignored and scheduling proceeds as if the request had succeeded.

retry_limit (optional):

If a request fails, up to this many attempts will be made using the retry_interval. If the number of retries exceeds this limit, further retries will fall back to interval or at timing. This must be a positive integer or zero; the default is zero, meaning retries on retry_interval continue indefinitely.

quiet_failure (optional):

If an HTTP request fails and this value is boolean true, VirtualEntityController will never issue a notification (shown in the Status page's Current Alerts widget, if present). If this value is an integer, then no notification is issued until the number of attempts exceeds the given value (e.g. if 3 is given, no notification is sent for the first three request attempts, but if the fourth attempt fails, a notification is then sent. Request failures are always logged in Reactor's log file(s) regardless of this setting. See Handling Request Failures below.

skip_update_on_error (optional):

If a request fails and this flag is true, all of the capabilities and attributes of the entity will be left as-is (i.e. the entity will be treated as if the request had not occurred). The default is false, meaning each attribute will be updated and you need to handle the null response value in your expressions. See Handling Request Failures below.

dump_response (optional):

This boolean flag, when true, logs the entire response from the server to the Reactor log file so that you can inspect it. This is helpful for debugging and navigating JSON objects for specific elements of data. JSON data may not be formatted nicely for human reading, but you can copy-paste it into tools like jsonlint.com to improve their readability.

ignore_invalid_certs (optional):

When true, HTTPS requests to servers using self-signed or invalid certificates are permitted. Default is false.

force_json (optional):

If your server returns a JSON response but does not declare it as such (i.e. by failing to provide the required Content-Type: application/json response header), this boolean flag can be used to force VirtualEntityController to handle the response as JSON. It can also be used in instances where a server-side failure produces a 200 status reply with a text or HTML error message rather than JSON containing an error indicator. Any non-JSON response will cause the request to fail (which can then be handled by other request flags or your expression(s)).

body (optional):

The HTTP request body, if any. This can be a string or an object. If an object, it will be converted to a JSON string. It is required that you supply a Content-Type header if you are providing a request body. You do not, however, need to supply Content-Length; VirtualEntityController will compute and supply it based on the body given.

Handling Request Failures

When an HTTP request fails, the reason for the failure is always logged, and the x_virtualentity.last_http_status attribute is updated. Since there is no valid response from the server, the response expression variable will be null.

The quiet_failure request configuration flag will control if (or when) a notification is sent to the Reactor UI (displayed in the Current Alerts widget on the Status panel). See the detail for that flag in the previous section, above.

If you have set the skip_update_on_error flag to true, your expressions will not be run and the attributes of your entity will not be affected. They will simply keep whatever value they last had until a successful request later updates them.

But if skip_update_on_error is not provided or is false, your expressions will still be run and receive the null value for response. Unless you handle this possibility, you will get additional errors from your expressions. To handle the null and avoid expression errors, here are some suggestions:

To avoid updating a single attribute, give it a if_expr: "! isnull( response )". This will cause that attribute's value expression to be skipped, and that attribute will be left with its current value until a later request succeeds.
For attributes where you want to provide a signalling value to your automations that no data is available, handle the null response variable in your value expression as you would for any other variable. For example, you may use the coalesce operators to provide a default value: response?.current?.temperature (the result value will be null if response is null). Or, you could use the ternary operator or an if-then-endif block:
```
isnull( response ) ? -1 : response.current.temperature

if isnull( response ) then -1 else response.current.temperature endif
```
Both of these examples return -1 when response is null. There are countless other ways to handle it, and the value that you use is entirely up to you.

Note that using skip_update_on_error: true is different from using if_expr in that the former applies to all attributes on the entity, where the latter applies only to the attributes having if_expr in their configurations. This gives you a lot of flexibility in handling request errors and what happens to the attributes.

Time Series

Experimental!

This feature is in the experimental stage and subject to breaking changes and, if it doesn't pan out, being completely changed or dropped. Feedback is welcome!

It is often useful to collect data for analysis as a series of values over a period of time, referred to as a time-series. VEC allows you to create a time-series from the value of another attribute on another entity, or the value of a global variable. Some instances where a time-series may be helpful in handling a device may include:

The device sends updates frequently and causes excessive rule evaluation. Using a time-series can dampen-out the frequent updates from the device and reduce them to a reasonable set to handle.
You want to respond to trends in data, and you need a way to detect them. For example, you have a humidity sensor in a bathroom, and you want to use it to control an exhaust fan. A fixed trigger value isn't working, because the natural change in humidity inside your home over the course of a day and from season to season changes gradually, but you notice that there's a sudden change over short period when showering/bathing occurs (so the rate of change is more important than the actual values).

To define a time-series in VEC, add the following config to your virtual entity's attribute definition:

    config:
      entities:
        -
          id: time_series
          name: "Time Series Example"
          capabilities:
            value_sensor:
              attributes:
                value:
                  model: time series
                  entity: "weather>home"
                  attribute: "wx.temperature"
                  interval: 60  # minutes
                  retention: 240  # minutes
                  aggregate: sma
                  depth: 3
                  precision: 2  # round the result to 2 digits right of decimal
          primary_attribute: value_sensor.value
          type: ValueSensor

The model key with the string value time series tells VEC that you want to collect data in a time-series. The source for the time-series is defined by entity and attribute, which are the entity (by canonical ID or name) and the attribute (specified as capabilityName.attributeName, e.g. temperature_sensor.value) on the entity for data collection. Alternately, you may use global_variable: name to use the named global variable as the source for the series.

The interval value, expressed in minutes, is the interval for sampling the source. On this interval, the source entity/attribute or global variable value will be collected and stored. The retention value, also in minutes, defines the length of time over which the data are stored. In the example given here, we are sampling the data on a 60-minute (one hour) interval, and keeping the data for 240 minutes (four hours). That means the storage will contain at most 5 samples: the most recent sample at time t, and samples at times t-60, t-120, t-180, and t-240 minutes (or n = floor(retention / interval) + 1). It is recommended, although not required, that the retention value be a multiple of the interval.

Info

Time is the essence of the time-series collection. Restarts of Reactor do not cause additional samples to be collected. Changes in the source value do not cause additional samples to be collected. Samples are only collected on the given interval, and the value collected is whatever the value is at that moment. That means sudden, short changes in value that occur between two intervals may be missed. Perhaps some day, Reactor will do event-based collection and resample the data, but for now, it's strictly managed by the clock.

The final step is to process (or aggregate) the time-series data to a scalar value, which is then stored on the virtual entity's designated attribute (in this case value_sensor.value). The aggregate configuration directive is given a (string) method name to define how that scalar value is computed. Additional configuration directives modify the method. The following table defines the supported methods:

`aggregate` value	Method Description
`sma`	Simple Moving Average — the average of the time-series values is computed and given as the scalar result. The optional `depth: <integer>` directive may be given to reduce the "lookback" of the average to the given number of samples. In our example configuration, we are keeping five samples in the series, but if we only want to consider the last three (i.e. from the last two hours), we can specify `depth: 3`.
`wa`	Weighted Average — the average is computed, weighting each value considered with a factor. Factors may be given in an array called `weight`, with the first element of the array applying to the most recent sample in the series, the second weight element applying to the most recent sample's predecessor, and so on. If no `weight` array is given, a default set that weights newer values higher than older values is used. To preserve the range of result values, the number of weights given should equal `depth` (i.e. if `depth` is 3, supply 3 weight values), and their sum should be 1.0 (the default set will follow these rules).
`ses`	Simple Exponential Smoothing — the time-series is put through an exponential smoothing algorithm with a given `alpha` (0.0 < `alpha` < 1.0; default 0.61803; closer to 0.0 means more aggressive smoothing). The scalar value is the last value in the smoothed series. This algorithm may be most useful for data that is given to sudden, short spikes in value. Hint: if you turn on debug level 5 for VirtualEntityController in `config/logging.yaml`, the time-series values and smoothed values will be written to the log in CSV format that you can import into Excel and plot in a graph. This may help you determine a suitable value for `alpha`.
`rate`, `derivative`	The rate of change of the time-series. The `depth` may be specified (see `sma`) to limit the "lookback" in the series; default `depth` for this method is 2 (meaning the reported rate of change will be that of the two most recent samples; if `depth` is 3, the rate will be determined from the most recent sample and its predecessor's predecessor; another way to look at this is to number the samples backwards, with the most recent sample being 1, its predecessor is 2, etc.). The rate is expressed in units of the source value per minute (i.e. if the source value is temperature reported in degrees Celsius, then the scalar value's units will be degrees Celsius per minute).
`accel`, `derivative2`	The acceleration of the recent samples (i.e. the rate of change of the rate of change, the second derivative). Here, too, `depth` may be applied, and like `rate`, the default `depth` is 2. The acceleration is computed by comparing the rate of change of the most recent `depth` values to the prior group of `depth` values. The time series must have at least `2 * depth - 1` samples. See the detailed example and explanation below.
`first`, `oldest`	The first (i.e. oldest) value in the time-series is returned.
`last`, `latest`	The last (i.e. latest) value in the time-series is returned.
`max`	The maximum of the time-series' values is returned.
`min`	The minimum of the time-series' values is returned.
`median`	The median of the time-series' values is computed and returned as the scalar result.
`raw`	The time-series itself is handed back as the value. This exposes the time-series data to an `expr` configuration directive that allows you to write an aggregate algorithm of your own using the expression language. The time-series data will appear in that context in the `value` variable as an array of objects with keys `time` (milliseconds since the Epoch) and `value` (in units of the source).

The configuration keys expr and precision can be used to affect the aggregate's result value before it saved to the target entity's attribute. When using expr, the result value of the aggregations is available in a local variable called value. Example: expr: "value*2" will double the result of the aggregation, and precision: 2 will round the result to two digits right of the decimal point.

Time Series Example 1: weighted average

Here's an example of using wa (weighted average) to look back at the last four samples, with significant custom weighting on the two most recent samples:

                  model: time series
                  ...
                  aggregate: wa  # Weighted moving average
                  depth: 4
                  weight:
                    - 0.50  # most recent sample, 50% weight
                    - 0.35  # 35%
                    - 0.10  # 10%
                    - 0.05  # 5% oldest sample of the four considered

Again, if you don't supply a weight array, Reactor will compute default weights. If you supply a weight array but it's too short (i.e. fewer than depth values in the array), missing weights are treated as zeroes (0.0); if the array is too long, the excess weights are simply ignored.

Sum of Weights = 1.0

In order for the scalar result to be in a similar range of values to those in the time-series, it's very important that you supply all of the weights needed, no more no less, and that the sum of the weights is 1.0. If either of these is not true, then your scalar result will be skewed out of the range. That may be OK, and you may even find crafty ways to use that fact to produce useful results, but for most users, sticking to perfect size/perfect sum is the way to go.

Time Series Example 2: `accel`

Here's an example of a time-series of 15 minute samples from am entity, comparing the acceleration of the most recent hour to that of the previous hour:

              attributes:
                value:
                  model: time series
                  entity: "mqtt>garage_humidity_sensor"
                  attribute: "humidity_sensor.value"
                  interval: 15  # minutes
                  retention: 120  # minutes
                  aggregate: accel
                  depth: 5

The interval for this example is 15 minutes, with a retention of 120 minutes (two hours), so Reactor will store 9 samples (as shown here, 9 is oldest and 1 is most recent):

Sample	9	8	7	6	5	4	3	2	1
time	t-120	t-105	t-90	t-75	t-60	t-45	t-30	t-15	t

Acceleration is the change in rate, so two rates must be computed to be able to compute the change in rate. The meaning of depth for this aggregate is the depth of the rate samples. Given the depth of 5 here, Reactor will compare the rate of change for the most-recent hour's samples 1 through 5 with the prior hour's samples 5 through 9. Notice these overlap at sample 5, t-60, because it is considered to be the start of the most recent hour and also the end of the prior hour. Because two rates are computed using depth, this aggregate requires at least (2*depth)-1 samples in the time series. If there are insufficient samples, the result of the aggregate will always be null.

Defining Actions

Yes, you can define actions for the capabilities extended to a virtual entity. Some day, this will get written. I'm not sure anyone will actually need to do this. But it's possible.

Updated: 2024-May-27