Background Tasks

Background tasks are run with the RQ (Redis Queue) library.

The Django-RQ package is used to integrate with Django.

The rq-scheduler package is used to schedule the periodic update tasks.

The background tasks code is: hub/meters_tasks/tasks.py.

Add Usage Records - Initial Call

When a meter is consented, it needs to call the function add_usage_records(meter_id) to fill our database with usage records from the api.

This function has the @job decorator, meaning it can be called with add_usage_records.delay(meter_id) and will work with RQ.

The meter_id is the primary key of the meter.

This function in turn calls get_next_usage_records(meter_id, last_elec_timestamp(meter_id), utility) for each utility present in the meter. This gets the next 90 days of records of usage for each utility (electricity or gas).

90 days is the maximum number of records available at one time from the API. The functions then recursively call themselves to get the next 90 days. Each call creating electricity usage records also calculates emissions using data from the carbon intensity API and catches specific error values which are defined in the Smart Energy Code, which result in an usage record of 0 flagged with an INVALID error. Implausibly high values are also detected and logged as OVERFLOW errors.

Update Usage Records Periodic Call

We also need to periodically update our database with the latest usage records. A second task is used for this: update_meter_records()

This task is run halfhourly. It will find all the consented meters in the database and call the add_usage_record() function on them all in turn.

This task will detect any calls which are rejected due to a removal of consent by N3RGY - this can occur if N3RGY detect that the user may not longer be the current occupant of the property. In this case, the MeterConsent object is flagged as unconsented, and the date and reason for the deconsenting is recorded, and an alert is sent to Slack.

Inaccessible Meters

If N3RGY reports that a meter cannot be reached we offer the user the opportunity to request a notification should the meter become available. These inaccessible meters are re-attempted once per week by check_inaccessibles(), a task which is run daily. This could be run more frequently to reduce the size of the batches being tried at once.

Membership check

A check is run each day at 5:30am to check for expired memberships among any meter owners. If a meter is accessed without a subscription payment by virtue of belonging to a Carbon Co-op member, then the meter will be deconsented at the expiration of the grace period for membership payment (three months after the expiration date of the membership).

Garbage Collector

The garbage collector runs at 3am each day to purge abandoned Signups and MeterConsents. Signups are considered abandoned after 24 hours if no data has been entered, or after 3 months if data was entered but the signup never resulted in a consented meter or a subscription.

The garbage collector also checks for meters for which no data has been received. A warning is triggered to admin if no data has been received for a new meterconsent two days after consenting. This is a chance for an admin to use n3rgy’s separate tools to attempt to establish communications with the meter. If no data has been received after seven days the meter is automatically deconsented, the subscription cancelled and a refund is attempted.

Gap Filling

The gap filler is run daily at 6am. It is performed separately for gas and electricity data.

  • Any halfhours within the data record for which no usagerecords exist are identified.

  • A request is sent to n3rgy for these gaps. If they can be filled in, they are. Otherwise the halfhourly record is completed by inserting a NODATA zero valued error record for each missing halfhour. The date at which the retry took place is stored.

  • Each NODATA record is then retried according to a schedule defined in meters_tasks.selectors.get_nodata_records_to_retry(): #. Three days after the first retry #. Two weeks after the second retry #. Four weeks after the third retry

  • After this fourth retry the gap is abandoned, though the administrator can trigger a further attempt to fill gaps through the admin system at any time.

In order to minimise calls to the n3rgy API these tasks are not performed separately in this order. Instead, all the gaps which are going to be reattempted are compiled first and then one round of queries is made to n3rgy to cover these gaps in as few calls as possible bearing in mind the 90 day maximum period that can be queried.

Note that because meter data can lag by up to a few days - and in the case where meters are prodded back into communications by the n3rgy tools, months or even years - the actual lag between the retry and the missing timestamp could be anything.