Jobs

In addition to the synchronous APIs in the Eliq Data Management API, an asynchronous approach called Jobs is available. Jobs are designed and optimised to handle operations on larger datasets. Examples of such operations include upserting or deleting a set of users, locations, or meters.

Jobs are managed through the Data Management API, which provides endpoints for creating jobs and checking their status. A job is a background process that processes operations in bulk. Each job is associated with an entity and an operation, such as upserting users. Each entity-operation pair has an associated schema, which is described further below.

The supported file format is .ndjson (Newline Delimited JSON).

Life cycle

A job goes through various states between being received until finished. When a job is sent in to the Data Management API, the state is set to queued. This means that the job was successfully created and ready to be processed. When Eliq’s asynchronous job-processors starts processing the job, the state is set to processing. When the job finishes, it enters one of the states completed or failed. If all lines were imported successfully, the state is set to completed. If one or more rows in the import failed, the state is set to failed. Failed files are retried, more information about this can be found in section Error handling.

A job in state processing is updated (by best effort) every 30 seconds with the number of handled rows. This can be used to follow up on the process, and verify that the processing is ongoing.

Processing order

All jobs created in Eliq’s system are processed sequentially, and Eliq ensures that the processing order remains the same as it was received. This is because jobs may have dependencies, e.g., a user must be created before a location can be created.

NOTE: While Eliq ensures that jobs are processed in order, Eliq DOES NOT ensure processing order within a file. Sending multiple updates for the same entity in a job may result in race conditions. To avoid issues, ensure that the IDs are unique within a job.

Example: If the file associated with the job contains two rows where the name of a user with ID x is updated, Eliq does not ensure that the last row in the file will be processed last.

Error handling

As mentioned in the "Life Cycle" section, the state of the job is set to "failed" if one or more rows fail during the import. Successful rows in a failed job will not be rolled back, so there may be partial success in a job. A job in the "failed" state will be retried up to three times before moving on to the next job. If the job is not successful after three attempts, its state remains "failed."

Please note that a failed job may result in a cascading effect. For example, if a user fails to be created due to a badly formatted row, this may cause a location in a later job to fail during import. Therefore, it is important for the client to monitor the process of these jobs to be aware of such issues.

All indexes of the rows that failed, along with an error code and description, are stored. This information is available for retrieval from the Data Management API. Possible error codes are described at the endpoint level below, but in short, they will be the same as the error codes from Eliq’s synchronous endpoints in the Data Management API.

Note: If a file has more than 1000 rows failing during processing, the process will stop and the state will be set to "failed." This is a precautionary measure to save computing power in case of formatting issues or temporary internal problems. The file will be retried according to the retry behaviour explained above.

Supported jobs

Each job is linked to a entity and operation combination. The currently supported combinations can be found below

API

API Documentation of the jobs endpoint can be found in Eliq Data Management API documentation.

A file us uploaded to the Data Management API when creating a job, and example files can be found below. This is sent to the Data Management API via a POST request using Content-Type: multipart/form-data, and is done like described below.

Request header

Content-Type: multipart/form-data;
boundary=EXAMPLE_BOUNDARY_VALUE

Request body

--EXAMPLE_BOUNDARY_VALUE
Content-Disposition: form-data; name="content"; filename="<your-reference>.ndjson"

(Content of your .ndjson file)
--EXAMPLE_BOUNDARY_VALUE--

Notes

What this endpoint expects is the same as what a browser will send when posting a form that contains a file input named content.

HTML example (will not actually work because of missing Authorization header):

<form method="post" action="{base-uri}/integration/api/v1/jobs/user/upsert" enctype="multipart/form-data">
	<input type="file" name="content" />
	<input type="submit" />
</form>

curl example (without Authorization header):

curl -F "content=@/tmp/users221019.ndjson" {base-uri}/integration/api/v1/jobs/users/upsert

File format and examples

The currently supported format is .ndjson (Newline Delimited Json). Newline Delimited JSON is a file format where each line consists of a valid JSON object, and the line separator is '\n'. These files are convenient for streaming and storing structured data where each line can be handled individually. More information can be found here: ndjson

Example files

User upsert

{"email":"example-user-1@domain.com","language_code":"en-GB","name":"Forename 1","id":"example-user-id-1","phone":null}

See user model definition in Data Management API docs for model documentation

User delete

{"user_id":"example-user-id-1"}

User export

{"limit":"1000", "after_user_id":"example-user-id-1", "created_after":"2021-01-01T01:01:01.00Z"}

Location upsert

{"user_id":"example-user-id-1","id":"example-location-id-1","address":{"country_code":"SE","postal_code":"41537","street_address":"Address","city":"Gothenburg"}}

See location model definition in Data Management API docs for model documentation

Location delete

{"location_id":"example-location-id-1"}

Meter upsert

{"id":"example-meter-id-1","fuel":"elec","location_id":"example-location-id-1", "user_id": "example-user-id-1", "supply_start_date":"2021-01-01","type":"import","model":"Eliq Internal - Standard Energy V1"}

See meter model definition in Data Management API docs for model documentation

Meter delete

{"meter_id":"example-meter-id-1"}

Energy-daily upsert

OBS! The maximum number of elements in the ‘values’ list is the same as in Data Management API (366 elements). Hence, in historic imports, one meter might need to be split up into two rows in.

{"meter_id":"example-meter-id-1","values":[{"date":"2022-11-01","is_estimated":false,"energy_wh":690}]}

Energy-daily delete

{"meter_id":"example-meter-id-1","from":"2022-10-01","to":"2022-10-01"}

Highres-energy-{resolution} upsert

OBS! The maximum number of elements in the ‘values’ list is the same as in Data Management API (500 elements). Hence, in historic imports, one meter might need to be split up into two rows in.

{"meter_id":"example-meter-id-1","values":[{"interval_start":"2022-10-31T23:00:00Z","energy_wh":100}]}

Highres-energy-{resolution} delete

{"meter_id":"example-meter-id-1","from":"2022-10-01T00:00:00Z","to":"2022-11-01T00:00:00Z"}

Highres-energy-{resolution} export

{"meter_id":"example-meter-id-1","from":"2022-10-01T00:00:00Z","to":"2022-11-01T00:00:00Z"}

Homeprofile upsert

{"location_id":"example-location-id-1","values":[{"key":"house_type","value":"apartment"}]}

Homeprofile delete

{"location_id":"example-location-id-1"}

Homeprofile export

{"location_id":"example-location-id-1"}

Priceformula upsert

{"meter_id":"example-meter-id-1","from":"2021-01-01","to":"2023-10-01","currency_code":"SEK","elements":[{"name":"unit_rate_fixed","type":"unit_rate","unit_rate":{"type":"fixed","fixed":{"unit_rate":1.25}}}]}

The model is extending price formula model with parameter meter_id. Please find the rest of the price formula model definition in Data Management API docs.

Priceformula delete

{"meter_id":"example-meter-id-1","from":"2021-01-01"}

Insights-energy-{resolution} export

{"location_id": "example-location-id-1", "from": "2021-01-01", "to": "2021-01-01", "fuel": "elec", "direction": "import", "include_incomplete": true}

Insights-advice

{"location_id": "example-location-id-1", "state": "new", "limit": 5}

Similar-homes-{resolution}

{"meter_id":"example-meter-id-1","from":"2022-10-01T00:00:00Z","to":"2022-11-01T00:00:00Z"}

Highres-energy-{resolution} export

{"meter_id":"example-meter-id-1","from":"2022-10-01T00:00:00Z","to":"2022-11-01T00:00:00Z"}

Homeprofile upsert

{"location_id":"example-location-id-1","values":[{"key":"house_type","value":"apartment"}]}

Homeprofile delete

{"location_id":"example-location-id-1"}

Homeprofile export

{"location_id":"example-location-id-1"}

Priceformula upsert

{"meter_id":"example-meter-id-1","from":"2021-01-01","to":"2023-10-01","currency_code":"SEK","elements":[{"name":"unit_rate_fixed","type":"unit_rate","unit_rate":{"type":"fixed","fixed":{"unit_rate":1.25}}}]}

The model is extending price formula model with parameter meter_id. Please find the rest of the price formula model definition in Data Management API docs.

Priceformula delete

{"meter_id":"example-meter-id-1","from":"2021-01-01"}

Insights-energy-{resolution} export

{"location_id": "example-location-id-1", "from": "2021-01-01", "to": "2021-01-01", "fuel": "elec", "direction": "import", "include_incomplete": true}

Insights-advice

{"location_id": "example-location-id-1", "state": "new", "limit": 5}

Similar-homes-{resolution}

{"location_id": "example-location-id-1", "from": "2021-01-01", "to": "2021-01-01", "fuel": "elec", "unit": "energy"}

Location-forecasts-{resolution}

{"location_id": "example-location-id-1", "fuel": "elec", "unit": "energy", "from": "2021-01-01", "to": "2021-01-01"}

breakdown – euc export

{"location_id": "example-location-id-1", "from": "2023-01-01 01:00", "to": "2023-11-01 01:00", "included_monetary_value": false, "fuel": "elec" }

Consent export

{"user_id": "example-user-id-1", "from": "2023-01-01", "consent_name": "example-consent" }

Energy-period delete

{"meter_id":"example-meter-id-1", "period": { "from":"2022-10-01T00:00:00Z","to":"2022-11-01T00:00:00Z"}}

Energy-period export

{"meter_id":"example-meter-id-1", "period": { "from":"2022-10-01T00:00:00Z","to":"2022-11-01T00:00:00Z"}}

Energy-period upsert

{"meter_id":"example-meter-id-1","from":"2022-10-01T00:00:00Z","to":"2022-11-01T00:00:00Z","energy_wh":10.5,"is_estimated":false,"registers":[{"id":"example-register-1","energy_wh":10.5,"energy_m3":null}]}

Notification-history export

{"user_id":"example-user-id-1","notification_id":1,"number_of_notifications":10}