Aggregates
Vessel Insight apply both real time aggregation and pre-aggregation to provide performant responses to data queries used in the Aggregates, vesselsensorevents and OnGrid endpoints.
The aggregate interval options are:
1s, 5s, 10s, 1m, 5m, 10m, 1h, 6h, 1d, 1w, 1M, 3M, 1y
Real time aggregate freshness
Any aggregate levels below 5m utilize the raw data to compute the aggregates to the API consumer, thus these will always be fresh within few minutes if raw data is available.
Pre-aggregation freshness
Aggregates from 5m and up utilize the pre-aggregates which is depending on the pre-aggregation policy. Calculation of pre-aggregates are initiated on schedule every 10 minutes, and will calculate the previous 10 minute window with a 10 minute buffer offset. This means that aggregates are not generated until after 10-19 minutes for fresh data.
Lets look at a few examples:
- 5m aggregates for the period of 11:00 to 11:09. Policy starts after 10 minutes buffer time, which could be any time after 11:20 (but before 11:30), depending on system resources. Aggregates will start coming for the period 11:00 to 11:09 any time after 11:20.
- 5m aggregates for the period of 11:00 to 11:10. Policy for 11:00 to 11:09 starts as per above example. Policy for period 11:10 and onwards starts after 10 minutes buffer time, which could be any time after 11:30 (but before 11:40), depending on system resources. Aggregates will start coming for the period 11:10 any time after 11:30.
- The same concept for freshness will apply for all aggregates above 5m
- 6h and 1d aggregates are starting from 00:00
- 1w aggregates are starting from Monday at 00:00
- 1M, 3M and 1y aggregates are starting from the 1. day of the month at 00:00
Freshness and backfill
For backfilled data, or delayed data coming from data sources, there are 2 different processes that manages the aggregates.
Backfill newer than 30 days
When backfills with data from less than 30 days are received, the backfill is processed immediately by the real time ingest, and aggregate freshness is the same as described above.
Backfill older than 30 days
Any backfill older than 30 days will be added to a scheduled backfill queue, and will undergo a delayed processing. Aggregates are refreshed after this delayed ingest.
The timing of the scheduled backfill process is not fixed, but in general it is initiated every 48 hours.
Time Weighted Average aggregate
In cases where we are interested in accumulating values from e.g. a flow rate sensor or a power sensor and the sampling frequency is irregular, the standard "Average" aggregate is not sufficient. We are providing a Time weighted average as a new aggregate in the vesselsensorevents endpoint. This new aggregate will interpolate values to ensure a proper average for accumulates. The last observed value is applied as interpolated values which is appropriate to use in cases where sensor values updates at change.
As an example, we will use the time information from a fictitious Voyage to get the data from a path mapped to a mass flow sensor FO_Mass_Flow :
Curl --location 'https://api.kognif.ai/galore/v2/timeseries/vesselsensoreventsbypath'
\--header 'Ocp-Apim-Subscription-Key: [YOUR-OCP-APIM-SUBSCRIPTION-KEY]'
\--header 'Authorization: ••••••'
\--data '{<body>]}
This mass flow sensor gives us time irregular data, in this case kg/h, and we are using the preview API endpoint to get the data:
We are using a time_weighted_average aggregation to ensure we are getting values for each period within the time interval, this is how body looks like:
{
"from":"2024-11-01T01:01:01Z",
"to":"2024-11-04T10:00:30Z",
"interval":"1m",
"resamplingMethods":[
"time_weighted_average"
],
"paths":[
"/Fleet/imoxxxxxxx/Engines/Main/1/FO_Mass_Flow"
]
}
The response, shortened for readability:
{
"intervalInSeconds":60,
"resamplingMethods":[
"time_weighted_average"
],
"time":"2024-11-01T01:00:00Z",
"interval":"1m",
"columns":[
{
"path":"/Fleet/imoxxxxxxx/Engines/Main/1/FO_Mass_Flow",
"unit":"kg/h",
"samples":{
"time_Weighted_Average":[
741.6849276974416,
738.2358333333333,
"…"
],
"dataValidationErrors":{}
}
}
],
"errors":[]
}
We can now summarize all 1-hour time weighted averages and divide by 60 to get the accumulated kg.
Limitations
When one of the advanced aggregation methods is used, the endpoint will accept a maximum of 31 days as time period. This also apply for any other aggregate methods in combination.
Preview - Time Weighted Average integral
We are introducing a new advanced aggregate method: "time_weighted_integral", which is using the same underlying processing as the "time_weighted_average" and the same endpoint: vesselsensorevents
This new method gives us the area under the graph, the integral, and can be used to retrieve accumulations from instant values e.g. get distance from a speed sensor, or tonnes from a mass flow sensor etc.
The POST body is the same as the example above, only use "time_weighted_integral" as "resamplingmethod".
Limitations
When one of the advanced aggregation methods is used, the endpoint will accept a maximum of 31 days as time period. This also apply for any other aggregate methods in combination.