The previous sections covered methods to identify paths and fetching data through the API. In this section we'll put the capabilities together and explore best practices to set up pipelines and workflows.
The API is designed and aimed at covering the following use cases:
- Fetch close-to-realtime data from the vessel and show it in operational dashboards.
- Fetch a small batch of recent historic data and store it in a client database for further processing, analysis and usage.
- If needed, pagination can be used to fetch data for longer time periods.
For querying larger batches of data with time periods exceeding 1 month, it is not recommended to use the API.
Workflow
As a data consumer, the main steps in setting up a data pipeline would be:
- Identify which paths to query data. Either by getting a list from KDI or by fetching asset model info. This is typically a one time job.
- Query data using the OnGridByPath method.
Queries
We recommend:
- Using the OnGridByPath endpoint to get data.
- Set up a recurring scheduled API call with an interval of 1 hour.
- Using 1 minute aggregate intervals in the queries.
- Implement exponential backoff to minimize unnecessary API calls:
- If no new data is available in the API, flag the timestamp and sleep the requests. You can use the Get Latest endpoint to check if new data has arrived. When data stream is back online, start pulling data from the flagged timestamp.
- If no new data appears, contact data owner and support.
Common fault scenarios
As described in the Overall Dataflow section, the Vessel Insight provides raw data as-is from the source systems onboard the vessels. Some common fault scenarios are:
- The sources are not streaming any data, so there is no data available in the cloud API.
- The sources are streaming data to the edge, but the connectivity between vessel and cloud is down. In this case, the data is buffered on the edge and sent once connectivity is back up.
- The data values are faulty - typical issues are frozen values, outliers, drifting sensors or correlated signals are behaving unexpectedly.
Requesting additional sensors
If there are sensors or paths you need that are not available, please contact the data or vessel owner and KDI Customer Success to explore possibilities to interface the necessary sensors.
Pagination
For the OnGridByPath endpoint , it is possible to use pagination if the number of items in response exceed 10 000, by sending a new request using the latest timestamp (plus the interval period) in the previous response as from date. Then repeat this until the latest timestamp is equal to or greater than the overall end date.