As machine learning models grow in sophistication, their increased complexity often reduces their interpretability. Standard methods of assessing model performance, such as mean squared error and mean absolute error for regression tasks, often overlook the nuanced insights that could be gained from a more comprehensive evaluation. This is particularly true for complex models like time series forecasting, where a simple accuracy number might not capture the full story.

The real-world impact of trusting a model without understanding its inner workings can lead to mistakes and potentially damaging results. Thankfully, the importance of interpretability in machine learning has gained significant attention recently, and with it, the need for more robust evaluation methods. In this blog post, we delve into the importance of using more detailed visualization techniques like plots and graphs for assessing the performance of multivariate time series models. By exploring innovative evaluation metrics designed specifically for time series forecasting, we shed light on local interpretability, allowing for more informed decision-making.

When attempting to forecast occupancy in larger areas, there are several factors to consider. When building a forecasting system, we have to factor in all the external parameters that can (or can’t) be added as variables. There are some typical variables that we can account for …

Firstly, seasonality plays a significant role with multiple cycles affecting occupancy rates. In public spaces, the number of visitors is closely related to the seasons and weather conditions.

Secondly, the quality and quantity of data can vary depending on the time of day, with some periods potentially having more and better data than others. This is also heavily impacted by the measurement systems that are in place (eg. computer vision systems might have shortcomings in the dark, systems with radio waves might be more robust, but less precise).

Lastly, there can be more variability in occupancy at specific parts of the day. The challenge lies in effectively measuring all these aspects to ensure accurate predictions. Unexpected events can also have a big impact on the consistency of the measurements.

To mitigate these effects and improve the accuracy of occupancy predictions, several methods can be employed:

• Seasonality Adjustment: To handle the impact of seasonality, methods such as time series decomposition can be employed. This involves breaking down a time series into its components (trend, seasonal, and residual). Seasonal adjustment can help in identifying the underlying trends more accurately.
• Data Aggregation: For times of the day where data is sparse, one solution could be to aggregate the data over larger time intervals. For example, instead of considering hourly data, one could aggregate data at a daily level. This can help in making the data more robust and less prone to fluctuations.
• Smoothing Techniques: To handle the variability in specific parts of the day, smoothing techniques such as moving averages or exponential smoothing can be used. These techniques can help in reducing the noise and uncovering the underlying patterns in the data.
• Utilizing Advanced Forecasting Models: Today there are quite some tools and frameworks out there for time series forecasting, varying from univariate approaches (like ARIMA models) to extended multi-variate models that can account for a lot of interaction effects (like custom LSTM models or other model architectures).

So, long story short: There are a lot of possibilities, but the key component will remain to have proper evaluation methods, which assess a model’s usability in the business context in which it is supposed to run.

For the current case, we eventually went with an LSTM model with over 50 input features over 24 time steps. The picture below shows a quick overview of the model input:

This model primarily uses data from occupation metrics in and around the city, such as parking occupancy, high-level telecom information, and other specific occupancy measurement tools. It also considers contextual information about events that may affect occupancy levels.

While these input features do not cover all city activities, they offer a comprehensive overview of the current situation.

In assessing the complexity of the forecasting model, it’s crucial to examine its performance at different parts of the day. This is particularly important for ensuring the model’s effectiveness during critical periods such as mornings and noons. These are crucial periods of the day, to make sure the traffic conditions are adapted properly and to optimize the staffing in shops and restaurants over noon (and other shopping hours).

Next to that, we also wanted to know how much into the future we can predict. The amount of timesteps that we can predict into the future had an immediate impact on the (use) cases that can be enabled by this model.

In a best attempt to compare the accuracy for different forecasting horizons, we trained 24 models with 1 to 24 timesteps into the future. We stopped training at 20 epochs, to have an early indication of where the accuracy is going. On the test set we used the best_model_weights.

When plotting the Mean Absolute Percentage Error (MAPE) in this manner, we observe a gradually increasing MAPE. Which shows that the accuracy reduces as the forecasting horizon increases. This makes sense, and is what we expected. Nevertheless, we still observed quite some variability in the MAPE, which means the model didn’t entirely capture the complexity. This is also strongly related to the fact that we stopped training after 20 epochs, independently of the learning curve of the model.

We also wanted to know whether we were better during some times of the day. In a somewhat different fashion than the tests above, we trained a model to predict the upcoming 24 hours of the day.

To evaluate our performance through-out the day we combined the hour in which we did the prediction (X-axis), the number of hours in the future we predicted in relation to the hour of the day (Y-axis) and the Mean Absolute Percentage Error (colour).

The plot above led us to the conclusion that we are somewhat better at predicting the first half of the day at the beginning of the day. This is quite useful to manage crowdedness in the morning.

Next to that, we observed that we’re less accurate in predicting the first hours of the afternoon at noon, and it gets better in the afternoon.

It’s not uncommon for ML Engineers to try to jump straight to model development to solve a problem (yes, let’s admit it). However, model development and the different flavours and set-ups are an important part of capturing and evaluating the possibilities with the data at hand. So take your time to develop different model architectures in the light of the (business) evaluations you want to do.

It’s generally known that neural networks are incredibly good at solving well-defined, narrow problems (despite the perceived general problem-solving capabilities of Large Language Models — we’re not just yet at the level of AGI).

So this also goes for this use case: It’s less effective to have one model that forecasts the upcoming 24 hours and to build multiple use cases on top of that. Instead we opt to build dedicated models per specific use case (eg. morning shift predictions, lunch-staffing predictions, …).

We still observed quite some variance in our test results, which shows that our model(s) didn’t (yet) capture all complexity. Next to that, we observed some systematic, lower accuracy of measurements during the night. This got (partly) resolved by moving to multi-variate models (the LSTM with the features displayed above), which was able to get the best of the different measurements at hand. Yet again, we learned that data quality is key in building proper data solutions.

At Sensr we recently had the opportunity to serve as an implementation partner on the DAP²CHEM project , an innovative project aiming to stimulate the transition of chemical and life science companies towards Industry 4.0. This project was a collaboration between top Flemish research institutes and top players in the chemical and life space (with a Flemish presence), Procter and Gamble, Ajinomoto Bio-Pharma Services and Janssen. Through this article we share our learnings about machine learning at the edge from that project.

Most of our interactions with machine learning (ML) backed systems happen through communication over the internet. We send data to an ML model in the cloud for inferencing, wait a few (milli)seconds, and get a response with the results. At Sensr we often encounter Industrial Internet of Things (IIoT) systems where this pattern for ML inference can be prohibitive due to:

• The latency inherent to communication over the internet. The (milli)second delay that we mentioned might be enough for the window of opportunity where the output of the model is useful to pass. For example, a faulty item on a production line might be too far down the conveyor belt to be discarded.
• The absence of stable connectivity to the internet. This can be due to the environment the system operates in, or due to the nature of the device producing the data. ML-backed systems must be able to provide at least the same level of reliability as the systems they are replacing or improving upon in order to be viable. Hence, they must be able to properly function even when connectivity drops (un)expectedly.
• The high velocity and volume at which data is produced. In addition to amplifying the issues caused by the previous two points. Sending all of this data to the cloud for processing, whether in batch or real-time, would incur quite a cost. This is especially the case when the number of edge devices that produce data grow into the thousands.
• Although less relevant for industrial applications of IoT, laws and regulations may be in place for sensitive data, in order to protect the privacy of individuals. In this case, sending certain information to the cloud might be prohibited, or the risk of interception and leakage may outweigh the benefit of processing the data in the cloud.

By deploying ML models closer to the source of the data they are processing we can eliminate the need for internet communication in our flow. This increases the viability of some interesting industrial use cases of ML including:

• The integration of computer vision models in visual inspection, security, and safety systems, where inclusion of ML into the loop can only be beneficial if the rest of the system can react to the output of the models in real time.
• Performing predictive maintenance using data collected by audio and vibration sensors attached to industrial machinery. When the data collected by a sensor indicates anomalous behavior, real-time contingency measures can be taken to prevent damage to the machine or the environment it operates in.

Most of our interactions with machine learning (ML) backed systems happen through communication over the internet. We send data to an ML model in the cloud for inferencing, wait a few (milli)seconds, and get a response with the results. At Sensr we often encounter Industrial Internet of Things (IIoT) systems where this pattern for ML inference can be prohibitive due to:

• The latency inherent to communication over the internet. The (milli)second delay that we mentioned might be enough for the window of opportunity where the output of the model is useful to pass. For example, a faulty item on a production line might be too far down the conveyor belt to be discarded.
• The absence of stable connectivity to the internet. This can be due to the environment the system operates in, or due to the nature of the device producing the data. ML-backed systems must be able to provide at least the same level of reliability as the systems they are replacing or improving upon in order to be viable. Hence, they must be able to properly function even when connectivity drops (un)expectedly.
• The high velocity and volume at which data is produced. In addition to amplifying the issues caused by the previous two points. Sending all of this data to the cloud for processing, whether in batch or real-time, would incur quite a cost. This is especially the case when the number of edge devices that produce data grow into the thousands.
• Although less relevant for industrial applications of IoT, laws and regulations may be in place for sensitive data, in order to protect the privacy of individuals. In this case, sending certain information to the cloud might be prohibited, or the risk of interception and leakage may outweigh the benefit of processing the data in the cloud.

By deploying ML models closer to the source of the data they are processing we can eliminate the need for internet communication in our flow. This increases the viability of some interesting industrial use cases of ML including:

• The integration of computer vision models in visual inspection, security, and safety systems, where inclusion of ML into the loop can only be beneficial if the rest of the system can react to the output of the models in real time.
• Performing predictive maintenance using data collected by audio and vibration sensors attached to industrial machinery. When the data collected by a sensor indicates anomalous behavior, real-time contingency measures can be taken to prevent damage to the machine or the environment it operates in.

We boiled down Amazon Web Services’ (AWS) and Microsoft Azure’s reference architectures for enabling ML inference at the edge to their essence; the pattern described by the following sequence diagram:

A central part of the flow is some entity that represents the desired state of the edge device. This entity is part of your “always available” IT infrastructure and can thus be reliably updated as part of a continuous deployment flow. The associated edge device periodically polls this entity for state changes and updates its own state when necessary. This approach solves the three deployment related challenges we listed earlier:

• The desired state entity can be updated programmatically, that is, very efficiently, for thousands of devices.
• Since the edge device initiates any communication it is considered to be outbound. Outbound communication is often less strictly policed by corporate firewalls than inbound communication, making it easier to comply with existing firewall rules.
• If the device is offline while the desired state is updated, the change will be picked up by the device once it is back online.

You could, in theory, roll your own implementation, but both AWS and Azure have a suite of IoT services that can help you quickly facilitate this pattern as we will show in the following section.

The Azure IoT suite of services can give you a running start when trying to implement this pattern. Azure IoT Hub and Azure IoT Edge together provide a comprehensive solution for managing IoT devices and processing IoT data at scale:

• Azure IoT Hub is a cloud-based service that enables bidirectional communication between IoT devices and the cloud, allowing for device management, data ingestion, and monitoring at scale. Device and module twins in IoT Hub are used to maintain the desired state of the IoT devices and their associated modules.
• Azure IoT Edge extends the capabilities of Azure IoT Hub by providing a way to run cloud workloads locally on IoT Edge devices, enabling them to perform edge computing and analytics, even when the devices are offline or have limited connectivity. One of the responsibilities of the runtime that comes with IoT Edge is handling communication between the IoT Edge device and the cloud.

In what follows we assume that you already:

1. Set up an IoT Hub. Be sure to choose a tier that supports Azure IoT Edge.
2. Created and provisioned an IoT Edge device.
3. Have a basic understanding of how to develop IoT Edge modules.

As an example we set up a free tier IoT Hub and provisioned an IoT Edge device that we set up in a local virtual machine. To serve as a starting point we have also created a custom Python IoT Edge module, called model_loader, based on the Python Cookiecutter template.

By setting up these Azure-provided building blocks we have already “implemented” three components crucial to our pattern; an edge device, a way to set its desired state via device and module twins in the cloud, and communication between the edge device and the cloud. As such most of the heavy lifting has been done for us by Microsoft. All that is left to do for us is some fine-tuning to enable the rest of the pattern.

IoT edge modules run in containerized environments. Hence when a module, or the device needs to restart any files that have been created in a module’s filesystem are lost, e.g., downloaded model artifacts. We can provide our custom module with access to persistent storage on the host file system by extending the module’s create options in the deployment manifest with a storage binding, where Source and Target contain your host and module storage path, respectively; both must be absolute paths. The deployment manifest for our model_loader module looks as follows:

In Azure IoT Hub we can set the desired state for a device/module through the desired properties object in the device/module twin. For example, we set the following desired properties on the twin of our model_loader module to indicate which model we want the module to load, where to save it, and which checksum to use for validating the downloaded model:

For the purpose of this article we are using a publicly available model, but under normal circumstances you would host your model in a private repository.

The module template that we used to create our model_loader module does not implement any handling of changes in the desired state, so called twin patches. Further down we provide a basic implementation of an incoming twin patch handler, which is invoked when desired properties are updated on the module twin. Inside the handler we also set a reported property, to provide some form of feedback to IoT Hub.

The handler shown below performs the following steps when a twin patch is received:

1. Check whether the twin patch is valid, that is, contains all expected keys.
2. Extract the model’s download url, MD5 checksum, and local file name from the twin patch.
3. Check whether the model already exists locally. If so, returns in order to prevent an unnecessary download.
4. Downloads the model and validates the downloaded file using the MD5 checksum from the twin patch.
5. Sets the reported property lastModelUpdate to the current time to indicate when the model.

After implementing and deploying the handler, the logs of our model_loadermodule look as follows:

We can see that the twin patch handler was invoked twice. The first time, triggered by the module starting up, the handler finds an existing model file, which was downloaded during development of the module. The second invocation was triggered by editing modelFileName in the module twin through the Azure Portal UI. A file with that name does not yet exist and therefore a download is triggered.

If we then look in the edge device’s local folder that we set as mount source in the deployment manifest, we see both files mentioned in the logs. These won’t be lost if the device or the module’s container is restarted and can be accessed by other modules through the same mounting mechanism.

In this article, we cover the need for ML at the edge and associated deployment challenges. We also describe a generic pattern for tackling said challenges and provide an example implementation using the Azure IoT Edge framework. I hope that this article helped you in a meaningful way and that you had as much fun reading it as I had writing it.

Cheers,

— Andrei