How To Efficiently Store High-Frequency Vibration Sensor Data

Managing and accurately storing high-frequency sensor data is critical to many industries, as it directly impacts the effectiveness of predictive maintenance and data optimization. Efficient storage solutions are essential to ensure this data remains reliable and accessible for analysis.

 

Figure 1. Data storage must be reliable and accessible to support analysis. Image used courtesy of Adobe

 

This guide will explore methods for handling pre-processed and raw high-frequency vibration sensor data, focusing on optimal strategies for effective data replication and storage.

 

Storing Raw Data and Pre-Processed Metrics

Maintaining both raw and pre-processed vibration sensor data is critical for efficient data management and comprehensive analysis. Let’s look at how each data type serves a unique purpose in the broader context of system diagnostics and predictive maintenance.

 

Raw Data

Raw data is captured with each signal, providing a detailed and unfiltered representation of the sensor's output. This data is essential for in-depth diagnostics because it preserves all the nuances of the original signal, allowing for thorough analysis of any anomalies or patterns.

With raw data, industries have the flexibility to revisit and re-analyze the data as new algorithms and diagnostic techniques are developed. This capability is particularly valuable for developing future algorithms and models to enhance predictive maintenance strategies, ensuring potential problems are identified and addressed before they result in costly downtime.

Raw data storage allows industries to maintain a comprehensive record of sensor activity, providing a rich foundation for continuous improvement and innovation in data analysis.

 

What Are the Benefits of Storing Raw Data?

Raw data storage is a useful approach that has the advantage of preserving all the information from high-frequency vibration sensors. This flexibility is essential for the development of many new diagnostic tools and data loggers. Raw data can be used to perform a variety of analyses. These include frequency analysis with Fast Fourier Transform (FFT), time domain analysis, wavelet transform, envelope analysis, and modal analysis.

 

Figure 2. Industrial processes and the expansion of IIoT systems are generating huge volumes of raw data. Image used courtesy of Adobe.

 

For example, the Fast Fourier Transform is useful for identifying all types of signal frequencies. It also shows potential problems such as misalignment, data imbalance, or bearing failure in rotating equipment.

 

Why to Consider Pre-Processing Before Storage?

Preprocessing is an important step that involves summarizing the raw data into essential metrics. The goal and primary benefit of preprocessing is to reduce the amount of storage required. For example, if a signal is sampled at 1-second intervals, it results in a 1 Hz signal.

For an original data chunk sampled at 10 kHz, the resulting data size can be reduced by a factor of 10,000. This approach is useful for managing high-frequency vibration sensor data with minimal storage requirements.

 

Pre-Processed Metric Examples

Pre-processed metrics are crucial in simplifying raw vibration data into easily interpretable indicators, making them more manageable to store and analyze. These metrics distill complex data into key figures that are essential for quick decision-making in industrial settings, where timely responses are often critical to maintaining operational efficiency.

Examples of pre-processed metrics include:

• RMS (Root Mean Square): This metric calculates the average magnitude of an alternating signal by measuring the square root of the average of the squared values.
• P2P (Peak-to-Peak): This metric calculates the contrast between the minimum and maximum values of each signal data. A value represents the amplitude range.
• Crest Factor: The metric is a ratio of peak to RMS. This ratio indicates how much the peaks in the signal exceed the average (RMS) level.

By focusing on these pre-processed metrics, you can streamline the analysis process. This enables faster, more informed decisions without continuously processing large volumes of raw data. These metrics strike a balance between simplifying data and preserving critical information, ensuring that key indicators of asset health are always at hand.

 

Use of Time Series Databases—Storing Vibration Sensor Data

Time series databases are functional and advanced systems that properly manage time-stamped data. They are efficient at handling large volumes of data points in sequential form. For example, these databases can store vibration measurements and perform instant read and write operations.

Let’s examine the two types of databases that can store vibration sensor data.

 

Traditional Time Series Databases

Traditional time series databases are designed to store time-stamped data points in scalar form, making them suitable for many time series applications. However, their read and write speeds can become a limiting factor when dealing with high-frequency vibration data, where rapid data generation and retrieval are critical. Because these databases often struggle with the sheer volume and velocity of data produced by high-frequency sensors, they can create performance bottlenecks in systems requiring real-time analysis.

 

Time Series Object Store Databases

On the other hand, time series object storage databases can handle complex data structures along with associated timestamps and unique metadata. In these systems, vibration sensor data is stored as binary objects, allowing efficient management of high-frequency data streams. Attaching rich metadata to each data chunk provides greater context and flexibility for subsequent analysis.

 

Figure 3. Saving key metadata with the binary sensor data provides greater context and flexibility for subsequent analysis. Image used courtesy of ReductStore [click to enlarge]

 

ReductStore uses these time series object store databases to provide a robust solution for storing high-frequency vibration data in binary format. This capability ensures that data is stored efficiently and remains easily accessible for detailed analysis and diagnostics, making it ideal for industrial applications that require high performance and scalability.

 

Adopting Data Retention and Replication Techniques

By storing sensor data locally, we can minimize data processing latency. We can store the information for immediate diagnosis. However, local storage can also lead to disk overrun problems.

To overcome these problems, you need to handle the data efficiently and use retention and replication techniques to ensure the critical information is properly retained. Let us examine each method in detail.

 

Volume-Based Retention

First-In-First-Out (FIFO) data retention policies help manage disk space by retaining data based on volume. This type of retention policy is especially important in edge applications where devices can run out of disk space.

For example, ReductStore follows a volume-based retention policy, ensuring that data is retained based on its volume. This method prioritizes the most recent data and ensures that older, less relevant data is automatically overwritten when storage limits are reached.

By implementing volume-based retention, organizations can optimize storage resources, maintain operational efficiency, and prevent data loss, especially in environments where real-time processing and minimal latency are critical.

 

Data Replication

With a data replication strategy, we can replicate important chunks of data based on key metrics such as RMS values, peak amplitude, or other critical indicators. This approach allows important information to be stored locally while ensuring that important raw data remains accessible for further analysis.

By selectively replicating data, we can reduce the burden on storage systems while maintaining the integrity and availability of critical data for diagnostic purposes. This strategy is particularly valuable in predictive maintenance applications, where access to specific, high-fidelity data points can lead to more accurate predictions and improved uptime.

 

Vibration Data Storage Best Practices

Implementing a valuable and efficient vibration data storage strategy is essential for effective data monitoring and diagnostics in many industries. Time series object storage databases offer significant advantages over traditional databases, particularly when handling complex, high-frequency data. These systems can efficiently manage vibration data in chunk form, allowing for more flexible and scalable storage solutions.

By incorporating volume-based retention policies and replication techniques, industrial managers can ensure that critical information is retained without overwhelming storage resources. This approach also increases the reliability and accuracy of predictive maintenance efforts. By following these best practices, industries can optimize their data storage strategies and ensure that vibration sensor data is efficiently stored and readily available for accurate predictive analysis and operational decision-making.