Introduction to Pandas for Physics - Data Cleaning & Missing Experimental Data

Program 1

Data Cleaning & Missing Experimental Data

Python 3.11 Ready

import pandas as pd
import numpy as np

# Simulated telemetry stream from an atmospheric physics observatory
# np.nan simulates temporary sensor dropouts or wireless connection drop
sensor_stream = {
    'time_min': [0, 5, 10, 15, 20, 25, 30, 35, 40, 45],
    'solar_irradiance_W_m2': [842.1, 846.5, np.nan, 852.0, 855.4, np.nan, np.nan, 863.1, 866.7, 869.2],
    'barometric_pressure_hPa': [1013.2, 1013.1, 1013.0, np.nan, 1012.8, 1012.7, 1012.5, 1012.4, np.nan, 1012.1],
    'sensor_flag': ['VALID', 'VALID', 'DROPOUT', 'VALID', 'VALID', 'ERROR', 'ERROR', 'VALID', 'VALID', 'VALID']
}

df_telemetry = pd.DataFrame(sensor_stream)

print("=== Raw Telemetry with Sensor Dropouts ===")
print(df_telemetry)

# 1. Null Value Diagnostics: Tally missing physical readings
print("\nMissing Measurement Count per Sensor Channel:")
print(df_telemetry.isna().sum())

# 2. Create clean working copy
df_clean = df_telemetry.copy()

# 3. Forward-Fill (ffill) for high-inertia thermodynamic state (Barometric Pressure)
df_clean['barometric_pressure_hPa'] = df_clean['barometric_pressure_hPa'].ffill()

# 4. Linear Interpolation for continuous physical fields (Solar Irradiance)
df_clean['solar_irradiance_W_m2'] = df_clean['solar_irradiance_W_m2'].interpolate(method='linear')

# 5. Type Casting and Filtering
df_clean['time_min'] = df_clean['time_min'].astype(int)
df_clean['barometric_pressure_hPa'] = df_clean['barometric_pressure_hPa'].astype(float)
df_clean['is_reconstructed'] = df_telemetry['sensor_flag'] != 'VALID'

print("\n=== Reconstructed & Cleaned Physics Dataset ===")
print(df_clean[['time_min', 'solar_irradiance_W_m2', 'barometric_pressure_hPa', 'is_reconstructed']].to_string(index=False))

# Verify all numerical gaps are resolved
remaining_nans = df_clean[['solar_irradiance_W_m2', 'barometric_pressure_hPa']].isna().sum().sum()
print(f"\nVerification: {remaining_nans} missing values remaining. Data stream continuous and valid.")

Click "Run Code" to execute script and render figures.

Mathematical Problem Formulation

Physical Model: Reconstruction of Missing Experimental Observables:

In experimental astrophysics and environmental telemetry, sensors experience intermittent dropouts where recorded data is missing ($\text{NaN}$).

1. Physical Continuity & Linear Interpolation:

For smoothly varying continuous physical fields (such as atmospheric solar irradiance $I(t)$), a missing value at time $t \in [t_0, t_1]$ is reconstructed via the linear interpolation scheme:

$$I(t) = I(t_0) + \frac{I(t_1) - I(t_0)}{t_1 - t_0} (t - t_0)$$

2. Thermodynamic Persistence (Forward-Fill):

For high-inertia thermodynamic variables such as barometric pressure $P(t)$, when sensor disconnection occurs for brief intervals ($\Delta t \ll \tau_{\text{relax}}$), the previous equilibrium value provides a reliable estimate:

$$P(t) = P(t - \Delta t)$$

Theoretical Background & Explanation

1. Why Experimental Data Cleaning is Critical:

Raw physics laboratory streams rarely arrive in pristine condition. Loose probe connections, voltage sags, ADC overflow errors, and wireless packet drops introduce NaN (Not a Number) values into data arrays.

2. Methods of Handling Missing Data in Pandas:

.isna().sum(): Rapid diagnostic tool to tally missing sensor readings per physical channel.
.dropna(): Removes records containing missing values. In physics time series, blind dropping can distort temporal continuity and step-sizes $\Delta t$.
.interpolate(method='linear'): Smoothly estimates missing samples based on physical continuity.
.ffill() / .bfill(): Forward or backward propagation of persistent states.

3. Data Type Validation:

Sensor loggers often store numbers as text strings due to formatting anomalies. Using .astype() or pd.to_numeric() ensures physical values are true 64-bit IEEE floating-point numbers ready for numerical computation.

Ch. 8.4 Data Cleaning & Missing Experimental Data

Data Cleaning & Missing Experimental Data

Mathematical Problem Formulation

Theoretical Background & Explanation