NumPy & Pandas: Data Engineering Foundations

Data Handling: Why NumPy & Pandas Matter

Python’s native data structures, while flexible, become inefficient when processing large datasets, particularly for numerical operations. A standard Python list stores references to objects, not raw values. Each integer, float, or string in a list is a distinct Python object, carrying overhead for type information and reference counting.

Consider a list of one million integers:

data = list(range(1_000_000))

Each element in data is a full Python int object. This design choice provides flexibility, allowing a single list to hold mixed data types, but it consumes significantly more memory than the raw numerical values require. Accessing and operating on these elements involves dereferencing pointers and dynamic type checks, which slows down computations and leads to poor CPU cache use. For analytical tasks involving millions or billions of data points, these overheads quickly become prohibitive.

NumPy solves the numerical inefficiency problem through its ndarray (N-dimensional array) object. NumPy arrays store homogeneous, fixed-type data contiguously in memory, similar to C arrays. This design eliminates per-element object overhead and enables direct memory access. Operations on ndarray objects are implemented in C, providing vectorized computations that are orders of magnitude faster than iterating through Python lists. This makes NumPy the standard for high-performance numerical computing in Python.

Pandas extends NumPy’s capabilities to structured, tabular data. Its core DataFrame object uses NumPy arrays internally for each column, where columns are homogeneous in type. This allows Pandas to manage datasets with mixed data types efficiently while retaining the performance benefits of NumPy for numerical columns. Pandas adds rich indexing, data alignment, and a powerful set of data manipulation tools, making it the primary library for data cleaning, transformation, and analysis.

Together, NumPy and Pandas form the bedrock for efficient data handling in Python. They provide the necessary performance and structure to move from raw data to production-ready insights, overcoming the inherent limitations of native Python for large-scale data engineering tasks.

NumPy Arrays: Efficient Numerical Operations

NumPy’s ndarray provides a memory-efficient container for numerical data, enabling high-performance operations on large datasets. Unlike Python lists, all elements within an ndarray must be of the same data type, which allows NumPy to optimize storage and computation. This homogeneous nature is critical for its speed advantages.

Create a NumPy array from a standard Python list. The dtype attribute shows the data type of the array elements.

import numpy as np

data = [10, 20, 30, 40, 50]
arr = np.array(data)
print(arr)
print(arr.dtype)

[10 20 30 40 50]
int64

Accessing elements uses standard Python indexing. Slicing allows selecting subsets of the array, similar to list slicing.

print(arr[0])      # First element
print(arr[-1])     # Last element
print(arr[1:4])    # Elements from index 1 up to (but not including) 4
print(arr[::2])    # Every second element

10
50
[20 30 40]
[10 30 50]

NumPy excels at vectorized operations, applying functions or arithmetic operators to all elements of an array without explicit loops. This internal C-level implementation is significantly faster than iterating through Python lists.

arr_doubled = arr * 2
print(arr_doubled)

arr_sum = arr + 100
print(arr_sum)

[ 20  40  60  80 100]
[110 120 130 140 150]

Arrays can be multi-dimensional. A 2D array, often called a matrix, is common. Its shape attribute returns a tuple indicating the size of each dimension.

matrix = np.array([[1, 2, 3], [4, 5, 6]])
print(matrix)
print(matrix.shape)

[[1 2 3]
 [4 5 6]]
(2, 3)

Indexing multi-dimensional arrays requires specifying an index for each dimension, separated by commas. Slicing can also be applied across dimensions.

print(matrix[0, 1])   # Element at row 0, column 1
print(matrix[:, 0])   # All elements in column 0
print(matrix[1, :])   # All elements in row 1

2
[1 4]
[4 5 6]

Pandas DataFrames: Tabular Data Structures

Pandas DataFrames provide a labeled, two-dimensional data structure for tabular data. They resemble spreadsheets or SQL tables, where data is organized into rows and columns. Each column in a DataFrame is a Pandas Series, a one-dimensional labeled array capable of holding any data type.

To begin, import the pandas library, typically aliased as pd. Loading data from common formats like CSV is a frequent first step.

import pandas as pd

# Create a sample CSV file for demonstration
csv_data = """Name,Age,City,Salary
Alice,30,New York,70000
Bob,24,Los Angeles,60000
Charlie,35,Chicago,85000
David,29,New York,72000
"""
with open("employees.csv", "w") as f:
    f.write(csv_data)

# Load data from a CSV file into a DataFrame
df = pd.read_csv("employees.csv")
print(df.head())

       Name  Age        City  Salary
0     Alice   30    New York   70000
1       Bob   24  Los Angeles   60000
2   Charlie   35      Chicago   85000
3     David   29    New York   72000

Accessing specific columns uses bracket notation. A single column returns a Series, while a list of columns returns a new DataFrame.

# Select a single column (returns a Series)
ages = df['Age']
print(type(ages))

# Select multiple columns (returns a DataFrame)
names_cities = df[['Name', 'City']]
print(names_cities)

<class 'pandas.core.series.Series'>
      Name        City
0    Alice    New York
1      Bob  Los Angeles
2  Charlie    Chicago
3    David    New York

Rows can be selected using label-based indexing with .loc[] or integer-location based indexing with .iloc[]. .loc[] uses row labels and column labels, while .iloc[] uses integer positions.

# Select rows by label (index 0 and 2)
first_and_third = df.loc[[0, 2]]
print(first_and_third)

# Select rows by integer position (first two rows)
first_two_rows = df.iloc[0:2]
print(first_two_rows)

      Name  Age      City  Salary
0    Alice   30  New York   70000
2  Charlie   35   Chicago   85000
       Name  Age        City  Salary
0    Alice   30    New York   70000
1      Bob   24  Los Angeles   60000

Filtering data uses boolean indexing. Provide a Series of boolean values to the DataFrame, where True selects the row and False excludes it.

# Filter employees older than 30
older_employees = df[df['Age'] > 30]
print(older_employees)

# Filter employees from New York with salary over 70000
specific_employees = df[(df['City'] == 'New York') & (df['Salary'] > 70000)]
print(specific_employees)

      Name  Age     City  Salary
2  Charlie   35  Chicago   85000
Empty DataFrame
Columns: [Name, Age, City, Salary]
Index: []

New columns can be added directly by assigning a Series or an array to a new column name. This operation modifies the DataFrame in place.

# Add a new column 'Bonus'
df['Bonus'] = df['Salary'] * 0.10
print(df.head())

      Name  Age        City  Salary    Bonus
0    Alice   30    New York   70000   7000.0
1      Bob   24  Los Angeles   60000   6000.0
2  Charlie   35      Chicago   85000   8500.0
3    David   29    New York   72000   7200.0

Data Transformation: Practical Pandas & NumPy Patterns

Applying conditional logic to DataFrame columns is a frequent transformation, such as categorizing numerical values or flagging specific rows. A common initial approach involves DataFrame.apply() with a Python function or lambda expression. This method processes each row individually, repeatedly entering and exiting the Python interpreter.

Consider deriving a Category column from an Amount column: “Low” if Amount is under 500, “Medium” for 500-1000, and “High” above 1000. Using df.apply() for this logic incurs significant overhead, especially with large datasets, due to its row-by-row execution model.

import pandas as pd
import numpy as np

data = {'ID': range(7), 'Amount': [400, 750, 1200, 300, 900, 1500, 600]}
df = pd.DataFrame(data)

# Inefficient approach for demonstration
def get_category_apply(amount):
    if amount < 500:
        return 'Low'
    elif 500 <= amount <= 1000:
        return 'Medium'
    else:
        return 'High'

df['Category_Apply'] = df['Amount'].apply(get_category_apply)
print(df)

   ID  Amount Category_Apply
0   0     400            Low
1   1     750         Medium
2   2    1200           High
3   3     300            Low
4   4     900         Medium
5   5    1500           High
6   6     600         Medium

A more performant method employs NumPy’s vectorized operations. For binary conditions, np.where() evaluates a condition on an entire array and returns elements chosen from two other arrays. This operation executes at C-speed, avoiding Python’s interpreter overhead.

For multiple conditions, np.select() extends this concept. It accepts lists of conditions and corresponding choices. This function provides a vectorized alternative to chained if/elif/else statements or nested np.where() calls, maintaining high performance across the entire Series.

# Vectorized approach using np.select
conditions = [
    df['Amount'] < 500,
    (df['Amount'] >= 500) & (df['Amount'] <= 1000),
    df['Amount'] > 1000
]
choices = ['Low', 'Medium', 'High']
df['Category_NumPy'] = np.select(conditions, choices, default='Unknown')
print(df)

   ID  Amount Category_Apply Category_NumPy
0   0     400            Low            Low
1   1     750         Medium         Medium
2   2    1200           High           High
3   3     300            Low            Low
4   4     900         Medium         Medium
5   5    1500           High           High
6   6     600         Medium         Medium

The performance difference between apply() and vectorized NumPy functions like np.select() is substantial with increasing data volume. While apply() offers flexibility for highly complex, non-vectorizable logic, its cost for large datasets is often prohibitive. Prioritize vectorized solutions whenever possible.

NumPy’s universal functions (ufuncs) also extend directly to Pandas Series and Dataframes, offering efficient element-wise operations. For example, calculating the natural logarithm of a column is efficient using np.log(). When combining data, ensure data types are consistent. Operations between Series and NumPy arrays are optimized when dtypes align; mismatched dtypes can force Python object operations, negating vectorized performance benefits.

Library Integration: Data Cleaning Exercise

Data collected from disparate sources often contains inconsistencies. This is a common challenge when integrating information from various sensors, user inputs, or external APIs. We will clean a dataset containing missing values and incorrect data types, preparing it for analysis.

To illustrate, consider a dataset simulating sensor readings. It includes SensorID, Timestamp, and Value. We generate a DataFrame with common issues: NaN values, string representations for numbers, and mixed types.

import pandas as pd
import numpy as np

data = {
    'SensorID': [101, 102, 101, 103, 102, 101, 104, 103, 102, 101],
    'Timestamp': pd.to_datetime([
        '2023-01-01 10:00:00', '2023-01-01 10:05:00', '2023-01-01 10:10:00',
        '2023-01-01 10:15:00', '2023-01-01 10:20:00', '2023-01-01 10:25:00',
        '2023-01-01 10:30:00', '2023-01-01 10:35:00', '2023-01-01 10:40:00',
        '2023-01-01 10:45:00'
    ]),
    'Value': [23.5, 24.1, np.nan, '22.8', 25.0, np.nan, 23.9, 'invalid', 24.5, 23.7],
    'Status': ['OK', 'OK', 'WARN', 'OK', 'OK', 'ERROR', 'OK', 'WARN', 'OK', 'OK']
}
df = pd.DataFrame(data)

First, inspect the DataFrame’s structure and data types using df.info(). This reveals the presence of non-numeric data in the Value column and potential missing entries. The Value column is object type due to the mixed numerical and string data.

print(df.info())

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 10 entries, 0 to 9
Data columns (total 4 columns):
 #   Column     Non-Null Count  Dtype         
---  ------     --------------  -----         
 0   SensorID   10 non-null     int64         
 1   Timestamp  10 non-null     datetime64[ns]
 2   Value      8 non-null      object        
 3   Status     10 non-null     object        
dtypes: datetime64[ns](1), int64(1), object(2)
memory usage: 448.0+ bytes
None

The Value column shows 8 non-null entries out of 10, confirming missing data.

The Value column contains np.nan and the string 'invalid'. Convert this column to a numeric type, coercing non-numeric values to NaN. This uses pd.to_numeric with errors='coerce', which allows conversion despite invalid entries, but discards the original problematic data.

df['Value'] = pd.to_numeric(df['Value'], errors='coerce')
print(df['Value'].isnull().sum())

3

Now, three values are NaN (the original two np.nan and the 'invalid' string). Fill these missing numeric values. A common strategy is to use the median, as it is less sensitive to outliers than the mean.

median_value = df['Value'].median()
df['Value'] = df['Value'].fillna(median_value)
print(df['Value'].head())

0    23.5
1    24.1
2    23.9
3    22.8
4    25.0
Name: Value, dtype: float64

Filling with the median (23.9) for missing values ensures that the central tendency is maintained without distortion from extreme values.

After cleaning, the data is ready for further operations. For instance, filter out sensor readings marked with ERROR status. This removes potentially unreliable data points.

df_cleaned = df[df['Status'] != 'ERROR']
print(df_cleaned.shape)

(9, 4)

The DataFrame now contains 9 rows, excluding the entry where Status was ERROR.

Confirm the data types and overall structure of the cleaned DataFrame.

print(df_cleaned.info())

<class 'pandas.core.frame.DataFrame'>
Index: 9 entries, 0 to 9
Data columns (total 4 columns):
 #   Column     Non-Null Count  Dtype         
---  ------     --------------  -----         
 0   SensorID   9 non-null      int64         
 1   Timestamp  9 non-null      datetime64[ns]
 2   Value      9 non-null      float64       
 3   Status     9 non-null      object        
dtypes: datetime64[ns](1), float64(1), int64(1), object(1)
memory usage: 360.0+ bytes
None

The Value column is now float64, and all columns show 9 non-null entries, indicating successful cleaning and type conversion.