Jupyter Notebooks Week 5: Your Strategic Study Guide for Real-World Data Analysis
Here’s the thing about Week 5 of Jupyter notebooks - this is where the training wheels come off and you start working with real, messy, interesting data. The notebooks in the ZCW-Summer25/JupyterNotebooks.Week5 repository aren’t just exercises; they’re mini data science projects that mirror what you’ll encounter in the professional world. But here’s the catch: without a strategic approach, you can easily get overwhelmed by the complexity.
That’s exactly why you need this study guide. I’m going to walk you through a systematic approach to tackle these notebooks that’ll not only help you complete them successfully, but actually learn the deeper patterns of data analysis that make you valuable as a programmer.
As a group, you will need to make sure everyone is moving thru the notebooks in a coordinated fashion, and that everyone is getting comfortable with the material. We suggest that you work with an AI to generate a few quizzes about the material, and have everyone in the group take the quizzes. Also, once you have finished a notebook, use AI to summarize the key points, concepts and topics you should be carrying forward from the contents of the finished notebook. Add those summaries to the repo, and mark them as AI generated.
Use an AI to explain things you don’t understand. Make sure you express yourself fully when prompting the AI.
Repository Overview: What You’re Getting Into
The Week 5 collection contains six distinct data analysis projects, each focusing on different types of real-world datasets from Delaware:
- CityOfNewarkDETreeSurvey - Municipal tree inventory analysis
- NOAADailySummaries - Weather data time series analysis
- NOAALocations - Geographic weather station data
- NOAAMonthlySummaries - Long-term climate pattern analysis
- RegisteredVotersFileWilmingtonDE - Civic data demographic analysis
- DataAcquisitionLab - Comprehensive data collection project
What makes this collection special is that it moves beyond toy datasets to real municipal, environmental, and civic data - the kind of messy, interesting datasets you’ll work with in actual data roles.
Prerequisites: What You Need Before Starting
Before diving in, make sure you’re solid on these intermediate Python concepts:
Core Python Skills
- Data structures mastery: Lists, dictionaries, sets, and when to use each
- List/dict comprehensions: You should be comfortable reading and writing them
- File I/O operations: Reading CSV, JSON, and handling file paths
- Error handling: Try/except blocks and debugging strategies
- Functions and modules: Writing reusable code and importing libraries
Essential Libraries
1# You should be comfortable with these imports and basic usage
2import pandas as pd
3import numpy as np
4import matplotlib.pyplot as plt
5import seaborn as sns
6from datetime import datetime, timedelta
7import json
8import osData Analysis Fundamentals
- Pandas basics: DataFrames, Series, indexing, filtering, grouping
- Data cleaning concepts: Handling missing values, data types, duplicates
- Basic visualization: Creating meaningful charts that tell stories
- Statistical thinking: Understanding distributions, correlations, trends
Strategic Approach: The Three-Phase Method
Here’s how I recommend approaching each notebook project. This method will keep you organized and ensure you’re learning, not just completing exercises.
Phase 1: Reconnaissance (15-20 minutes per notebook)
Before writing any code, understand what you’re working with:
1# Start every notebook with this exploration pattern
2import pandas as pd
3import numpy as np
4
5# Load the dataset
6df = pd.read_csv('your_data.csv')
7
8# The "Big Four" questions to answer first:
9print("Dataset Overview:")
10print(f"Shape: {df.shape}")
11print(f"Columns: {list(df.columns)}")
12print(f"Data types:\n{df.dtypes}")
13print(f"Missing values:\n{df.isnull().sum()}")
14
15# Quick peek at the data
16print("\nFirst 5 rows:")
17df.head()Key Questions to Answer:
- What is this dataset about? (Read any documentation first)
- How big is it? (Rows and columns)
- What are the main variables?
- What quality issues do I see? (Missing data, weird values)
- What questions might this data answer?
Phase 2: Systematic Exploration (30-45 minutes per notebook)
Now dive deeper with a structured exploration:
1# Numerical columns analysis
2numeric_cols = df.select_dtypes(include=[np.number]).columns
3if len(numeric_cols) > 0:
4 print("Numerical summaries:")
5 display(df[numeric_cols].describe())
6
7# Categorical columns analysis
8categorical_cols = df.select_dtypes(include=['object']).columns
9for col in categorical_cols[:5]: # Limit to first 5
10 print(f"\n{col} value counts:")
11 print(df[col].value_counts().head())
12
13# Time-based analysis (if applicable)
14date_cols = [col for col in df.columns if 'date' in col.lower()]
15if date_cols:
16 for col in date_cols:
17 try:
18 df[col] = pd.to_datetime(df[col])
19 print(f"Date range for {col}: {df[col].min()} to {df[col].max()}")
20 except:
21 print(f"Could not convert {col} to datetime")Documentation Pattern: Create markdown cells to document your findings:
1## Initial Findings
2
3### Data Quality Issues
4- Missing values in X column (Y% of data)
5- Potential outliers in Z column
6- Date format inconsistencies
7
8### Interesting Patterns
9- Seasonal trends visible in time series
10- Geographic clustering in location data
11- Demographic patterns in voter data
12
13### Analysis Questions
141. What are the trends over time?
152. Are there geographic patterns?
163. What factors correlate with outcomes?Phase 3: Focused Analysis (45-60 minutes per notebook)
Now tackle specific analysis questions with purpose:
1# Pattern: Always start with a hypothesis
2"""
3Hypothesis: Tree species diversity varies by neighborhood in Newark
4Test: Group by location and analyze species counts
5"""
6
7# Group and analyze
8location_analysis = df.groupby('location_column').agg({
9 'species_column': ['count', 'nunique'],
10 'other_metric': ['mean', 'std']
11}).round(2)
12
13# Visualize findings
14plt.figure(figsize=(12, 6))
15plt.subplot(1, 2, 1)
16location_analysis['species_column']['count'].plot(kind='bar')
17plt.title('Tree Count by Location')
18plt.xticks(rotation=45)
19
20plt.subplot(1, 2, 2)
21location_analysis['species_column']['nunique'].plot(kind='bar')
22plt.title('Species Diversity by Location')
23plt.xticks(rotation=45)
24
25plt.tight_layout()
26plt.show()Notebook-Specific Study Strategies
1. CityOfNewarkDETreeSurvey
Focus Areas:
- Geographic data analysis (if coordinates included)
- Species diversity and distribution
- Tree health metrics and patterns
- Urban forestry insights
Key Skills to Practice:
1# Grouping and aggregation
2species_summary = df.groupby('species').agg({
3 'diameter': ['count', 'mean', 'std'],
4 'condition': lambda x: x.mode()[0] if len(x) > 0 else 'Unknown'
5})
6
7# Geographic clustering (if lat/lon available)
8if 'latitude' in df.columns:
9 plt.scatter(df['longitude'], df['latitude'],
10 c=df['diameter'], alpha=0.6, cmap='viridis')
11 plt.colorbar(label='Tree Diameter')
12 plt.title('Tree Distribution and Size')Success Metric: Can you identify which neighborhoods have the healthiest/largest trees and why?
2. NOAADailySummaries & NOAAMonthlySummaries
Focus Areas:
- Time series analysis and trends
- Seasonal pattern identification
- Data cleaning for weather data
- Statistical summaries over time
Key Skills to Practice:
1# Time series preparation
2df['date'] = pd.to_datetime(df['date'])
3df = df.sort_values('date')
4df.set_index('date', inplace=True)
5
6# Rolling averages for trend analysis
7df['temp_7day_avg'] = df['temperature'].rolling(window=7).mean()
8df['temp_30day_avg'] = df['temperature'].rolling(window=30).mean()
9
10# Seasonal analysis
11df['month'] = df.index.month
12monthly_patterns = df.groupby('month')['temperature'].agg(['mean', 'std'])
13
14# Visualization
15fig, axes = plt.subplots(2, 1, figsize=(15, 10))
16df['temperature'].plot(ax=axes[0], title='Daily Temperature')
17monthly_patterns['mean'].plot(kind='bar', ax=axes[1], title='Average Temperature by Month')Success Metric: Can you identify long-term trends and explain seasonal variations?
3. NOAALocations
Focus Areas:
- Geographic data visualization
- Station network analysis
- Spatial relationships
- Data completeness across locations
Key Skills to Practice:
1# Geographic analysis
2if 'latitude' in df.columns and 'longitude' in df.columns:
3 # Station distribution map
4 plt.figure(figsize=(10, 8))
5 plt.scatter(df['longitude'], df['latitude'], alpha=0.7)
6 plt.xlabel('Longitude')
7 plt.ylabel('Latitude')
8 plt.title('NOAA Weather Station Locations')
9
10 # Calculate distances between stations (if needed)
11 from math import radians, sin, cos, sqrt, atan2
12
13 def haversine_distance(lat1, lon1, lat2, lon2):
14 R = 6371 # Earth's radius in km
15 dlat = radians(lat2 - lat1)
16 dlon = radians(lon2 - lon1)
17 a = sin(dlat/2)**2 + cos(radians(lat1)) * cos(radians(lat2)) * sin(dlon/2)**2
18 return 2 * R * atan2(sqrt(a), sqrt(1-a))Success Metric: Can you create a meaningful map and analyze station coverage?
4. RegisteredVotersFileWilmingtonDE
Focus Areas:
- Demographic analysis and patterns
- Geographic voting patterns
- Data privacy and anonymization
- Civic data insights
Key Skills to Practice:
1# Demographic analysis (while respecting privacy)
2age_groups = pd.cut(df['age'], bins=[18, 30, 45, 65, 100],
3 labels=['18-30', '31-45', '46-65', '65+'])
4df['age_group'] = age_groups
5
6# Cross-tabulation analysis
7demo_breakdown = pd.crosstab(df['ward'], df['age_group'], normalize='index')
8demo_breakdown.plot(kind='bar', stacked=True, figsize=(12, 6))
9plt.title('Age Distribution by Ward')
10plt.xticks(rotation=45)
11
12# Registration trends over time (if date data available)
13if 'registration_date' in df.columns:
14 df['reg_date'] = pd.to_datetime(df['registration_date'])
15 monthly_registrations = df.groupby(df['reg_date'].dt.to_period('M')).size()
16 monthly_registrations.plot(title='Voter Registrations Over Time')Success Metric: Can you identify meaningful demographic patterns while maintaining data privacy principles?
5. DataAcquisitionLab
Focus Areas:
- Data collection methodologies
- API usage and web scraping
- Data validation and quality
- Integration of multiple data sources
Key Skills to Practice:
1# Data source validation
2def validate_data_source(df, expected_columns, expected_row_range):
3 """Validate that acquired data meets expectations."""
4 issues = []
5
6 # Check columns
7 missing_cols = set(expected_columns) - set(df.columns)
8 if missing_cols:
9 issues.append(f"Missing columns: {missing_cols}")
10
11 # Check row count
12 if not (expected_row_range[0] <= len(df) <= expected_row_range[1]):
13 issues.append(f"Row count {len(df)} outside expected range {expected_row_range}")
14
15 # Check for completely empty columns
16 empty_cols = df.columns[df.isnull().all()].tolist()
17 if empty_cols:
18 issues.append(f"Completely empty columns: {empty_cols}")
19
20 return issues
21
22# Data integration patterns
23def integrate_datasets(df1, df2, join_key, how='inner'):
24 """Safely integrate multiple datasets."""
25 print(f"Before integration: df1={len(df1)}, df2={len(df2)}")
26
27 # Check join key exists in both
28 if join_key not in df1.columns or join_key not in df2.columns:
29 raise ValueError(f"Join key '{join_key}' not found in both datasets")
30
31 # Perform join
32 result = df1.merge(df2, on=join_key, how=how)
33 print(f"After integration: {len(result)} rows")
34
35 return resultSuccess Metric: Can you successfully acquire, validate, and integrate data from multiple sources?
Common Pitfalls and How to Avoid Them
Pitfall 1: Jumping Into Analysis Too Quickly
Problem: Starting complex analysis before understanding the data structure.
Solution: Always complete the reconnaissance phase first. Spend 15-20 minutes just exploring before any serious analysis.
Pitfall 2: Not Documenting Your Thinking
Problem: Creating notebooks that are just code without explanation.
Solution: Use markdown cells liberally. Future you will thank current you.
1## Why This Analysis Matters
2
3I'm exploring X because...
4I expect to find Y because...
5This analysis will help answer the question: ZPitfall 3: Ignoring Data Quality Issues
Problem: Proceeding with analysis despite obvious data problems.
Solution: Create a data quality checklist:
1# Data Quality Checklist
2def data_quality_report(df):
3 """Generate comprehensive data quality report."""
4 print("=== DATA QUALITY REPORT ===")
5 print(f"Total rows: {len(df):,}")
6 print(f"Total columns: {len(df.columns)}")
7
8 # Missing data
9 missing_pct = (df.isnull().sum() / len(df) * 100).round(2)
10 missing_issues = missing_pct[missing_pct > 0]
11 if len(missing_issues) > 0:
12 print(f"\n⚠️ Missing Data Issues:")
13 for col, pct in missing_issues.items():
14 print(f" {col}: {pct}% missing")
15
16 # Data type issues
17 print(f"\n📊 Data Types:")
18 for dtype in df.dtypes.unique():
19 cols = df.select_dtypes(include=[dtype]).columns
20 print(f" {dtype}: {len(cols)} columns")
21
22 # Duplicates
23 dup_count = df.duplicated().sum()
24 if dup_count > 0:
25 print(f"\n⚠️ Duplicate rows: {dup_count}")
26
27 print("========================")
28
29# Use it on every dataset
30data_quality_report(df)Pitfall 4: Creating Unmaintainable Code
Problem: Writing one-off scripts instead of reusable functions.
Solution: Wrap common operations in functions:
1def create_summary_stats(df, group_col, metric_col):
2 """Create standardized summary statistics."""
3 return df.groupby(group_col)[metric_col].agg([
4 'count', 'mean', 'median', 'std', 'min', 'max'
5 ]).round(2)
6
7def plot_time_series(df, date_col, value_col, title="Time Series"):
8 """Create standardized time series plot."""
9 plt.figure(figsize=(12, 6))
10 plt.plot(df[date_col], df[value_col])
11 plt.title(title)
12 plt.xlabel(date_col)
13 plt.ylabel(value_col)
14 plt.xticks(rotation=45)
15 plt.grid(True, alpha=0.3)
16 plt.tight_layout()
17 plt.show()Success Metrics: How to Know You’re Learning
For each notebook, you should be able to answer these questions:
Technical Mastery
- Can you load and clean the data without errors?
- Can you create meaningful visualizations that tell a story?
- Can you identify and explain patterns in the data?
- Can you write functions that others could reuse?
Analytical Thinking
- Can you formulate good questions about the data?
- Can you design appropriate analyses to answer those questions?
- Can you interpret results and draw reasonable conclusions?
- Can you identify limitations in your analysis?
Communication Skills
- Can you explain your findings to someone without a technical background?
- Can you document your process clearly enough for others to follow?
- Can you justify your analytical choices?
Time Management Strategy
Here’s a realistic time allocation for each notebook:
- Week Day 1-2: Complete 2-3 notebooks using the three-phase method
- Week Day 3: Focus on the most challenging notebook (probably DataAcquisitionLab)
- Week Day 4: Review and refine all notebooks, improve documentation
- Week Day 5: Create a summary analysis comparing insights across datasets
Total Time Investment: 8-12 hours of focused work across the week.
The Bottom Line: Building Real Data Science Skills
These Week 5 notebooks aren’t just assignments - they’re your gateway to understanding how data analysis works with real, messy datasets. The patterns you learn here - systematic exploration, careful documentation, reproducible analysis - are exactly what separates junior developers from confident data professionals.
The key is to approach each notebook not as something to complete, but as something to master. Take your time with the exploration phase. Document your thinking. Ask good questions. Build reusable code. When you do this consistently, you’re not just finishing assignments - you’re developing the analytical mindset that makes you valuable as a programmer.
Trust me, the skills you develop working through these notebooks systematically will serve you for years. You’re not just analyzing trees and weather data - you’re learning to think like a data scientist. And that’s a skill that’ll make you stand out no matter where your programming career takes you.