🏠 RI Real Estate Report

Technical Documentation & Learning Guide

Last updated: July 18, 2025

1. Architecture Overview

2. Data Flow & Loading

Parallel Loading Strategy

Instead of loading data sequentially, we use Promise.all() to load both the property data and SQL.js WASM simultaneously:

Promise.all([
  // Fetch + decompress gzipped JSON
  fetch('ri-sales.json.gz?v=' + new Date().toISOString().split('T')[0])
    .then(r => r.arrayBuffer())
    .then(buf => JSON.parse(pako.inflate(new Uint8Array(buf), {to:'string'}))),
  
  // Load SQL.js WASM
  initSqlJs({ locateFile: f => `https://cdnjs.cloudflare.com/ajax/libs/sql.js/1.8.0/${f}` })
]).then(([data, SQL]) => {
  // Both ready - initialize everything
});

Savings: ~2-3 seconds compared to sequential loading

Batch SQL Insert

Instead of inserting 36K rows one-by-one, we batch in chunks of 500:

const BATCH = 500;
for (let i = 0; i < data.length; i += BATCH) {
  const chunk = data.slice(i, i + BATCH);
  const placeholders = chunk.map(() => '(?,?,?...)').join(',');
  const values = chunk.flatMap(p => [p.address, p.city, ...]);
  db.run(`INSERT INTO properties VALUES ${placeholders}`, values);
}

~70x faster than row-by-row insertion

Cache Busting

The data URL includes today's date to ensure fresh data while allowing same-day caching:

fetch('ri-sales.json.gz?v=2025-07-18')

3. Performance Optimizations

Precomputed City Stats

Instead of recalculating city statistics every time insights render, we compute them once on data load:

// On data load:
window.cityStats = {};
rawData.properties.forEach(p => {
  if (!window.cityStats[p.city]) {
    window.cityStats[p.city] = { prices: [], doms: [], soldPrices: [], yearlyPrices: {} };
  }
  const cs = window.cityStats[p.city];
  if (p.price > 50000) cs.prices.push(p.price);
  if (p.dom > 0 && p.dom < 365) cs.doms.push(p.dom);
  // ... collect yearly data for trends
});

// Compute aggregates once
Object.keys(window.cityStats).forEach(city => {
  const cs = window.cityStats[city];
  cs.medianPrice = calculateMedian(cs.prices);
  cs.avgDom = calculateAverage(cs.doms);
});

// Later, in updateInsights():
const topCities = Object.entries(window.cityStats)
  .filter(([c, stats]) => stats.count >= 10)
  .sort((a,b) => b[1].medianPrice - a[1].medianPrice)
  .slice(0, 5);

4. Service Worker & Offline Cache

How Offline Caching Works

// sw.js (Service Worker)
const CACHE_NAME = 'ri-report-v1';
const ASSETS = [
  '/ri-report.html',
  '/ri-sales.json.gz',
  'https://cdn.jsdelivr.net/npm/chart.js',
  'https://cdnjs.cloudflare.com/ajax/libs/pako/2.1.0/pako.min.js'
];

// Install: Cache all assets
self.addEventListener('install', event => {
  event.waitUntil(
    caches.open(CACHE_NAME).then(cache => cache.addAll(ASSETS))
  );
});

// Fetch: Try cache first, then network
self.addEventListener('fetch', event => {
  event.respondWith(
    caches.match(event.request).then(cached => {
      // Return cached version if available
      if (cached) return cached;
      
      // Otherwise fetch from network and cache for next time
      return fetch(event.request).then(response => {
        const clone = response.clone();
        caches.open(CACHE_NAME).then(cache => cache.put(event.request, clone));
        return response;
      });
    })
  );
});

Registration in Main Page

// In ri-report.html
if ('serviceWorker' in navigator) {
  navigator.serviceWorker.register('/sw.js').then(reg => {
    console.log('Service Worker registered:', reg.scope);
  });
}

For RI Report: Recommended Strategy

// Cache-first for static assets
// Stale-while-revalidate for data (return cached, fetch fresh in background)

self.addEventListener('fetch', event => {
  const url = new URL(event.request.url);
  
  // Data file: stale-while-revalidate
  if (url.pathname.includes('ri-sales.json.gz')) {
    event.respondWith(
      caches.match(event.request).then(cached => {
        const fetchPromise = fetch(event.request).then(response => {
          caches.open(CACHE_NAME).then(cache => cache.put(event.request, response.clone()));
          return response;
        });
        return cached || fetchPromise;
      })
    );
    return;
  }
  
  // Everything else: cache-first
  event.respondWith(
    caches.match(event.request).then(cached => cached || fetch(event.request))
  );
});

5. Feature Deep Dives

Explore Neighborhood (Haversine Formula)

Finds properties within a radius using the Haversine formula for great-circle distance:

function getDistance(lat1, lon1, lat2, lon2) {
  const R = 3959; // Earth radius in miles
  const dLat = (lat2 - lat1) * Math.PI / 180;
  const dLon = (lon2 - lon1) * Math.PI / 180;
  const a = Math.sin(dLat/2) * Math.sin(dLat/2) +
            Math.cos(lat1 * Math.PI/180) * Math.cos(lat2 * Math.PI/180) *
            Math.sin(dLon/2) * Math.sin(dLon/2);
  const c = 2 * Math.atan2(Math.sqrt(a), Math.sqrt(1-a));
  return R * c;
}

function getNearbyProperties(lat, lng, radiusMiles, excludeAddress) {
  return rawData.properties.filter(p => {
    if (!p.lat || !p.lng) return false;
    if (p.address === excludeAddress) return false;
    return getDistance(lat, lng, p.lat, p.lng) <= radiusMiles;
  });
}

Dynamic Insights

The Key Market Insights cards update dynamically from real data:

• Top 5 Inland Cities: Excludes 14 coastal cities, sorts by median price
• Fastest to Sell: Cities with lowest average Days on Market (DOM)
• YoY Appreciation: Compares current year median to last year

Deal Score Formula (House Hunter)

// Discount from market value
const avgPpsf = comps.reduce((s, c) => s + c.ppsf, 0) / comps.length;
const marketValue = sqft * avgPpsf;
const discount = (1 - askingPrice / marketValue) * 100;

// DOM bonus (sitting longer = more negotiating power)
const domBonus = Math.min(20, Math.floor(dom / 15));

// Final deal score
const dealScore = discount + domBonus;

Pagination with Page Size Selector

Users can choose 25, 50, 75, 100, or All results per page:

let propPageSize = 25; // Default

function setPageSize(size) {
  propPageSize = size;
  propPage = 1; // Reset to first page
  renderPropResults();
}

// In pagination HTML:
'' +
  [25, 50, 75, 100].map(n => 
    '' + n + ''
  ).join('') +
  'All' +
''

6. Future Improvements

// Example: Box plot data structure
const boxPlotData = {
  min: prices[0],
  q1: prices[Math.floor(prices.length * 0.25)],
  median: prices[Math.floor(prices.length * 0.5)],
  q3: prices[Math.floor(prices.length * 0.75)],
  max: prices[prices.length - 1],
  outliers: prices.filter(p => p < q1 - 1.5*iqr || p > q3 + 1.5*iqr)
};

Built with ✨ by Supernova for Nick | Live Report | GitHub