The System That Couldn't Fail

Seven months. A hundred pages of documentation. A live system serving rural clinics in Delta State, Nigeria. The Delta Health Information and Appointment Booking System was a final-year project at the University of Port Harcourt that became a real production system for real patients. Patients who travel hours to reach a clinic, only to find it full or their appointment cancelled. The pressure wasn't a good grade—it was people's health.

The stack: React.js, Node.js, PHP with Laravel, MySQL, AWS, Docker. Communication channels included SMS gateways, WhatsApp Business API, and USSD because not every patient has a smartphone. The system had to run on 2G, on entry-level Android phones with 2GB RAM, where electricity and internet are unreliable.

Three bugs tested everything.

Bug 1: The Clinic That Appeared in the Ocean

The location feature was a point of pride. Patients could book appointments by proximity, using GPS coordinates stored in the database. Clinics table used DECIMAL(10,7) for latitude and longitude. In development, everything worked. During field testing, a clinic in Delta State appeared off the coast of Ghana. On the map. In the ocean.

MySQL accepted every value because they were technically valid decimals. The problem was human: administrators entering coordinates manually from printed sheets made transcription errors. Swapped lat/lng, missing digits, misread numbers. Six degrees of longitude and five degrees of latitude are valid decimals—they just point to the ocean.

The fix: server-side validation in Node.js checking coordinates against Nigeria's geographic bounding box before writing to the database. Plus a one-time audit script to flag outliers.

function isValidNigerianCoordinate(lat, lng) {
  const NIGERIA_BOUNDS = {
    minLat: 4.0,
    maxLat: 14.0,
    minLng: 2.7,
    maxLng: 15.0
  };
  return (
    lat >= NIGERIA_BOUNDS.minLat &&
    lat <= NIGERIA_BOUNDS.maxLat &&
    lng >= NIGERIA_BOUNDS.minLng &&
    lng <= NIGERIA_BOUNDS.maxLng
  );
}

if (!isValidNigerianCoordinate(lat, lng)) {
  return res.status(400).json({
    error: 'Coordinates fall outside expected geographic boundary. Please verify and resubmit.'
  });
}

Six clinics had swapped coordinates. Three had misread digits. All nine placed facilities where they had no business being.

Lesson: Validate coordinates against geography, not just data types. A number can be valid and completely wrong.

Bug 2: The Notifications That Went Nowhere

The communication layer worried the author most. Patients needed appointment confirmations via SMS, WhatsApp, or USSD. The notifications table tracked every outgoing message. During integration testing, API calls returned 200, status showed "sent". Field testing revealed: none of the patients received their reminders. None.

The SMS gateway had a two-stage delivery model. When the server sent a batch, the gateway accepted them and returned 200. That confirmed receipt by the gateway, not delivery to the recipient. Actual delivery was asynchronous—the gateway would call a webhook URL to update final status. They hadn't registered a webhook. Messages entered the queue, some delivered, some failed, and they had zero visibility.

// Before: wrong assumption
const sendNotification = async (userId, message) => {
  const response = await smsGateway.send({ to: phone, body: message });
  if (response.status === 200) {
    await db.notifications.update({ status: 'sent' });
    // WRONG: this is gateway receipt, not delivery
  }
};

// After: correct with webhook const sendNotification = async (userId, message) => { const response = await smsGateway.send({ to: phone, body: message, webhookUrl: ${process.env.BASE_URL}/webhooks/delivery-receipt }); await db.notifications.update({ status: 'queued' }); };

app.post('/webhooks/delivery-receipt', async (req, res) => { const { messageId, status } = req.body; await db.notifications.update({ messageId, status: status === 'delivered' ? 'delivered' : 'failed' }); if (status === 'failed') { await retryQueue.add({ messageId }); } res.sendStatus(200); });


After the fix, confirmed delivery went from catastrophically low to 94%. The remaining 6% were phones switched off or unreachable—a real baseline in rural environments.

**Lesson:** A 200 from a messaging API is an acknowledgement, not a guarantee. Implement webhooks. Never trust a status you didn't actively confirm.

## Bug 3: The Bug That Only Existed in the Real World

This was the most frustrating because it couldn't be reproduced in development. The system was bandwidth-tolerant with offline caching and optimized UI. What wasn't accounted for: what happens when a patient starts a booking, the connection drops mid-transaction, connectivity returns, and the patient hits submit again.

Duplicate appointments appeared. Same patient, same clinic, same time slot, two confirmed bookings. Reproducing required unstable internet—exactly the environment the system was built for.

The sequence: patient initiates booking, client generates request, connection drops before request reaches server, client shows loading, connection returns, patient re-submits, both requests eventually reach server, both are accepted because nothing detected they were the same.

Fix: idempotency keys.

```javascript
// Client side
const initiateBooking = async (bookingData) => {
  const idempotencyKey = `booking_${userId}_${clinicId}_${slotId}_${Date.now()}`;
  localStorage.setItem('pendingBookingKey', idempotencyKey);
  const response = await fetch('/api/appointments', {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'Idempotency-Key': idempotencyKey
    },
    body: JSON.stringify(bookingData)
  });
  if (response.ok) {
    localStorage.removeItem('pendingBookingKey');
  }
};

// Server side
app.post('/api/appointments', async (req, res) => {
  const idempotencyKey = req.headers['idempotency-key'];
  const existing = await db.appointments.findByKey(idempotencyKey);
  if (existing) {
    return res.status(200).json(existing);
  }
  const appointment = await db.appointments.create({
    ...req.body,
    idempotencyKey
  });
  res.status(201).json(appointment);
});

Also added client-side request state machine and local caching for retry on connectivity return.

Lesson: Test in the conditions the system will run in. Stable dev environments hide entire categories of bugs. Idempotency is the difference between one booking and two.

What Seven Months Felt Like

These bugs weren't exotic. They were hard because each required stepping outside assumptions built into the mental model. Administrators made transcription errors because data entry had no validation. Patients re-submitted because nothing visually confirmed their action. A gateway returning 200 seemed enough—in most environments it would be.

The author documented everything because the system had to outlast their involvement. The final presentation to the university panel included a live demo on a 2G connection. It worked. The system is still running today.

The takeaway: Build for the real world, not your development environment. Validate beyond data types. Confirm delivery, don't trust acknowledgements. Make operations idempotent. And test where your users actually are.