I Built a Local-First HSA Receipt Tracker with Flask, Google Drive, and AI

Source: Dev.to

Introduction

A tiny local Python app that turns HSA receipts into structured records in Google Drive and Sheets in about five seconds.

What is an HSA?

HSAs are unusual in the U.S. tax system because they’re triple‑tax advantaged:

• contributions are tax‑deductible
• investments grow tax‑free
• withdrawals for qualified medical expenses are tax‑free

The Bogleheads wiki has a great explanation if you’re curious:

Because of those advantages, some people treat their HSA as a long‑term investment account instead of reimbursing medical expenses right away.

The “Save Receipts and Reimburse Later” Strategy

1. Contribute to the HSA
2. Invest the money
3. Pay medical expenses out‑of‑pocket
4. Save the receipts
5. Reimburse yourself years (or decades) later, or don’t
6. Profit

There’s no deadline to reimburse yourself as long as:

• the expense happened after the HSA was opened
• you kept documentation

The Problem

If you’re saving receipts for potential reimbursement later, you need to keep track of:

• provider
• date
• amount
• proof of payment
• the original receipt

Manually this usually turns into:

• PDFs scattered across downloads folders
• random email attachments
• a spreadsheet you forget to update

Solution: A Local‑First Receipt Tracker

I built a tiny local‑first web app that:

• accepts a medical receipt PDF
• extracts useful fields automatically
• stores the document in Google Drive
• logs the expense in Google Sheets

The whole flow takes about 5 seconds of actual work.

Workflow Overview

receipt → upload → text extraction → AI field parsing → Drive storage → Sheets entry

When a receipt is submitted, the app:

1. Saves the uploaded PDF locally
2. Computes a SHA‑256 hash to detect duplicates
3. Extracts text using pdfplumber
4. (Optional) Calls OpenAI to extract fields such as vendor, service date, amount, payment date, payment method, notes
5. Creates (or reuses) a Google Drive folder for the month (e.g., 2026-03/) and uploads the receipt there
6. Appends a row to Google Sheets with a link to the file

Example Spreadsheet Row

Now every expense has:

• structured data
• the original document
• a searchable log

Duplicate Detection

Each uploaded file is hashed and stored in a local receipt_hashes.json. If the same receipt is uploaded twice, the app catches it before cluttering the spreadsheet or Drive folder.

Technical Stack

• Flask – local web app
• pdfplumber – extract text from receipts
• OpenAI (optional) – prefill receipt fields
• Google Drive API – store receipts
• Google Sheets API – expense log

Why a Local‑First Approach?

• No extra account to maintain
• Easier to hack on
• Files stay in your own Google account
• No database or infrastructure to run

I also set it up as a macOS launchd service, so it’s always available on my laptop.

Getting Started

The project is open source:

If you’re using the “save receipts and reimburse later” HSA strategy and have built something similar, I’d love to hear about it. Or if you have a better way to automate this, please let me know before I add OCR and accidentally build an entire product.