Building a Local-First Data Exploration Tool with DuckDB

Source: Dev.to

What

I’m building streaksight, a GUI‑based data analysis tool that runs locally on your machine.

The project is open source and available on GitHub.

The goal is to minimize the time required to turn data into insights. streaksight lets you load, analyze, and visualize data in a single local application.

Note: streaksight is still a work in progress and hasn’t been released yet.

What you can do

Data ingestion

By simply selecting a file, streaksight automatically detects the schema and imports the data into its embedded database. Currently, CSV and JSON formats are supported. In the future, the following data sources are planned:

• Object storage such as AWS S3
• Databases such as MySQL
• SaaS platforms such as Google Analytics

Query building and visualization with nodes

Queries can be constructed by connecting nodes. Execution results are shown in real time, allowing you to iteratively explore data and gain insights interactively while refining your queries.

Who

streaksight is best suited for working with relatively small datasets.

• Indie developers and small teams running their own products
• People who want to analyze small‑to‑medium‑sized log or event data locally
• Teams who want to validate the value of their data before building a full‑scale data platform

Why

I first encountered DuckDB while investigating logs at work. Its execution speed was surprisingly fast and left a strong impression on me.

After that experience, I started following the DuckDB ecosystem more closely, paying particular attention to projects like MotherDuck and DuckLake.

At the same time, I was responsible for operating BigQuery, including cost‑optimization efforts. Through this work I became increasingly interested in the technologies behind building and operating data platforms. While exploring various SaaS and OSS projects related to data infrastructure, I noticed several recurring problems.

Problem 1: Setup complexity

Most data‑infrastructure OSS projects focus on a single responsibility and are designed to be composed together.

• ETL: Airbyte, Apache NiFi, …
• Query engines: DuckDB, ClickHouse, …
• Visualization: Redash, Metabase, Apache Superset, …

This separation improves extensibility and stability, but it also increases setup and maintenance costs. Selecting the right tools and wiring them together can add noticeable overhead, especially for small‑scale data analysis.

Problem 2: Difficulty of building correct queries

With the rise of AI, writing SQL queries has become much easier. However, debugging those queries and verifying that they actually produce the intended results remains difficult.

A query is a sequence of set operations and joins. To quickly validate AI‑generated queries, I felt there was a strong need for an interface that allows users to inspect intermediate results, such as sub‑queries and join outputs.

streaksight addresses these problems by enabling fast, iterative query experimentation and result inspection in a local environment, without having to combine multiple tools.

Technology stack

Design principle: run efficiently on a single machine

• DuckDB – an OLAP‑focused database that delivers excellent performance on a single machine.
• Tauri – enables building cross‑platform desktop applications; produces lightweight binaries and allows the frontend to be built with web technologies.
• Deno.js – a JavaScript runtime that integrates well into Rust‑based projects; used to allow users to add custom JavaScript plugins without rebuilding the application.
• Singer.io – used as a reference when designing the interface for ETL plugins.

Roadmap

Because data sources vary widely, flexibility in connectors is a key focus. The next steps are to increase the number of supported connectors and to finalize a plugin interface. Ultimately, the goal is to allow users to write custom connectors in JavaScript, enabling direct connections to a wide variety of data sources.

In addition, the following features are being explored:

• Sharing insights
  • Dashboards
• AI integration
  • Model Context Protocol
• Cloud integration

The scope and direction of these features will be refined based on real‑world use cases.

Final words

If you find this project interesting, I would really appreciate it if you could give it a star on GitHub.

If you have any questions or feedback about the project, please feel free to start a discussion here:
https://github.com/whtsht/streaksight/discussions