Building Intelligent Retail Signage with BrightSign's NPU: A Deep Dive into Real-Time Gaze Detection

Source: Dev.to

Source: Dev.to

Overview

Every Series 5 BrightSign player (except the XC line) ships with an onboard Neural Processing Unit (NPU). Most people don’t know this, and even fewer have tried to use it. We think that should change.

The NPU can run inference at up to 6 TRILLION operations per second (TOPS)—more than enough for real‑time computer‑vision workloads. To demonstrate what’s possible, we built a showcase application: a smart retail‑signage system that detects when people are looking at the display, adapts content in real time, and unlocks engagement metrics that content managers and marketing teams would love to have.

This post walks through what we built, how it works, and what it means for developers building on BrightSign hardware.

## Person Detection in Action

*Demo Analytics Dashboard*

*(Insert demo video / GIF here)*

---

The Hardware Story

BrightSign’s Series 5 and later players integrate Rockchip NPUs directly into the SoC. The hardware supports multiple precision formats (INT4, INT8, INT16, FP16), giving developers flexibility to balance speed and accuracy depending on the use case.

Gaze‑Detection Demo

• Model: RetinaFace (trained on the WIDER FACE dataset)

• Why RetinaFace?

  • Detects faces across varying scales, lighting conditions, and angles.
  • Ideal for retail environments where people approach displays from all directions.

• Implementation:

  • Runs through our open‑source gaze‑detection extension.
  • Compiled specifically for the BrightSign NPU using Rockchip’s RKNN toolchain.

• Performance:

  • 30 + FPS inference from a USB‑camera feed, running entirely on the player.

Try the Demo

GitHub – brightsign‑npu‑gaze‑extension

Application State Machine

Transitions

• Persistence logic is used to prevent flickering.
• A single frame of no gaze detected does not drop you out of SPEAKING; several consecutive frames without engagement are required before transitioning.
• This smooths out the noise inherent in any real‑world camera setup.

Gaze‑Detection Extension

The extension runs as a BrightSign extension package. It:

1. Captures frames from a video source (USB camera, RTSP stream, or video file on the SD card).
2. Pushes them through the NPU for inference.
3. Publishes results as UDP packets on localhost:5002.

{
  "faces_attending": 1,
  "faces_in_frame_total": 3,
  "timestamp": 1756854339
}

Your application subscribes to this UDP stream and does whatever you want with the data. The extension handles the ML heavy lifting; you handle the business logic. This separation means you can build sophisticated signage applications in TypeScript, JavaScript, or any language you’re comfortable with—without touching any neural‑network code.

Configuration

The extension is configurable through BrightSign registry keys:

Metrics & Dashboard

The demo aggregates inference results into Prometheus‑compatible metrics that can be scraped from the player’s /metrics endpoint.

A Grafana dashboard (included in the repository) visualises these metrics in real time. It lets you:

• View engagement funnels.
• Compare content performance across variants.
• Monitor system health (CPU, memory, inference latency, etc.).

---

Why This Matters for Content Managers

• Actionable data – Answer questions such as:

  • “Does the purple idle screen attract more attention than the green one?”
  • “How long do people actually watch our product demo before walking away?”

• Data‑driven optimisation – Iterate on creative assets based on real‑world engagement numbers, enabling continuous improvement of the viewer experience.

Edge Inference Benefits

Getting Started

1. Hardware – A Series 5 BrightSign player (XT5, XD5, HD5, LS5, or HS5 – not the XC5 line).
2. Camera – A USB camera (or an RTSP‑capable network camera).
3. Software – Clone the repository, follow the README to install the extension, and configure the registry keys for your setup.

git clone https://github.com/brightdevelopers/brightsign-npu-gaze-extension.git
cd brightsign-npu-gaze-extension
# Follow the installation steps in the README

Resources

• Extension source: https://github.com/brightdevelopers/brightsign-npu-gaze-extension
• Demo application & Grafana dashboard: Included in the same repository.
• Rockchip RKNN toolchain: https://github.com/rockchip-linux/rknn-toolkit
• RetinaFace model & WIDER FACE dataset: https://github.com/deepinsight/insightface

---

TL;DR

BrightSign Series 5 players have a powerful NPU that can run real‑time vision models. By using our open‑source gaze‑detection extension you can:

• Detect faces and gaze on‑device at >30 FPS.
• Drive dynamic content based on audience attention.
• Export lightweight metrics for analytics, A/B testing, and compliance.

Give it a try and start building the next generation of intelligent digital signage!

Overview

• Hardware requirements

  • Camera
  • Node.js

• Extension behavior

  • The extension auto‑starts and begins publishing gaze data immediately.

---

Purpose

The demo showcases what’s possible today with hardware you may already own.
It is not a finished product—it’s a starting point. Feel free to:

• Take it, break it apart, and improve it.
• Use the open‑source extensions as‑is or as a foundation for custom applications.

Both the Gaze Detection and YOLOX Object Detection extensions are available on GitHub.

Important Note

Fair Warning: Building custom NPU extensions is non‑trivial.
You’ll essentially be porting or writing AI applications for an embedded edge device.

• Python support is limited.
• Much (or all) of your code may need to be ported to C++.

---

Resources

• [Gaze Detection Extension]
• [Object Detection Extension]
• [Extension Template]

(All links point to the corresponding GitHub repositories.)

Support

We encourage you to reach out to BrightSign for assistance:

• See the Contact Us section below.
• If you already have a BrightSign contact, feel free to ask them about the NPU.

---

Future Directions

The NPU on BrightSign players unlocks application categories that were previously impractical on dedicated signage hardware. We’re exploring use cases beyond computer vision, such as:

• Intelligent log analysis
• Predictive error handling
• Automated content optimization

We’re still in the ideation phase for many of these ideas. If you’re building on BrightSign hardware and have concepts for NPU‑accelerated features that would make your workflow easier, we want to hear from you.

---

Get in Touch

Find the proper channels for contacting us [here].

We’re curious to see what you’ll create!