Bokeh

Making sense of the mess in my head

The PAL Project

Several weeks ago, I created the nelcea/PAL: Wearable AI exploration repository on GitHub. I want to take a minute to explain in slight more detail what this open-source project is about.

PAL is the vehicle for my research in the world of Wearable AI, with an initial focus on the fields of “Personal Knowledge Management” and “Building a Second Brain” (BASB).
The initial use case is to capture conversations I’m having, store their text somewhere, and make it possible for me to consult, process, or interact with them.

I have plenty of other ideas extending the use cases to connect many other domains, but I’ll address those in a separate post.

The solution I’m currently working on is based on:

  • A Friend device to capture the conversations.
  • An iPhone app to record them, transcribe them and interact with the notes.
  • A server-side component for heavier processing.
  • Potentially some desktop/web app for further interaction.

The goal of this project is for me to explore and learn in public, not to build a polished project suitable for a large user base.
I encourage typical users to check out what Based Hardware is doing—they’re addressing the most typical use cases. I think they’re doing a great job moving quickly to provide useful features to their early adopters, and I’m happy to help with their project whenever I can.

While learning, I intend to delve deeply into all required topics, publishing the code and writing blog posts about my discoveries. I will link to those here as they become available.

This will cover a broad range of topics, as the use case above will already require:

  • BLE connectivity between the Friend and iPhone.
  • Audio recording and codec usage.
  • Audio processing to get the cleanest sound possible for transcription.
  • Automatic speech recognition, including diarisation.
  • Text processing for e.g. summarisation or extracting actions.
  • Chat interface to interact with the content, using RAG and vector databases.

In addition, I have the following requirements:

  • Use the Swift programming language for everything: iPhone app, server component and eventually firmware (at first I’ll use stock Friend firmware)
  • Run everything locally, either on the iPhone or on a local server, and avoid sending any data out to a cloud service. A conclusion might be that it’s not possible to get acceptable results with that constraint, but that’s also part of the experiment.

Do not hesitate to spread the word and contact me on X or Mastodon to join me on this journey.