Dipping A Toe Into The AI Waters

Ok, I admit it, I’ve been tinkering with AI again.

Not using it, this time, but trying to add it to my app The Stack.

Yet another person shamelessly trying to shoe-horn some AI into their app, so that they can jump on the bandwagon?

Well yeah, guilty as charged, I guess. Anyhoo…

Summarising Notes

Individual notes in the stack are generally quite short. Things like “add a blog post talking about adding AI to The Stack”. There’s probably not a lot of mileage in summarising them.

Given a long list of them though, it’s easy to get swamped.

Could we use some sort of AI to generate a summary of all items, or of all items with a given tag? Maybe it could pull out one or two emerging themes? Perhaps also suggest the item to tackle next?

On-Device

For iOS and macOS 26.0, Apple offer a new on-device Foundation Model.

This seems like it’s tailor made for my use case.

The query doesn’t leave your device, so there are no privacy issues to contend with.

So I decided to give it a go.

Adding The Code

On the plus side, adding the code for this was ridiculously easy. About three lines of actual Swift:

  model = LanguageModelSession(instructions: summarizerInstructions)
  let response = try await model.respond(to: prompt)
  return response.content.trimmingCharacters(in: .whitespacesAndNewlines)

It’s no exageration to say that the instructions and prompt that I feed to the AI is a lot longer than the code I use to invoke it.

Therein lies one problem, however. The prompt. Getting a prompt that actually produces useful output is difficult.

I’m initialising the model with these instructions:

    let summarizerInstructions = """
      You summarize a list of to‑do notes for planning.

      Rules:
      - Output: one concise paragraph, no more than 50 words
      - Use plain text, no headings or lists.
      - Weigh recently updated items slightly more, but consider all items.
      - Use tags only as background context; never mention tag names in the output.
      - Prefer synthesis over enumeration; use actionable, neutral language.
      - Summarise for the current user only. Do not refer to teams or other users.
      - Write in the imperative mood where possible. Do not address the user directly.
      - Do not invent notes or details.
      - Write in the same language as the notes.
      """

Then giving it the following prompt:

  var prompt = """
      Summarize the following to‑do items into a single paragraph that captures the main themes and priorities.

      Each item is in the form <date>: <body>, where: 
      - <date> is the date that the note was last modified (more recent = higher weight)
      - <body> is the note body, including any hashtags

      Items:
      \(summarisedItems.joined(separator: "\n\n"))
      """

This took quite a lot of tweaking. Inevitably, I asked Copilot to help 😆.

I am sure it could be greatly improved.

So Far, So Meh…

The current prototype sometimes does a decent job, but the output has a tendency to end up a bit wordy.

I haven’t had it completely hallucinate, but it does sometimes completely ignores parts of its instructions!

I understand enough about the process to not be surprised by this, but the level of unpredicability makes it quite difficult to rely on.

Worse, I’m not entirely sure that the output is particularly helpful!

More testing required I think, particular with large lists of items.

There’s a bigger problem with that, however, when using the on-device model.

Unfortunately the context window is small. It only supports 4096 tokens, where each token is a few characters of text.

Even with a modest collection of a couple of hundred notes, it’s easy to exceed that window, resulting in an error.

I suspect that this means that the on-device approach is a non starter, and it will be necessary to send the entire item list to an online model. Which opens up a whole other can of worms…

Oh well.

Conclusion

Interesting experiment.

I’m not yet convinced, but will keep playing…