In earlier posts, I wrote about helping agents see your photos and knowing who's in them. At a conference in September, I found myself describing the bigger idea behind that work as an "agentic intelligence layer for photos."

That sounds a little abstract. What I mean is that Gumnut gives an agent the information it needs to understand who's in your photos, what's happening, where and when they were taken, and how they were taken. It can use that information to organize a collection or tell a story without access to the original files.

This isn't something that agents can do on their own. A capable vision model can describe a photo if you give it the image. But "a child roasting a marshmallow" isn't the same as knowing which child that is, finding them across years of photos, or knowing which camping trip the picture came from.

The missing piece is a persistent understanding of the library. That's what I've been building in Gumnut.

For example, I could ask an agent: "Write a short account of our Hendy Woods camping trip, with a few representative photos."

If I've already organized the trip into an album, the agent can start there. Otherwise, it can search by place and date. It can follow who appears across the results, read what happened, give favorites extra weight, and choose representative photos from groups of similar shots.

Here's an anonymized example of a real response from Claude:

The dates give the story its sequence. Locations connect the campsite, river, and stop on the drive home. People and descriptions connect names with activities and small details, so the agent can bring the collection together into an account of the trip.

Agents can also propose an album and draft captions for selected moments, asking for clarification where the records leave something uncertain.

The agent brings the reasoning. Gumnut brings a reusable understanding of the collection.

An agent builds a story like that by connecting information across photos.

Here's an example response from Gumnut's get_asset endpoint for a real photo from another camping trip. I've replaced personal information with made-up names and IDs, and omitted image links and unrelated fields. I've also trimmed the description and the faces and people lists for readability.

There's no image in that JSON. Yet an agent already has a lot to work with. Each part comes from a different kind of processing, and each answers a different question.

For a photo eligible for face detection under user preferences and applicable local regulations, face locations and named people can give an agent more context. Here's one of the faces in this example and its corresponding person:

The person_id tells the agent this is Milo. The bounding box tells it where his face is in the photo. Together with the image dimensions, it places him on the left side.

Now connect that with the description: a young boy on an adult's lap, on the left, holding a marshmallow stick. An agent can associate Milo with that activity instead of only knowing that he appears somewhere in the photo. If the scene is ambiguous, it can ask for clarification or inspect the image.

Person IDs also connect appearances across photos, so an agent can follow the same person through a trip or find their pictures across years. A description alone says "a young boy." The library gives that person an identity.

Gumnut can connect new faces to people already known in the library, or group consistent faces into people you can name. Uncertain matches can stay unassigned rather than acquiring a confident but wrong identity.

I wrote more about that challenge in Knowing who's in your photos. My kids and their cousins were a useful reminder that similar-looking relatives, hats, and awkward angles can make matching difficult. The goal is to give an agent useful identities and room to ask for clarification, with your corrections becoming part of the library it works with.

Gumnut uses a vision-language model (VLM), an AI model that can interpret images and describe them in words, to generate a description after a photo is uploaded or edited. It stores that description and returns it alongside the other asset information, so an agent can use it without opening the photo and running its own vision model each time.

The descriptions cover the scene, subjects, actions, lighting, and composition. They need enough detail to be useful, while staying short enough for an agent to compare many photos together. In this example, Gumnut gives the agent a marshmallow-roasting moment rather than a filename it has to guess from.

I wrote more about that balance in Helping agents "see" your photos in Gumnut. One lesson was to ask the model for observable details rather than invented mood or artistic intent: a cat sitting in a toy wagon is visible; a story about what the cat is feeling isn't. The agent can draw its own interpretations by combining the description with the people, places, dates, and organization in the library.

Gumnut also provides a quality_score from 0 to 1, giving the agent a signal for how well a photo is focused, exposed, and composed. A higher score can help it choose between similar shots, but doesn't tell it how meaningful a photo is to you. Evaluating photo quality has been a deep and interesting journey in itself that I intend to write about in a future blog post.

Latitude and longitude place the photo on a map; altitude gives its elevation in meters. Gumnut also reverse-geolocates coordinates into useful place names. "Hendy Woods State Park" gives the agent context that coordinates alone wouldn't communicate.

Getting those names right took some work. I compared several reverse-geocoding services against coordinates from real photos, looking at both accuracy and whether the label was a place I'd actually recognize. Some services were better at street addresses than at naming parks, stadiums, or landmarks.

Gumnut also filters the provider's response before using it. It considers the kind of place and its distance from the photo's coordinates, excluding unrelated shops or restaurants and points of interest that are too far away. When the response doesn't give enough information to support a specific place name, Gumnut falls back to a broader location label. I'd rather give an agent a useful region than a misleading name for the business down the street.

Photo dates take more work than simply reading EXIF. Timezones can be missing, exports can change file dates, and scanned photos may have no capture metadata at all.

Gumnut preserves reliable capture metadata. When it's missing, it can draw on dates in filenames, file timestamps, and timezone clues from nearby photos taken with the same device to fill gaps. The agent gets a capture-local time to help order photos and keep an evening's memories on the right day. Where the evidence is incomplete, Gumnut avoids inventing precision or treating a missing timezone as UTC.

Gumnut also reads the information that cameras and editing tools store inside files, across different formats. The agent gets consistent fields for the camera, lens, and capture settings.

In this example, it can see that the photo came from a Pixel 9 Pro, along with its aperture, ISO, and exposure time. That can help it find photos from a particular camera or understand the conditions a photo was taken in.

An agent can also learn from the way you've organized and selected your photos.

Stacks tell it which photos belong together. In an automatically grouped burst, the frames are often very similar. An agent choosing photos for an album can pick a representative from that stack instead of filling the album with nearly identical shots. A stack you've made yourself may have another purpose, so it's a clue rather than a guarantee.

Favorites tell it which photos you've singled out. A favorite may be your best shot, or a blurry picture that means more to you than the technically better one. That gives an agent a personal signal that an image-quality score can't supply.

Albums tell it which groups of photos were meaningful enough to organize. An album's name, description, and membership can provide context about a trip, a person, or a project. Instead of reconstructing every event from individual images, the agent can build on the organization you've already done.

These signals are available through Gumnut's library tools alongside asset metadata. They help an agent connect the details of individual photos with the way you've organized the collection.

Many tasks can start with descriptions and metadata, then open only the images where visual detail matters — and when you approve. Gumnut makes that context available through its API and Model Context Protocol (MCP), the standard agents use to call tools.

Connected applications authenticate and work within library permissions. They can also receive image links or request visual inspection, so working from metadata doesn't mean an agent can never access an image.

The processing behind that context matters too. Generating descriptions uses a hosted model with zero data retention routing; looking up place names shares coordinates with a geocoding service. Those controls are separate from the policies of the agent you connect.

Names, descriptions, and locations are private information in their own right. My goal is to let agents work with the context a task needs, while making those access and processing boundaries clear.

Sign up for Gumnut and try it with a few photos from a trip, a family gathering, or an everyday moment you want to remember. Ask an agent to help tell the story or choose pictures for an album. I'd love to hear what it gets right, what it misses, and what you'd ask it to do next.

If you're building a product that works with photos, have a use case in mind, or just want to compare notes, get in touch. You can also leave a comment below.

What would you want an agent to do with your photo library?