Enabling image AI analysis in Sitecore

Integrating AI image analysis into Sitecore requires connecting it with an external AI service capable of analyzing images. Although Sitecore does not offer native AI image analysis capabilities, it can be linked with several AI services, such as Azure Cognitive Services, Google Cloud Vision, and others.

Recently, we received a requirement to analyze images uploaded to the media library using an AI tool and capture the resulting tags in a custom field.

The primary goals of incorporating Image Analysis into our Sitecore project are to enhance the content management process, improve user experience, and automatically generate SEO-friendly tags and descriptions to boost search engine visibility.

We’ve implemented Azure Cognitive Services to integrate Image Analysis into our project. Here are the steps that cover this implementation:

Step -1: Azure Setup

• Sign up / Sign In to Azure portal for Azure Cognitive Services.
• Create a Computer Vision resource.

• After setting up the Computer Vision API, you will get an endpoint URL and an API key that will be used to make requests.

Step-2: Sitecore Custom Module

• Open your project in Visual Studio.
• Install necessary NuGet packages for calling Azure Cognitive Services.

Install-Package Microsoft.Azure.CognitiveServices.Vision.ComputerVision

• Create the Event Handler — Create a class that will handle media item events, such as when an image is uploaded.

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using Microsoft.Azure.CognitiveServices.Vision.ComputerVision;
using Microsoft.Azure.CognitiveServices.Vision.ComputerVision.Models;
using Sitecore.Data.Items;
using Sitecore.Events;
using Sitecore.SecurityModel;

namespace Demo.AzureAIServices
{
    public class MediaItemUploadedHandler
    {
        private static readonly string subscriptionKey = "your-subscription-key";
        private static readonly string endpoint = "your-endpoint-url";

        public void OnItemSaved(object sender, EventArgs args)
        {
            var item = Event.ExtractParameter(args, 0) as Item;
            if (item == null || !item.Paths.IsMediaItem)
                return;

            using (new SecurityDisabler())
            {
                using (new EditContext(item))
                {
                    var mediaItem = new MediaItem(item);
                    using (var stream = mediaItem.GetMediaStream())
                    {
                        var analysisResult = AnalyzeImage(stream);

                        item["AI Analysis Result"] = analysisResult;
                    }
                }
            }
        }

        private string AnalyzeImage(Stream imageStream)
        {
            ComputerVisionClient client = new ComputerVisionClient(new ApiKeyServiceClientCredentials(subscriptionKey))
            {
                Endpoint = endpoint
            };

            var features = new List<VisualFeatureTypes?> { VisualFeatureTypes.Description };
            var analysis = client.AnalyzeImageInStreamAsync(imageStream, features).Result;
            return string.Join(", ", analysis.Description.Captions.Select(c => c.Text));
        }
    }
}

• Update Sitecore Configuration patch in the App_Config\Include folder.

<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
    <sitecore>
        <events>
            <event name="item:saved">
                <handler type="Demo.AzureAIServices.MediaItemUploadedHandler, Demo.ScWebsite" method="OnItemSaved"/>
            </event>
        </events>
    </sitecore>
</configuration>

• Add Custom Field — In Sitecore, go to the Media Library template and add a new field called “AI Analysis Result”.
• Deploy the DLL and config changes.

Step-3: Test the Integration

• Upload an image to the Sitecore Media Library.
• Open the media item in Sitecore Content Editor and verify that the “AI Analysis Result” field is populated with the analysis results from Azure Cognitive Services.

Following these steps enables you to develop a custom module in Sitecore that seamlessly integrates Azure Cognitive Services for image analysis. This approach leverages Sitecore XP and XM capabilities to handle content and events efficiently while integrating external AI services seamlessly.

You can find my other blogs here.

Thank you!