Scalable Webhooks for XM Cloud Forms: Capturing Data Using Vercel, MongoDB and Node.js

The combination of XM Cloud, Vercel, MongoDB, and Node.js creates a robust and scalable architecture for capturing form data. Here’s why they are suitable for this task:

1. Scalability: All four technologies are designed to handle growth. XM Cloud’s flexible content management, Vercel’s automatic scaling, MongoDB’s sharding capabilities, and Node.js’s non-blocking I/O operations ensure that the application can scale efficiently as traffic and data volume increase.
2. Speed and Performance: Vercel optimizes frontend delivery, while Node.js allows for fast processing of incoming data. This results in a responsive user experience, essential for capturing form submissions effectively.
3. Flexibility: The headless nature of XM Cloud allows for easy integration with other services, while MongoDB’s schema-less design supports dynamic data structures, making it easy to adapt to changing data requirements.
4. Developer-Friendly: The JavaScript ecosystem allows developers to use a unified language (JavaScript) across the stack. This reduces complexity and enhances productivity, making it easier to develop, deploy, and maintain the application.

Step 1: Setting Up XM Cloud Forms

Setting up forms in XM Cloud is a straightforward process that allows you to collect user data effectively. This article will guide you through creating a form. Let’s start by setting up a basic contact information form to capture the following details:

Set Up Form Submission

• Settings: Look for settings related to form submission.
• Webhook URL: To create a new webhook, click on the “Manage webhooks” option highlighted in the screenshot. This will be the URL for your Node.js serverless function deployed on Vercel. You can skip this step for now; we will provide the URL once the API is deployed to Vercel, which will be discussed in the following steps.
• Success and Error Messages: Customize the messages that users will see after form submission. Provide feedback for successful submissions and guidance for errors.

Step 2: Set Up MongoDB (MongoDB Atlas)

• Create a MongoDB Atlas Account: Sign up for a free MongoDB Atlas account at https://www.mongodb.com/cloud/atlas.
• Create a Cluster: After logging in, create a new cluster in MongoDB Atlas. Choose the free tier to get started.
• Create a Database User: Once your cluster is set up, create a database user with a username and password.
• Obtain the Connection String: In MongoDB Atlas, click Connect on your cluster, and then select “Connect your application.” You’ll see a MongoDB connection string like this:

mongodb+srv://<username>:<password>@cluster0.mongodb.net/<dbname>?retryWrites=true&w=majority

• Replace <username>, <password>, and <dbname> with the credentials and name you set up.
• Allow Access from Anywhere: Under the Network Access tab, add your IP address or allow access from anywhere (0.0.0.0/0) for testing purposes.

Step 3: Create the Node.js Webhook Project

Create a Project Directory: Create a new directory for your project if you don’t already have one:

mkdir webhook-project
cd webhook-project

Initialize the Project: Initialize your project using npm:

npm init -y

Install Dependencies: You’ll need mongodb for interacting with MongoDB, and other dependencies like express for creating the webhook:

npm install express mongodb

Create the Webhook Function: Inside the project, create an api/ folder (required for Vercel serverless functions) and then add the webhook logic inside api/webhook.js:

const express = require('express');
const { MongoClient } = require('mongodb');

const app = express();
app.use(express.json());

const uri = process.env.MONGODB_URI;

let client;
let db;

async function connectToDatabase() {
    if (!client) {
        client = new MongoClient(uri, { useNewUrlParser: true, useUnifiedTopology: true });
        await client.connect();
        db = client.db(process.env.DB_NAME);
    
    }
    return db;
}

app.post('/api/webhook', async (req, res) => {
    const { name, email, message } = req.body;

    try {
        const db = await connectToDatabase();
        const collection = db.collection('contact_submissions');

        const result = await collection.insertOne({ name, email, message, date: new Date() });
        res.status(200).json({ message: 'Data inserted successfully', id: result.insertedId });
    } catch (err) {
        console.error('Error inserting data into MongoDB:', err);
        res.status(500).json({ error: 'Database insert failed' });
    }
});

module.exports = app;

Configure Environment Variables:

• The MongoDB connection string and database name should be kept in environment variables for security.
• In the api/webhook.js, you will use process.env.MONGODB_URI and process.env.DB_NAME to refer to them.

Step 4: Deploy to Vercel

• Install Vercel CLI: If you haven’t installed the Vercel CLI yet, install it globally via npm:

npm install -g vercel

• Login to Vercel: Log in to Vercel. If you’re new to Vercel, create an account by following the sign-up process before logging in.

vercel login

• Deploy the Project: Deploy your project to Vercel:

vercel

• Vercel will guide you through the deployment process.
• Choose the appropriate project name or create a new one.
• Select “Yes” for the api/ directory to deploy it as serverless functions.

• Add Environment Variables: After deploying, you need to add the MongoDB connection string and database name as environment variables in Vercel.

vercel env add MONGODB_URI
vercel env add DB_NAME

• You’ll be prompted to add the connection string and database name. For example:

MONGODB_URI=mongodb+srv://<username>:<password>@cluster0.mongodb.net/<dbname>?retryWrites=true&w=majority
DB_NAME=your-database-name

• Alternatively, you can add these environment variables via the Vercel Dashboard:
• Go to your project’s dashboard on Vercel.
• Navigate to Settings > Environment Variables.

• Redeploy the Project: After setting environment variables, redeploy your project using the following command:

vercel --prod

Step 5: Test the Webhook

After deployment, Vercel will provide a public URL for your project, such as https://your-app.vercel.app/api/webhookYou can test the webhook using Postman or curl:

curl -X POST https://your-app.vercel.app/api/webhook \
  -H "Content-Type: application/json" \
  -d '{"name": "S Kumar", "email": "skumar@example.com", "message": "This is a test message"}'

In case you get Authentication error in Postman, try to change this setting in Vercel.

Step 6: Verify Data in MongoDB

After the data has been successfully inserted, you can verify the new record in MongoDB Atlas. Alternatively, you can use MongoDB Compass to connect locally.

Step 7: Assign Webhook URL to XM Cloud Form:

Now, return to Step 1 and create a new webhook endpoint by clicking on the “Manage webhooks” option. Enter the Vercel URL that we just tested in Postman.

That’s it. Setting up forms in XM Cloud is an essential first step in capturing user data effectively. With the right configuration, you can ensure a smooth data collection process that feeds directly into your backend system, ready for further processing and storage.

You can find my other blogs here.

Thank you!