Automatically scale Sitecore on Azure PaaS

Since Sitecore 8.2 update 1, we can simply scale our App Service plans up or out. You can find these options on the App Service Plan as well as the App Service themselves.

Scaling up means selecting a different tier for our subscriptions – more RAM and cores for instance.

scaleup

As you can see, I’m currently running on the S1 tier – which I can’t recommend to anyone as it’s quite slow (of course, it is way below the minimum specs found in the installation guides)…

Our other option is to scale out. That means we can add (or remove) a number of instances based on… something. Let’s take a look.

When scaling out, we have a couple of options:

  1. Scale manually. I can do this by setting the ‘scale by’ field to the option ‘an instance count that I enter manually’. I can select a number of instances using a slider.
    scalemanually
    In the above image, I’m simply making sure that we’re going to run on 3 instances.
  2. Scale on CPU percentage. This option is a bit smarter, we can now automatically scale up.
    scalecpu.png
    As the above image shows, I’ve set Azure up to use 2 instances at minimum and 7 maximum. Azure will automatically scale this based on the target range of the CPU – if that falls between 80 and 100, a new instance will be added (up to the maximum of 7 instances). If it falls below that minimum in the target range, it will automatically scale down as well.
  3. Scale on schedule and performance rules. The performance rules are pretty similar to the previous option, but gives us a bit more flexibility and power.
    scaleout
    As you can see in the above image, we’ve set a Default profile up, with a scale of 1 as minimum and 10 as maximum. We then have 2 rules: One to scale up, every time the CPU percentage hits 80% on average, over the past 10 minutes (see the right hand side), it will auto-scale up. In addition to that, it will also auto-scale down when on average over the past 10 minutes the CPU percentage is <1. Which might be a bit too low…
    There are other metrics that we can use rather than CPU percentage there as well,  and these rules can be set to scale up differently, for instance spinning up a different number based on the metric selected.
    We can also scale on schedule. If I know that I have very high traffic during the week and lower during the weekend, or if my traffic is high during work hours but low outside of them, or even if I have a fixed date that I know I have high traffic I can make sure I scale based on that as well. For this we need to select the scaling profile.
    recurrence
    For the first couple of options mentioned above I can use the ‘recurrence’ tab. For the fixed date, I can (obviously) use the ‘fixed date’ tab, where I can set a start date/time and end date/time. I’ll need to create a profile for my high traffic and low traffic as well, and build out the rules for both.
    schedulerules

There is one really important point here that I have yet to mention: Licensing. If you are on a perpetual licensing model and you want this auto scaling functionality, I would highly recommend putting in the maximums to make sure you won’t be in breach of the license.

Application Insights for Sitecore (or, where have my logfiles gone?)

So with Sitecore’s new Azure PaaS offering, you might’ve noticed in your Azure resource group that there’s an Application Insights resource there.

Application Insights can be used to monitor the live web applications and helps with diagnosing issues and figuring out what users are doing with your app. More information here. Application Insights is quite a big and powerful tool, and I won’t be able to cover everything juts quite yet. I will however run through some of the cool features.

health

Even by itself that’s a pretty cool (and useful) tool to have. But the Application Insights have actually also been integrated with Sitecore.

In the Application Insights overview blade we can get some additional information by clicking the Search button. We can filters the results there as well, by type (such as trace, request or page view) or property. For Sitecore’s log entries we are interested in the Trace filter.

trace

To query the log files in more depth, we can actually hit the Analytics button in that same overview blade. This will open a new window, where we can begin writing queries to find issues logged in our Sitecore logs.

insights

Any of the properties you see in the bottom right we can filter on – you can find those properties on the left hand side as well, in the Schema. There’s also some very useful documentation on writing queries.

In the Performance blade, I can also create my own charts based on Sitecore statistics, as well as see how my application is performing.

performance

We can also do things like set up alerts when there’s over 5 server exceptions in the application (see image below), and if this happens send an email to the administrators.

alerts

This type of reactive analytics isn’t the only thing you can do with Application Insights though – we can use some more advanced, proactive features. In the linked document, it states “For example, if your app loads slowly on some OS versions, or has slower responses in particular geographical location, you’ll get an email about it. Proactive Detection uses machine learning and data mining to help you find issues that would be hard to detect – and are often only discovered when customers complain.”

For more information about this type of diagnostics, see this Microsoft documentation.

Lastly, there’s also the Application Map, which allows you to see a diagram of your setup, showing the dependencies of your components.

appmap.png

By clicking through on the various components, we can see which parts of our setup are working correctly and which have issues. As you can see in the image above, there seems to be something wrong with my search. I can then click through on that to see some detailed information.

searchfailures.png

By clicking through on in the chart, I can also see even more detail (in the below image, I clicked the ‘Dependency Failures’ chart)

searchfailuredetail.png

From there I can then even create a new Work Item, either in Visual Studio Team Services or in GitHub, making sure the Work Items have all relevant data embedded.

I’ve covered only a small portion of what Application Insights can do for us, but I do hope I’ve shown its power and its usefulness. I’m pretty excited, in any case🙂

Deploying code to Sitecore on PaaS

Now that we have our Sitecore environment on Azure PaaS, lets get some code on there!

I’m going to use Visual Studio to deploy my code, in the same way we can publish code locally to the website if we’re not working in the web root of the project. Of course, you can have your deployment process fully automated using things like FTP, PowerShell and more.

In the Azure portal, I can select the server I’d like to deploy to and select the ‘…More’ option to be able to download the PublishSettings file.

publishsettings

Little side note, if you wanted to upload your code through FTP you can get the username and password to use from the PublishSettings file as well – just open the file in your favourite text editor.

publishsettings2.png

All details required for the FTP connection are there. Also, if you were silly enough to put your username and password on the Internet, you can reset your publish settings by going going back to the Azure portal and in the server blade select the ‘Reset publish profile’ options.

Back to deploying our code through Visual Studio: We now can open (or create) our solution in Visual Studio and select ‘<New Custom Profile>’ in the Publish dropdown. Just give the profile a name, go back to the Profile button and select the ‘Import’ option. Here, you can select the PublishSettings file we downloaded earlier.

importprofile

At this point you’ll get an overview of the connection Visual Studio created for you, which we can then validate.

validateprofile

Now all that’s left is to click the Publish button, and we’ll have our Sitecore environment all up-to-date with the latest and greatest code!

Awesome. So now I have my code online – but oh no, an error. How can I debug this, when I just get a simple yellow screen? Well, first of all you could dive into the logfiles to get some more information. In addition I could turn off remote errors, but neither of those might actually tell me what’s happening within my code. What I’d really like to do is to start debugging. This first needs to be enabled in the Azure portal. You can find it in the Application Settings of your app service.

debug

Simply set Remote Debugging in the Debugging section to On, and set your Remote Visual Studio version to the version you use. Alternatively, Visual Studio also asks if it can enable remote debugging if you skip setting  it manually.

Then all that’s left is to publish your website with the Debug Configuration. In your Server Explorer in Visual Studio you can then right click on the Azure website and select Attach Debugger. The browser will then automatically open to the home page, so you might have to browse around a bit to find your error page🙂.

Don’t forget to deploy your code using the Debug profile rather than the Release profile if you do want to debug.

PS. If any of your code has a dependency on the data folder, it has set it to the /App_Data folder by default.

Update

It has been pointed out to me that we don’t actually need to download the publish settings from the portal. Instead, when creating a new Publishing Profile, click the ‘Microsoft Web Apps’ publishing target. Sign in the pop up window, and there’s an option to select a web app (or create a new one)

publishprofile

Getting started with Sitecore on Azure PaaS

So now that Sitecore 8.2 update 1 has been released, we can finally stop using the Sitecore Azure module and go use the true power of Azure PaaS.

Be aware that at this point in time Sitecore only has an XP1 and XM1 delivery using the Azure Resource Manager (ARM) templates – more on ARM here – and XM1 only on the Azure Marketplace. Keep your eyes peeled for further updates on this, as other options will be released.

Before we get into this, make sure you have the following installed on your machine:

By now, you should have downloaded the ARM templates for Sitecore – or alternatively you can also use the Azure Marketplace to install Sitecore on Azure as well, so lets see how this works.

First of all, Sitecore’s ARM templates will not create the MongoDBs for us, so we need to start by getting that set up. I’m going to use mLab‘s free Sandbox tier for that in this blog post, but feel free to use other options (such as the xDB Cloud offering by Sitecore themselves).

mongodbsetup

At the moment of writing I can’t select Azure’s West Europe (Amsterdam) location, as mLabs does not provide a single node there (the only option that’s free).

mlabsdb

I’ll need to create the database for each of the MongoDBs. After the creation of my database is complete, I just add a user to each as well and copy and paste the connectionstring it’s giving me into the appropriate xX.Template.params.json file. In this file, I need to change:

  • The connection strings to all 3 MongoDB connection strings: analytics, tracking_live and tracking_contact – these need to be set to the correct connection strings given to us in the mLabs overview
  • sqlserver.login – the username for the sql server user. This user will be created in the databases automatically
  • sqlserver.password – the password to use for the user created (make sure this has 8 or more characters or an error will be thrown)

Lastly, I need to update the PowerShell script Run.ps1 to use the correct parameters. I need to set the following options:

  • $ArmTemplatePath – needs to be set to the correct xX.Template.json file location
  • $ArmParametersPath – needs to be set to the file location of our updated .params.json file
  • $licenseFileContent – this needs to point to a valid license.xml file
  • $Name – the name for the deployment
  • $location – the Azure region to deploy in. A good overview can be found here, including which functionality is or isn’t available in the region. Make sure the region you select supports Azure Search – not all of them do.
  • $AzureSubscriptionId – Your subscription ID to use for the deployment. You can find this in the ‘Billing’ blade in the Azure portal.

As an additional (optional) change, one could set up a principle service and use that. I just want to get up-and-running ASAP, so I’ll take the manual login option instead for which I don’t have to make any changes.

I also noticed that in my xP1.Template.json file the Application Insights were set to the location of “Central US”:

{
  "type": "Microsoft.Insights/Components",
  "name": "[variables('appInsightsNameTidy')]",
  "apiVersion": "[variables('appInsightsApiVersion')]",
  "location": "Central US",
  "properties": {
    "ApplicationId": "[variables('appInsightsNameTidy')]",
    "Application_Type": "web"
  },
  "tags": {
    "provider": "[parameters('sitecoreTags').provider]"
  }
}

That’s technically fine, but I’d rather have it in the same region as my other resources, so changed the “location” line to the following instead:

"location": "[variables('resourceGroupLocation')]",

Of course, you’ll need to make sure that location supports the Application Insights.

After this I’m all set so I can run the PowerShell Script. It is at this point that you can safely go get a cup of tea, as this will take a little while. In fact, my deployment took 23 minutes and 49 seconds. I didn’t bother moving the blobs used to install Sitecore in my own data center so I could’ve had an even faster deployment – but just under 24 minutes is still pretty impressive, no?

environment

Improving the usability of Sitecore’s bucket facets

My dear colleague Steve McGill pointed something out to me I never noticed before. However after noticing it I could not help but be extremely bothered about it.

A true case of:

what has been seen

(I might or might not be exaggerating just a tad here)

In any case, I’m talking about the search facets when searching for bucketable content in the Sitecore Content Editor:

search.png

So far so good. Lets select a couple of facets.

facets.png

It’s still not too bad – but now imagine if we’ve have multiple custom facets selected – especially booleans. The facet, when selected, would just say ‘0’ or ‘1’ – but we don’t know which facet this would be.

facets2

This is definitely less useful… So how can we make this more useful? The file that we’d be interested in changing here is webroot/sitecore/shell/Applications/Buckets/Scripts/ItemBucket.js.

A quick fix might be to change one line of code in that JavaScript. In line 263 (on Sitecore 8.1 update 2 at least) the line is:

facetList += '<li class="filter"><a href="javascript:void(0);" onclick="javascript:RemoveFacet(\'' + this.value + '\');" title="' + escapedText + '" class="facetClick facetClickSelected">' + innerText + "</a></li>";

Which we can change to (I’ve added the bold for emphasis):

facetList += '<li class="filter"><a href="javascript:void(0);" onclick="javascript:RemoveFacet(\'' + this.value + '\');" title="' + escapedText + '" class="facetClick facetClickSelected">' + this.value.split('|')[0] + ':' + innerText + "</a></li>";

‘this.value’ is simply the field name Sitecore facets on. I added ‘.split’ as well because it passes the value in there as well (so for English language facet it would pass ‘culture|en’).

facets3

This isn’t flawless either, since the ‘Author’ facet is using the ‘parsedcreatedby’ field in the index. If we wanted to change anything there we’d have to change the AppendFacet method in the same file to take in the display name of the facet as well (rather than the field).

One big consideration to keep in mind here is that we’re overriding a Sitecore file, which of course might change with future updates in Sitecore, so use with caution.

Campaigns, querystrings and FXM

So I saw that in Sitecore’s Federated Experience Manager (FXM) campaigns get added to a page by adding a page filter. I came across this great article that shows in detail how to set goals, campaigns and the like.

In any case, I found it a bit strange that a campaign would have to be set as a page filter, rather than being able to link to an affiliate site with the campaign tracker in  the URL like we’re used to in our normal use of Sitecore (e.g. use sc_camp=someguid).

Of course, as with most things Sitecore, we can customize FXM and I figured this would be a nice way for me to get some more experience with FXM.

The most obvious step first of course: Adding the site to FXM. This can be done through the Federated Experience Manager link on the Launch Pad, and you can then click ‘add external website’. In my case I’ve added the website Sandbox FXM with the domain sandbox.fxm.

fxm-new-site

FXM will then give you a beacon (in the ‘Generated script’ field) which you can parse into your external site. I’ve just dropped mine right before the closing body tag.

Don’t forget to create a site in IIS and update your hosts file as well to make sure you can browse it.

After I pasted that I ran into my first issue: I kept getting the error “Error: Failed to load resource: the server responded with a status of 400 (DomainInvalid)” when loading the FXM site. This has a simple fix: Publish your FXM site, which you can do from the FXM screen.

So, that resolved we can start to write some code. By looking at the TrafficTypeProcessor I found I need to inherit from ITrackPageVisitProcessor and ITrackPageVisitProcessor<ITrackPageVisitArgs>.

Then, in the code I will grab the URL of the page, get the sc_camp querystring from it and register that campaign

public class RegisterCampaignProcessor : ITrackPageVisitProcessor, ITrackPageVisitProcessor<ITrackPageVisitArgs>
{
    public void Process(ITrackPageVisitArgs args)
    {
        var url = System.Web.HttpContext.Current.Server.UrlDecode(WebUtil.GetRawUrl());
        var campaign = string.Empty;
        if (url.IndexOf("sc_camp=") > 0)
        {
            campaign = url.Substring(url.IndexOf("sc_camp=") + 8, 38);
        }
        if (!string.IsNullOrEmpty(campaign))
        {
            var campaignItem = args.Context.Database.Database.GetItem(campaign);
            if (campaignItem != null)
            {
                CampaignItem item = new CampaignItem(campaignItem);
                args.CurrentPageVisit.TriggerCampaign(item);
            }
        }
    }
}

When we do WebUtil.GetRawUrl() we will actually get back something similar to this:
/sitecore/api/ssc/Beacon/Service/beacon/trackPageVisit/?contactId=6b19088ea2c54bdab775231697a5e0a3|False&sessionId=44ir0jmxxilktj3xiaqlidt5&page=http://sandbox.fxm/?sc_camp={87925D56-34D9-4080-A725-6EC8B3B4CB0A}&referrer=&rt=1453220938704
So I’ll grab the campaign from there – although in a production scenario you might want to be a bit more careful about this because this particular code requires you to have the accolades as part of the campaign id (as well as not having any checks on whether it’s a GUID etc). From that GUID I resolve the campaign item as an Item, create a CampaignItem object from that and trigger that on the campaign.

We can use our Sitecore include files to patch this processor in in the correct place – in my case I’ve patched it right before the TrafficTypeProcessor:

<?xml version="1.0"?>
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
    <sitecore>
        <pipelines>
            <group groupName="FXM" name="FXM">
                <pipelines>
                    <tracking.trackpagevisit>
                        <processor patch:before="*[@type='Sitecore.FXM.Pipelines.Tracking.TrackPageVisit.TrafficTypeProcessor, Sitecore.FXM']" type="Sandbox.FXM.TriggerCampaign.RegisterCampaignProcessor, Sandbox" />
                    </tracking.trackpagevisit>
                </pipelines>
            </group>
        </pipelines>
    </sitecore>
</configuration>

So that’s now all well and good but how do I figure out whether it’s working? If I had an empty xDB it would be easy enough but I actually have a lot of dummy data in there so had to be a bit more creative. In Mongo I wrote a query to figure out which data came from FXM:

db.Interactions.find({ "SiteName" : "Sandbox_FXM" }, { "ContactId" : 1 } )

In there, I can see that a campaign has been triggered, but I’d also like to see my xFile, andof course the ContactId is then shown as a binary type 3 value. I’m using MongoVue to view my Mongo data, which actually comes with a very handy option:

mongo-vue-preferences.png

Using that I can find the GUID I had for that visitor. I can then open any random xFile and copy/ paste the GUID of this particular Contact in the URL.

fxm-contact-xfile.png

Although for the people paying attention: The contact ID is also in the URL we had from running WebUtil.GetRawUrl(). Sadly I couldn’t actually use that in my example as it was a new InPrivate window and I had closed it.

Saving WFFM form result as PDF

In the Web Forms For Marketers module I wanted to be able to save the responses as PDF, to then be able to send that PDF with the responses, save the PDF in the Media Library or whatever else.

So there are 2 parts to this:

  1. Create a new custom WFFM Save Action
  2. Create a PDF using some kind of library. In my case I used PDFSharp, which is open source and easy to use but I haven’t done any research into PDF libraries so I couldn’t say whether there’s better ones out there

First up is creating the new WFFM Save Action, which is fairly simple. We’ll need a Sitecore definition item and a bit of code as usual. The WFFM Save Actions can be found in /sitecore/system/Modules/Web Forms for Marketers/Settings/Actions/Save Actions. I’ve created a ‘Save PDF’ item there based on the ‘Submit Action’ template.

wffm-save-action

Don’t forget to set the Assembly and Class fields.

Then the code. When we’re creating a WFFM save action we need to make sure our class inherits from the WffmSaveAction class. We’ll also have to reference Sitecore.WFFM.Abstractions and Sitecore.WFFM.Actions. For our PDF creation functionality we’ll also need to add a reference to the PdfSharp assembly.

In my case, this is how my class looks:

namespace Sandbox.WFFM.Actions
{
    using Sitecore.WFFM.Abstractions.Actions;
    using Sitecore.WFFM.Actions.Base;
    using Sitecore.Data;
    using PdfSharp.Pdf;
    using PdfSharp.Drawing;

    public class SavePDF : WffmSaveAction
    {
        public override void Execute(ID formId, AdaptedResultList adaptedFields, ActionCallContext actionCallContext = null, params object[] data)
        {
            // Our code goes here.
        }
    }
}

Now in the Execute method we’ll start by creating a new PDF document, adding a new page and writing some content there.

PdfDocument pdf = new PdfDocument();
PdfPage page = pdf.AddPage();
XGraphics graph = XGraphics.FromPdfPage(page);
XFont font = new XFont("Verdana", 20, XFontStyle.Bold);
var i = 0; // Used for positioning the text
foreach (AdaptedControlResult field in adaptedFields) // Loop through all fields in the form
{
    graph.DrawString(string.Format("{0}: {1}", field.FieldName, field.Value), font, XBrushes.Black,
new XRect(40, i, page.Width.Point, page.Height.Point), XStringFormats.TopLeft);

    i = i + 40; // Make sure to increment i so we don't write on the same line over and over again
}

pdf.Save(@"C:\temp\sandboxtest.pdf"); // Probably shouldn't write this here... If we do we need something smart for the naming

Then to test. I’ve added a form on one of our news articles:
wffm-form
And this is the resulting PDF after filling in the form:
pdf-result

Sitecore – Retrieve data from MongoDB

As we all know, Sitecore’s xDB collects a TON of data. So if we want to pull some of that data from the MongoDBs, how do we go about that? I had never worked with Mongo before, so in case you haven’t either I hope this is of use to you.

Initially, to view the documents in the Mongo collections I used the free edition of MongoVue. It is great for things like viewing how many contacts you have in your Contacts collection, or digging into the details of the Interactions.

Mongo Collections Overview

Now, if I want to see some of the Interactions, that’s completely possible as well. For instance, I can either use the View option on the Contacts collection, or I can write a query.

Viewing:

Contact details

Writing a query:

Sample query in MongoVue

Of course, you can do the same thing in the Mongo shell as well, if you prefer.

I’ve found that I keep re-using a couple of different queries, so here are a couple of the ones I found useful:

Finding interactions with a certain amount of Engagement Value points:


{ "Value" : { $gt : 10 } }

This will get all interactions with more than 10 Engagement Value points. Instead of $gt (greater than), you can also use other operators such as $lt (less than), $lte (less than or equal to) or $gte (greater than or equal to).

For more query operators, view the Mongo Docs page.

In case you’re using the Mongo shell, you might want to add a limit to the results as well, and format the documents. In the Mongo shell, your query might look like this (assuming you’ve set the db to be your Mongo Analytics database):


db.Interactions.find
( 
    { "Value" : { $gt : 10 } } 
)
.limit(1)
.pretty()

Limit here is pretty obvious, it will only result the first result in our case. Pretty will format the document, in a similar way to CTRL+K, CTRL+D in Visual Studio. Another thing you might want to add in the query for the Mongo shell is a projection of the fields you’d like to be displayed or hidden. Projections can be used as follows:


db.Interactions.find
(
    { 
        "Value" : { $gt: 10 } 
    }, 
    { "ContactId" : 1 } 
)
.limit(1)
.pretty()

In this case, we’ll only be displaying the ContactId field. To exclude a field, you can set it to 0 instead. For more information on Projection in Mongo, I again refer you to the Mongo Docs page.

Another query I tend to use is to figure out how many contacts I have from various countries. From MongoVue:


{ "GeoData.Country" : "United Kingdom" }

I have to admit that I’m not sure how to count the number of records in MongoVue.

From the Mongo shell:


db.Interactions.find( { "GeoData.Country" : "United Kingdom" } ).count()

Lastly, finding interactions with (for instance) 2 page events:

Again, in MongoVue:


{ “Pages.PageEvents” : { $size: 2 } }

In the Mongo shell:


db.Interactions.find( { “Pages.PageEvents” : { $size: 2 } } ).pretty()

On a last note, it’s important to realize the fieldnames are case sensitive.

On the very last note, for some more (beginners) details on MongoDB, not only writing queries but also architectural information, I would very highly recommend the Introduction to MongoDB course on PluralSight.

I hope this will be of some use🙂

Commerce Server – Cart not persisting

So I was trying to get add the ‘add to cart’ functionality to my website. Basically I would get the cart, then create a cart line based on the product and add that to the cart object:


var cartLine = new CommerceCartLine(product[CommerceConstants.KnownFieldIds.CatalogName], product.Name, null, quantity); // null because I don't currently support variants.
var cart = GetCart();
var request = new AddCartLinesRequest(cart, new[] { cartLine });
myServiceProvider.AddCartLines(request);

And while I was stepping through the code I could see the cart being updated. The only problem was after a reload (or after adding another product to the cart) the cart would be empty again.

At first I thought the problem was my code, but no matter what I changed, it didn’t seem to get any better. I figured I’d better check the Commerce Server Customer and Order Manager tool to find out what was happening to my baskets, and it was there that I found the issue. It turns out that by default the Customer and Order Manager wants to have the Profiles webservice and the Orders webservice to run in https mode. As I was working on my local development environment that wasn’t really much of an option, so I had to figure something else out. Luckily I found a blog post on MSDN which explains what to do.

In short: There’s a CustomerAndOrdersManager.exe.config file, in the install directory of Commerce Server (by default C:\Program Files (x86)\Commerce Server 11\Business User Applications) . In the <applicationSettings> you’ll be able to find the setting “AllowHTTP” which is set to False by default. Set it to True, recycle the app pool and you’re in business!


<applicationSettings>
...
<setting name="AllowHTTP" serializeAs="String">
<value>True</value>
</setting>
...
</applicationSettings>