Sitecore – Retrieve data from MongoDB

As we all know, Sitecore’s xDB collects a TON of data. So if we want to pull some of that data from the MongoDBs, how do we go about that? I had never worked with Mongo before, so in case you haven’t either I hope this is of use to you.

Initially, to view the documents in the Mongo collections I used the free edition of MongoVue. It is great for things like viewing how many contacts you have in your Contacts collection, or digging into the details of the Interactions.

Mongo Collections Overview

Now, if I want to see some of the Interactions, that’s completely possible as well. For instance, I can either use the View option on the Contacts collection, or I can write a query.

Viewing:

Contact details

Writing a query:

Sample query in MongoVue

Of course, you can do the same thing in the Mongo shell as well, if you prefer.

I’ve found that I keep re-using a couple of different queries, so here are a couple of the ones I found useful:

Finding interactions with a certain amount of Engagement Value points:


{ "Value" : { $gt : 10 } }

This will get all interactions with more than 10 Engagement Value points. Instead of $gt (greater than), you can also use other operators such as $lt (less than), $lte (less than or equal to) or $gte (greater than or equal to).

For more query operators, view the Mongo Docs page.

In case you’re using the Mongo shell, you might want to add a limit to the results as well, and format the documents. In the Mongo shell, your query might look like this (assuming you’ve set the db to be your Mongo Analytics database):


db.Interactions.find
( 
    { "Value" : { $gt : 10 } } 
)
.limit(1)
.pretty()

Limit here is pretty obvious, it will only result the first result in our case. Pretty will format the document, in a similar way to CTRL+K, CTRL+D in Visual Studio. Another thing you might want to add in the query for the Mongo shell is a projection of the fields you’d like to be displayed or hidden. Projections can be used as follows:


db.Interactions.find
(
    { 
        "Value" : { $gt: 10 } 
    }, 
    { "ContactId" : 1 } 
)
.limit(1)
.pretty()

In this case, we’ll only be displaying the ContactId field. To exclude a field, you can set it to 0 instead. For more information on Projection in Mongo, I again refer you to the Mongo Docs page.

Another query I tend to use is to figure out how many contacts I have from various countries. From MongoVue:


{ "GeoData.Country" : "United Kingdom" }

I have to admit that I’m not sure how to count the number of records in MongoVue.

From the Mongo shell:


db.Interactions.find( { "GeoData.Country" : "United Kingdom" } ).count()

Lastly, finding interactions with (for instance) 2 page events:

Again, in MongoVue:


{ “Pages.PageEvents” : { $size: 2 } }

In the Mongo shell:


db.Interactions.find( { “Pages.PageEvents” : { $size: 2 } } ).pretty()

On a last note, it’s important to realize the fieldnames are case sensitive.

On the very last note, for some more (beginners) details on MongoDB, not only writing queries but also architectural information, I would very highly recommend the Introduction to MongoDB course on PluralSight.

I hope this will be of some use :-)

Commerce Server – Cart not persisting

So I was trying to get add the ‘add to cart’ functionality to my website. Basically I would get the cart, then create a cart line based on the product and add that to the cart object:


var cartLine = new CommerceCartLine(product[CommerceConstants.KnownFieldIds.CatalogName], product.Name, null, quantity); // null because I don't currently support variants.
var cart = GetCart();
var request = new AddCartLinesRequest(cart, new[] { cartLine });
myServiceProvider.AddCartLines(request);

And while I was stepping through the code I could see the cart being updated. The only problem was after a reload (or after adding another product to the cart) the cart would be empty again.

At first I thought the problem was my code, but no matter what I changed, it didn’t seem to get any better. I figured I’d better check the Commerce Server Customer and Order Manager tool to find out what was happening to my baskets, and it was there that I found the issue. It turns out that by default the Customer and Order Manager wants to have the Profiles webservice and the Orders webservice to run in https mode. As I was working on my local development environment that wasn’t really much of an option, so I had to figure something else out. Luckily I found a blog post on MSDN which explains what to do.

In short: There’s a CustomerAndOrdersManager.exe.config file, in the install directory of Commerce Server (by default C:\Program Files (x86)\Commerce Server 11\Business User Applications) . In the <applicationSettings> you’ll be able to find the setting “AllowHTTP” which is set to False by default. Set it to True, recycle the app pool and you’re in business!


<applicationSettings>
...
<setting name="AllowHTTP" serializeAs="String">
<value>True</value>
</setting>
...
</applicationSettings>

Hidden settings: MinimalProfileScoreCount

I was running through the Sitecore Experience Editor the other day, where I wanted to check whether my various patterns would be showed the correct personalized content. The easiest thing to use is the Experience Explorer, which you can get to through the Other button in the Mode chunk.

Opening the Experience Explorer

This will open the Experience Explorer on the left-hand side of the website with the Experience Viewer on the right-hand side. In the Onsite Behavior tab I set the proper scores to match a particular pattern card. After clicking ‘Apply’ I didn’t see any difference until I checked the Onsite Behavior tab in the Experience Viewer.

Experience Viewer

Easily managed by just clicking ‘Apply’ 2 more times, but a bit annoying. Turns out, there’s a way of going around this: The Analytics.Patterns.MinimalProfileScoreCount setting. By default it won’t show up in your settings so you’ll have to manually add it. After applying this setting the profile personalization will be applied directly after setting the value of Analytics.Patterns.MinimalProfileScoreCount to 1.

Before you change it though, keep in mind that this will not only affect the Experience Explorer, but also the ‘normal’ website – and as such might have adverse effects.

Quick tip: Sitecore Security packages

If you ever want to move Sitecore items over to different environments the easiest way to do this is to create a Sitecore package. This is no different for Sitecore Users and Sitecore Roles. After opening the Package Designer, click the ‘Security Accounts’ button in the ribbon.

The package designer screen

Using the pop-up, you’ll be able to view which roles/ users you have in your package, or add/ remove them using the buttons. Clicking the Add button will pop up another window, which allows you to select your users and roles.

Adding roles and users

After you’ve created your package you can install this on your other environment, using the Package Installer. If the users or roles already exist you’ll get a message that Sitecore will skip the installation of that user or role.

Roles are now all ready for use. Users however need two additional steps to be active:

Firstly, they need to be activated.

Sitecore User Manager showing disabled users

As you can see, in the Locked column, they’ll show up as Disabled. To enable the users the Enable button in the ribbon can be used.

Secondly, the users need their password to be reset. Do this using the Change Password button.

The Reset Password popup

Clicking ‘Generate’ will randomly create a new password, which you can then select and copy to the Old Password field. Fill in the new password, and you’ll be ready to go!

Commerce Server – Missing ProductId in Product Definition

For one of my Sitecore projects using Commerce Server I wanted to add a new Product Definition (to see how that’s done, go to my previous blogpost on just that subject).

So after I created a new Product Definition and had the definition to my specifications I updated my data templates in Sitecore and started working on some products and product pages. To work on products I’d use the SPEAK application Merchandising Manager (can be downloaded here for Sitecore 7.5 or here for Sitecore 7.2), which is where I ran into an issue. Opening a product based on my new product definition an exception was thrown. The error is ‘Sequence contains no matching element’, with the stacktrace showing that a renderer was trying to get a field called ‘ProductId’. Looking at my definition, the ProductId property was indeed missing – and what’s more, it wasn’t possible to add it either as displayed in the screenshow below.

The New Definition window

There is an extremely simple workaround for this: in Sitecore, we can go to our data templates, by default located in /sitecore/templates/Commerce/Catalog Generated. Find the template generated for the product (or category) definition, in our case the MyDefinition template, and add a Single-Line Text field called ProductId.

Added ProductId field

Save that and that’ll take care of the issue. One thing left though. If we were to browse to the Merchandising Manager and open our product now it would load up, but the ProductId would still be blank and there would be an error on the top of the page because of this. Of course, we could go ahead and fill in the ProductId and save it, or we could open the Content Editor, browse to our product and hit Save. This will automatically set the ProductId field we created and will fix the general error we were having. Of course, if you have a lot of products, you might consider creating a script to do set the ProductId for you.

Sitecore Commerce Server – Products and Definitions

I’ve been working with Commerce Server now for a while, trying to enable it on our dev environment. After the installation process (which might require its own post as well), I figured I’d write something about how to get product definitions, products, category definitions and categories from Commerce Server into Sitecore.

Creating definitions

The Product Definitions and Category Definitions (and I’m guessing Property Definitions also, but I haven’t used custom properties yet) work in a similar way. They can be created in the Commerce Server Catalog and Inventory Schema Manager, using the appropriate view in the Views panel on the bottom left in the screenshot below.

CategoryDefinitions

Creating a new definition is reasonably straightforward. Just give the definition a name and optionally a description, and select the properties you’d like that particular definition to use. After this, you’re done! You can now create new products or categories based on that new definition.

Creating products and categories

Creating new categories and products can be done in the Commerce Server Catalog Manager. Right-click where you’d like to add a new category or product, and use the context menu to create a new category or product. Alternatively, you could use the SPEAK app Merchandising Manager, on which I’m sure more blog posts will be coming as well. In the meantime, more on the Merchandising Manager can be found here. When using the Merchandising Manager, you can skip to the section about getting the definitions to Sitecore. When using the Commerce Server Catalog Manager, you can follow the steps below to get create products or categories.

ContextMenu

In this screenshot you can see I’m using the MVC demo store (of which you can download the source on the Sitecore Marketplace or the Sitecore package from SDN if you don’t need the source).

Creating a new category and/ or new product will pop up with a wizard which will run through the various steps (Selecting the definition, entering the properties followed by the custom properties, selecting primary and additional categories and then selecting relations for the category or product). This is all very easy to do.

All done, we have our products and categories in Commerce Server.
Now, how do we get this to go into Sitecore? If the product and category definitions already exist in Sitecore,  products and categories will be synced automatically, although caching might get in your way sometimes.

Syncing products and categories

If they don’t get synced automatically, and the definitions exist, there’s two options:

  1. In the Commerce Server Catalog Manager, in the Tasks pane, click ‘Refresh Site Cache’.
    SiteCache
  1. After logging into Sitecore, you can go to the Content Editor, go to the Sitecore Commerce tab in the ribbon, and click ‘Refresh Sitecore Cache’.
    SitecoreSiteCache

Syncing definitions

To make sure your definitions go into SItecore (they will be used as the data templates), you need to go into the Content Editor and click ‘Update Data Templates’ in the Sitecore Commerce tab in the ribbon.

DataTemplates
Don’t forget to publish – in fact, when there’s new definitions and new products/ categories I’ve had to to a full republish in some cases- and don’t forget to re-index as well.

Sitecore Rocks – SitecoreCop

SitecoreCop is a cool little feature in Sitecore Rocks, which can check a lot of the Sitecore configurations. It can be executed by connecting to a Sitecore instance using Sitecore Explorer, then right-clicking on a Sitecore Item -> SitecoreCop -> Run SitecoreCop.

By default there are already a whole lot of rules available, from checking whether fields contain Lorem Ipsum text to whether the license is about to expire and from whether media items have actual media attached to them to whether the admin password is still the default – and a whole lot more still.

Even so, if one would want to add their own rules to it, that’s also possible but requires a bit more than just the code (which I didn’t find out until @TheRocksGuy gave me some pointers): The assemblyname has to start with Sitecore.Rocks.Server, so for instance Sitecore.Rocks.Server.CustomRules. Also, the assembly needs to reference Sitecore.Rocks.Server.Core and Sitecore.Rocks.Server.Validation.

The class then needs to be marked with the [Validation] attribute, in which you can specify a rule name and the rule category.

namespace Sitecore.Rocks.Server.CustomRules
{
  [Validation("This is a custom rule", "Items")]
  public class SomeCustomRule : ItemValidation
  {
    ...
  }
}

There are multiple existing Validation types to inherit from, such as ItemValidation or RenderingValidation.

You will then need to implement the CanCheck and Check methods. CanCheck will execute to see if the rule can be executed in the context you’re running it in and the Check method has the actual logic.

...
public bool CanCheck(string contextName, Sitecore.Data.Items.Item item)
{
  Assert.ArgumentNotNull(contextName, "contextName");
  return contextName == "Site";
}

public void Check(ValidationWriter output)
{
  // You'd probably want some logic here to make sure it only flags the problem in the right cases.
  output.Write(SeverityLevel.Warning, "Title", "This is the problem", "This is the solution");
}
...

When you then execute SitecoreCop (after making sure your assembly is copied across to the bin folder of your webroot) you might have your problem flagged up:

SitecoreCop custom rule

Adding FxCop rules to Sonarqube

In my posts about custom FxCop rules I mention creating rules for FxCop. In this post I’ll explain how to add the created rules to Sonarqube.

First of all, the assembly needs to be put on the server somewhere. The location doesn’t really matter too much, as long as it’s accessible for Sonarqube. In my case I just dropped it into the default FxCop Rules folder, which is located by default in C:\Program Files (x86)\Microsoft FxCop 10.0\Rules.

After that, Sonarqube needs to be told about the rules as well. The XML is actually fairly similar to the XML file we create for the FxCop rules in the first place.

<?xml version="1.0" encoding="utf-8" ?>
<rules>
    <rule key="Classname">
        <name><![CDATA[Name for the rule]]></name>
        <configKey><![CDATA[Classname@Location\Of\Assembly\Assemblyname.dll]]></configKey>
        <description><![CDATA[Rule description]]></description>
    </rule>
</rules>

So, to take the example of the GetItemUsingID rule, the XML could look something like this:

<?xml version="1.0" encoding="utf-8" ?>
<rules>
    <rule key="GetItemUsingID">
        <name><![CDATA[Enforce calling the GetItem method using an ID]]></name>
        <configKey><![CDATA[GetItemUsingID@$(FxCopDir)\Rules\Sitecore.FxCop.dll]]></configKey>
        <description><!CDATA[Use GUIDS where possible instead of path / name. This will improve performance as well as prevent code breaking if content is renamed or moved in the content tree.]]></description>
    </rule>
</rules>

When the XML has been created it has to be put in the FxCop custom rules field in Sonarqube. You can get to this by going to Sonarqube -> Settings -> Configuration -> .NET FxCop:
Sonarqube configuration settings

Before the rules can be used Sonarqube does have to be restarted. Then we can go into Sonarqube -> Settings -> Quality Profiles. Select the active code profile, and go to the inactive FxCop rules. Select the newly added rules to be active, and we’re all done.

I guess this would also be a good time to mention the source code of the 11 rules I initially created is available on Bitbucket and Sitecore’s marketplace, so feel free to create a fork and add your own rules!

If you just want to use the rules, that’s possible as well, just go to the download page on Bitbucket and download the zipfile. The zipfile contains a dll of the initial release at the moment, although I hope to steadily add new rules to the project.

Sitecore and FxCop – ItemAxes and Database Rules

This is the fourthpost in the Sitecore and FxCop series

I’ve been working on a bunch of easier to implement rules, which I figured I’d run through here. As always, I’m trying to improve the code, either increasing the performance of the code or making it a little more safe to use.

There’s a couple of categories in which I’ve added rules, and I’ll be creating a post per category, as it’ll be a huge post otherwise.

The categories I added rules in are:

ItemAxes & Database

This is one of the most important areas for performance issues. When we are using the ItemAxes to get all ancestors or descendants that can take a very big performance hit. It is often much better and faster to use Sitecore.ContentSearch (granted, this can only be done using Sitecore 7 and up). The same goes for the SelectItems() with Xpath as well.

So the four rules I put in on the ItemAxes are:

  1. Don’t use SelectSingleItem (accessible as Sitecore.Data.Items.ItemAxes.SelectSingleItem as well as Sitecore.Data.Database.SelectSingleItem)
  2. Don’t use SelectItems (accessible as Sitecore.Data.Items.ItemAxes.SelectItems as well as Sitecore.Data.Database.SelectItems)
  3. Don’t use GetAncestors
  4. Don’t use GetDescendants

On this blogpost I saw some stats for Lucene and fast query, which is not really a fair comparison I think because fast query will always go straight to the database, I figured I’d get a bit of data together.

Consider the following code:

var index = Sitecore.ContentSearch.ContentSearchManager.GetIndex("sitecore_master_index")
{
    using (var context = index.CreateSearchContext())
    {
        // ContentSearch
        Stopwatch sw = new Stopwatch();
        sw.Start();
        var indexed = context.GetQueryable<Sitecore.ContentSearch.SearchTypes.SearchResultItem>().Where(item => item.Name.Contains("s"));
        sw.Stop();
        Sitecore.Diagnostics.Log.Info(string.Format("ContentSearch. Elapsed time: {0}. Items found: {1}", sw.Elapsed, indexed.ToList().Count));
    }

    // Fast query
    Stopwatch sw2 = new Stopwatch();
    sw2.Start();
    var fQuery = Sitecore.Configuration.Factory.GetDatabase("master").SelectItems("fast://*[@@name='%s%']");
    sw2.Stop();
    Sitecore.Diagnostics.Log.Info(string.Format("Fast query. Elapsed time: {0}. Items found: {1}", sw2.Elapsed, fQuery.Length), this);

    // Normal query
    Stopwatch sw3 = new Stopwatch();
    sw3.Start();
    var query = Sitecore.Configuration.Factory.GetDatabase("master").SelectItems("//*[contains(@@name, 's')]");
    sw3.Stop();
    Sitecore.Diagnostics.Log.Info(string.Format("Normal query. Elapsed time: {0}. Items found: {1}", sw3.Elapsed, query.Length), this);
}

This gave me the following output in my Sitecore log file:

INFO ContentSearch. Elapsed time: 00:00:00:0178173. Items found: 8723
INFO Fast query. Elapsed time: 00:00:05.2718443. Items found: 5194
INFO Normal query. Elapsed time: 00:00:09.7881149. Items found: 4320

Those are some pretty big differences. These results are for just under 13000 items. Of course, it’s still not completely honest because the normal query does need to call the Contains function, which I don’t know the performance implications for…

Another big plus of using ContentSearch is that Linq is a lot easier to read in more complex queries than Xpath / Fast query.

From a custom rule perspective, those four rules are extremely simple to create. Almost all rules I added to my project so far use a lot of the same code – If the corresponding classes get called, flag them as warnings on the page.

Of course, as with all rules, there’s bound to be some exceptions and actual legitimate good uses of these classes. The rules are only there to point out that it might be a good idea to re-think that.

Sitecore and FxCop – Field Rules

This is the fourthpost in the Sitecore and FxCop series

I’ve been working on a bunch of easier to implement rules, which I figured I’d run through here. As always, I’m trying to improve the code, either increasing the performance of the code or making it a little more safe to use.

There’s a couple of categories in which I’ve added rules, and I’ll be creating a post per category, as it’ll be a huge post otherwise.

The categories I added rules in are:

Fields

I actually added a couple of field rules. In a previous post I was talking about accessing raw field value properly and getting items from the database with IDs rather than a path.

Now, of course it would also make sense to get the field values based not on the fieldname, but the GUID of the particular fields. This of course has the advantage of getting the correct field always, even when the fieldname changes.

I’ll admit this is a rule I had some difficulty with, although it ended up being very easy.

public override void VisitMethodCall(MethodCall methodCall)
{
    base.VisitMethodCall(methodCall);

    var memberBinding = methodCall.Callee as MemberBinding;
    if (memberBinding != null)
    {
        var methodCalled = memberBinding.BoundMember as Method;
        if (methodCalled != null)
        {
            if (methodCalled.FullName == "Sitecore.Collections.FieldCollection.get_Item(System.String)")
            {
                Expression parameter = methodCall.Operands[0];
                CheckValue(methodCall, parameter);
            }
        }
    }
}

In which CheckValue just tries to match the value of the parameter with a GUID, using a Regular Expression. The thing I had difficulties with was that I was accessing a FieldCollection, and could not inspect the actual line which was making the call.

Another rule which I think will be particularly useful is one that flags where people are trying to get the value from a CheckboxField – to check whether it’s been checked or not. The utils that come with Sitecore seem to be a little under-used, and one of the things that’s possible with the utils is exactly this: Get the value of a checkbox.

Consider the following code snippet:

namespace SomeNamespace
{
    public class SomeClass
    {
        public void SomeMethod()
        {
            var item = Sitecore.Context.Database.GetItem("{66EC0398-1E67-4D43-B2EB-ED29E3E8E291}");
            if (item != null)
            {
                CheckboxField field = (CheckboxField)item.Fields["checkbox"];
                if (field != null)
                {
                    var isChecked = field.Checked;
                }
            }
        }

        public void SomeOtherMethod()
        {
            var item = Sitecore.Context.Database.GetItem("{66EC0398-1E67-4D43-B2EB-ED29E3E8E291}");
            if (item != null)
            {
                var isChecked = MainUtil.GetBool(item["checkbox", false);
             }
        }
    }
}

Of course, instead of calling the field with its fieldname we should do that with the ID – but for this example it’s a bit easier to read like this.
We don’t have to first get the checkboxfield, then check whether it even exists followed by finally getting the value, we can just call Sitecore’s MainUtil.GetBool, pass in the value we want to convert to bool and the default value. A lot safer and it’s a couple lines shorter which should appeal to us lazy programmers ;-)

I figured I’d skip the same FxCop rule code this time, as it’s the same as the examples in the other posts. The method we’re checking for this time is Sitecore.Data.Fields.CheckboxField.get_Checked . If that method is indeed called, flag that as a possible problem.