Various Contact Lock Stages in Sitecore xDB

This blog is based on one of the topic covered in my recent presentation in Auckland Sitecore user group where we discussed on Contact lock stages in Sitecore xDB while I presented on xDB and Personalization at Scale.

It was demonstrated for Sitecore 8.2(update 7 instance)

Assuming user has already visited website and the contact data and corresponding Interactions data are already stored in xDB(Collection Database in Mongo DB). So What happens next once same Contact visits website again and how is Contact data passed at different stages and how are the locking and unlocking of Contact data is done at various stages-

Pictorial Representation of the entire Process below-

Stage 1:
Assuming there are no active sessions of the Contact and at this point of time , Contact data is stored in Mongo DB (Collection Database).
Contact data lies in Contact Collection.
User Interactions data lies in Interaction Collection.

Stage 2:
As soon user session starts – Contact data gets locked in collection database and a copy of Contact data is loaded in Shared Session database. So at this Stage Contact data cannot be updated in Collection database.

Stage 3:
On each individual Page request by same user – Contact Data then gets locked in Shared Session database and a copy of Contact Data gets loaded in Tracker object. At this point new user Interaction data is loaded as well in Tracker object. This is the point where Contact data is accessible in the Page Code.

Stage 4:
When the Page request ends – Contact is removed from the Tracker object and at this point now Contact data gets unlocked in Shared Session database.

Stage 5:
And in end of this process – as soon user Session ends – Contact data is removed from Shared Session database and is saved back to Collection database. To be noted: now Contact is unlocked in Collection database (Contact collection in mongo DB).

References : Sitecore documentation and Sugcon EU presentation from Dmytro Shevchenko

Verify Contacts and Page Visits in XDB|Sitecore implementaion

In last couple of weeks I was working to resolve XDB issue in a Sitecore implementation (V8.2).
For one of my analysis I was supposed to verify if all the anonymous contacts and their corresponding page visits are tracked in MongoDB.
After lot of findings and querying against MongoDBcollections came across excellent blog from jonathanrobbins which helped me to figure out the contacts in MongoDB and then I went ahead to query all the corresponding user visits.

Below is the summary of my findings which I would like to document so that its a handy documentation helpful for others and for me too if i want to query MongoDB for similiar verification. 

        1. Sitecore relies on cookies to track anonymous contacts – cookie name:SC_ANALYTICS_GLOBAL_COOKIE
          • Sample sc_analytics value : “43dd8a5d41c347cbb2bb0255da48d783|True”
          • cookie value contains Guid which is ContactId and a bool with a pipe identifier.
          • bool value is set by Sitecore as – IsClassificationGuessed (true or false)
          • for a new visitor its set to false(as can be human or robot). When visitor classification is determined , its set to true and now contact information can be pushed to MongoDB.
        2. For this blog purpose I used my sitecore 8.2 demo site- and visited below pages.
          Page VisitedDate Time
          http://sitecore8/08-06-2019| 22:28
          http://sitecore8/mypage108-06-2019| 22:29
          http://sitecore8/mypage208-06-2019|22:30
          http://sitecore8/mypage308-06-2019|22:31

          Cookie value for SC_ANALYTICS_GLOBAL_COOKIE:959d3c1789754dbcb9ff9757172af6df|True

        3. My intention is to capture behavior as same user returns to demo site later on and visits below Pages.(Technically speaking same user comes back to site in new browser session)
          Page VisitedDate Time
          http://sitecore8/08-06-2019| 22:34 
          http://sitecore8/mypage208-06-2019| 22:37 
          http://sitecore8/mypage308-06-2019| 22:37 
          http://sitecore8/mypage608-06-2019| 22:37 

          Cookie value for SC_ANALYTICS_GLOBAL_COOKIE:959d3c1789754dbcb9ff9757172af6df|True

          • As you see from Date time in # 2 and #3-  I have immediately visited the site before the default session would have cleared. To mimic session clear-  I have one custom admin Page added which just kills the current browser session with a line of C# code.  Session.Abandon();
          • Feel free to grab sessionkill.aspx file and place it in your web-root folder – (Website\sitecore\admin\sessionkill.aspx)
          • I was able to kill current session using- http://sitecore8/sitecore/admin/sessionkill.aspx 
        4. Next I need to verify if my contact and corresponding page visits in both sessions were saved in Mongo DB.
            •  

              • I have used Robo 3T to connect to my mongo DB
                https://kumarsaurabhrajpoot.files.wordpress.com/2019/06/robo-3t-1.3-2019-06-08-23.03.43.png
              • Contact is saved inside Contacts collection. 
              • Now we need to have mongoDB query to fetch data from Contacts Collection and verify it for the above anonymous user.
                • Contact Id from sc_analytics needs to be encoded to be used in the mongoDB query.
                • Grab SC_ANALYTICS_GLOBAL_COOKIE value and get the query using C# snippet available at dotnetfiddle.netcontactRoboQuery
              • Grab the query from output window and execute it against Contacts Collection. Robomongodb.getCollection('Contacts').find({_id:new BinData(3, 'FzydlXWJvE25/5dXFyr23w==')})
                contactCollection
                Contact Collection

                So we can now confirm that our anonymous Contact has been saved in Contact Collection which we have verified based on sc_analytics_global_cookie value.

              • All the user Page visits are saved to Interactions collection. Grab the ContactId from above output and place it in below snippet(just replace the orange text: #Your ContactId based on above output) to generate query for retrieval of user interactions. db.getCollection('Interactions').find({"ContactId" : LUUID("173c9d95-7589-bc4d-b9ff-9757172af6df")})
                Execute Query:
                interactions
                Interaction Collection

                Above output confirms both sessions has been tracked in interaction Collection.

              • Expand each interaction record in the above output window to verify if the corresponding Pages were tracked.
                interactions
                Interaction Collection:Record 1

                Verified:All the Pages user visited in #2 – can be verified from this output.


                interactions
                Interaction Collection:Record 2

                Verified:All the Pages user visited in #3 – can be verified from this output.

So, in this blog I tried penning down my experience of figuring out the corresponding anonymous contact and their corresponding Page visits in Mongo DB.

Happy Sitecoring !!

Install Sitecore 9 on Dev Box | A Step by Step Guide.

I know I am quite late. – “Better Late than Never” – This is what I said myself when I started exploring Sitecore 9 since last 1 week.
Had some glitches as well during my installation with initial version (Sitecore Experience Platform 9.0).

Many Thanks to below blogs/tutorials which helped me in my way, as and when I was stuck at various steps.

Prerequisites

  • Windows 8.1/10.
  • .Net framewrok 4.6.2 or later.
  • SQL Server 2016  SP1
  • IIS 8.5 or 10
  • JRE(Java run time environment) is required for running SOLR.
  • Microsoft Powershell 5.1 +
  • Web Platform Installer 5.0
  • Web Administration Module with (IIS)- This should be installed by default with IIS 8.5 or 10.
  • Web deploy 3.6 for Hosting Servers. – via Web Platform Installer.
  • URL Rewrite 2+ . – Via Web Platform Installer.
  • Microsoft SQL Server Data – Tier Application Framework (17.1) – Via Web Platform Installer.
  • Microsoft SQL Server Transact-SQL ScriptDom Library- Via Web Platform Installer.

Let us follow below steps for installing Sitecore 9.0 in our local development Environment.

  1. Install SOLR 6.6.2 as Windows service. Earlier blog can be referred .
  2. Configure SSL for SOLR. Earlier blog can be referred .
  3. Setup an installation folder which will contain all required files which we will need to get sitecore 9 working with SIF(Sitecore Install Framework). In my case – I have created – D:\InstallSitecore9
  4. For SC 9.0 to be installed on local developer instance – We need On Prem XP0 instance packages. -> Sitecore 9.0.0 rev. 171002 (WDP XP0 packages).zip
    Get it from – dev.sitecore.net
  5. Extract Sitecore 9.0.0 rev. 171002 (WDP XP0 packages).zip – which gives you 3 files.
    • Sitecore 9.0.0 rev. 171002 (OnPrem)_single.scwdp (.zip)
    • Sitecore 9.0.0 rev. 171002 (OnPrem)_xp0xconnect.scwdp (.zip)
    • XP0 Configuration files rev.171002 (.zip)
  6. Extract -XP0 Configuration files rev.171002 file – gives you below files.
    • sitecore-solr
    • sitecore-XP0
    • xconnect-createcert
    • xconnect-solr
    • xconnect-xp0
  7. Copy below files from the above extractions to the setup folder created in above step 3.
    • Sitecore 9.0.0 rev. 171002 (OnPrem)_single.scwdp.zip
    • Sitecore 9.0.0 rev. 171002 (OnPrem)_xp0xconnect.scwdp.zip
    • sitecore-solr.json
    • sitecore-XP0.json
    • xconnect-createcert.json
    • xconnect-solr.json
    • xconnect-xp0.json
  8. Grab the sitecore license file(license.xml) and place it in your setup folder as well.
  9. This is how the setup folder (D:\InstallSitecore9) should look like now-Blog4_Image1
  10. Setup SQL Server User – In past for all Sitecore installations I have been using sa user credential. In this step I will be setting up a standard user for all our Sitecore installations(having a ‘sysadmin’ role).
    • Open SSMS (SQL Server Management Studio) and connect to your local database.
    • Go to Security -> Login -> Add new Login
      • Your login name – I have given – “sclogin” for me.
      • Set your password.
      • Un-check Enforce Password policy.Blog4_Image3
      • From Server Roles – Give it a “sysadmin” server role and Click OkBlog4_Image4
    • Reconnect with newly created user – “sclogin” to verify if user is created successfully.
  11. Enable Contained Database Authentication- script is taken from Sitecore installation guide.
    • Launch MS SQL Server Management Studio and log in as an administrator
    • Run the following new query: – Can grab .sql file.
      sp_configure 'contained database authentication', 1;
      GO
      RECONFIGURE;
      GO
      Blog4_Image5
  12. Now its SIF time  : “Sitecore Install Framework”
    • Open PS prompt as an administrator.
    • Register Sitecore repositories.
      Register-PSRepository -Name sitecoreRepo -SourceLocation https://sitecore.myget.org/F/sc-powershell/api/v2
      Blog4_Image6
    • Install SIF Module
      Install-Module SitecoreInstallFramework
      Blog4_Image7
    • Install Sitecore Fundamentals Module.
      Install-Module SitecoreFundamentals
      Blog4_Image8
  13. At last – Now we are ready to run SC 9 installation script.
    • I have taken the script from SC9 installation guide and placed it in my setup folder – installsc9.ps1 . You can grab script I used from here: installsc9.ps1
    • Update all of the parameters accordingly in power shell script file- installsc9.ps1

      #define parameters
      $prefix = "learnsc9" ==> name for your sitecore instance
      $PSScriptRoot = "D:\InstallSitecore9" => setup folder which you have created in Step 3
      $XConnectCollectionService = "$prefix.xconnect"
      $sitecoreSiteName = "$prefix.sc"
      $SolrUrl = "https://localhost:8983/solr" => SOLR url - which I have setup in Step 1 and 2.
      $SolrRoot = "D:\SOLR\solr-6.6.2" => SOLR root folder which I have setup in Step 1.
      $SolrService = "solr662" => SOLR service name which I have setup in Step 1.
      $SqlServer = "LAPTOP-FT2VPMF2\SQLDEV" => Connection to my local database.
      $SqlAdminUser = "sclogin" => new login user which was created as part of Step 10
      $SqlAdminPassword="passwordforsclogin" => password which was setup as part of Step 10
    • Open Windows Power shell ISE as an administrator
      • Lets import both modules(SitecoreInstallFramework and SitecoreFundamentals) into our current session which we have installed in Step 12.Blog4_Image9
      • Browse for script file – installsc9.ps1  from the setup folder – D:\InstallSitecore9 (in my case)
      • Execute PS Script and Fingers Crossed.
      • Take a break – and you should be having your installation complete in next few minutes.Blog4_Image10
    • Verify SC 9 installation by opening up the instances.
    • Happy Me 🙂 – SC9 instance is up and running nowBlog4_Image11
    • Post Installation Steps. – Make sure the perform all the post installation steps mentioned in SC installation guide- Chapter 6.
      • Rebuild the Search Indexes
        • From Sitecore Launchpad, click Control Panel =>Indexing section => click Indexing manager.
        • Indexing Manager dialog box => click Select all => click RebuildBlog4_Image12
      • Build the Link Databases
        • Sitecore Launchpad=> Control Panel => Database section => click Rebuild Link Databases.
        • Select the Master and Core databases and then click Rebuild.Blog4_Image13
      • Add a Recognized User to the xDB Shard Databases
        • Query can be grabbed from SC installation guide
        • We need to update DatabasePrefix as per our installation.
        • I have updated script it as per the current instance we have installed.:SETVAR DatabasePrefix learnsc9
        • Execute the Query in SSMS.Blog4_Image14
      • Deploy Marketing Definitions
        • Sitecore Launchpad => Control Panel => Analytics => click Deploy Marketing Definitions.
        • select all the definitions and taxonomies and click Deploy.Blog4_Image15
  14. Wow !!!- We have successfully installed – up and running SC 9.0 instance in local developer box.

Happy Sitecoring !!

Secure SOLR Service for Sitecore 9 | Configure SSL for SOLR.

This blog is my experience of taking me one step closer to Sitecore 9 installation.  Its an extension of my previous post of setting up SOLR as Windows Service . After setting up SOLR as windows service I verified SOLR admin page with URL -http://localhost:8983/solr/

Purpose of this blog is to make it https => an extra s , that’s it!!! 🙂

In SC 9 SOLR by default needs to be setup in secure way. As this is for my local dev box I am going ahead with self signed certificates. Credit to Kam for providing us with a handy Power shell Script for generating self signed certificate and installing it .

Follow the below Steps for securing your SOLR Service in your local dev box.

  1. Go to Kams webiste – and get the original PS script.(solrssl.ps1) . Many Thanks to Kam Figy for it.
  2. Script makes use of Java keytool which is a certificate manager.
  3. Figure out your  keyttol.exe path from java directory in your PC. I have mine at  C:\Program Files\Java\jre1.8.0_181\bin\keytool.exe Blog3_Image1
  4. Open the script(solrssl.ps1) in an editor – and go to Line # 37
    $keytool = (Get-Command 'keytool.exe').Source
  5. Update Line #37 as below depending on your keytool.exe path figured out in previous step
    $keytool = (Get-Command 'C:\Program Files\Java\jre1.8.0_181\bin\keytool.exe').Source
  6. Check for updated PS File : solrssl.ps1
  7. Open Power shell as administrator and navigate to folder where you have kept PS file – solrssl.ps1 (from previous step)Blog3_Image2
  8. Pass in location for the keystore file.
    .\solrssl.ps1 -KeystoreFile D:\SOLR\solr-6.6.2\server\etc\solr-ssl.keystore.jks
  9. Execute PS and your keys are generated.Blog3_Image3
  10. In case of Error- ” “File [.ps1] cannot be loaded because the execution of scripts is disabled on this system.”  – Execute below line on Power shell window and re-run Step 8
    Set-ExecutionPolicy -Scope CurrentUser Unrestricted
  11. Verify if keystore files are created in location which we provided . In my case – i checked at path – D:\SOLR\solr-6.6.2\server\etcBlog3_Image4
  12. Based on Next Steps from PS output of Step 9 – Open your solr.in.cmd file which should be inside bin folder of your SOLR installation folder. In my case its in – D:\SOLR\solr-6.6.2\bin\
  13. Search for below 4 lines in solr.in.cmd file , un-comment all 4 lines and save the changes.
    set SOLR_SSL_KEY_STORE=etc/solr-ssl.keystore.jks
    set SOLR_SSL_KEY_STORE_PASSWORD=secret
    set SOLR_SSL_KEY_STORE_TYPE=JKS
    set SOLR_SSL_TRUST_STORE=etc/solr-ssl.keystore.jks
  14. Restart your SOLR Service from Service Manager.
  15. Check SOLR admin page – http URL- http://localhost:8983/solr/   – Yes this Page should not work now.
  16. Add an extra “s” – and open it securely – https://localhost:8983/solr/Blog3_Image7

With this I am done securing my SOLR Url and excited – as now I am ready to install Sitecore 9.

Happy Sitecoring !!

Install SOLR and run as Windows Service

This blog will focus on setting up SOLR and running it as Windows Service using Non Sucking Service Manager tool. I have tried to list down each steps I did during my installation of SOLR 6.6.2 in my windows 10 dev machine.

Install SOLR

  1. Grab solr-6.6.2.zip from the SOLR archivesImage14
  2. Unzip it to any common location. I have placed it inside my directory    D:\SOLR\solr-6.6.2
    SOLRFolderStructure
  3. Verify if SOLR is working.
    • Open Command Prompt as admin and navigate to bin folder of Solr 6.6.2  As per above directory structure it will D:\SOLR\solr-6.6.2\bin
    • Start SOLR on port 8983.
      Image2
    • Open the SOLR admin page to verify – http://localhost:8983/solr/Imag4
    • Awesome!! SOLR is up and running on port 8983.
    • After verification Stop SOLR from Command Prompt window.Image7

Setup SOLR as Windows Service 

We will be using NSSM tool for setting up SOLR as windows service. NSSM is Non Sucking Service Manager tool.

  1. Grab NSSM from https://nssm.cc/download
  2. Download the Featured pre-release :  nssm-2.24-101-g897c7ad.zip (which is said to have better compatibility with Windows 10)
  3. Unzip the content in a common folder structure. I have placed it in same directory where I have placed my SOLR files. I have renamed folder to nssm(just to keep it simple – it can be left as is)Image6
  4. Open Command prompt as admin and navigate to – D:\SOLR\nssm\win64 (Folder structure may vary for you if you have placed files in different location)
  5. Run the nsmm install commandimage8
  6. Update – application tab of nssm service installer pop-up.Image 9
    • Path – D:\SOLR\solr-6.6.2\bin\solr.cmd (Point to solr.cmd from in your installed SOLR 6.6.2\bin folder.
    • Startup directory- D:\SOLR\solr-6.6.2\bin (Point it to your SOLR 6.6.2\bin folder)
    • Arguments – start -f -p 8983 (port on which SOLR will run)
    • Service Name – solr662 (Provide the name for your SOLR service)
  7. Update – details tab of nssm service installer pop-up.Image10
    • Startup – Automatic
    • Service Name – solr662
  8. Click Install Service and you should be getting confirmation as below.Image11
  9. Open Services manage to verify (Run -> services.msc)Image13
  10. Start the Service for the first time. – Yay!!Now SOLR is running as Windows Service in your machine.
  11. Verify by going to local host SOLR admin URL – http://localhost:8983/solr/#/

With this I am done with my SOLR 6.6.2 instance up and running in my Windows 10 dev box !!

Happy Sitecoring !!