Stress test the AEM capability using Tough day 2

What is 'Tough day 2'?
In real scenarios we need to stress test the AEM to find out its limits. Tough day 2 is a tool provided by Adobe to stress test AEM instances.
It has default test suite or can be customized as per needs to run various stress tests in AEM.

How do we get the Tough Day 2 Application?

We can get it from Adobe Repository

How to run the Tough Day 2?

We need to run the Tough Day 2 jar using below command; where localhost is the host where AEM is running.

java -jar toughday2.jar --host=localhost


How to get help?
To enable the Help, run Tough day 2 using below command,
java -jar toughday2.jar --help_full

What are the default test options available for Tough day 2?

Some of the default cases are given below,
1) Getting the Home page
2) Triggering the queries in Query builder
3) Delete the assets from DAM
4) Create pages
5) Create live copies and run roll outs

Can we customize Tough Day 2?
Yes as mentioned previously, we have complete customization option here. We can do it by either providing command line parameters or yaml configurations. Command line has the highest priority, which means, it can override the configuration files or default parameters.
Configuration files has the second priority, which can override the default configurations.
Remember this, configuration files are used for large custom suites to make it easier.

What are all the various run modes available in Tough Day 2?

1) Normal: This generates constant number of threads. It has 2 params
    a) Concurrency
    b) waittime

2) Constant Load:This generates constant number of started test executions.

How do we analyze Test Day 2 Output?
The output of Test Day 2 consists of logs and test metrics(9 of them).

Test Metrics
The test metrics are available in CSV (CSVPublisher), and in console (ConsolePublisher ) form.
 
Logs
The logs are also created under same folder of Tough Day 2 installation.
2 Files are created,
  a) generic(toughday.log)- all generic & global messages
  b) custom(toughday_<testname>.log) - related to our specific test

Do we need load test in AEM Authoring instance?

In usual scenarios, the Author performance has been less of a problem in most cases. But the number of concurrent users and the size of the assets are the biggest factors in sizing the author. The general practice is load-test content authoring workloads only when a) number of content authors is high (>50) AND b) they use AEM ASSETS.

Load test is done using Tools like JMeter(Author/Publish instance in our Jenkins pipeline), LoadRunner etc. Recommendation is always to test with production size content! AEM author will not have less number of pages w.r.t production( For e.g. 10 pages vs 100.000+ pages are a big change to compare)

Summary:
The test suite comprise of 85% read and 15% write. When we install and run Tough Day 2, it installs all the content packages, which can be avoided by passing parameters while load run.
The parameters list can be found here for AEM 6.5

Since Tough Day 2 once run cannot clean up the things from AEM by itself, it is always recommended to run Tough day on a cloned AEM instance.

Storage concepts in AEM 6.5

AEM server holds both content and binary data. For a better deployment isolation and organizing data considering performance and scalability of AEM, the binary data(for eg media files) can be stored independently from the content nodes by configuring it.

Generally binary data is stored in a data store, whereas content nodes are stored in a node store. Both 'data stores' and 'node stores' can be configured by updating persistent identifier (PID) in OSGi configuration.

Say when we are working with large number of binaries, Adobe recommends to use an external data store instead of the default node stores. This provides better performance.

The external data store can be File Data Store, Amazon S3 Data Store, Azure Data Store which works well when we need to store binaries like media files.

Node store can be of two types 'Segment Node' Store & 'Document node' store. Segment node store is the basis of Adobe's TarMK implementation & document node store is the basis of AEM's MongoMK implementation in AEM6x.

Configuration PID's for each storage types are given below.

Document node store
org.apache.jackrabbit.oak.plugins.document.DocumentNodeStoreService.config

Segment Node Store
org.apache.jackrabbit.oak.segment.SegmentNodeStoreService.config


File Data Store
org.apache.jackrabbit.oak.plugins.blob.datastore.FileDataStore.config

Amazon S3 Data Store
org.apache.jackrabbit.oak.plugins.blob.datastore.S3DataStore.config PID

Azure Data Store
org.apache.jackrabbit.oak.plugins.blob.datastore.AzureDataStore.config

Summary: Thus by selecting the recommended configurations based on requirement, we can achieve a high performing AEM instance.

Note: The concepts are same for all AEM 6x versions, except a few minor configuration changes.


More Like This:

AEM 6.5 Site related new features

AEM 6.5 Specific new features in Assets section

New Features in AEM 6.5 Forms

Foundation updates in AEM 6.5 which a developer should be aware of

Cloud Manager for AEM 6.5 New features

Storage concepts in AEM 6.5   

HTML Template language updates which needs to be observed with AEM 6.5

AEM With MongoMK - Options

We can have 2 types of MongoMK configurations for AEM Oak Clusters.

1) Failover for High Availability in a Single Datacenter

Here a primary MongoDB will be used for Read/Write. So the multiple Oak instances accessing a MongoDB replica set within a single data center.
This setup helps for high availability and redundancy in the event of a hardware or network failure

Major advantages are,

• Scalability
• High availability
• redundancy
• automated failover of data layer

One of the disadvantage of this setup is, there are some performance issues when compared with TarMK.

2) Failover Across Multiple Datacenters

Here multiple Data Centers are involved. Primary MongoDB will be replicated to secondary systems. So multiple Oak instances accessing a MongoDB replica set across multiple data centers. In this configuration, MongoDB replication provides the same high availability and redundancy as comapared to previous configuration but it also includes the ability to handle a data center outage.

Major advantages are,

• Scalability
• High availability
• Redundancy
• Automated failover of data layer

Final notes:
Always remember this. Adobe highly recommends TarMK as the default persistence technology for both the AEM Author and Publish instances, except in the use cases outlined here.

Foot notes:
What is MongoDB Arbiter. Why we need an Arbiter?

In software the CAP theorem says, "If there are equal number of servers on either side of the partition, the database cannot maintain CAP (Consistency, Availability, and Partition tolerance). An Arbiter is specifically designed to create an -imbalance- or majority on one side so that a primary can be elected in this case."

As per MongoDB site(https://docs.mongodb.com/v3.4/tutorial/add-replica-set-arbiter/) "Arbiters are mongod instances that are part of a replica set but do not hold data. Arbiters participate in elections in order to break ties. If a replica set has an even number of members, add an arbiter. Arbiters have minimal resource requirements and do not require dedicated hardware. You can deploy an arbiter on an application server or a monitoring host."

If you get an even number of nodes on both sides, MongoDB will not elect a primary and your set will not accept writes.

Videos on AEM & Persistence
AEM Persistence TarMk & MongoMK Series 1
AEM Persistence TarMk & MongoMK Series 2  AEM with TarMK
AEM Persistence TarMk & MongoMK Series 3  AEM with MongoMK
AEM Persistence TarMk & MongoMK Series 4  TarMK Vs MongoMK
Read More
Rule Engine integration with AEM

Subscribe Our YouTube Channel Here.

AEM Microkernels - MongoMK vs TarMK

Latest AEM comes with Oak storage. Oak(Apache Jackrabbit Product) is a new JCR implementation with a completely new internal architecture.
There are two options for the AEM persistence layer back-end used by Oak : TarMK and MongoMK. Let us see the scenarios where each one is feasible and what are all the main differences.


TarMK Vs MongoMK

Adobe recommendation says we must consider MongoMK when we come across below metrics

• Number of named users connected in a day: in the thousands or more.
• Number of concurrent users: in the hundreds or more.
• Volume of asset ingestion per day: in hundreds of thousands or more.
• Volume of page edits per day: in hundreds of thousands or more (including automated updates via Multi Site Manager or news feed ingestion for example).
• Volume of searches per day: in tens of thousands or more.

Prerequisites for considering MongoMK as persistence layer.

Once we identified MongoMK as persistence layer below factors are mandatory.
1. Ensure involving MongoDB architects/ Adobe MongoDB Specialists for designing the solution.
2. AEM Developers should work hand in hand with MongoDB developers to implement a successful solution.
3. For a better turn around time, always ensure to have Mongo DB Maintenance License.
4. Ensure the proposed design is validated by an Adobe Certified Experience Manager Architect to avoid any future issues.

Advantages of MongoMK in Authroing Instances

Horizontal Scalability Support – MongoMK supports multiple AEM instances share the same MongoDB instance.
Efficient Data Replication – MongoMK effectively delegates replication functionality to the MongoDB, which has mature model to maintain replica sets. Major advantage here is MongoDB replicas and AEM instances are independent each other.
Distributed Authoring Team Support – Content authoring across different geographical regions are supported by Mongo MK. At first data gets saved int the primary MongoDB and then gets replicated to the secondary DB replica sets.
Automated Failover Recovery – MongoDB supports automated failover. The efficient configuration can helps in automated system recovery from data center failure.

Advantages of TarMK in Publish Instances
Automated Failover Recovery - Efficient configuration of TarMK provides Automated recovery during system failures.
24/7 availability: Code changes are released on one data center, validate the release and then release it to the other data center. TarMK supports such rolling release efficiently.
Efficent upgrades: AEM Upgrades and patch fixes needs rolling release, thus TarMK provides better support.

Huge Data considerations: Which one to use?
If most of the Data in (TB) are binaries, TarMK is a good fit, and a data store on the file system. If the data (in multiple TB) is structured content / nodes, MongoMK is the better choice because we can shard the data.

Sharding can be found at MongoDB Documentation : Overview / Adobe Experience Manager 6.0 / Deploying and Maintaining / or The MongoDB Manual.

Final Notes:
TarMK is optimized for single node performance whereas MongoMK is designed for scalability and clustered deployments. Another factor is the volume of the data. Also keep in mind MongoMK setup/configuration/maintenance requires additional knowledge about MongoDB.

In my next blog I will be talking about MongoMK Recommended Deployments(AEM on MongoDB )
Videos on AEM & Persistence
AEM Persistence TarMk & MongoMK Series 1
AEM Persistence TarMk & MongoMK Series 2  AEM with TarMK
AEM Persistence TarMk & MongoMK Series 3  AEM with MongoMK
AEM Persistence TarMk & MongoMK Series 4  TarMK Vs MongoMK
Read More
Rule Engine integration with AEM

Subscribe Our YouTube Channel Here.