Regarding Integrating AEM Cloud Manager and Jenkins – Diagrams & How-To’s

If you are considering moving your Adobe Experience Manager-powered site to AEM as a Cloud Service or Adobe’s AEM Managed Services (AMS) the only way to get code onto these systems is via Adobe’s Cloud Manager CI/CD toolset. As you may know from experience, or from other writings on the topic, Cloud Manager is meant to be a one-size-fits-many solution rather than a fully-customizable CI/CD platform like Jenkins or Bamboo. For small companies or for projects without robust development & QA teams, Cloud Manager’s feature set can be acceptable – as it does indeed take the code out of a git repo and put it on a server and performs some basic testing & validation on the code as well.

The larger and more-complicated your site is, however, the more your team will run into the acute limitations of the Cloud Manager product. The purpose of this post is to illustrate some of these limitations, and illustrate an approach that one can take to vastly extend what Cloud Manager can do.

Regarding Scoping your AEM Cloud Migration Project

Let’s get the tough news out of the way first: you’re very likely going to need to scope some more DevOps and CI/CD work into your AEM Cloud Migration plan than you originally thought. Purpose of my post here is to try to get your wheels turning on what that scope might need to be.

Let’s say that you have a 3-environment setup like many of us do (Dev / Stage / Prod). First of all, if you have more envs than that and plan to move to AEM as a Cloud Service, you’ll need to contract that just to Dev/Stage/Prod as that is all that is currently supported by the service, but Adobe AEM Managed Services (AMS) does support more environments than that.

Migrating a 3-environment setup into Cloud Manager gives you a pipeline which generally would look like this (explanation follows):

Diagram of Manual Steps in OOTB AEM Cloud Manager Release Process

• 

As noted before, Cloud Manager was never intended to be a Jenkins or a Bamboo or a CircleCI. It was meant to be a simple, easy-to-use, straightforward CI solution that would just work, and where all of the heavy lifting for configuration is done behind the scenes by Adobe employees. As a result there are a few very key areas that Cloud Manager lacks:

• Repository Syncing: As you may have read in prior articles on AEM as a Cloud Service, the only way to get code in to Cloud Manager is from an Adobe Git repo. There’s no OOTB automation to automate moving the code from your own repo into Adobe Git, and it’s very much not recommended to use Adobe Git as your source of truth for code, as it has no SLA around its availability. So, unless you want every dev deploy to also include a manual sync of your Github/Bitbucket to Adobe git, you’ll need an outside process for this.
• Adding Conditional Build Steps: There is no way to add steps into the build process in Cloud Manager either before or after the build fires. If there is a step that needs doing, this needs to be done outside of Cloud Manager. This means everything from smoke testing, load testing, UAT approvals, external system manipulation, reindexing, etc – that would all have to be done OUTSIDE of Cloud Manager.
• Dispatcher Configuration Linting & Testing: In Cloud Manager, dispatcher (caching config, apache config, rewrites, filters, etc) code is deployed along with the application code. However, very little testing is done on that code on its way in, so the only way you find out that there is something wrong with the Dispatcher code is when it fails and has to revert to the previous code. It presently gives you no UI feedback aside from “FAILED” as well as an email from your support engineer.
• Slack/Teams or Email Notifications: Currently, the system only supports email notifications in the Stage / Prod deployments, and nothing in the Dev or other non-prod deployments. This can be rough for dev teams, as while the deployment is going on, the system will go through waves of unavailability which can be rough for users who aren’t in the know. Slack/Teams notifications in Cloud Manager are not supported out of the box

Flipping the Script: Having Jenkins Run Cloud Manager

If your project would benefit by automating the manual touchpoints of an out-of-the-box Cloud Manager setup, you’ll really need to jump feet first into flipping things around and not using Cloud Manager as your central source of truth for CI/CD, but instead to use a standalone tool like Jenkins which itself will reach out & talk to Cloud Manager where needed.

Here is a diagram of what a more-ideal workflow would look like, centering around Jenkins as the main source of truth on the CI pipeline, rather than Cloud Manager:

(Please zoom in, there’s a bunch of detail on there)

Basically though, in this case, the big difference is that you are telling Jenkins to make Cloud Manager deploy things, and have Cloud Manager notify you back using Adobe IO, rather than the reverse of trying to start from Cloud Manager, and then figure out how to insert conditional build steps into it (which you can’t do anyhow).

The Pipeline illustrated above would be:

1. Develop Locally, Commit to Local Git: Developer creates locally, and then commits into the corporate git repo (i.e. Bitbucket).
2. Jenkins Dev Deployment: Jenkins jobs then:
   1. Sync the Dev branch of the git repo to the dev branch of Adobe CM git
   2. Make an Adobe IO call out to kick off the Dev Deployment Pipeline in Cloud Manager. Adobe IO is then configured to get notifications on completion events in Cloud Manager.
   3. Jenkins then gets the result of that Cloud Manager build and passes it on to a Notification job which can send webhooks out to Slack or Microsoft Teams.
   4. On a successful build, jobs can then kick off to run an automated smoke test or UI Functional Test in a framework like Saucelabs or Selenium Grid, or even just run some very basic smoke testing scripts.
3. Sign-Off: Ideally, you would then pipeline your Jenkins jobs to the next environment – with either a Jira integration or a manual trigger so that the Stage Pipeline can’t get kicked off until (a) a successful build from Dev completes, and (b) it gets appropriate sign-off.
4. Sync to Release Branch: Then you kick off a Jenkins job that syncs the approved state of the Dev branch into the release branch (obviously this could be rejiggered based on your preferred branching strategy). This would sync from your Bitbucket (or Github) Dev into Bitbucket Release, and then Bitbucket Release into Cloud Manager release branch.
5. Staging Release: When you’ve got a release candidate, you then trigger the “Stage/Prod” pipeline in Jenkins which again reaches out via Adobe IO to trigger a Cloud Manager “production” build, because Cloud Manager’s Stage and Prod builds are a part of the same pipeline, and are the only truly “pipelined” builds in CM. The Cloud Manager build then goes through its normal steps of creating a build, running tests, creating a build artifact and deploying that to Staging, and then doing its own internal set of security tests and load tests. Whatever its result is, it notifies Adobe IO which then notifies Jenkins.
6. Regression / Load and other Automated tests: This is where you would plug into Jenkins any further automated load tests, regression tests, UI Functional Tests, or other security scans and / or jobs that need doing after the Stage deployment is done.
7. Once these get the green light, you can hit the button to notify to the CSE to deploy to prod.

How do we do all this?

Thankfully a lot of the real heavy lifting on this has been done already by some great folks at Perficient and Cognifide among others.

I would fully recommend going through this how-to by Wojciech Blachowski who’s put together a solid step-by-step on making the bare bones of this work. Just make sure to first have your organization’s Adobe account provisioned for Firefly and Adobe IO Runtime, or else the glue that sticks it all together won’t work. Example code for making this work is on github here: GitHub – wblachowski/cm-jenkins-helpers: A set of auxiliary sources for CM+Jenkins solution

But a lot of the hard work on this comes before implementation, and will be the result of some quality time in front of a whiteboard figuring out what your ideal Dev/Ops pipeline for AEM really is.

Hit me up if you want to chat about it!