The Continuous Integration Legend

The Continuous Integration Legend

The importance of achieving continuous quality visibility during Continuous Integration is growing these days. Release cycles are getting more frequent, there is much more to test in order to meet today’s coverage metrics (Did anyone said “DIGITAL”?) and the number of test cases is also growing (service logic/mechanism is built to scale and requires us to test more components).

Walking the Agile path, we are required to calibrate our efforts to stabilize everything during a sprint cycle. In simple words: we need to decrease risk levels during a shorter SDLC towards release.

Do Something Meaningful With Your Quality Insights

One of the main challenges organizations are facing today is around the need to fit the Quality Feedback Loop Cycle into this high speed race. “Fail Fast – Fix Fast” is not a cliché anymore but a desired state for teams that are looking to find bugs early enough in the process in order to increase efficiency and ensure optimized User Experience to their users. The equation is very simple: If you run tests just for the sake of running tests – you’re wasting time and money. You need to do something meaningful with the test results that you get (and also talented people worked hard to build sustainable automation tests…). Many teams merge code that contains defects in order to keep up the pace of continuous integration: “we run specific builds multiple times a day when we get closer to release. We cannot analyze thousands of tests in such a short time frame. So even if some builds fail, code is still being pushed to Master branches (and hopefully/usually things will be found in Regression cycles later on…)”.

Visibility is a Big Expression – What You Really Need is Actionable Data

Visibility is all about taking “Go/No-Go” decisions based on an educated basis – Quality insights. A senior DevOps Manager that I’ve met recently told me that in many cases he feels that his teams are shifting left but bugs are shifting right…”.

The long stabilization phase, known from Waterfall practices, is being replaced in Continuous Integration. with a managed process. “Continuous” means that we keep asking ourselves “is this thing going on the right direction?” literally with every new piece of code (Just like using building blocks for a house – Each brick is relaying on the previous one). If things are breaking from some reason, we can quickly take a decision and revert to the last point they were stable and take it from there.  So while we keep building hopefully a solid code structure, we need to have the right visibility in motion at each and every decision point.

Green, Green, Green, RED!…….. Trends Mean Something in Continuous Integration – Listen to Them!

So, first I’d like to know as a manager whether my latest build passed or failed. At that point, I would usually compare the build status to the previous executions (Remember? We’re talking about a continuous process – step-by-step). But wait a second, how can I know that? Do I have to review all my builds one-by-one and manually track their history? This is not scalable.

The common script structure will include a “try and catch” mechanism that helps us to highlight exceptions very easily. The same concept should be adopted here: if you were doing something good (writing stable code) all day/week long, but skipping your afternoon coffee caused some focus issues (and potentially some bugs in your code) – you should either drink more coffee, go home earlier (and know your productivity limits) or more seriously get back to last stable position of your build and take things back from there.

Stabilizing your builds on the green side (“Pass”) should be always on top priority. In order to do that, you also need to know what is failing your tests and isolate that.

This is easily said, but tracking this per multiple builds (that, in some cases, run a dozen times per day) on multiple digital platform requires some computing power.

If you have spare time or unlimited budget – you’d probably build your own dashboards. But, if you still read these lines, maybe the result isn’t good enough or the process is hard to maintain. The good news is that help is on its way. The less-good news are there few more challenges.

My Build Failed – But Why?

Once your build failed, you were fortunate enough to get a Slack notification regarding that and you are now officially a member of the early defects identifiers club.

As you’re still required to run as fast as you can and continuously speed-up – you need to find the needle in needlestack. Let’s say that your build is running a responsive web design test case over 7 different Digital platforms – can you specify QUICKLY enough what has failed your build? Is it a flaky test? Maybe an unstable environment (backend) that couldn’t be reached? A new browser version that is not fully covered by your service? Or maybe your iOS object locators weren’t defined properly?

At this point, we need to differentiate between recurring or standard Continuous Integration iterations like smoke tests or short regression cycles to builds that are testing new functionality during development (mainly unit tests). Some test cannot suffer the Red color; in case of a failure, you either fix the bug immediately or skip/comment out the problematic test in case it is not important enough (as the continuous SUCCESS of critical builds is above any new functionality).

The clear objective when a failure occurs in Continuous Integration is to eliminate the False Negatives ASAP:

Among the common actions you can use a retry mechanism on your FW, check Device-state related issues including Network, App status and device recovery. 

As a last resort, redundant devices can be also used in order to eliminate the false negative which is related to the device itself.

Continuous Integration Release Cadences are Solid-Rock – How Can I Assure I’ll Deliver on Time?

Yeah, yeah, I get it. You release on a quarterly basis, can you be more predictable than that? My answer is – YES.

Let’s say that you do release once a month and everything is known upfront, but what happens if a major bug is found in your service and you need to release a patch to production? How long will it take you to run a full regression suite?

Can you state the exact duration of a smoke test running across your entire Digital lab?

By sizing the major quality process and tests that are running on your digital lab, you can actually plan an optimized SDLC. You can be more accurate about deliveries (“are we about to release on time or not?”) and plan dedicated slots to test new functionality (instead of doing that on free-space basis).

Finally, by making a deep dive a part of a manageable process, maybe you’ll find out that your regression cycle can be shorten (as the process is much more effective). This will create a room for additional development & testing of new functionality.

Perfecto’s DigitalZoom Reporting introduces a new Continous Integration dashboard for Optimized Quality Management of your DevOps. All the above-mentioned challenges are addressed easily without any hassle; the Dashboard provides an opportunity to have real-time tracking of Build’s Quality status and Trends, and perfectly fits into any automation at scale.

Continuous Integration users can view aggregation of test results status identified by Job Name/Build Number as generated by their Continuous Integration (CI) tool.

The Dashboard does not require any special implementation and only requires the identifiers for Job Name and Number (Build).


The Continuous INtegration (CI) Jobs Dashboard


Examining a test day on the Continuous Integration (CI) Dashboard


Examining test results on the Continuous Integration (CI) Dashboard

Tzvika Shahaf is a Director of Product Management at Perfecto, focusing on the quality intelligence and reporting and analytics capabilities. He previously worked in business development and product leader roles at venture capital and startup firms.

Leave A Comment