Improving build times with Gradle build scans

Waiting for code to compile is something that Java developers experience daily.

Waiting for a full build to finish before continuing on your next task can take even longer time. There must be a better way to increase consumption of computing power and reduce waiting times. Let me show you one way to make this happen with a concrete example.

The Sentinel project by Alibaba is a very popular project in China, self described as

A lightweight powerful flow control component enabling reliability and monitoring for microservices

In particular this project defines capabilities that are time sensitive (sync and async) that are verified in tests, which means handling of time is quite important. Often times this means ensuring that a particular condition has been met before continuing evaluation of tests, or as some would have it, invoking Thread.sleep() for a fixed period of time. Sentinel is built with Maven. The build files are pretty clean (though some files contain duplicate dependency definitions and verbose plugin configurations). Executing the build with mvn clean package at commit 83f6de9 yields the following picture

The build takes 3:15 mins to complete. The execution summary shows that the sentinel-core module takes the longest time (at 2:26 mins). This build was executed sequentially. Let's see what happens when we instruct Maven to run a Thread per CPU core (I'm running the build on a MacBook Pro with 2.6 GHz Intel Core i7 => 4 cores) by invoking mvn -T 1C clean package

The result is 2:49 mins without changing a single line of code. We can appreciate that sentinel-core continues to be the dominant factor in build times. Well that's a bit better, however we can do more. The next step was to migrate the build from Maven to Gradle to figure out if there are any gains by switching. The details of this migration will be the subject of another post as there are some points you should keep in mind however if you're curious you can find the changes here. An equivalent build was executed by invoking the following command: ./gradlew test jacocoTestReport jar --scan. As it turns out the Maven build had configured JaCoCo for code coverage; the --scan flag enables the build scans feature which allow us to drill down on all aspects of the build, making it easier to find the places that require optimization. The full build scan is available at https://gradle.com/s/uqhajjlrris6s

Total build time is 3:07 mins (3:05 mins is only for execution). This is just 7 seconds better when compared with the sequential Maven build. Next I turned on the parallel flag by setting org.gradle.parallel=true on my local settings (~/.gradle/gradle.properties) and measured again with the same command as before (build scan available at https://gradle.com/s/y7cknonftf2jc).

The build went down to 2:32 mins, which is 17 secs better than the parallel Maven build. But were not done yet. Looking at the top tasks we can see that tests are the major contributors to build times, particularly sentinel-core. We knew already that this module was a hotspot but it wasn't clear what part was taking the most time, whether it was compiling the code, processing resources, or executing tests. Now we know for sure and can continue the investigation. Performing a grep for "sleep" on test code yields the following files as candidates for optimization

sentinel-extension/sentinel-datasource-redis/src/test/java/com/alibaba/csp/sentinel/datasource/redis/StandaloneRedisDataSourceTest.java
sentinel-extension/sentinel-datasource-redis/src/test/java/com/alibaba/csp/sentinel/datasource/redis/SentinelModeRedisDataSourceTest.java
sentinel-extension/sentinel-datasource-zookeeper/src/test/java/com/alibaba/csp/sentinel/datasource/zookeeper/ZookeeperDataSourceTest.java
sentinel-core/src/test/java/com/alibaba/csp/sentinel/AsyncEntryIntegrationTest.java
sentinel-core/src/test/java/com/alibaba/csp/sentinel/slots/logger/EagleEyeLogUtilTest.java
sentinel-core/src/test/java/com/alibaba/csp/sentinel/slots/statistic/base/LeapArrayTest.java
sentinel-core/src/test/java/com/alibaba/csp/sentinel/slots/block/degrade/DegradeTest.java
sentinel-core/src/test/java/com/alibaba/csp/sentinel/slots/block/flow/controller/WarmUpControllerTest.java
sentinel-core/src/test/java/com/alibaba/csp/sentinel/slots/block/flow/FlowPartialIntegrationTest.java
sentinel-core/src/test/java/com/alibaba/csp/sentinel/base/metric/MetricsLeapArrayTest.java
sentinel-cluster/sentinel-cluster-server-default/src/test/java/com/alibaba/csp/sentinel/cluster/ClusterFlowTestUtil.java
sentinel-cluster/sentinel-cluster-server-default/src/test/java/com/alibaba/csp/sentinel/cluster/flow/ClusterFlowCheckerTest.java
sentinel-dashboard/src/test/java/com/alibaba/csp/sentinel/dashboard/discovery/AppInfoTest.java

As it turns out some of those tests had a fixed pauses to allow certain conditions to be met before continuing. The problem with using fixed pauses is that those conditions could be resolve much earlier but the tests will wait the full specified time regardless. Fortunately we can expect the Java ecosystem to provide us with a solution: Awaitility


Updating the test code with this new library and running the Maven build once more yields the following result

Impressive, 1:06 mins, we saved 1:43 mins by updating test code. The Gradle build also saw an increase in build speed (build scan available at https://gradle.com/s/hlccweqnxjwl2)

With a grand total of 57 secs. That's 1:35 mins better than its previous measure and 9 secs better than its equivalent Maven build. I should mention that during all Gradle measurements the Gradle daemon was deactivated, to make a fair comparison with Maven. What happens when we activate the daemon and also enable the Build Cache feature? (build scan available at https://gradle.com/s/3dpiznc5ntwg2)

Hmm build time goes up to 1:04 mins. Understandable as now Gradle is performing extra work to check the local build cache and store results. The actual gains of using the Gradle daemon in combination with the Build Cache are visible the next time we issue a build. If we were to run the build as it currently stands then the result would be almost immediate as we haven't changed a single source file and most task outputs have been cached. Updating a single dependency version should do the trick, from example pushing nacos from 0.6.2 to 0.8.0 (build scan available at https://gradle.com/s/hbd3hz7qrd3oe)

7 secs is all it took to make a full build with a simple change. Yes, you read that right, just 7 secs for a full build. We can appreciate that the sentinel-datasource-nacos module was recompiled as well as some other modules that depend on it, however all other tasks remained unchanged. Let's go further and make an actual change to a test (build scan available at https://gradle.com/s/onq5wj4bsfedo)

It took the build 5 secs to execute the updated tests found in the sentinel-transport-netty-http module. That's pretty fast. Sentinel, like many Open Source projects, has automated checks when a Pull Request is submitted, such as checking for a signed CLA, code coverage numbers, and if the build succeeds or not. Running a build on every commit push and/or PR takes time; and if we can reduce those times then we reduce the waiting period for a PR to be reviewed/approved as well as the energy consumption at the data center where the builds are run. The Build Cache feature can take advantage of multiple nodes providing build results, allowing full builds to be faster. One more thing I'd like to show is the time spent when making a build that's intended for publication. The Maven build generates additional artifacts (-javadoc.jar and -sources.jar) when the following command is invoked: mvn -T 1C -Poss javadoc:jar source:jar package

Resulting in a total of 1:29 mins. That's still pretty decent, then again the equivalent Gradle build invoking the following command: ./gradlew build (build scan available at https://gradle.com/s/3eipi7wnv36ss)

Takes about 1:23 mins. Hmm just 6 secs better? As it turns out the full Gradle build performs more work than the Maven build. If you have a look at the build scan you'll see additional tasks that have run, not just for creating additional javadoc and source JARs but also tasks that

  • check license headers
  • generate an aggregate JaCoCo report
  • configure additional capabilities such as
    • calculate source statistics
    • generate prettified source reports with Java2Html

These capabilities are added to the build thanks to the kordamp-gradle-plugins. Have a look at the guide to learn more about the features provided by these plugins.

If you think build scans are a good feature to have but for some reason you're unable to switch from Maven to Gradle, no worries, Gradle is bringing this feature to the Maven world via its Gradle Enterprise product. You can see a recording of a recent screencast that explains how this feature works in combination with Maven.

Keep on coding!

Liked it? Take a second to support aalmiray on Patreon!
Become a patron at Patreon!

1 comment

Trackbacks/Pingbacks

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

ˆ Back To Top