AS4 large file performance issue in 5.5.0

[DraganDordevic]

There is a performance issue when processing large files using AS4. We've been trying to process an incoming file 100 MB big and locally it was taking us 23 minutes on a fairly powerful laptop.

The issue was the security provider which somehow doesn't seem to be a BouncyCastle one as the fix in #84 promised. It looks like the fix in #104 somehow affected it.

Debugging revealed that the bottleneck was in AttachmentContentSignatureTransform at the line number 217.:

while ((numBytes = inputStream.read(buf)) != -1)

To fix it, as discovered by my colleague @javierestevez, we introduced the following code:

@PostConstruct public void init() { Security.insertProviderAt(new BouncyCastleProvider(), 1); }

to the configuration class in the spring boot project. By doing that the processing time went down to 11 seconds locally.

We are using:

• Spring boot 2.7.7
• oxalis-as4 5.5.0
• jdk 11

Worth noting is there is no issue when using AS2.

[aaron-kumar]

Usage of BouncyCastleProvider as the preferred security provider (Security.insertProviderAt(new BouncyCastleProvider(), 1)) was First introduced as part of Oxalis release 4.1.7 with following code:

    // Make sure that BouncyCastle is the preferred security provider
    final Provider[] providers = Security.getProviders();
    if (providers != null && providers.length > 0)
        Security.removeProvider(BouncyCastleProvider.PROVIDER_NAME);
    log.debug("Registering BouncyCastle as preferred Java security provider");
    Security.insertProviderAt(new BouncyCastleProvider(), 1);

It was later removed as part of Oxalis 4.1.8 when some Oxalis users started facing problem because of setting BouncyCastleProvider as the preferred security provider. Refer issue #118 with comment: - "Earlier oxalis-as4 put BouncyCastle as the preferred SecurityProvider, but that introduced several problems for some of our users. Generally it is not recommended to reorder the security provider list."

But if you can set BouncyCastleProvider as the preferred security provider within your integration code without any problem/interference with your other application then please go ahead and set that. I myself don't have visibility as what issue/s was faced by Oxalis users when it was first introduced as part Oxalis release 4.1.7 so can't comment further on this. But we will try to reintroduce this as part of some minor release after Oxalis 6.0.0 major release to understand the impact.

[FrodeBjerkholt]

If you know which algorithm it is, you can use something like this in As4CommonModule.init:

Security.setProperty("jdk.security.provider.preferred", "AES/GCM/NoPadding:BC");

or generally:

Security.setProperty("jdk.security.provider.preferred", "<algorithm>:BC");

[javierestevez]

Hello @FrodeBjerkholt, @aaron-kumar ,

Despite that property being set, when I run the application with the option -Djava.security.debug=provider, I see that the Sun provider is being picked up instead:

Provider: Signature.SHA256withRSA verification algorithm from: SunRsaSign
Provider: MessageDigest.SHA-256 algorithm from: SUN
Provider: MessageDigest.SHA-256 algorithm from: SUN
Provider: MessageDigest.SHA-256 algorithm from: SUN
Provider: Cipher.AES/GCM/NoPadding decryption algorithm from: SunJCE

However, if we insert Bouncy Castle as the first provider, I then see the following:

Provider: Signature.SHA256withRSA verification algorithm from: BC
Provider: MessageDigest.SHA-256 algorithm from: BC
Provider: MessageDigest.SHA-256 algorithm from: BC
Provider: MessageDigest.SHA-256 algorithm from: BC
Provider: Cipher.AES/GCM/NoPadding decryption algorithm from: BC

I am not aware of anything overriding the preferred provider property on our end, and as far as I can tell, the property keeps its value throughout the execution. And in case something was overriding the property, then it wouldn't pick up the Bouncy Castle provider either when we insert it as the first one.

I am a bit clueless at the moment.

[FrodeBjerkholt]

I am not working on the Oxalis project anymore, my note was just a tip for @aaron-kumar. He probably should debug to check why the SUN provider is being used for AES/GCM/NoPadding. I added the line with
Security.setProperty("jdk.security.provider.preferred", "AES/GCM/NoPadding:BC");
some years ago to fix exactly the problem with large files. The SUN provider has a really bad implementation that has O(N^2) for the AES/GCM/NoPadding algorithm.

[javierestevez]

@aaron-kumar

A few more insights. It seems that changing the jdk.security.provider.preferred property at runtime has no effect (at least in my case, using version 11 of both the HotSpot and OpenJDK implementations). If I change it in the $JDK_HOME/conf/security/java.security file, then it does work.

[dladlk]

There can be multiple reasons, why Security.setProperty("jdk.security.provider.preferred", "AES/GCM/NoPadding:BC"); does not help to switch to BC provider in

Oxalis-AS4/src/main/java/network/oxalis/as4/common/As4CommonModule.java

Line 65 in ff21c65

Security.setProperty("jdk.security.provider.preferred", "AES/GCM/NoPadding:BC");

1. this security property was introduced only in Java 9 and is absent in Java 8 (it is absent in "lib\security\java.security", even in latest Java 8 builds (see https://github.com/openjdk/valhalla/blob/9a9add8825a040565051a09010b29b099c2e7d49/jdk/src/share/lib/security/java.security-windows and compare with https://github.com/openjdk/valhalla/blob/78e63c03e2549e120ac1aa4e62a0f7a40dd2b8c8/jdk/src/java.base/share/conf/security/java.security?plain=1#L109 )

2. it is read only once - on first initialization of sun.security.jca.Providers class, as it stores them in private static providerList field: (example of the code from OpenJDK-9):

https://github.com/openjdk/valhalla/blob/801281100cbbec62d41c643ee4d79537a8e8992c/jdk/src/java.base/share/classes/sun/security/jca/Providers.java?plain=1#L50

It is read together with the rest of security.provider.X properties in sun.security.jca.ProviderList.ProviderList() constructor, although it could be different between java versions and build, example of openjdk-9:

https://github.com/openjdk/valhalla/blob/801281100cbbec62d41c643ee4d79537a8e8992c/jdk/src/java.base/share/classes/sun/security/jca/ProviderList.java?plain=1#L201

As a result, setting Security.setProperty AFTER Providers.providerList = ProviderList.fromSecurityProperties(); is invoked, has no effect even on JDK which supports it.

And Providers.providerList can be initialized by a lot of reasons:

1. Loading BouncyCastle signed jars by classloader - as it uses JCA for a signature verification
2. Registration of BC provider - e.g. in As4OutboundModule:
   

   Oxalis-AS4/src/main/java/network/oxalis/as4/outbound/As4OutboundModule.java

   Lines 23 to 25 in ff21c65

   	if (Security.getProvider(BouncyCastleProvider.PROVIDER_NAME) == null) {
   	Security.addProvider(new BouncyCastleProvider());
   	}

3. Just simple TestNG tests initialization (when TestNG XmlTest generates default name and calls UUID.randomUUID()
4. Many other cases...

It means, that setting this property should be done on the earliest phase of the application initialization.

But Guice framework does not guarantee any predictable Modules loading order by design - as Modules are expected to have no side effects, and if so, the order of modules loading is considered not relevant.

So it looks like this setting should be set either at java.security file or on earliest possible startup of each distribution separately or user application.

Here you can see a performance comparison of SunJCE in different Java versions versus BouncyCastle 1.7 at encryption and decryption by Peppol used algorithm http://www.w3.org/2009/xmlenc11#aes128-gcm (AES/GCM/NoPadding) on different payload sizes: https://github.com/dladlk/sunjce-vs-bc-performance/blob/main/README.md

And some conclusions: https://github.com/dladlk/sunjce-vs-bc-performance/blob/main/README.md#some-conclusions

[dladlk]

It looks like Java 21 SunJCE significantly improved performance of decryption by AES/GCM/NoPadding algorithm, so it works almost as fast as BouncyCastle (for 100 MB payload it is 1,8 second vs 1,5 second by BouncyCastle, but Java 20 did it in 11 minutes), and 5 times faster encrypts 100 MB payload (255 ms vs. 1435 ms by BouncyCastle).