unsafe memory access in AWS Lambda since 3 days

[Mortinke]

We get Unhandled Exceptions in our AWS Lambda functions for 3 days (2019-07-24). This exception occurs with version 1.0.7 and 1.0.11. We have not touched the Lambda function in the last months, so we suspect that AWS has changed something in its runtime environment. Unfortunately, with these changes, the library will no longer work.

Environment: we use java8 lambda environment.

a fault occurred in a recent unsafe memory access operation in compiled Java code: java.lang.InternalError
java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code
at java.nio.DirectByteBuffer.getLong(DirectByteBuffer.java:766)
at java.nio.DirectByteBuffer.getLong(DirectByteBuffer.java:772)
at io.logz.sender.com.bluejeans.common.bigqueue.BigArray.initArrayIndex(BigArray.java:243)
at io.logz.sender.com.bluejeans.common.bigqueue.BigArray.commonInit(BigArray.java:157)
at io.logz.sender.com.bluejeans.common.bigqueue.BigArray.<init>(BigArray.java:141)
at io.logz.sender.com.bluejeans.common.bigqueue.BigQueue.<init>(BigQueue.java:83)
at io.logz.sender.com.bluejeans.common.bigqueue.BigQueue.<init>(BigQueue.java:67)
at io.logz.sender.DiskQueue.<init>(DiskQueue.java:32)
at io.logz.sender.DiskQueue.<init>(DiskQueue.java:10)
at io.logz.sender.DiskQueue$Builder.build(DiskQueue.java:154)
at io.logz.sender.LogzioSender$Builder.getLogsQueue(LogzioSender.java:306)
at io.logz.sender.LogzioSender$Builder.build(LogzioSender.java:298)
at io.logz.log4j2.LogzioAppender.start(LogzioAppender.java:347)
at org.apache.logging.log4j.core.config.AbstractConfiguration.start(AbstractConfiguration.java:265)
at org.apache.logging.log4j.core.LoggerContext.setConfiguration(LoggerContext.java:545)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:617)
at org.apache.logging.log4j.core.LoggerContext.reconfigure(LoggerContext.java:634)
at org.apache.logging.log4j.core.LoggerContext.start(LoggerContext.java:229)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:153)
at org.apache.logging.log4j.core.impl.Log4jContextFactory.getContext(Log4jContextFactory.java:45)
at org.apache.logging.log4j.LogManager.getContext(LogManager.java:194)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getContext(AbstractLoggerAdapter.java:122)
at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:43)
at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:46)
at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:29)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:284)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:304)
at com.mercateo.image.processing.lambda.ImageProcessingLambda.<clinit>(ImageProcessingLambda.java:43)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)

[idohalevi]

@Mortinke Can you provide the appender configuration file? Our log4j appender, by default, uses a persistence queue and I'm not sure it will be the best approach for serverless applications

[Mortinke]

@idohalevi: sorry for the delay, I was on vacation the last days.

Here you can find our log4j2.xml:

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN"
	packages="com.amazonaws.services.lambda.runtime.log4j2.LambdaAppender,io.logz.log4j2.LogzioAppender"
	shutdownHook="disable">
	<Appenders>
		<Lambda name="Lambda">
			<PatternLayout>
				<pattern><![CDATA[%d{yyyy-MM-dd HH:mm:ss} <%X{AWSRequestId}> %-5p %c{1}:%L - %m%n]]></pattern>
			</PatternLayout>
		</Lambda>
		<LogzioAppender name="Logzio">
			<logzioToken>${env:LOGZIO_TOKEN}</logzioToken>
			<logzioType>Java</logzioType>
			<logzioUrl>https://listener-eu.logz.io:8071</logzioUrl>
		</LogzioAppender>
	</Appenders>
	<Loggers>
		<Logger name="com.mercateo" level="debug" additivity="false">
			<AppenderRef ref="Lambda" />
			<AppenderRef ref="Logzio" />
		</Logger>
		<Root level="info">
			<AppenderRef ref="Lambda" />
			<AppenderRef ref="Logzio" />
		</Root>
	</Loggers>
</Configuration>

[idohalevi]

@Mortinke Can you use the in-memory option of this appender? The persistence queue is redundant in a serverless environment and it will discard the usage of the bigqueue library that I believe causes the issue