When the agent of opentelemetry is executed on jdk21, it will cause the CarrierThread to stop executing.

[origwang]

Describe the bug

Our program makes extensive use of virtual threads, and tomcat has also enabled virtual threads.

When the program runs on jdk21, we find that there are occasional instances of the entire service being unresponsive.

At the very beginning, I suspected that it was some dependent jar packages that caused the virtual thread to be "pinned" (such as synchronized, etc.).
However, upon checking the stack, it was found that it was not due to the "pin" issue:
None of the virtual threads is in the state of PARKED or PINNED;
On the contrary, a large number of virtual threads are in the STARTED state;
There are exactly a number of virtual threads in the RUNNABLE state with Runtime.getRuntime().availableProcessors().

Further, I examined the stack of CarrierThread, and the stack of each CarrierThread stays here:

jdk.internal.misc.CarrierThread @ 0x684b4f1a0 : ForkJoinPool-1-worker-22
  at jdk.internal.misc.Unsafe.park(ZJ)V (Unsafe.java(Native Method))
  at java.util.concurrent.locks.LockSupport.park(Ljava/lang/Object;)V (LockSupport.java:221)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Ljava/util/concurrent/locks/AbstractQueuedSynchronizer$Node;IZZZJ)I (AbstractQueuedSynchronizer.java:754)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(I)V (AbstractQueuedSynchronizer.java:990)
  at java.util.concurrent.locks.ReentrantLock$Sync.lock()V (ReentrantLock.java:153)
  at java.util.concurrent.locks.ReentrantLock.lock()V (ReentrantLock.java:322)
  at java.lang.ref.ReferenceQueue.poll()Ljava/lang/ref/Reference; (ReferenceQueue.java:182)
  at io.opentelemetry.javaagent.shaded.instrumentation.api.internal.cache.weaklockfree.AbstractWeakConcurrentMap.expungeStaleEntries()V (AbstractWeakConcurrentMap.java:235)
  at io.opentelemetry.javaagent.shaded.instrumentation.api.internal.cache.weaklockfree.WeakConcurrentMap$WithInlinedExpunction.getIfPresent(Ljava/lang/Object;)Ljava/lang/Object; (WeakConcurrentMap.java:193)
  at io.opentelemetry.javaagent.shaded.instrumentation.api.internal.cache.WeakLockFreeCache.get(Ljava/lang/Object;)Ljava/lang/Object; (WeakLockFreeCache.java:26)
  at io.opentelemetry.javaagent.bootstrap.field.VirtualFieldImpl$java$lang$Runnable$io$opentelemetry$javaagent$bootstrap$executors$PropagatedContext.mapGet(Ljava/lang/Object;)Ljava/lang/Object; (VirtualFieldImplementationsGenerator.java:297)
  at io.opentelemetry.javaagent.bootstrap.field.VirtualFieldImpl$java$lang$Runnable$io$opentelemetry$javaagent$bootstrap$executors$PropagatedContext.realGet(Ljava/lang/Object;)Ljava/lang/Object; (VirtualFieldImplementationsGenerator.java)
  at io.opentelemetry.javaagent.bootstrap.field.VirtualFieldImpl$java$lang$Runnable$io$opentelemetry$javaagent$bootstrap$executors$PropagatedContext.get(Ljava/lang/Object;)Ljava/lang/Object; (VirtualFieldImplementationsGenerator.java:279)
  at io.opentelemetry.javaagent.bootstrap.executors.ExecutorAdviceHelper.attachContextToTask(Lio/opentelemetry/javaagent/shaded/io/opentelemetry/context/Context;Lio/opentelemetry/javaagent/shaded/instrumentation/api/util/VirtualField;Ljava/lang/Object;)Lio/opentelemetry/javaagent/bootstrap/executors/PropagatedContext; (ExecutorAdviceHelper.java:51)
  at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(Ljava/lang/Runnable;JLjava/util/concurrent/TimeUnit;)Ljava/util/concurrent/ScheduledFuture; (ScheduledThreadPoolExecutor.java:556)
  at java.lang.VirtualThread.scheduleUnpark(J)Ljava/util/concurrent/Future; (VirtualThread.java:705)
  at java.lang.VirtualThread.parkNanos(J)V (VirtualThread.java:632)
  at java.lang.VirtualThread.sleepNanos(J)V (VirtualThread.java:807)
  at java.lang.Thread.sleep(J)V (Thread.java:507)

Please note that the last instruction execution, there is at java.util.concurrent.locks.LockSupport.park(Ljava/lang/Object;)V (LockSupport.java:221).

In jdk21, LockSupport.java:221 is a "park" for non-virtual threads：https://github.com/openjdk/jdk/blob/jdk-21%2B35/src/java.base/share/classes/java/util/concurrent/locks/LockSupport.java

Therefore, I suspect that other code was invaded during the scheduling process of the virtual thread task. So the relevant classes in the jvm were printed for analysis.

Through the print in the JVM java.util.concurrent.ScheduledThreadPoolExecutor, I found that the proxy in the schedule virtual thread task,will perform io.opentelemetry.javaagent.bootstrap.executors.ExecutorAdviceHelper.attachContextToTask.

            /*
             * WARNING - void declaration
             */
            @Override
            public ScheduledFuture<?> schedule(Runnable runnable, long l, TimeUnit timeUnit) {
                PropagatedContext propagatedContext;
                Object object;
                Context context;
                try {
/*556*/             context = Java8BytecodeBridge.currentContext();
                    if (ExecutorAdviceHelper.shouldPropagateContext(context, runnable)) {
                        object = VirtualFieldImpl.java.lang.Runnable.io.opentelemetry.javaagent.bootstrap.executors.PropagatedContext.getVirtualField(Runnable.class, PropagatedContext.class);
                        propagatedContext = ExecutorAdviceHelper.attachContextToTask(context, object, runnable);
                    } else {
                        propagatedContext = null;
                    }
                }
                catch (Throwable throwable) {
                    try {
                        ExceptionLogger.logSuppressedError((String)"Failed to handle exception in instrumentation for java.util.concurrent.ScheduledThreadPoolExecutor", (Throwable)throwable);
                    }
                    catch (Throwable throwable2) {
                    }
                    propagatedContext = null;
                }
                context = propagatedContext;
                object = this;
                Runnable runnable2 = runnable;
                long l2 = l;
                TimeUnit timeUnit2 = timeUnit;
                try {
                    void delay;
                    void unit;
                    void command;
                    if (command == null || unit == null) {
                        throw new NullPointerException();
                    }
/*558*/             RunnableScheduledFuture<Object> t = ((ScheduledThreadPoolExecutor)object).decorateTask((Runnable)command, (ScheduledThreadPoolExecutor)object.new ScheduledFutureTask<Object>((Runnable)command, null, ((ScheduledThreadPoolExecutor)object).triggeUnit)unit), sequencer.getAndIncrement()));
/*562*/             ((ScheduledThreadPoolExecutor)object).delayedExecute(t);
/*563*/             object = t;
                    return object;
                }
                finally {
                    runnable2 = null;
                }
            }

However, because virtual thread execute java.lang.VirtualThread.parkNanos. In jdk21, this will be "Sets the current thread to the current carrier thread."

And coincidentally, when the next virtual thread task is submitted in, the attachContextToTask method of the proxy should be executed first. After the attach, when performing the "AbstractWeakConcurrentMap. ExpungeStaleEntries", may be in the "java.lang.ref.ReferenceQueue.poll", Run "LockSupport.park(Ljava/lang/Object;)" again.
Please note that at this time,"Thread.currentCarrierThread() "returns the CarrierThread itself.
So the CarrierThread have entered "jdk.internal.misc.Unsafe.park(ZJ)V".

If all CarrierThreads execute up to here simultaneously, and the wake-up of park requires the execution of logic in other virtual threads. Then, all the virtual threads will freeze.

Steps to reproduce

The following are the code and operation steps for problem reproduction:

1. code:
   AgentTest class:

package com.example;

import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

public class AgentTest {
    public static void main(String[] args) {
        System.out.println("[Main] Starting AgentTest application");

        // Create a virtual thread per task executor
        ExecutorService executor = Executors.newVirtualThreadPerTaskExecutor();

        // Submit tasks that will trigger scheduleUnpark
        Thread.ofVirtual().name("submit-thread").start(() -> {
            for (int i = 0; i < Runtime.getRuntime().availableProcessors() * 2; i++) {
                final int taskNumber = i;
                executor.submit(() -> {
                    System.out.println("[Main] Thread: " + Thread.currentThread() + " - Task " + taskNumber + " is running");
                    
                    try {
                        // This will trigger VirtualThread.scheduleUnpark
                        // which internally uses ScheduledThreadPoolExecutor
                        Thread.sleep(1000);
                    } catch (InterruptedException e) {
                        System.err.println("[Main] Task " + taskNumber + " interrupted");
                        e.printStackTrace();
                    }
                    
                    System.out.println("[Main] Thread: " + Thread.currentThread() + " - Task " + taskNumber + " is done");
                });
            }
        });

        try {
            System.out.println("[Main] Main thread sleeping");
            Thread.sleep(100000000000L);
        } catch (InterruptedException e) {
            System.err.println("[Main] Main thread interrupted");
            e.printStackTrace();
        }
    }
} 

ExecutorAgent:

package com.example;

import java.lang.instrument.Instrumentation;
import java.lang.instrument.ClassFileTransformer;
import java.security.ProtectionDomain;
import javassist.ClassPool;
import javassist.CtClass;
import javassist.CtField;
import javassist.CtMethod;

public class ExecutorAgent {
    public static void premain(String agentArgs, Instrumentation inst) {
        System.out.println("[Agent] Initializing ExecutorAgent");
        
        inst.addTransformer(new ClassFileTransformer() {
            @Override
            public byte[] transform(ClassLoader loader, String className, Class<?> classBeingRedefined,
                                  ProtectionDomain protectionDomain, byte[] classfileBuffer) {
                if ("java/util/concurrent/ScheduledThreadPoolExecutor".equals(className)) {
                    try {
                        System.out.println("[Agent] Transforming ScheduledThreadPoolExecutor class");
                        ClassPool pool = ClassPool.getDefault();
                        CtClass cc = pool.get("java.util.concurrent.ScheduledThreadPoolExecutor");
                        
                        // Add a static counter field
                        CtField counterField = CtField.make("private static final java.util.concurrent.atomic.AtomicInteger SCHEDULE_COUNTER = new java.util.concurrent.atomic.AtomicInteger(0);", cc);
                        cc.addField(counterField);
                        
                        // Modify the schedule method
                        CtMethod scheduleMethod = cc.getDeclaredMethod("schedule");
                        scheduleMethod.insertBefore("java.lang.Thread currentThread = java.lang.Thread.currentThread();" +
                                "int currentTask = SCHEDULE_COUNTER.getAndIncrement();" +
                                "System.out.println(\"[ScheduledThreadPoolExecutor] Task \" + currentTask + \" scheduled by \" + currentThread);" +
                                "if (currentTask > 0) {" +
                                "    System.out.println(\"[ScheduledThreadPoolExecutor] Task \" + currentTask + \" will park\");" +
                                "    java.util.concurrent.locks.LockSupport.park(currentThread);" +
                                "    System.out.println(\"[ScheduledThreadPoolExecutor] Task \" + currentTask + \" unparked\");" +
                                "}");
                        
                        byte[] byteCode = cc.toBytecode();
                        cc.detach();
                        System.out.println("[Agent] Successfully transformed ScheduledThreadPoolExecutor class");
                        return byteCode;
                    } catch (Exception e) {
                        System.err.println("[Agent] Error transforming ScheduledThreadPoolExecutor class");
                        e.printStackTrace();
                    }
                }
                return null;
            }
        });
    }
} 

MANIFEST.MF:

Manifest-Version: 1.0
Premain-Class: com.example.ExecutorAgent

pom:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.example</groupId>
    <artifactId>jvm-agent-demo</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <maven.compiler.source>21</maven.compiler.source>
        <maven.compiler.target>21</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.javassist</groupId>
            <artifactId>javassist</artifactId>
            <version>3.29.2-GA</version>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.11.0</version>
                <configuration>
                    <compilerArgs>
                        <arg>--add-exports=java.base/jdk.internal.misc=ALL-UNNAMED</arg>
                    </compilerArgs>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-assembly-plugin</artifactId>
                <version>3.6.0</version>
                <configuration>
                    <descriptorRefs>
                        <descriptorRef>jar-with-dependencies</descriptorRef>
                    </descriptorRefs>
                    <archive>
                        <manifest>
                            <mainClass>com.example.AgentTest</mainClass>
                            <addClasspath>true</addClasspath>
                        </manifest>
                        <manifestEntries>
                            <Premain-Class>com.example.ExecutorAgent</Premain-Class>
                            <Can-Redefine-Classes>true</Can-Redefine-Classes>
                            <Can-Retransform-Classes>true</Can-Retransform-Classes>
                        </manifestEntries>
                    </archive>
                </configuration>
                <executions>
                    <execution>
                        <id>make-assembly</id>
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project> 

2. operation
   mvn clean package
   and
   java -javaagent:jvm-agent-demo-1.0-SNAPSHOT-jar-with-dependencies.jar -jar jvm-agent-demo-1.0-SNAPSHOT-jar-with-dependencies.jar

3. output

java  -javaagent:target/jvm-agent-demo-1.0-SNAPSHOT-jar-with-dependencies.jar -jar target/jvm-agent-demo-1.0-SNAPSHOT-jar-with-dependencies.jar

[Agent] Initializing ExecutorAgent
[Main] Starting AgentTest application
[Agent] Transforming ScheduledThreadPoolExecutor class
[Agent] Successfully transformed ScheduledThreadPoolExecutor class
[Main] Main thread sleeping
[Main] Thread: VirtualThread[#37]/runnable@ForkJoinPool-1-worker-6 - Task 5 is running
[Main] Thread: VirtualThread[#35]/runnable@ForkJoinPool-1-worker-4 - Task 3 is running
[Main] Thread: VirtualThread[#42]/runnable@ForkJoinPool-1-worker-11 - Task 10 is running
[Main] Thread: VirtualThread[#43]/runnable@ForkJoinPool-1-worker-12 - Task 11 is running
[Main] Thread: VirtualThread[#44]/runnable@ForkJoinPool-1-worker-13 - Task 12 is running
[Main] Thread: VirtualThread[#36]/runnable@ForkJoinPool-1-worker-5 - Task 4 is running
[Main] Thread: VirtualThread[#47]/runnable@ForkJoinPool-1-worker-16 - Task 15 is running
[Main] Thread: VirtualThread[#40]/runnable@ForkJoinPool-1-worker-9 - Task 8 is running
[ScheduledThreadPoolExecutor] Task 7 scheduled by Thread[#32,ForkJoinPool-1-worker-2,5,CarrierThreads]
[ScheduledThreadPoolExecutor] Task 7 will park
[Main] Thread: VirtualThread[#33]/runnable@ForkJoinPool-1-worker-1 - Task 1 is running
[ScheduledThreadPoolExecutor] Task 8 scheduled by Thread[#75,ForkJoinPool-1-worker-14,5,CarrierThreads]
[ScheduledThreadPoolExecutor] Task 8 will park
[Main] Thread: VirtualThread[#39]/runnable@ForkJoinPool-1-worker-8 - Task 7 is running
[Main] Thread: VirtualThread[#34]/runnable@ForkJoinPool-1-worker-3 - Task 2 is running
[ScheduledThreadPoolExecutor] Task 10 scheduled by Thread[#77,ForkJoinPool-1-worker-16,5,CarrierThreads]
[ScheduledThreadPoolExecutor] Task 10 will park
[Main] Thread: VirtualThread[#46]/runnable@ForkJoinPool-1-worker-15 - Task 14 is running
[ScheduledThreadPoolExecutor] Task 11 scheduled by Thread[#73,ForkJoinPool-1-worker-12,5,CarrierThreads]
[ScheduledThreadPoolExecutor] Task 11 will park
[Main] Thread: VirtualThread[#45]/runnable@ForkJoinPool-1-worker-14 - Task 13 is running
[Main] Thread: VirtualThread[#38]/runnable@ForkJoinPool-1-worker-7 - Task 6 is running
[Main] Thread: VirtualThread[#41]/runnable@ForkJoinPool-1-worker-10 - Task 9 is running
[Main] Thread: VirtualThread[#31]/runnable@ForkJoinPool-1-worker-2 - Task 0 is running
[ScheduledThreadPoolExecutor] Task 15 scheduled by Thread[#65,ForkJoinPool-1-worker-4,5,CarrierThreads]
[ScheduledThreadPoolExecutor] Task 15 will park

4. Execute AgentTest alone，output：

[Main] Starting AgentTest application
[Main] Main thread sleeping
[Main] Thread: VirtualThread[#47]/runnable@ForkJoinPool-1-worker-15 - Task 14 is running
[Main] Thread: VirtualThread[#50]/runnable@ForkJoinPool-1-worker-2 - Task 17 is running
[Main] Thread: VirtualThread[#63]/runnable@ForkJoinPool-1-worker-9 - Task 30 is running
[Main] Thread: VirtualThread[#38]/runnable@ForkJoinPool-1-worker-6 - Task 5 is running
[Main] Thread: VirtualThread[#48]/runnable@ForkJoinPool-1-worker-16 - Task 15 is running
[Main] Thread: VirtualThread[#35]/runnable@ForkJoinPool-1-worker-3 - Task 2 is running
[Main] Thread: VirtualThread[#44]/runnable@ForkJoinPool-1-worker-13 - Task 11 is running
[Main] Thread: VirtualThread[#41]/runnable@ForkJoinPool-1-worker-9 - Task 8 is running
[Main] Thread: VirtualThread[#37]/runnable@ForkJoinPool-1-worker-5 - Task 4 is running
[Main] Thread: VirtualThread[#36]/runnable@ForkJoinPool-1-worker-4 - Task 3 is running
[Main] Thread: VirtualThread[#40]/runnable@ForkJoinPool-1-worker-8 - Task 7 is running
[Main] Thread: VirtualThread[#43]/runnable@ForkJoinPool-1-worker-11 - Task 10 is running
[Main] Thread: VirtualThread[#32]/runnable@ForkJoinPool-1-worker-2 - Task 0 is running
[Main] Thread: VirtualThread[#45]/runnable@ForkJoinPool-1-worker-12 - Task 12 is running
[Main] Thread: VirtualThread[#34]/runnable@ForkJoinPool-1-worker-1 - Task 1 is running
[Main] Thread: VirtualThread[#39]/runnable@ForkJoinPool-1-worker-7 - Task 6 is running
[Main] Thread: VirtualThread[#46]/runnable@ForkJoinPool-1-worker-14 - Task 13 is running
[Main] Thread: VirtualThread[#42]/runnable@ForkJoinPool-1-worker-10 - Task 9 is running
[Main] Thread: VirtualThread[#52]/runnable@ForkJoinPool-1-worker-11 - Task 19 is running
[Main] Thread: VirtualThread[#51]/runnable@ForkJoinPool-1-worker-9 - Task 18 is running
[Main] Thread: VirtualThread[#49]/runnable@ForkJoinPool-1-worker-13 - Task 16 is running
[Main] Thread: VirtualThread[#53]/runnable@ForkJoinPool-1-worker-11 - Task 20 is running
[Main] Thread: VirtualThread[#54]/runnable@ForkJoinPool-1-worker-9 - Task 21 is running
[Main] Thread: VirtualThread[#56]/runnable@ForkJoinPool-1-worker-13 - Task 23 is running
[Main] Thread: VirtualThread[#57]/runnable@ForkJoinPool-1-worker-11 - Task 24 is running
[Main] Thread: VirtualThread[#58]/runnable@ForkJoinPool-1-worker-9 - Task 25 is running
[Main] Thread: VirtualThread[#55]/runnable@ForkJoinPool-1-worker-12 - Task 22 is running
[Main] Thread: VirtualThread[#59]/runnable@ForkJoinPool-1-worker-13 - Task 26 is running
[Main] Thread: VirtualThread[#60]/runnable@ForkJoinPool-1-worker-11 - Task 27 is running
[Main] Thread: VirtualThread[#61]/runnable@ForkJoinPool-1-worker-9 - Task 28 is running
[Main] Thread: VirtualThread[#62]/runnable@ForkJoinPool-1-worker-13 - Task 29 is running
[Main] Thread: VirtualThread[#64]/runnable@ForkJoinPool-1-worker-11 - Task 31 is running
[Main] Thread: VirtualThread[#36]/runnable@ForkJoinPool-1-worker-7 - Task 3 is done
[Main] Thread: VirtualThread[#41]/runnable@ForkJoinPool-1-worker-13 - Task 8 is done
[Main] Thread: VirtualThread[#43]/runnable@ForkJoinPool-1-worker-15 - Task 10 is done
[Main] Thread: VirtualThread[#38]/runnable@ForkJoinPool-1-worker-8 - Task 5 is done
[Main] Thread: VirtualThread[#56]/runnable@ForkJoinPool-1-worker-2 - Task 23 is done
[Main] Thread: VirtualThread[#50]/runnable@ForkJoinPool-1-worker-13 - Task 17 is done
[Main] Thread: VirtualThread[#57]/runnable@ForkJoinPool-1-worker-8 - Task 24 is done
[Main] Thread: VirtualThread[#59]/runnable@ForkJoinPool-1-worker-15 - Task 26 is done
[Main] Thread: VirtualThread[#62]/runnable@ForkJoinPool-1-worker-14 - Task 29 is done
[Main] Thread: VirtualThread[#35]/runnable@ForkJoinPool-1-worker-11 - Task 2 is done
[Main] Thread: VirtualThread[#63]/runnable@ForkJoinPool-1-worker-10 - Task 30 is done
[Main] Thread: VirtualThread[#44]/runnable@ForkJoinPool-1-worker-11 - Task 11 is done
[Main] Thread: VirtualThread[#37]/runnable@ForkJoinPool-1-worker-16 - Task 4 is done
[Main] Thread: VirtualThread[#48]/runnable@ForkJoinPool-1-worker-7 - Task 15 is done
[Main] Thread: VirtualThread[#47]/runnable@ForkJoinPool-1-worker-10 - Task 14 is done
[Main] Thread: VirtualThread[#40]/runnable@ForkJoinPool-1-worker-3 - Task 7 is done
[Main] Thread: VirtualThread[#45]/runnable@ForkJoinPool-1-worker-4 - Task 12 is done
[Main] Thread: VirtualThread[#39]/runnable@ForkJoinPool-1-worker-13 - Task 6 is done
[Main] Thread: VirtualThread[#46]/runnable@ForkJoinPool-1-worker-11 - Task 13 is done
[Main] Thread: VirtualThread[#34]/runnable@ForkJoinPool-1-worker-14 - Task 1 is done
[Main] Thread: VirtualThread[#51]/runnable@ForkJoinPool-1-worker-6 - Task 18 is done
[Main] Thread: VirtualThread[#32]/runnable@ForkJoinPool-1-worker-2 - Task 0 is done
[Main] Thread: VirtualThread[#42]/runnable@ForkJoinPool-1-worker-1 - Task 9 is done
[Main] Thread: VirtualThread[#49]/runnable@ForkJoinPool-1-worker-15 - Task 16 is done
[Main] Thread: VirtualThread[#52]/runnable@ForkJoinPool-1-worker-5 - Task 19 is done
[Main] Thread: VirtualThread[#53]/runnable@ForkJoinPool-1-worker-2 - Task 20 is done
[Main] Thread: VirtualThread[#54]/runnable@ForkJoinPool-1-worker-2 - Task 21 is done
[Main] Thread: VirtualThread[#58]/runnable@ForkJoinPool-1-worker-9 - Task 25 is done
[Main] Thread: VirtualThread[#60]/runnable@ForkJoinPool-1-worker-6 - Task 27 is done
[Main] Thread: VirtualThread[#55]/runnable@ForkJoinPool-1-worker-5 - Task 22 is done
[Main] Thread: VirtualThread[#61]/runnable@ForkJoinPool-1-worker-1 - Task 28 is done
[Main] Thread: VirtualThread[#64]/runnable@ForkJoinPool-1-worker-2 - Task 31 is done

Expected behavior

All virtual thread tasks on jdk21 are running normally

Actual behavior

All the virtual threads suddenly stopped executing.
This is the stack information of CarrierThread:

jdk.internal.misc.CarrierThread @ 0x684b4f1a0 : ForkJoinPool-1-worker-22
  at jdk.internal.misc.Unsafe.park(ZJ)V (Unsafe.java(Native Method))
  at java.util.concurrent.locks.LockSupport.park(Ljava/lang/Object;)V (LockSupport.java:221)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(Ljava/util/concurrent/locks/AbstractQueuedSynchronizer$Node;IZZZJ)I (AbstractQueuedSynchronizer.java:754)
  at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(I)V (AbstractQueuedSynchronizer.java:990)
  at java.util.concurrent.locks.ReentrantLock$Sync.lock()V (ReentrantLock.java:153)
  at java.util.concurrent.locks.ReentrantLock.lock()V (ReentrantLock.java:322)
  at java.lang.ref.ReferenceQueue.poll()Ljava/lang/ref/Reference; (ReferenceQueue.java:182)
  at io.opentelemetry.javaagent.shaded.instrumentation.api.internal.cache.weaklockfree.AbstractWeakConcurrentMap.expungeStaleEntries()V (AbstractWeakConcurrentMap.java:235)
  at io.opentelemetry.javaagent.shaded.instrumentation.api.internal.cache.weaklockfree.WeakConcurrentMap$WithInlinedExpunction.getIfPresent(Ljava/lang/Object;)Ljava/lang/Object; (WeakConcurrentMap.java:193)
  at io.opentelemetry.javaagent.shaded.instrumentation.api.internal.cache.WeakLockFreeCache.get(Ljava/lang/Object;)Ljava/lang/Object; (WeakLockFreeCache.java:26)
  at io.opentelemetry.javaagent.bootstrap.field.VirtualFieldImpl$java$lang$Runnable$io$opentelemetry$javaagent$bootstrap$executors$PropagatedContext.mapGet(Ljava/lang/Object;)Ljava/lang/Object; (VirtualFieldImplementationsGenerator.java:297)
  at io.opentelemetry.javaagent.bootstrap.field.VirtualFieldImpl$java$lang$Runnable$io$opentelemetry$javaagent$bootstrap$executors$PropagatedContext.realGet(Ljava/lang/Object;)Ljava/lang/Object; (VirtualFieldImplementationsGenerator.java)
  at io.opentelemetry.javaagent.bootstrap.field.VirtualFieldImpl$java$lang$Runnable$io$opentelemetry$javaagent$bootstrap$executors$PropagatedContext.get(Ljava/lang/Object;)Ljava/lang/Object; (VirtualFieldImplementationsGenerator.java:279)
  at io.opentelemetry.javaagent.bootstrap.executors.ExecutorAdviceHelper.attachContextToTask(Lio/opentelemetry/javaagent/shaded/io/opentelemetry/context/Context;Lio/opentelemetry/javaagent/shaded/instrumentation/api/util/VirtualField;Ljava/lang/Object;)Lio/opentelemetry/javaagent/bootstrap/executors/PropagatedContext; (ExecutorAdviceHelper.java:51)
  at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(Ljava/lang/Runnable;JLjava/util/concurrent/TimeUnit;)Ljava/util/concurrent/ScheduledFuture; (ScheduledThreadPoolExecutor.java:556)
  at java.lang.VirtualThread.scheduleUnpark(J)Ljava/util/concurrent/Future; (VirtualThread.java:705)
  at java.lang.VirtualThread.parkNanos(J)V (VirtualThread.java:632)
  at java.lang.VirtualThread.sleepNanos(J)V (VirtualThread.java:807)
  at java.lang.Thread.sleep(J)V (Thread.java:507)

Javaagent or library instrumentation version

opentelemetry-api: 1.38.0; javaagent: Implementation-Version: 2.13.0

Environment

JDK:
OpenJDK 64-Bit Server VM (21.0.4+7-LTS mixed mode, sharing)
OS:
Ubuntu 24.04 LTS

Additional context

No response

[laurit]

opentelemetry-api: 1.38.0; javaagent: Implementation-Version: 2.13.0

According to the stack trace you must be using a significantly older version of the agent, perhaps 2.0.0 or 2.1.0. I believe this issue should be fixed in a later version, please update your agent.