Skip to content

example/rdma_performance性能测试偶发报错bad_weak_ptr,导致core dumped #3288

Open

Description

@daming6

Describe the bug
AgentCombiner类继承自enable_shared_from_this,内部会有一个weak_ptr成员,这个成员是在对象被第一个std::shared_ptr管理时才会被初始化,相当于必须要有make_shared出来的shared_ptr管理才能正常使用weak_ptr成员,所以在类似偶发的Reducer对象/IntRecorder对象/Percentile对象 销毁后或初始化前(没有shared_ptr管理weak_ptr成员)再调用该类的operator<<函数录入样本数据, operator<<函数调用_combiner->get_or_create_tls_agent(),_combiner->get_or_create_tls_agent()再调用this->shared_from_this()就会报错bad_weak_ptr
所以长时间重复执行下述命令就会偶发报错bad_weak_ptr
taskset -c 0-95 ./kpl_tools server -use_rdma 0 -thread_num 96
taskset -c 48-95 ./kpl_tools client -thread_num 48 -queue_depth 85 -attachment_size 131072 -use_rdma 0 -connection_type pooled

terminate called after throwing an instance if 'std::bad_weak_ptr'
what(): bad_weak_ptr
Aborted (core dumped)

To Reproduce
在两个shell窗口分别执行如下命令:
taskset -c 0-95 ./kpl_tools server -use_rdma 0 -thread_num 96

taskset -c 48-95 ./kpl_tools client -thread_num 48 -queue_depth 85 -attachment_size 131072 -use_rdma 0 -connection_type pooled

Expected behavior
预期长时间重复执行上述命令不会偶发报错bad_weak_ptr

Versions
OS: openEuler 24.03 (LTS-SP2)
Compiler: gcc 12.3.1
brpc: 1.16
protobuf: protobuf-25.1-12.oe2403sp2.aarch64

Additional context/screenshots

Activity

  1. chenBright commented on May 10, 2026

    @chenBright
    Contributor

    Reducer对象/IntRecorder对象/Percentile对象 销毁后或初始化前(没有shared_ptr管理weak_ptr成员)再调用该类的operator<<函数录入样本数据

    对象销毁后或初始化前调用该类的operator<<,这个操作本身就有问题吧。

  2. chenBright commented on May 10, 2026

    @chenBright
    Contributor

    会不会是rdma_performance bvar全局变量生命周期的问题?能用 gdb 看看堆栈吗?

  3. daming6 commented on May 11, 2026

    @daming6
    Author

    会不会是rdma_performance bvar全局变量生命周期的问题?能用 gdb 看看堆栈吗?

    这个偶发问题复现概率有点低,目前手动gdb能看不core dumped时调用make_shared()代码中有3处,具体gdb打断点调试如下:
    (gdb) c
    Continuing.
    [New Thread 0x4000037cfdc0 (LWP 2118472)]

    Thread 1 "client" hit Breakpoint 1.5, std::make_shared<bvar::detail::AgentCombiner<bvar::Stat, unsigned long, bvar::IntRecorder::AddToStat>>() () at /usr/include/c++/12/bits/shared_ptr.h:1010
    1010 std::forward<_Args>(__args)...);
    (gdb) bt
    #0 std::make_shared<bvar::detail::AgentCombiner<bvar::Stat, unsigned long, bvar::IntRecorder::AddToStat>>() () at /usr/include/c++/12/bits/shared_ptr.h:1010
    #1 bvar::IntRecorder::IntRecorder (this=0x431090 <g_latency_recorder>) at ./src/bvar/recorder.h:127
    #2 bvar::detail::LatencyRecorderBase::LatencyRecorderBase (this=this@entry=0x431090 <g_latency_recorder>, window_size=window_size@entry=-1) at src/bvar/latency_recorder.cpp:170
    #3 0x0000000000414c5c in bvar::LatencyRecorder::LatencyRecorder (this=0x431090 <g_latency_recorder>, prefix=...) at ../../output/include/bvar/latency_recorder.h:80
    #4 0x000000000040a490 in __static_initialization_and_destruction_0 (__initialize_p=1, __priority=65535) at client.cpp:50
    #5 _GLOBAL__sub_I_client.cpp(void) () at client.cpp:312
    #6 0x0000400002f27828 in __libc_start_main () from /usr/lib64/libc.so.6
    #7 0x000000000040a730 in _start ()
    (gdb) c
    Continuing.

    Thread 1 "client" hit Breakpoint 1.6, std::make_shared<bvar::detail::AgentCombiner<long, long, bvar::detail::MaxTo >, long&, long&, bvar::detail::MaxTo const&> ()
    at /usr/include/c++/12/bits/shared_ptr.h:1010
    1010 std::forward<_Args>(__args)...);
    (gdb) bt
    #0 std::make_shared<bvar::detail::AgentCombiner<long, long, bvar::detail::MaxTo >, long&, long&, bvar::detail::MaxTo const&> () at /usr/include/c++/12/bits/shared_ptr.h:1010
    #1 bvar::Reducer<long, bvar::detail::MaxTo, bvar::detail::VoidOp>::Reducer (inv_op=..., op=..., identity=0, this=0x4310f0 <g_latency_recorder+96>) at ./src/bvar/reducer.h:204
    #2 bvar::Maxer<long, void>::Maxer (default_value=0, this=0x4310f0 <g_latency_recorder+96>) at ./src/bvar/reducer.h:412
    #3 bvar::detail::LatencyRecorderBase::LatencyRecorderBase (this=this@entry=0x431090 <g_latency_recorder>, window_size=window_size@entry=-1) at src/bvar/latency_recorder.cpp:158
    #4 0x0000000000414c5c in bvar::LatencyRecorder::LatencyRecorder (this=0x431090 <g_latency_recorder>, prefix=...) at ../../output/include/bvar/latency_recorder.h:80
    #5 0x000000000040a490 in __static_initialization_and_destruction_0 (__initialize_p=1, __priority=65535) at client.cpp:50
    #6 _GLOBAL__sub_I_client.cpp(void) () at client.cpp:312
    #7 0x0000400002f27828 in __libc_start_main () from /usr/lib64/libc.so.6
    #8 0x000000000040a730 in _start ()
    (gdb) c
    Continuing.

    Thread 1 "client" hit Breakpoint 1.7, std::make_shared<bvar::detail::AgentCombiner<bvar::detail::PercentileSamples<254ul>, bvar::detail::PercentileSamples<30ul>, bvar::detail::detail::AddPercentileSamples>>() ()
    at /usr/include/c++/12/bits/shared_ptr.h:1010
    1010 std::forward<_Args>(__args)...);
    (gdb) bt
    #0 std::make_shared<bvar::detail::AgentCombiner<bvar::detail::PercentileSamples<254ul>, bvar::detail::PercentileSamples<30ul>, bvar::detail::detail::AddPercentileSamples>>() ()
    at /usr/include/c++/12/bits/shared_ptr.h:1010
    #1 bvar::detail::Percentile::Percentile (this=this@entry=0x431140 <g_latency_recorder+176>) at src/bvar/detail/percentile.cpp:89
    #2 0x000040000063d3dc in bvar::detail::LatencyRecorderBase::LatencyRecorderBase (this=this@entry=0x431090 <g_latency_recorder>, window_size=window_size@entry=-1) at src/bvar/latency_recorder.cpp:170
    #3 0x0000000000414c5c in bvar::LatencyRecorder::LatencyRecorder (this=0x431090 <g_latency_recorder>, prefix=...) at ../../output/include/bvar/latency_recorder.h:80
    #4 0x000000000040a490 in __static_initialization_and_destruction_0 (__initialize_p=1, __priority=65535) at client.cpp:50
    #5 _GLOBAL__sub_I_client.cpp(void) () at client.cpp:312
    #6 0x0000400002f27828 in __libc_start_main () from /usr/lib64/libc.so.6
    #7 0x000000000040a730 in _start ()

    然后看代码Reducer对象/IntRecorder对象/Percentile对象对应文件中都有operator<<函数,operator<<函数调用_combiner->get_or_create_tls_agent()
    现在我不知道如何直接证明“Reducer对象/IntRecorder对象/Percentile对象 销毁后或初始化前(没有shared_ptr管理weak_ptr成员)再调用该类的operator<<函数录入样本数据” 这一点猜测,这个猜测是基于weak_ptr的分析

  4. chenBright commented on May 13, 2026

    @chenBright
    Contributor

    得复现后看现场的coredump堆栈

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions