Navigation

    数据用户治理组

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups

    写入 StarRocks 时发生 close index channel 异常怎么办?

    官方FAQ
    starrocks
    1
    1
    1402
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • CloudCanal-万少
      CloudCanal-万少 last edited by junyu-cloudcanal

      简述

      写入 StarRocks 时发生 close index channel 异常,任务日志堆栈堆栈如下。

      Caused by: java.lang.RuntimeException: Failed to flush data to StarRocks, Error response:
      {“Status”:“Fail”,“BeginTxnTimeMs”:0,“Message”:“close index channel failed,load_id=8742fa5d-3ab6-7fe5-6615-35c50e90d488”,“NumberUnselectedRows”:0,“CommitAndPublishTimeMs”:0,“Label”:“ffbeed8c-01c6-46df-a4be-c1e4742c6fb3”,“LoadBytes”:20871,“StreamLoadPutTimeMs”:1,“NumberTotalRows”:0,“WriteDataTimeMs”:3,“TxnId”:1540574,“LoadTimeMs”:4,“ReadDataTimeMs”:0,“NumberLoadedRows”:0,“NumberFilteredRows”:0} {}

      这个问题,一般是由于导入数据过于频繁,StarRocks 来不及 compaction导致, 可以通过以下步骤解决。

      步骤

      • 修改 StarRocks be.conf 配置文件中参数, 提升 compaction 能力
         cumulative_compaction_num_threads_per_disk = 4
         base_compaction_num_threads_per_disk = 2
         cumulative_compaction_check_interval_seconds = 2
        
      • 全量阶段报错
        • 调大 CloudCanal 参数 fullBatchSize、fullRingBufferSize,需要注意过大可能导致迁移进程 OOM
        • 调大 CloudCanal 参数 fullBatchWaitTimeMs ,以增加全量写入停顿时间, 推荐 1~10 秒
      • 增量阶段报错
        • 调大 CloudCanal 参数 increBatchSize、increRingBufferSize,同样需要注意过大可能导致同步进程 OOM
        • 调大 CloudCanal 参数 increBatchWaitTimeMs ,以增加增量写入停顿时间,推荐 1~10 秒
      1 Reply Last reply Reply Quote 0
      • 1 / 1
      • First post
        Last post
      Copyright © 2020 ClouGence, Inc.备案号:浙ICP备20007605号-2