Rate Limit 방지를 위한 배치 처리 구현 by hye-on · Pull Request #26 · hye-on/Saebom

hye-on · 2025-03-04T12:37:16Z

🔗 관련 이슈

resolves #17

📝 작업 내용

Discord.js가 내부적으로 Rate Limit을 처리하지만, 라이브러리의 내부 동작에 의존하는 것보다 애플리케이션 레벨에서도 대응을 하는 것이 적절하다고 생각했습니다. 대규모 메시지 전송 시 안정성을 보장하기 위해, Batch 단위로 메시지를 전송하며 1초씩 딜레이를 적용하도록 구현했습니다.

MessageBatchSenderService 구현
한 번에 30개 메시지를 보내고, 1초 딜레이를 적용하여 구현 (이벤트 루프를 막지 않도록 구현)
성공 개수 로깅 추가

🍇 고민한 부분

Discord.js의 자동 Rate Limit 처리 시 문제 점 & 대응 방법

Discord.js가 내부적으로(SequentialHandle) 큐를 사용해 Rate Limit(채널별: 초당 5개, 봇 전체: 초당 50개)을 대응하지만, 대량 메시지를 동시에 큐에 넣을 경우 메모리 부하 문제등이 생길 수 있습니다.
애플리케이션 레벨에서 전송 속도를 제어하도록 구현하여, Discord.js의 내부 Rate Limit 처리와 함께 더 안정적인 메시지 전송이 가능하게 하였습니다.
해결 방안으로 BullMQ와 Redis 기반 큐 시스템도 검토했으나, 현재 소규모 사용자(0명..) 임을 고려해 배치 처리를 채택했습니다.
로깅 시스템을 통해 모니터링하며, 사용자 증가 시 Redis 기반 처리로 확장할 계획입니다.

Discord에서 Rate Limit 대응 방식

공식 문서를 보면, 응답 헤더를 통해 애플리케이션이 직접 Rate Limit을 감지하고 대응해야 하는 것 처럼 보일 수 있는데

고수준 API인 discord.js를 사용하고 있어 직접 @discordjs/rest을 이용한 rate limit 처리를 하지 않아도 됩니다.

X-RateLimit-Limit: 5
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1470173023
X-RateLimit-Reset-After: 1
X-RateLimit-Bucket: abcd1234

graph TD
    A[discord.js] -->|의존| B[@discordjs/rest]
    B -->|HTTP 요청 처리| C[Discord API]

Discord.js의 SequentialHandler는 API 요청의 레이트 리밋을 관리하는 핸들러입니다. 요청을 큐에 넣고 Discord API의 레이트 리밋 헤더(위에 나온 )에 따라 처리하고 ,429 응답을 받을 경우 재시도 로직을 구현합니다.

Discord.js Rate Limit 테스트

하지만 실제로 discord.js 문서에 이에 관한 내용이 없어 rate limit이 신뢰하기 위해 간단하게 테스트 해봤습니다.
Discord.js 공식 문서에서는 Rate Limit 처리 방식에 대한 내용이 없어서 Rate Limit 대응이 실제로 동작하는지 확인하기 위해 간단한 테스트를 진행했습니다.

20개와 100개를 같은 채널에 보내보았고, 공식 문서에 명시된대로 약 5개, 약 50개 이상이 되었을때 딜레이가 걸렸습니다.

채널별: 초당 5개, 봇 전체: 초당 50개

20개와 100개 메시지를 동시에 전송 & 타임스탬프를 분석

20개 로그 파일

[15:35:00.123] INFO (53171):
    message: "Starting concurrent rate limit test"
    messageCount: 20
    channelId: "1338425043352420352"
[15:35:00.549] INFO (53171):
    message: "Message 0 sent successfully"
    messageIndex: 0
[15:35:01.015] INFO (53171):
    message: "Message 1 sent successfully"
    messageIndex: 1
[15:35:01.648] INFO (53171):
    message: "Message 2 sent successfully"
    messageIndex: 2
[15:35:02.033] INFO (53171):
    message: "Message 3 sent successfully"
    messageIndex: 3
[15:35:02.391] INFO (53171):
    message: "Message 4 sent successfully"
    messageIndex: 4
[15:35:02.761] INFO (53171):
    message: "Message 5 sent successfully"
    messageIndex: 5
[15:35:03.059] INFO (53171):
    message: "Message 6 sent successfully"
    messageIndex: 6
[15:35:07.950] INFO (53171):
    message: "Message 7 sent successfully"
    messageIndex: 7
[15:35:08.335] INFO (53171):
    message: "Message 8 sent successfully"
    messageIndex: 8
[15:35:08.611] INFO (53171):
    message: "Message 9 sent successfully"
    messageIndex: 9
[15:35:08.897] INFO (53171):
    message: "Message 10 sent successfully"
    messageIndex: 10
[15:35:09.186] INFO (53171):
    message: "Message 11 sent successfully"
    messageIndex: 11
[15:35:09.555] INFO (53171):
    message: "Message 12 sent successfully"
    messageIndex: 12
[15:35:14.446] INFO (53171):
    message: "Message 13 sent successfully"
    messageIndex: 13
[15:35:14.860] INFO (53171):
    message: "Message 14 sent successfully"
    messageIndex: 14
[15:35:15.153] INFO (53171):
    message: "Message 15 sent successfully"
    messageIndex: 15
[15:35:15.469] INFO (53171):
    message: "Message 16 sent successfully"
    messageIndex: 16
[15:35:15.804] INFO (53171):
    message: "Message 17 sent successfully"
    messageIndex: 17
[15:35:16.115] INFO (53171):
    message: "Message 18 sent successfully"
    messageIndex: 18
[15:35:20.923] INFO (53171):
    message: "Message 19 sent successfully"
    messageIndex: 19
[15:35:20.923] INFO (53171):
    message: "Concurrent rate limit test completed"

100개 로그 파일


[15:53:00.083] INFO (54494):
    message: "Starting concurrent rate limit test"
    messageCount: 100
    channelId: "1338425043352420352"
[15:53:00.524] INFO (54494):
    message: "Message 0 sent successfully"
    messageIndex: 0
[15:53:00.858] INFO (54494):
    message: "Message 1 sent successfully"
    messageIndex: 1
[15:53:01.150] INFO (54494):
    message: "Message 2 sent successfully"
    messageIndex: 2
[15:53:01.548] INFO (54494):
    message: "Message 3 sent successfully"
    messageIndex: 3
[15:53:01.852] INFO (54494):
    message: "Message 4 sent successfully"
    messageIndex: 4
[15:53:02.263] INFO (54494):
    message: "Message 5 sent successfully"
    messageIndex: 5
[15:53:06.997] INFO (54494):
    message: "Message 6 sent successfully"
    messageIndex: 6
[15:53:07.383] INFO (54494):
    message: "Message 7 sent successfully"
    messageIndex: 7
[15:53:07.789] INFO (54494):
    message: "Message 8 sent successfully"
    messageIndex: 8
[15:53:08.097] INFO (54494):
    message: "Message 9 sent successfully"
    messageIndex: 9
[15:53:08.408] INFO (54494):
    message: "Message 10 sent successfully"
    messageIndex: 10
[15:53:08.816] INFO (54494):
    message: "Message 11 sent successfully"
    messageIndex: 11
[15:53:13.528] INFO (54494):
    message: "Message 12 sent successfully"
    messageIndex: 12
[15:53:13.975] INFO (54494):
    message: "Message 13 sent successfully"
    messageIndex: 13
[15:53:14.434] INFO (54494):
    message: "Message 14 sent successfully"
    messageIndex: 14
[15:53:14.741] INFO (54494):
    message: "Message 15 sent successfully"
    messageIndex: 15
[15:53:15.092] INFO (54494):
    message: "Message 16 sent successfully"
    messageIndex: 16
[15:53:15.363] INFO (54494):
    message: "Message 17 sent successfully"
    messageIndex: 17
[15:53:20.070] INFO (54494):
    message: "Message 18 sent successfully"
    messageIndex: 18
[15:53:20.403] INFO (54494):
    message: "Message 19 sent successfully"
    messageIndex: 19
[15:53:20.751] INFO (54494):
    message: "Message 20 sent successfully"
    messageIndex: 20
[15:53:21.077] INFO (54494):
    message: "Message 21 sent successfully"
    messageIndex: 21
[15:53:21.408] INFO (54494):
    message: "Message 22 sent successfully"
    messageIndex: 22
[15:53:21.718] INFO (54494):
    message: "Message 23 sent successfully"
    messageIndex: 23
[15:53:26.542] INFO (54494):
    message: "Message 24 sent successfully"
    messageIndex: 24
[15:53:26.871] INFO (54494):
    message: "Message 25 sent successfully"
    messageIndex: 25
[15:53:27.172] INFO (54494):
    message: "Message 26 sent successfully"
    messageIndex: 26
[15:53:27.563] INFO (54494):
    message: "Message 27 sent successfully"
    messageIndex: 27
[15:53:27.980] INFO (54494):
    message: "Message 28 sent successfully"
    messageIndex: 28
[15:53:28.292] INFO (54494):
    message: "Message 29 sent successfully"
    messageIndex: 29
[15:53:32.897] INFO (54494):
    message: "Message 30 sent successfully"
    messageIndex: 30
[15:53:33.275] INFO (54494):
    message: "Message 31 sent successfully"
    messageIndex: 31
[15:53:33.603] INFO (54494):
    message: "Message 32 sent successfully"
    messageIndex: 32
[15:53:33.968] INFO (54494):
    message: "Message 33 sent successfully"
    messageIndex: 33
[15:53:34.313] INFO (54494):
    message: "Message 34 sent successfully"
    messageIndex: 34
[15:53:34.644] INFO (54494):
    message: "Message 35 sent successfully"
    messageIndex: 35
[15:53:39.442] INFO (54494):
    message: "Message 36 sent successfully"
    messageIndex: 36
[15:53:39.841] INFO (54494):
    message: "Message 37 sent successfully"
    messageIndex: 37
[15:53:40.150] INFO (54494):
    message: "Message 38 sent successfully"
    messageIndex: 38
[15:53:40.452] INFO (54494):
    message: "Message 39 sent successfully"
    messageIndex: 39
[15:53:40.773] INFO (54494):
    message: "Message 40 sent successfully"
    messageIndex: 40
[15:53:41.077] INFO (54494):
    message: "Message 41 sent successfully"
    messageIndex: 41
[15:53:45.893] INFO (54494):
    message: "Message 42 sent successfully"
    messageIndex: 42
[15:53:46.404] INFO (54494):
    message: "Message 43 sent successfully"
    messageIndex: 43
[15:53:46.687] INFO (54494):
    message: "Message 44 sent successfully"
    messageIndex: 44
[15:53:46.984] INFO (54494):
    message: "Message 45 sent successfully"
    messageIndex: 45
[15:53:47.301] INFO (54494):
    message: "Message 46 sent successfully"
    messageIndex: 46
[15:53:47.609] INFO (54494):
    message: "Message 47 sent successfully"
    messageIndex: 47
[15:53:52.557] INFO (54494):
    message: "Message 48 sent successfully"
    messageIndex: 48
[15:53:52.873] INFO (54494):
    message: "Message 49 sent successfully"
    messageIndex: 49
[15:53:53.286] INFO (54494):
    message: "Message 50 sent successfully"
    messageIndex: 50
[15:53:53.769] INFO (54494):
    message: "Message 51 sent successfully"
    messageIndex: 51
[15:53:54.061] INFO (54494):
    message: "Message 52 sent successfully"
    messageIndex: 52
[15:53:54.486] INFO (54494):
    message: "Message 53 sent successfully"
    messageIndex: 53
[15:53:58.994] INFO (54494):
    message: "Message 54 sent successfully"
    messageIndex: 54
[15:53:59.397] INFO (54494):
    message: "Message 55 sent successfully"
    messageIndex: 55
[15:53:59.705] INFO (54494):
    message: "Message 56 sent successfully"
    messageIndex: 56
[15:54:00.038] INFO (54494):
    message: "Message 57 sent successfully"
    messageIndex: 57
[15:54:00.319] INFO (54494):
    message: "Message 58 sent successfully"
    messageIndex: 58
[15:54:00.633] INFO (54494):
    message: "Message 59 sent successfully"
    messageIndex: 59
[15:54:05.453] INFO (54494):
    message: "Message 60 sent successfully"
    messageIndex: 60
[15:54:05.852] INFO (54494):
    message: "Message 61 sent successfully"
    messageIndex: 61
[15:54:06.142] INFO (54494):
    message: "Message 62 sent successfully"
    messageIndex: 62
[15:54:06.450] INFO (54494):
    message: "Message 63 sent successfully"
    messageIndex: 63
[15:54:06.757] INFO (54494):
    message: "Message 64 sent successfully"
    messageIndex: 64
[15:54:07.129] INFO (54494):
    message: "Message 65 sent successfully"
    messageIndex: 65
[15:54:11.935] INFO (54494):
    message: "Message 66 sent successfully"
    messageIndex: 66
[15:54:12.253] INFO (54494):
    message: "Message 67 sent successfully"
    messageIndex: 67
[15:54:12.657] INFO (54494):
    message: "Message 68 sent successfully"
    messageIndex: 68
[15:54:13.136] INFO (54494):
    message: "Message 69 sent successfully"
    messageIndex: 69
[15:54:13.535] INFO (54494):
    message: "Message 70 sent successfully"
    messageIndex: 70
[15:54:13.893] INFO (54494):
    message: "Message 71 sent successfully"
    messageIndex: 71
[15:54:18.517] INFO (54494):
    message: "Message 72 sent successfully"
    messageIndex: 72
[15:54:18.886] INFO (54494):
    message: "Message 73 sent successfully"
    messageIndex: 73
[15:54:19.366] INFO (54494):
    message: "Message 74 sent successfully"
    messageIndex: 74
[15:54:19.773] INFO (54494):
    message: "Message 75 sent successfully"
    messageIndex: 75
[15:54:20.091] INFO (54494):
    message: "Message 76 sent successfully"
    messageIndex: 76
[15:54:20.497] INFO (54494):
    message: "Message 77 sent successfully"
    messageIndex: 77
[15:54:25.053] INFO (54494):
    message: "Message 78 sent successfully"
    messageIndex: 78
[15:54:25.411] INFO (54494):
    message: "Message 79 sent successfully"
    messageIndex: 79
[15:54:25.714] INFO (54494):
    message: "Message 80 sent successfully"
    messageIndex: 80
[15:54:26.021] INFO (54494):
    message: "Message 81 sent successfully"
    messageIndex: 81
[15:54:26.320] INFO (54494):
    message: "Message 82 sent successfully"
    messageIndex: 82
[15:54:26.690] INFO (54494):
    message: "Message 83 sent successfully"
    messageIndex: 83
[15:54:31.663] INFO (54494):
    message: "Message 84 sent successfully"
    messageIndex: 84
[15:54:32.065] INFO (54494):
    message: "Message 85 sent successfully"
    messageIndex: 85
[15:54:32.483] INFO (54494):
    message: "Message 86 sent successfully"
    messageIndex: 86
[15:54:32.815] INFO (54494):
    message: "Message 87 sent successfully"
    messageIndex: 87
[15:54:33.162] INFO (54494):
    message: "Message 88 sent successfully"
    messageIndex: 88
[15:54:33.499] INFO (54494):
    message: "Message 89 sent successfully"
    messageIndex: 89
[15:54:38.118] INFO (54494):
    message: "Message 90 sent successfully"
    messageIndex: 90
[15:54:38.591] INFO (54494):
    message: "Message 91 sent successfully"
    messageIndex: 91
[15:54:39.044] INFO (54494):
    message: "Message 92 sent successfully"
    messageIndex: 92
[15:54:39.437] INFO (54494):
    message: "Message 93 sent successfully"
    messageIndex: 93
[15:54:39.746] INFO (54494):
    message: "Message 94 sent successfully"
    messageIndex: 94
[15:54:40.050] INFO (54494):
    message: "Message 95 sent successfully"
    messageIndex: 95
[15:54:40.462] INFO (54494):
    message: "Message 96 sent successfully"
    messageIndex: 96
[15:54:45.552] INFO (54494):
    message: "Message 97 sent successfully"
    messageIndex: 97
[15:54:45.856] INFO (54494):
    message: "Message 98 sent successfully"
    messageIndex: 98
[15:54:46.136] INFO (54494):
    message: "Message 99 sent successfully"
    messageIndex: 99
[15:54:46.136] INFO (54494):
    message: "Concurrent rate limit test completed"

hye-on added 2 commits March 4, 2025 20:36

feat: 대량 메시지 전송을 위한 배치 처리 구현

8f3c0bb

chore: 스케쥴러 시간 수정

382e09f

hye-on self-assigned this Mar 4, 2025

hye-on added the feature New feature or request label Mar 4, 2025

hye-on merged commit c6b7041 into main Mar 4, 2025
1 check passed

hye-on deleted the feat/rate-limit branch March 4, 2025 12:38

hye-on changed the title ~~feat: 대량 메시지 전송을 위한 배치 처리 구현~~ Rate Limit 방지를 위한 배치 처리 구현 Mar 4, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comments

Rate Limit 방지를 위한 배치 처리 구현#26

Rate Limit 방지를 위한 배치 처리 구현#26
hye-on merged 2 commits intomainfrom
feat/rate-limit

hye-on commented Mar 4, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Comments

Conversation

hye-on commented Mar 4, 2025

🔗 관련 이슈

📝 작업 내용

🍇 고민한 부분

Discord.js의 자동 Rate Limit 처리 시 문제 점 & 대응 방법

Discord에서 Rate Limit 대응 방식

Discord.js Rate Limit 테스트

20개와 100개 메시지를 동시에 전송 & 타임스탬프를 분석

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant