Skip to content

[Feat] Promtail, Loki 설정#203

Open
hyomee2 wants to merge 5 commits intodevelopfrom
feat/#195-monitoring
Open

[Feat] Promtail, Loki 설정#203
hyomee2 wants to merge 5 commits intodevelopfrom
feat/#195-monitoring

Conversation

@hyomee2
Copy link
Copy Markdown
Collaborator

@hyomee2 hyomee2 commented Mar 24, 2026

Related issue 🛠

  • closed #이슈넘버

Work Description 📝

1. Prometheus, Grafana 연동 확인

image
  • 위와 같이 기본적인 Prometheus을 통해 Grafana에서 애플리케이션 메트릭을 확인할 수 있도록 설정했던 것(지난 PR)이 정상적으로 동작함을 확인했습니다.
  • 우선 Micrometer 대시보드를 이용했고, 혹시 다른 정보도 확인하고 싶다면 추가하면 될 것 같습니다
  • Grafana ID, 비밀번호 등과 같은 설정은 추후 설정할 예정입니다. (현재는 기본값 이용)

2. 로그 수집 환경 설정 (Promtail + Loki)

  • 애플리케이션 로그를 애플리케이션(blue/green) 디렉토리로 분리하고, INFO/WARN/ERROR로 디렉토리별 분리를 해주었습니다.
  • promtail 설정 파일을 작성은 했으나, 운영 서버에 배포되고 정상작동하는지 확인 후 작업이 필요할 경우 추가 작업을 해줄 예정입니다.
  • 로그 수집 디렉토리 구조는 아래와 같습니다.
/app/deployment/logs/
├─ blue/
│   ├─ info/
│   ├─ warn/
│   └─ error/
└─ green/
    ├─ info/
    ├─ warn/
    └─ error/
  • 현재 EC2구조는 아래와 같습니다.
[메인(애플리케이션) 서버]
 ├─ Server (Docker)
 └─ Promtail (Docker)

[모니터링 서버]
 ├─ Loki (Docker)
 ├─ Prometheus (Docker)
 └─ Grafana (Docker)

ScreenShots 📷

To Reviewers 📢

Summary by CodeRabbit

  • 새로운 기능

    • 로그 인프라 개선: INFO/WARN/ERROR 로그를 레벨별 파일로 분리하고 일별 롤오버 및 보관을 적용했습니다.
    • 로그 수집 확장: Promtail 기반 로그 포워딩 구성 추가로 중앙 모니터링 연동이 가능해졌습니다.
  • 기타

    • 컨테이너 구성 업데이트: 애플리케이션에 로그 디렉터리 마운트 및 관련 환경변수가 추가되었습니다.
    • 모니터링 설정 수정: Prometheus 스크랩에 HTTPS와 식별 라벨이 명시되었습니다.

@coderabbitai
Copy link
Copy Markdown

coderabbitai bot commented Mar 24, 2026

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 495cd213-128c-483c-a6f5-819fcee58b53

📥 Commits

Reviewing files that changed from the base of the PR and between 4c87c94 and 0236973.

📒 Files selected for processing (3)
  • deployment/docker-compose-promtail.yml
  • deployment/docker-compose.yml
  • deployment/promtail/promtail-config.yml
✅ Files skipped from review due to trivial changes (2)
  • deployment/promtail/promtail-config.yml
  • deployment/docker-compose.yml
🚧 Files skipped from review as they are similar to previous changes (1)
  • deployment/docker-compose-promtail.yml

📝 Walkthrough

Walkthrough

Docker Compose에 로그 볼륨 마운트를 추가하고 Promtail/Loki용 설정과 Logback 파일 기반 로깅을 도입했으며, Prometheus scrape 설정에 scheme: httpsapplication 라벨을 추가했습니다.

Changes

Cohort / File(s) Summary
Docker Compose 변경
deployment/docker-compose.yml, deployment/docker-compose-monitoring.yml
kareer-blue/kareer-greenLOG_BASE=/app/logs 환경변수 추가 및 ./logs/{blue,green}/{info,warn,error}:/app/logs/{info,warn,error} 볼륨 마운트 추가. version 문자열 따옴표 형식 수정 ("3.8"'3.8').
Promtail 추가
deployment/docker-compose-promtail.yml, deployment/promtail/promtail-config.yml
promtail 서비스 정의 파일 추가 및 /app/logs/.../*.log 경로를 읽어 http://${MONITORING_HOST}:3100/loki/api/v1/push로 전송하도록 Promtail 구성 추가. 환경변수 및 포지션 파일 설정 포함.
애플리케이션 로깅 설정
src/main/resources/logback-spring.xml
Logback 설정 추가: LOG_BASE/APP_NAME 프로퍼티와 INFO/WARN/ERROR 수준별 롤링 파일 앱렌더(일일 롤오버, 7일 보관) 및 콘솔 앱렌더 구성.
Prometheus 구성
deployment/prometheus/prometheus.yml
kareer_server 스크랩 잡에 scheme: https 명시 및 static_configsapplication: kareer_server 라벨 추가 (타겟 표기 방식조정 포함).

Sequence Diagram

sequenceDiagram
    participant App as App<br/>(kareer-blue / kareer-green)
    participant FS as FileSystem<br/>(./logs/blue or ./logs/green)
    participant Promtail as Promtail
    participant Loki as Loki
    participant Prometheus as Prometheus

    App->>FS: Logback writes info/warn/error logs
    FS->>Promtail: Mounted volumes expose /app/logs/...
    Promtail->>Promtail: Tail files per scrape configs
    Promtail->>Loki: POST /loki/api/v1/push (labeled by job,deployment)
    Loki->>Loki: Store indexed logs
    Prometheus->>App: Scrape metrics (https, application=kareer_server)
Loading

Estimated Code Review Effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly Related PRs

Suggested Reviewers

  • eraser502
  • jeong1112

Poem

🐰
파일에 써진 하루의 발자국, 톡톡
파랑과 초록이 나뉘어도 한 그릇의 로그,
프롬테일이 수집해 띄우면,
로키에 별이 반짝이고,
엔지니어의 밤은 밝아지네 ✨

🚥 Pre-merge checks | ✅ 3
✅ Passed checks (3 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed PR 제목은 이 PR의 주요 변경 사항인 Promtail과 Loki 설정 추가를 명확하게 요약하고 있습니다.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/#195-monitoring

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 4

🧹 Nitpick comments (3)
src/main/resources/logback-spring.xml (2)

47-62: ERROR 로그는 더 긴 보관 기간을 고려해 보세요.

현재 모든 로그 레벨이 7일 보관으로 설정되어 있습니다. ERROR 로그는 장애 분석 및 디버깅에 중요하므로 14~30일 정도로 더 긴 보관 기간을 설정하는 것이 일반적입니다.

💡 ERROR 로그 보관 기간 연장 제안
     <appender name="ERROR_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
         <file>${LOG_BASE}/error/${APP_NAME}.error.log</file>
         <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
             <fileNamePattern>${LOG_BASE}/error/${APP_NAME}.error.%d{yyyy-MM-dd}.log</fileNamePattern>
-            <maxHistory>7</maxHistory>
+            <maxHistory>30</maxHistory>
         </rollingPolicy>
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/main/resources/logback-spring.xml` around lines 47 - 62, The ERROR_FILE
appender currently uses a TimeBasedRollingPolicy with
<maxHistory>7</maxHistory>; increase this to a longer retention (e.g., set
maxHistory to 14 or 30) to retain ERROR logs for a longer period for debugging;
update the RollingFileAppender configuration for ERROR_FILE (the
TimeBasedRollingPolicy tied to ERROR_FILE and its fileNamePattern) to the chosen
maxHistory value and keep the rest of the appender configuration unchanged.

3-3: LOG_BASE가 설정되지 않은 경우 기본값을 제공하세요.

환경 변수 LOG_BASE가 설정되지 않으면 로그 파일 경로가 _IS_UNDEFINED/info/...로 설정되어 오류가 발생할 수 있습니다.

♻️ 기본값 추가 제안
-    <property name="LOG_BASE" value="${LOG_BASE}"/>
+    <property name="LOG_BASE" value="${LOG_BASE:-/app/logs}"/>
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/main/resources/logback-spring.xml` at line 3, The LOG_BASE property is
currently used without a fallback which leads to paths like
"_IS_UNDEFINED/info/..."; update the <property name="LOG_BASE" value="..."/>
declaration so the value uses a Spring placeholder with a sensible default (e.g.
./logs) when LOG_BASE is not set; locate the property with name="LOG_BASE" in
logback-spring.xml and change its value attribute to include the default
fallback using the placeholder syntax so log file paths never resolve to an
undefined token.
deployment/promtail/promtail-config.yml (1)

2-3: positions 파일이 컨테이너 재시작 시 유실될 수 있습니다.

/tmp/positions.yaml은 컨테이너 재시작 시 초기화됩니다. 이로 인해 Promtail이 이미 전송한 로그를 다시 전송하거나, 로그 위치를 잃어버릴 수 있습니다.

♻️ 영구 볼륨 사용 권장

docker-compose-promtail.yml에 볼륨을 추가하고 경로를 변경하세요:

 positions:
-  filename: /tmp/positions.yaml
+  filename: /positions/positions.yaml

docker-compose-promtail.yml:

volumes:
  - promtail_positions:/positions
  # ... 기존 볼륨들
volumes:
  promtail_positions:
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@deployment/promtail/promtail-config.yml` around lines 2 - 3, 현재
promtail-config.yml의 positions.filename이 /tmp/positions.yaml로 설정되어 컨테이너 재시작 시
유실될 가능성이 있습니다; 변경 방법은 promtail-config.yml에서 positions.filename을 영구 볼륨 경로(예:
/positions/positions.yaml)로 수정하고, docker-compose-promtail.yml에
promtail_positions 같은 명시적 볼륨을 추가하여 컨테이너의 /positions 경로에 마운트하도록 설정하면 됩니다 — 참조할
심볼: positions.filename in promtail-config.yml and volume name promtail_positions
in docker-compose-promtail.yml.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@deployment/docker-compose-promtail.yml`:
- Around line 12-24: The mounted container log paths in the docker-compose
service (volumes entries mapping ./logs/blue/* and ./logs/green/* →
/app/logs/...) do not match the paths referenced in promtail-config.yml (e.g.
/var/log/info/*.log); update either the volume mounts or promtail-config.yml so
the file_glob targets the actual container paths (reference: volumes entries,
promtail-config.yml, and command: -config.file=/etc/promtail/config.yml), and
also ensure the service is attached to a network that can reach Loki (add a
network section or use host network) so Promtail can connect to the external
Loki endpoint.
- Line 5: The docker-compose service is using an unpinned image tag
"grafana/promtail:latest" which causes non-deterministic deployments; update the
image reference used in the compose file (the "image: grafana/promtail:latest"
line) to a fixed, supported version such as "grafana/promtail:3.6.0" to lock the
Promtail version and avoid using :latest.

In `@deployment/docker-compose.yml`:
- Line 15: 환경 변수 LOG_BASE의 컨테이너 경로가 잘못되어 있어 로그가 컨테이너 내부 위치와 매칭되지 않습니다;
docker-compose.yml에서 환경 변수 LOG_BASE 값을 호스트 상대 경로(./logs/blue)가 아닌 컨테이너 내부 경로인
/app/logs로 변경하고, 동일한 변경을 다른 서비스의 LOG_BASE 설정(예: kareer-green 환경 변수)에도 적용해 컨테이너 내
Logback이 /app/logs를 사용하도록 수정하세요.

In `@deployment/promtail/promtail-config.yml`:
- Around line 8-60: Promtail is not matching the Docker Compose volume paths:
update each job's __path__ entries (e.g., for job_name: info_blue, info_green,
warn_blue, warn_green, error_blue, error_green) to point to the actual mounted
directories used in docker-compose (for example change /var/log/info/*.log to
/app/logs/blue/info/*.log for info_blue and to /app/logs/green/info/*.log for
info_green) and similarly update warn and error jobs to
/app/logs/<blue|green>/warn/*.log and /app/logs/<blue|green>/error/*.log so
blue/green are distinct and paths match the compose mounts.

---

Nitpick comments:
In `@deployment/promtail/promtail-config.yml`:
- Around line 2-3: 현재 promtail-config.yml의 positions.filename이
/tmp/positions.yaml로 설정되어 컨테이너 재시작 시 유실될 가능성이 있습니다; 변경 방법은 promtail-config.yml에서
positions.filename을 영구 볼륨 경로(예: /positions/positions.yaml)로 수정하고,
docker-compose-promtail.yml에 promtail_positions 같은 명시적 볼륨을 추가하여 컨테이너의 /positions
경로에 마운트하도록 설정하면 됩니다 — 참조할 심볼: positions.filename in promtail-config.yml and
volume name promtail_positions in docker-compose-promtail.yml.

In `@src/main/resources/logback-spring.xml`:
- Around line 47-62: The ERROR_FILE appender currently uses a
TimeBasedRollingPolicy with <maxHistory>7</maxHistory>; increase this to a
longer retention (e.g., set maxHistory to 14 or 30) to retain ERROR logs for a
longer period for debugging; update the RollingFileAppender configuration for
ERROR_FILE (the TimeBasedRollingPolicy tied to ERROR_FILE and its
fileNamePattern) to the chosen maxHistory value and keep the rest of the
appender configuration unchanged.
- Line 3: The LOG_BASE property is currently used without a fallback which leads
to paths like "_IS_UNDEFINED/info/..."; update the <property name="LOG_BASE"
value="..."/> declaration so the value uses a Spring placeholder with a sensible
default (e.g. ./logs) when LOG_BASE is not set; locate the property with
name="LOG_BASE" in logback-spring.xml and change its value attribute to include
the default fallback using the placeholder syntax so log file paths never
resolve to an undefined token.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 00000463-e817-46a4-812f-3d61d1e9ecc1

📥 Commits

Reviewing files that changed from the base of the PR and between 7d7fce3 and 4c87c94.

📒 Files selected for processing (6)
  • deployment/docker-compose-monitoring.yml
  • deployment/docker-compose-promtail.yml
  • deployment/docker-compose.yml
  • deployment/prometheus/prometheus.yml
  • deployment/promtail/promtail-config.yml
  • src/main/resources/logback-spring.xml

Copy link
Copy Markdown
Collaborator

@jeong1112 jeong1112 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

고생하셨습니다~

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

저희가 현재 Blue, Green 컨테이너로 나눠놓은 것은 버전 관리가 아니고 무중단 배포를 하기 위함인데 로그를 분리해서 저장하는 이유가 있나요?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

장애 영향 범위를 최소화 하여 원인이 발생한 컨테이너에 대해서만 로그를 확인할 수 있어 더 유용하고, 롤백을 하게 될 경우에도 어떤 환경의 문제/로그인지 빠르고 정확하게 확인할 수 있다는 점에서 분리했습니당.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants