Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Jenkinsfile
Original file line number Diff line number Diff line change
Expand Up @@ -237,7 +237,7 @@ git clone https://${GIT_TOKEN}@github.com/Fac2Real/monitory-iac.git ${env.GIT_CO
cd ${env.GIT_COMMIT}
git checkout deploy
git fetch origin main
git merge --ff-only origin/main
git merge --no-ff origin/main -m "ci: merge main β†’ deploy"
yq -i ".image.tag = \\"${env.GIT_COMMIT}\\"" ${HELM_VALUES_PATH}
git config user.name "ci-bot"
git config user.email "ci-bot@monitory.space"
Expand Down
92 changes: 92 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
# Monitory Data Source

IoT, ν™˜κ²½, μ„€λΉ„, μ›¨μ–΄λŸ¬λΈ” λ“± λ‹€μ–‘ν•œ μ„Όμ„œ 데이터λ₯Ό AWS Kinesis, MQTT λ“±μ—μ„œ μˆ˜μ§‘ν•˜μ—¬ Flink둜 μ‹€μ‹œκ°„ μ²˜λ¦¬ν•˜κ³ , Kafka 및 S3 λ“±μœΌλ‘œ μ €μž₯/μ „μ†‘ν•˜λŠ” 데이터 슀트림 νŒŒμ΄ν”„λΌμΈμž…λ‹ˆλ‹€.

## πŸ—οΈ ν”„λ‘œμ νŠΈ ꡬ쑰

```
.
β”œβ”€β”€ src/main/java/com/monitory/data/
β”‚ β”œβ”€β”€ FlinkSourceApplication.java # 메인 μ‹€ν–‰ 파일
β”‚ β”œβ”€β”€ config/ # Kafka, MQTT μ„€μ • λ‘œλ”
β”‚ β”œβ”€β”€ sinks/ # S3, Kafka λ“± Sink κ΄€λ ¨ 클래슀
β”‚ β”œβ”€β”€ sources/ # Kinesis, MQTT λ“± Source κ΄€λ ¨ 클래슀
β”‚ β”œβ”€β”€ transformations/ # 데이터 λ³€ν™˜/가곡 둜직
β”‚ └── utils/ # μœ ν‹Έλ¦¬ν‹° 클래슀
β”œβ”€β”€ src/main/resources/
β”‚ β”œβ”€β”€ application.properties # ν™˜κ²½λ³„ μ„€μ • 파일 (gitignore)
β”‚ β”œβ”€β”€ certs/ # μΈμ¦μ„œ 폴더 (gitignore)
β”‚ β”œβ”€β”€ log4j.properties # 둜그 μ„€μ •
β”‚ └── flink-conf.yaml # Flink μ„€μ •
β”œβ”€β”€ build.gradle # Gradle λΉŒλ“œ 슀크립트
β”œβ”€β”€ Dockerfile # Docker 이미지 λΉŒλ“œ 파일
β”œβ”€β”€ Jenkinsfile # Jenkins CI/CD νŒŒμ΄ν”„λΌμΈ
└── ...
```

## πŸš€ μ£Όμš” κΈ°λŠ₯

- **AWS Kinesis, MQTT** λ“± λ‹€μ–‘ν•œ μ†ŒμŠ€μ—μ„œ μ‹€μ‹œκ°„ 데이터 μˆ˜μ§‘
- **Flink** 기반 μ‹€μ‹œκ°„ 슀트림 처리 및 μ΄μƒμΉ˜/라벨링 λ³€ν™˜
- **Kafka** 토픽별 데이터 λΆ„λ₯˜ 및 전솑
- **S3**둜 μ‹œκ°„/μž₯비별 집계 데이터 μ €μž₯
- **Prometheus** 기반 λͺ¨λ‹ˆν„°λ§ 지원

## βš™οΈ λΉŒλ“œ 및 μ‹€ν–‰

### 1. ν™˜κ²½ μ€€λΉ„

- Java 11 이상 (Amazon Corretto 11 ꢌμž₯)
- Gradle 8.x
- AWS 자격증λͺ… 및 S3, Kinesis, Kafka, MQTT 브둜컀 μ ‘κ·Ό κΆŒν•œ ν•„μš”

### 2. λΉŒλ“œ

```sh
./gradlew build
```

- μ‹€ν–‰ νŒŒμΌμ€ `build/libs/monitory-data-1.0-SNAPSHOT.jar`둜 μƒμ„±λ©λ‹ˆλ‹€.

### 3. μ‹€ν–‰

```sh
java -jar build/libs/monitory-data-1.0-SNAPSHOT.jar
```

λ˜λŠ” Docker둜 μ‹€ν–‰:

```sh
docker build -t monitory-data .
docker run --env-file .env -p 8081:8081 monitory-data
```

### 4. ν™˜κ²½μ„€μ •

- `src/main/resources/application.properties`에 Kafka, MQTT, AWS λ“± μ—°κ²° 정보λ₯Ό μ„€μ •ν•΄μ•Ό ν•©λ‹ˆλ‹€.
- μΈμ¦μ„œ νŒŒμΌμ€ `src/main/resources/certs/`에 μœ„μΉ˜ν•΄μ•Ό ν•˜λ©°, gitμ—λŠ” ν¬ν•¨λ˜μ§€ μ•ŠμŠ΅λ‹ˆλ‹€.

## πŸ“ μ£Όμš” 클래슀

- [`FlinkSourceApplication`](src/main/java/com/monitory/data/FlinkSourceApplication.java): Flink 슀트림 νŒŒμ΄ν”„λΌμΈμ˜ 메인 μ—”νŠΈλ¦¬ν¬μΈνŠΈ
- [`KinesisSourceUtil`](src/main/java/com/monitory/data/utils/KinesisSourceUtil.java): Kinesis Consumer 생성
- [`KafkaUtil`](src/main/java/com/monitory/data/utils/KafkaUtil.java): Kafka Sink 생성 및 ν† ν”½ λΆ„κΈ°
- [`S3SinkUtil`](src/main/java/com/monitory/data/utils/S3SinkUtil.java): S3 Sink 생성
- [`TimeStampAssigner`](src/main/java/com/monitory/data/transformations/TimeStampAssigner.java): UTC νƒ€μž„μŠ€νƒ¬ν”„ β†’ KST λ³€ν™˜
- [`EnvironmentDangerLevelAssigner`](src/main/java/com/monitory/data/transformations/EnvironmentDangerLevelAssigner.java): ν™˜κ²½ μ„Όμ„œ μœ„ν—˜λ„ 라벨링
- [`FaultyAssigner`](src/main/java/com/monitory/data/transformations/FaultyAssigner.java): μ„€λΉ„ μ„Όμ„œ μ΄μƒμΉ˜ 탐지
- [`WearableDangerLevelAssigner`](src/main/java/com/monitory/data/transformations/WearableDangerLevelAssigner.java): μ›¨μ–΄λŸ¬λΈ” μ„Όμ„œ μœ„ν—˜λ„ 라벨링

## πŸ› οΈ 개발/운영 μ°Έκ³ 

- CI/CD: Jenkins, GitHub Actions, ArgoCD, Slack 연동
- 둜그: log4j μ„€μ •(`log4j.properties`)
- Flink μ„€μ •: `flink-conf.yaml` μ°Έκ³ 

## πŸ“„ λΌμ΄μ„ μŠ€

λ³Έ ν”„λ‘œμ νŠΈλŠ” Apache 2.0 λΌμ΄μ„ μŠ€λ₯Ό λ”°λ¦…λ‹ˆλ‹€.

---

문의 및 κΈ°μ—¬λŠ” [Issues](https://github.com/Fac2Real/monitory-data-source/issues) λ˜λŠ” PR둜
6 changes: 4 additions & 2 deletions src/main/resources/flink-conf.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,10 @@ jobmanager:

taskmanager:
heap:
mb: 2048
mb: 4096
numberOfTaskSlots: 16

rest:
port: 8081
port: 8081

env.java.opts.taskmanager: "-XX:+UseG1GC"