Skip to content

Feature/convert data #6

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
DmitryDavydov1 wants to merge 7 commits into
base: main
Choose a base branch
from
Open

Feature/convert data #6

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
c5048c6
do method for dividing a line
DmitryDavydov1 Nov 27, 2024
bd1fec0
did method for getting text from pdf
DmitryDavydov1 Nov 27, 2024
d72e49f
did method for getting list string from pdf
DmitryDavydov1 Nov 28, 2024
e116be8
did method for getting list string from TXT
DmitryDavydov1 Nov 28, 2024
accfd91
did method for getting list string from Docx
DmitryDavydov1 Nov 28, 2024
82153c8
change structure
DmitryDavydov1 Nov 28, 2024
8bf154b
Update README.md
DmitryDavydov1 Nov 28, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 38 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1,38 @@
# HackathonWindowKnowledge

## API Reference


http://localhost:8080/question/ask/{assistant_id}/{conversation_id} -
Принимает id ассистента и id юзера и дает ответ
assistant_id - id ассистента
conversation_id - id usera




POST /data/pdf

http://localhost:8080/data/pdf - принимает pdf файл и преобразует его в текст деля на chunki по 200 слов
pdf - pdf файл
Принимает pdf файл и преобразует его в текст деля на chunki по 200 слов



http://localhost:8080/data/text - принимает текст и создает список по 200 слов



http://localhost:8080/data/docx-file - принимает docx файл и преобразует его в текст деля на chunki по 200 слов
docx - docx файл
Принимает docx файл и преобразует его в текст деля на chunki по 200 слов

http://localhost:8080/data/text-file - принимает text файл и преобразует его в текст деля на chunki по 200 слов
text - text файл
Принимает text файл и преобразует его в текст деля на chunki по 200 слов







180 changes: 96 additions & 84 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,100 +1,112 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>3.4.0</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
<groupId>ru.hackteam</groupId>
<artifactId>WindowOfKnowledge</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>WindowOfKnowledge</name>
<description>Demo project for Spring Boot</description>
<url/>
<licenses>
<license/>
</licenses>
<developers>
<developer/>
</developers>
<scm>
<connection/>
<developerConnection/>
<tag/>
<url/>
</scm>
<properties>
<java.version>17</java.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>3.4.0</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
<groupId>ru.hackteam</groupId>
<artifactId>WindowOfKnowledge</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>WindowOfKnowledge</name>
<description>Demo project for Spring Boot</description>
<url/>
<licenses>
<license/>
</licenses>
<developers>
<developer/>
</developers>
<scm>
<connection/>
<developerConnection/>
<tag/>
<url/>
</scm>
<properties>
<java.version>17</java.version>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>

<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springdoc</groupId>
<artifactId>springdoc-openapi-starter-webmvc-ui</artifactId>
<version>2.6.0</version>
</dependency>
<dependency>
<groupId>org.apache.pdfbox</groupId>
<artifactId>pdfbox</artifactId>
<version>2.0.29</version> <!-- Убедитесь, что версия актуальна -->
</dependency>

<dependency>
<groupId>org.apache.poi</groupId>
<artifactId>poi-ooxml</artifactId>
<version>5.2.3</version> <!-- Укажите актуальную версию -->
</dependency>

<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springdoc</groupId>
<artifactId>springdoc-openapi-starter-webmvc-ui</artifactId>
<version>2.6.0</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
<version>RELEASE</version>
<scope>compile</scope>
</dependency>

<dependency>
<groupId>com.vladmihalcea</groupId>
<artifactId>hibernate-types-52</artifactId>
<version>2.21.1</version>
</dependency>
<dependency>
<groupId>com.vladmihalcea</groupId>
<artifactId>hibernate-types-52</artifactId>
<version>2.21.1</version>
</dependency>

<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-redis</artifactId>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-redis</artifactId>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
</dependency>


</dependencies>
</dependencies>

<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<configuration>
<excludes>
<exclude>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
</exclude>
</excludes>
</configuration>
</plugin>
</plugins>
</build>
<build>
<plugins>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<configuration>
<excludes>
<exclude>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
</exclude>
</excludes>
</configuration>
</plugin>
</plugins>
</build>

</project>
64 changes: 64 additions & 0 deletions src/main/java/ru/hackteam/window_of_knowledge/api_openai/OpenAIEmbeddingsAPI.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
package ru.hackteam.window_of_knowledge.api_openai;

import com.fasterxml.jackson.databind.ObjectMapper;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;


import java.io.IOException;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.charset.StandardCharsets;
import java.util.HashMap;
import java.util.Map;


@Service
public class OpenAIEmbeddingsAPI {
private final ObjectMapper objectMapper;
private final String API_KEY = System.getenv("API_KEY");
@Value(value = "${api.url.embeddings}")
private String API_URL;

private final HttpClient httpClient;


public OpenAIEmbeddingsAPI() {
this.httpClient = HttpClient.newHttpClient();
this.objectMapper = new ObjectMapper();
}


public double[] getEmbeddings(String text) throws IOException, InterruptedException {
Map<String, Object> requestBody = new HashMap<>();
requestBody.put("input", text);
requestBody.put("model", "text-embedding-ada-002");

String jsonBody = objectMapper.writeValueAsString(requestBody);

HttpRequest request = HttpRequest.newBuilder()
.uri(URI.create(API_URL))
.header("Content-Type", "application/json")
.header("Authorization", "Bearer " + API_KEY)
.POST(HttpRequest.BodyPublishers.ofString(jsonBody, StandardCharsets.UTF_8))
.build();

HttpResponse<String> response = httpClient.send(request, HttpResponse.BodyHandlers.ofString());

if (response.statusCode() != 200) {
throw new RuntimeException("Ошибка при запросе к OpenAI API: " + response.body());
}

// Разбор JSON-ответа
Map<String, Object> responseMap = objectMapper.readValue(response.body(), Map.class);
Map<String, Object> data = (Map<String, Object>) ((java.util.List<?>) responseMap.get("data")).get(0);
java.util.List<Double> embeddingList = (java.util.List<Double>) data.get("embedding");

// Преобразование списка в массив
return embeddingList.stream().mapToDouble(Double::doubleValue).toArray();


}
}
24 changes: 24 additions & 0 deletions src/main/java/ru/hackteam/window_of_knowledge/configurations/RedisConfig.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
package ru.hackteam.window_of_knowledge.configurations;


import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.redis.connection.RedisConnectionFactory;
import org.springframework.data.redis.connection.lettuce.LettuceConnectionFactory;
import org.springframework.data.redis.core.RedisTemplate;

@Configuration
public class RedisConfig {
@Bean
public RedisTemplate<String, Object> redisTemplateEmbeddings(RedisConnectionFactory redisConnectionFactory) {

LettuceConnectionFactory factory = new LettuceConnectionFactory();
factory.setDatabase(0); // Подключение к базе 0
factory.afterPropertiesSet();

RedisTemplate<String, Object> template = new RedisTemplate<>();
template.setConnectionFactory(redisConnectionFactory);
return template;
}

}
44 changes: 0 additions & 44 deletions src/main/java/ru/hackteam/window_of_knowledge/controller/DataController.java
View file
Open in desktop

This file was deleted.

Loading