Skip to content

Add db population script, requirements, and README#52

Open
bengoh815 wants to merge 8 commits intomainfrom
awsscript
Open

Add db population script, requirements, and README#52
bengoh815 wants to merge 8 commits intomainfrom
awsscript

Conversation

@bengoh815
Copy link
Contributor

@bengoh815 bengoh815 commented Mar 14, 2025

  • Upload base script structure
  • Add requirements.txt for package dependencies
  • Create README.md with documentation
  • Update script to parse through all files
  • Improve documentation with clearer explanations

@bengoh815 bengoh815 requested a review from vithul7 March 14, 2025 16:23
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this being used by another file? if not, we can delete this

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is not but I think it should be kept somewhere for future reference, if database needs to be repopulated or backend needs a reference to the schema.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's delete it - we can repopulate the schema with a simple command so we shouldn't keep it in there.

@vithul7
Copy link
Contributor

vithul7 commented Mar 14, 2025

change your code to use the correct env variables that are outlined in our shared .env file in the drive

@sarthyparty
Copy link
Contributor

sarthyparty commented Mar 16, 2025

Hey Ben, commit names should be in present tense, like "add db population script". The PR name can be the same name as the commit

@bengoh815 bengoh815 changed the title Initial commit - Added script, requirements, and README Initial commit - Add db population script, requirements, and README Mar 16, 2025
@bengoh815 bengoh815 changed the title Initial commit - Add db population script, requirements, and README Add db population script, requirements, and README Mar 16, 2025
@vithul7
Copy link
Contributor

vithul7 commented Mar 16, 2025

why is there a db pipeline folder and a scripts folder within that, and also a scripts folder outside of this db_pipeline? I'm assuming db_pipeline folder is redundant as you don't mention it in the README.md. also, did you end up changing this to work with the shared .env? i would also rename the scripts folder to something more relevant like db_population

- Uploaded the base script structure
- Added requirements.txt for package dependencies
- Created README.md with documentation
- Updated script to parse through all files
- Improved documentation with clearer explanations
bengoh815 and others added 7 commits March 17, 2025 13:38
Updated for better indication of database usage
Updated logger in main for better indication on not found data
Updated dynamic models to match latest schema
Updated db_manager, add or update function to accommodate new schema changes.
Updated s3_manager, removed uncessary print statement and updated regions according to new schema changes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants