Add dataset parse/load library#13
Open
wzhao18 wants to merge 23 commits intoampersand-projects:masterfrom
Open
Add dataset parse/load library#13wzhao18 wants to merge 23 commits intoampersand-projects:masterfrom
wzhao18 wants to merge 23 commits intoampersand-projects:masterfrom
Conversation
27aaaab to
975ec9c
Compare
added 7 commits
February 22, 2022 19:52
b1959aa to
6f230d7
Compare
added 10 commits
February 28, 2022 01:10
anandj91
requested changes
Mar 28, 2022
Contributor
There was a problem hiding this comment.
struct Row {
string dataset;
};
struct CSVRow : public Row {
vector<string> cols;
};
template<typename R>
class DataParser {
protected:
virtual void run() = 0;
virtual void parse(fstream&) = 0;
virtual void decode(R&, stream::stream_event&) = 0;
bool write_serialized_to_ostream(stream::stream_event &t)
{
if (!google::protobuf::util::SerializeDelimitedToOstream(t, &cout)) {
cerr << "Fail to serialize data into output stream" << endl;
return false;
}
return true;
}
};
class CSVParser : public DataParser<CSVRow> {
protected:
void parse_file(fstream &file)
{
string line;
getline(file, line);
while (getline(file, line)) {
CSVRow row;
string word;
stringstream ss(line);
while (getline(ss, word, ',')) {
row.cols.push_back(word);
}
stream::stream_event data;
decode(row, data);
if (!write_serialized_to_ostream(data)) {
break;
}
}
}
};
consider something like the above organization of the base classes.
Additionally, try to follow some standard styling practice when you write code. Like the position of braces of classes and functions.
anandj91
reviewed
Mar 28, 2022
Contributor
There was a problem hiding this comment.
Consider using two different parsers for fare and trip datasets. You don't need to combine them in one.
Collaborator
Author
There was a problem hiding this comment.
The intention of combining the parsers for fare and trip datasets is that both can be streamed to std out at the same time. The taxi benchmark requires two streams.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.