Comma in a string field

[dosimont]

Example:

[...]
%EventDef PajeStartLink 40
% Time date
% Type string
% Container string
% StartContainer string
% Value string
% Key string
%EndEventDef
%EventDef PajeEndLink 41
% Time date
% Type string
% Container string
% EndContainer string
% Value string
% Key string
%EndEventDef
[...]
41 3.772031 "L_MPI_Coll" "C_Prog" "P#1_T#3626231744" "src=0, dest=1" "0_1_0x7f0df00f6390"
[...]
40 3.775296 "L_MPI_Coll" "C_Prog" "P#0_T#2910222272" "src=0, dest=1" "0_1_0x7f0df00f6390"

leads to this pjdumped output:

Link, Program, MPI collective communication, 3.7753, 3.77203, -0.003265, src=0, dest=1, P#0_T#2910222272, P#1_T#3626231744

where container fields are shifted of +1 because of the additional comma,

I see two solutions:

• the simplest : forbid commas in string and raise an error if a comma is present in the trace
• use " or ' in the output around the problematic string. However, according to the program that reads the csv, it could still be a problem to have a comma in the string.

[schnorr]

Can you provide a small but complete trace file so I can reproduce the error? Between the two solutions you mentioned, I'd favor the simplest one (forbid commas in strings), but wouldn't that be too much strict?

[dosimont]

I think it is preferable to be reliable/stable even if it implies to be strict. But it is just my opinion ;-)

[vdanjean]

Csv files can quote comma. There are several ways to do this. I think the best thing to do is to look at csv documentation and choose a well-used way to quote comma, simple and double quote.
It might be more tricky (and probably far less supported in csv readers) to handle return line in csv fields. Fot this, I think we can forbid them.

[schnorr]

Basically, the Paje format already accepts commas and spaces in fields, even when quoted, which is fine. The problem is that, in the pj_dump output, those commas become separators in the CSV. A small change would be needed to detect if there are commas, and if so, include the quotes in the output of pj_dump.

[rddtz]

The pj_dump standard parsing was ignoring the quotes in the fields, it was possible to use an alternative flex parser using the --flex flag that correctly handled the quotes in this fields. Anyway, this problem was fixed in this commit and now the standard parsing is handling quotes correctly.

[schnorr]

Reopen after commit 87d2d26