duplicate entries in csv

[mossburg]

Sometimes i found same entries with the same timestamps in 2 different csv :

Example:
0c9150d10cf718aeee1c99a6eb7db81c.csv:2011/10/28 12:00:00,auth_6b710d35-245d-4c2b-934e-27b8738b1abc,2922,0,0,0,0,2922,0,0,0,0,0,2922,2922,0,1146285,2922,0,0,0
e1b36fe2ca5a7ca9e08f850962de11c9.csv:2011/10/28 12:00:00,auth_6b710d35-245d-4c2b-934e-27b8738b1abc,2922,0,0,0,0,2922,0,0,0,0,0,2922,2922,0,1146285,2922,0,0,0

Is this a bug or a normal behavior ?

[notmyname]

this is normal in certain edge conditions (perhaps late-arriving log files?). if you have a repeatable test case, perhaps we can work on eliminating the duplicate lines.

[mossburg]

Thank you for you response.
We'll try to find a repeatable test case.

I'll keep you in touch.

[mossburg]

Here is another example :

zcat 2011/10/25/17/6245619f2bd486754e13f2f43af827a7.csv.gz
data_ts,account,2xx,4xx,5xx,COPY,DELETE,GET,HEAD,POST,PUT,account_requests,container_requests,object_requests,ops_count,public_bw_in,public_bw_out,public_request,service_bw_in,service_bw_out,service_request
2011/10/25 12:00:00,auth_6b710d35-245d-4c2b-934e-27b8738b1abc,836,0,0,0,0,836,0,0,0,0,4,832,836,0,615317,836,0,0,0
2011/10/25 12:00:00,auth_.auth,18,0,0,0,0,18,0,0,0,0,0,18,18,0,0,18,0,0,0

zcat 2011/10/25/18/89b6acb345416140bd4a747b42d92056.csv.gz
data_ts,account,2xx,4xx,5xx,COPY,DELETE,GET,HEAD,POST,PUT,account_requests,container_requests,object_requests,ops_count,public_bw_in,public_bw_out,public_request,service_bw_in,service_bw_out,service_request
2011/10/25 13:00:00,auth_6b710d35-245d-4c2b-934e-27b8738b1abc,2,0,0,0,0,2,0,0,0,0,2,0,2,0,167209,2,0,0,0
2011/10/25 12:00:00,auth_6b710d35-245d-4c2b-934e-27b8738b1abc,836,0,0,0,0,836,0,0,0,0,4,832,836,0,615317,836,0,0,0
2011/10/25 13:00:00,auth_.auth,6,0,0,0,0,6,0,0,0,0,0,6,6,0,0,6,0,0,0
2011/10/25 12:00:00,auth_.auth,18,0,0,0,0,18,0,0,0,0,0,18,18,0,0,18,0,0,0

I don't understand why the first log entries (12:00:00) are still present in the second csv. They seems to have been processed correctly ...

[briancline]

@mossburg, here are a few scenarios where you could see CSVs with data spanning multiple hours:

• the log uploader may be having trouble locally deleting successfully-uploaded hourly logs (check write permissions)
• the log processor may be having trouble updating the processed_files.pickle.gz in the container where it stores hourly CSVs
• multiple proxies are trying to process the same logs each hour and hitting a race condition in updating processed_files.pickle.gz (unlikely)
• rsyslog isn't configured to use hourly log files
• a proxy could have a system timezone set that is inconsistent with other proxies (the chances of this being the problem may depend more on your rsyslog config)
• a proxy is taken offline for service and therefore doesn't get to upload its pending logs on time
• a proxy maybe isn't uploading hourly logs on the same schedule as the others (check your crons for consistency)
• your cluster is busy enough that you're hitting a race condition at the top of an hour between the time swift generates the end timestamp of a completed request and the time that rsyslog determines that it's time to open up a new file (unlikely, but I've seen it happen when rsyslog is overworked)