fix clean copyright

[hust-nj]

I think there are 2 main problems in current clean_copyright_comments function

RedPajama-Data/data_prep/github/github_clean_dedup_local.py

Line 27 in 567ac9a

def clean_copyright_comments(content: str):

.

First, It cannot remove the copyright successfully in the following C-style code because of the early return in

RedPajama-Data/data_prep/github/github_clean_dedup_local.py

Line 37 in 567ac9a

return content

// Copyright

int main() {
    return 0;
    
    /* comment */
}

Second, I find that, when the file is large, the regex sometimes costs much time in my experiment, I think we only need to find the copyright in the first 100 lines.

[mauriceweber]

Hi @hust-nj ! Thanks for bringing this to our attention! I will review your PR asap.

[mauriceweber]

Hi @hust-nj , I had a look at your PR. Here's some feedback:

• I would prefer not to limit the search for copyright to the first 100 lines; based on what are you proposing 100 lines?
• Your current implementation also gets rid of comments in the beginning of any file, which we would like to keep. For example, this:

// A comment

int main() {
    return 0;
    
    /* comment */
}

yields

int main() {
    return 0;
    
    /* comment */
}