Discord Mentions are counted towards Language Tab

[UnduGT]

Hello,

This applies to Discord, don't now how it is on other Platforms.

I don't know if this can be classified as a bug. Here's the thing: The Language tab that shows common words, also tracks mentions with @ as text and includes them there.

For example, if a user calls himself Apple, then for every time this user has been tagged with @ in the past, the word Apple will be counted in the Language tab.

If the user now renames himself to, for example, Peach, then in the next report the word Apple will appear less but the word Peach will appear instead.

I don't know if that's intentional, but I can't imagine it, since it distorts the result a little every time.

Version:

[mlomb]

Interesting, since the way we are matching mentions should not allow mention texts to go to the language tab:

chat-analytics/pipeline/process/nlp/Tokenizer.ts

Lines 62 to 67 in c85098a

	{
	// match @mentions
	regex: /(^|\s)@[\p{L}_0-9]+/giu,
	tag: "mention",
	transform: (match) => match.trim().slice(1), // remove @
	},

---

Since you mentioned that changing the nickname of a user may cause this, this may be relevant:

chat-analytics/pipeline/parse/parsers/DiscordParser.ts

Lines 122 to 128 in c85098a

	let content = message.content;
	for (const mention of message.mentions) {
	// replace names by nicknames in mentions (to honor nicknames)
	// and just to make sure, replace spaces by underscores in the nickname and
	// add spaces in the sides so that it can be picked correctly up by the Tokenizer
	content = content.split(`@${mention.name}`).join(` @${mention.nickname.replace(/\s/g, "_")} `);
	}

chat-analytics/pipeline/parse/parsers/Discord.d.ts

Lines 101 to 107 in c85098a

	interface DiscordMention {
	id: Snowflake;
	name: string;
	discriminator: string;
	nickname: string;
	isBot: boolean;
	}

The original message content probably contains a name different from what the exporter is reading from the API, which is probably the latest nickname, so it does not get replaced (unsure about this). But still, the @ should be there and be handled like a mention, hmmm

[mlomb]

Can you check if the real usernames actually match the regex there? Maybe thats it

[UnduGT]

Can you explain what you mean with real usernames? (sorry for late answers, was away for a week)

[mlomb]

Yeah, you used Apple and Peach as examples; I mean the actual Discord usernames (unless those are lol)

[UnduGT]

Oh, while looking this up I kinda founded something weird, My Username on the Server is "Sad Bitch :3", after I changed the Name, I immediately noticed that the word "Bitch" spiked to almost 600 Words in the Report. If I use the Discord Search function and search "Bitch", there are only 128 Results, If I search the mention with <@userid> It finds almost 700 entries.
Whats funny about that is, that the word "Sad" is not recorded that many times, it's only recorded 41 times in the report and the search function shows 74 (but it also shows links containing the string sad)

I think it correctly filters out mentions but the space in the Name breaks everything, thats why the first word is not counted, but the second one is. But I have no Idea, thats only what I could find right now.

[UnduGT]

I tested the Regex above on regex101.com with JavaScript on my Username, I think thats the issue

[ctuncan]

Right, so I ran into this last year and never got around to raising a bug.
(And worked around it for my own purposes with a little pre-processing)

The issue is exactly here

chat-analytics/pipeline/parse/parsers/DiscordParser.ts

Lines 122 to 128 in c85098a

	let content = message.content;
	for (const mention of message.mentions) {
	// replace names by nicknames in mentions (to honor nicknames)
	// and just to make sure, replace spaces by underscores in the nickname and
	// add spaces in the sides so that it can be picked correctly up by the Tokenizer
	content = content.split(`@${mention.name}`).join(` @${mention.nickname.replace(/\s/g, "_")} `);
	}

Given a user with nickname Sad Bitch :3, and a discord name of UnduGT (Just to steal your example)

This code assumes that the content of the export will contain @UnduGT, and intends to replace it with @Sad_Bitch_:3.
However, in at least the current version of DCE (v2.43.3) the export will contain @Sad Bitch :3, which means this will not replace it!

---

Secondarily, the regex for mentions (^|\s)@[\p{L}_0-9]+ will not match the : in this nickname, so that will separately break it! Even if it were @Sad_Bitch_:3

Note that usernames are more restrictive, and will match [a-z0-9_.]{2,32} (With the additional restriction that you cannot have 2 consecutive .)

Meanwhile nicknames (Display Names) have almost no restrictions, lowercase, uppercase, spaces, emoji, anything goes! Just a 1-32 character length limit.

Source: https://support.discord.com/hc/en-us/articles/12620128861463-New-Usernames-Display-Names#h_01GXPQAGG6W477HSC5SR053QG1

---

My pre-processing workaround was to:

1. Make a list of all discord users
2. For each message, for each user do a replace.
3. Clear out the mentions array, so that it stays as the username, not the display name (due to special characters and emojis) Essentially preventing this block of code from running.

I used these two jq commands
cat *.json | jq '.messages[].author, .messages[].mentions[]' | jq -s 'unique_by(.nickname)' > users.json
(Where *.json matches all discord exports)

then in a for loop:

for f in *.json; do
    cat $f | jq --slurpfile authors users.json 'reduce $authors.[].[] as $item (.; .messages[].content |= sub($item.nickname; $item.name)) | .messages[].mentions |= []' > "$(basename "$f").new.json
done

---

If I have time, I'll raise a PR re-working this block of code.

content = content.split(`@${mention.nickname}`).join(`@${mention.name}`);

Would replicate me pre-process. But would change the functionality.
Something more robust would need to:

1. Identify if there are possible exports with @name that need to be processed (check DCE source)
2. Perhaps do 2 substitutions, both name and nickname
3. Maintain the nickname feature (aka what you replace with)
4. Improve the mentions regex to handle all the characters possible in a discord nickname (with regression checks, because the tokenizer is common to all the chat platforms)