thousands of duplicated emails

Support related problems and questions

Moderators: TerryRogers, Max

Steve Shank
Guru
Posts: 158
Joined: Sat Sep 08, 2018 3:17 am
Has thanked: 27 times
Been thanked: 11 times

Re: thousands of duplicated emails

Post by Steve Shank »

I just checked the picture I sent you. I have set it so that spams get labeled ***Spam*** prefixed to the subject. These are not labeled so. Also, they are in the inbox. But have a line through them indicating they are marked for deletion. - So, In the inbox as the picture shows and marked for deletion but not marked as spam.

I'm going to try turning off the check every 30 minutes and turning on the immediate idle support. This started with V11. It stopped when I turned off both checking every 30 minutes and idle support. It started again when I turned checking back on but left idle support off.
admin
Site Admin
Posts: 15601
Joined: Thu Nov 25, 2004 3:12 am
Has thanked: 1411 times
Been thanked: 984 times

Re: thousands of duplicated emails

Post by admin »

Then our point mentioned here: viewtopic.php?p=124294#p124294 is the one that explains what happens.
Android version of EssentialPIM. Keep all your data in sync!
Steve Shank
Guru
Posts: 158
Joined: Sat Sep 08, 2018 3:17 am
Has thanked: 27 times
Been thanked: 11 times

Re: thousands of duplicated emails

Post by Steve Shank »

What we have here is a failure to communicate. Are you truly saying, because this is what I am hearing, that if a spam email comes into my inbox, and your system marks it as spam or I mark it as spam and move it to the spam folder, then it will sometimes reproduce 3,000 times and that's by design? How do I get 3,000+ identical emails all marked for deletion and all the same spam email occurring every few days if the check every 30 minutes options is marked on Version 11 when not on version 10?

If that is not what you are saying, then help me understand why a single email gets replicated 3,000+ times every few days. It gets marked for deletion. It hasn't happened to a regular email, only to spam emails so far.
admin
Site Admin
Posts: 15601
Joined: Thu Nov 25, 2004 3:12 am
Has thanked: 1411 times
Been thanked: 984 times

Re: thousands of duplicated emails

Post by admin »

Please read my comment I'm referring to above. Forget about spam - only you know that particular message is spam. The problem here is that you move that message into a local folder in another account. Cease this action and all should be good or simply use the Compress option on Inbox.
Android version of EssentialPIM. Keep all your data in sync!
Steve Shank
Guru
Posts: 158
Joined: Sat Sep 08, 2018 3:17 am
Has thanked: 27 times
Been thanked: 11 times

Re: thousands of duplicated emails

Post by Steve Shank »

No. Please read my dozen or so messages. I do not move these messages to a local account.
1. MY local folders are all in this account so they are not moved to another account.
1b. I do not move what I call spam to local folders. I mark them as spam, I do not save spam.
2. The only moving that could occur, but I don't think does, is EPIM moving to the not local spam folder in this account or me marking it as spam and then EPIM moving it to the folder.
3. This does not occur unless automatic - read that again please. AUTOMATIC retrieving of messages is turned on. It occurs if either idle support or checking messages every 30 minutes is on. It did not occur for 8 day when those were turned off.
4. Let me be very specific. I leave the office. Everything is fine. There are only 130 emails in my inbox. I come in in the morning. I've left epim on all night because I sometimes sync to my android device. I check the account with webmail. There are 3,000 duplicate emails in my inbox. All but one or two of them are marked for deletion. I have not used epim. I didn't do anything. I didn't move anything.
5. Another scenario. I am working during the day. I get a notice, not from epim, that 500 messages have arrived in the last 10 minutes. I look in webmail because epim doesn't show them. There are now 2000 and they are deleted duplicates.

You do an excellent job of supporting people here. That's one reason I chose EPIM. I am not stupid or particularly dense. So, I can't figure out why we cannot communicate.

The only thing that I have which in some way might have something to do with what you've said is this:
I do have a local folder under this account, that 3 months ago I moved some older emails from another account into. So under old/2022/I have a folder for older emails I've pulled from that other account. Haven't touched it in 3 months. If you think that is actually the cause of this new problem starting with Version 11 I'll create a separate old/annual non-synced folder for that other account as well.

By local folder, I mean a folder that doesn't synchronize with the email server. Assuming you have the same definition.
admin
Site Admin
Posts: 15601
Joined: Thu Nov 25, 2004 3:12 am
Has thanked: 1411 times
Been thanked: 984 times

Re: thousands of duplicated emails

Post by admin »

When a message gets moved to a folder which is not synchronized with the server, the message is marked for deletion by the server. Then the server should automatically remove it. Let's see if you're able to get rid of all these messages marked for removal next time you experience this by right clicking in EPIM on the folder containing all these messages and choosing the Compress option. Will take it from there then. Thanks.
Android version of EssentialPIM. Keep all your data in sync!
Steve Shank
Guru
Posts: 158
Joined: Sat Sep 08, 2018 3:17 am
Has thanked: 27 times
Been thanked: 11 times

Re: thousands of duplicated emails

Post by Steve Shank »

The epim folder doesn't have the messages. the epim program does not download them. I do not move them. Let me describe what happened today.

I left the office last night. Epim is left running in the taskbar.
I came in this morning. Epim showed an error on a time out getting my messages.

I checked, not with EPIM, which was still sitting in on the taskbar. but with another program that looks at the server but only downloads headers.

There were 6,800 new bad false emails all marked for deletion. I can tell they are marked for deletion because there is a line through the topic.

About 3,400 duplicate spams came in at 2:49 am
Another 3,400 different duplicate spams came in at 5:15 AM

There were only 2 emails which duplicated to the same two messages, at the same time to the minute (at least) causing a clogging of 6,800 emails in the account's inbox. It is possible that not all were marked for deletion, but it is very hard to examine emails for a good one among 6,800 new emails. The 80 or 90 legitimate emails in my inbox are difficult to find.

I do not know if this is EPIM or the Mail host, or a combination, but somehow a single email is getting into a loop and reproducing. At about 3,000 the server decides there is an issue and stops it, and marks the duplicates for deletion. Because they are marked for deletion, EPIM never downloads them. It is possible that either your program decides one of them is spam and moves it to the spam folder. It is possible that Spam Assassin on their server decides it is spam and moves one to its spam folder. But, 3,400 duplicates remained in my inbox for each of the two problem emails.

When I change server settings, is it necessary to restart epim for them to take effect? Because yesterday, I unchecked both of the two auto checking with epim and was surprised the problem came back. I'm going to restart epim.
admin
Site Admin
Posts: 15601
Joined: Thu Nov 25, 2004 3:12 am
Has thanked: 1411 times
Been thanked: 984 times

Re: thousands of duplicated emails

Post by admin »

So did you try to use the Compress option like I suggested in my previous post?
Android version of EssentialPIM. Keep all your data in sync!
Steve Shank
Guru
Posts: 158
Joined: Sat Sep 08, 2018 3:17 am
Has thanked: 27 times
Been thanked: 11 times

Re: thousands of duplicated emails

Post by Steve Shank »

No I hadn't, but I did compress option just now. It did not appear to do anything at all. I had thought it a bizarre recommendation because the emails in question were never downloaded at all by epim, so there was nothing moved and nothing to compress. Is there any way to determine if the compress option which appears to do nothing, actually does something?
admin
Site Admin
Posts: 15601
Joined: Thu Nov 25, 2004 3:12 am
Has thanked: 1411 times
Been thanked: 984 times

Re: thousands of duplicated emails

Post by admin »

Sure, if you see a list with emails marked for deletion, the Compress option used in WinEPIM on a particular folder should among other things give the web server a command to actually delete those messages.
Android version of EssentialPIM. Keep all your data in sync!
Steve Shank
Guru
Posts: 158
Joined: Sat Sep 08, 2018 3:17 am
Has thanked: 27 times
Been thanked: 11 times

Re: thousands of duplicated emails

Post by Steve Shank »

OK. So that's a good way to clean up the mess after all the duplicates are created. What I'd like to determine is if there is a way to stop the duplication from occurring in the first place. So far, it seems like unchecking the check every x minutes and the idle support, seems to stop it. I've turned those off so I don't have to deal with it. Did you find anything useful in the logs I uploaded? I did those immediately after determining there was an attack.
admin
Site Admin
Posts: 15601
Joined: Thu Nov 25, 2004 3:12 am
Has thanked: 1411 times
Been thanked: 984 times

Re: thousands of duplicated emails

Post by admin »

Let's first see if the Compress option helps. In this case we'd know what to look for next.
Android version of EssentialPIM. Keep all your data in sync!
Steve Shank
Guru
Posts: 158
Joined: Sat Sep 08, 2018 3:17 am
Has thanked: 27 times
Been thanked: 11 times

Re: thousands of duplicated emails

Post by Steve Shank »

So, compress is pretty cool. It does delete the duplicates marked for deletion on the server. Here's how this latest one appeared. I had about 3, 700 duplicated emails looking at the server. They were marked for deletion. This time I paid attention to what it was. When I pulled up EPIM's inbox, it showed the message 1 time as a normal, not spam message in my inbox. It did not show the 3,700 duplicates that were on the server. Just the one message which it did not recognize as spam and showed as a normal message.

I marked it as spam and compressed and the 3,700 duplicates were removed. So this is better, but still, why are the duplicates appearing at all? I'd given you log files before, and a screenshot of how it looks on the server. I found a message, not this latest one, but an earlier one, that created lots of duplicates. I've attached in case the source code (as text file), might help you spot something if it is epim bouncing back and forth with the server.
Attachments
painfulknees.txt
(9.79 KiB) Downloaded 56 times
admin
Site Admin
Posts: 15601
Joined: Thu Nov 25, 2004 3:12 am
Has thanked: 1411 times
Been thanked: 984 times

Re: thousands of duplicated emails

Post by admin »

Ok, thanks for the update, now we know that compress works if executed manually for that particular mail server (normally it should work automatically).

Moving on with this problem. You meant to say that if EPIM was left running in the background then after some time there're thousands of duplicates marked for deletion on the server for just a random message specifically from the Inbox folder (neither auto message rules were applied to it, nor was it moved into other folder/mail account manually)?
Android version of EssentialPIM. Keep all your data in sync!
Steve Shank
Guru
Posts: 158
Joined: Sat Sep 08, 2018 3:17 am
Has thanked: 27 times
Been thanked: 11 times

Re: thousands of duplicated emails

Post by Steve Shank »

Yes. That's exactly what I'm saying. It seems to happen only if those two autocheck boxes are checked. There might be a few marked on the server not marked for deletion, but then a few thousand marked for deletion. It might go for 2 or 3 days without happening. It has not yet occurred with a "real" email. One from a client or friend. So far, it has only occurred with an email from a organization I've never heard of, like how to fix my bad knees or something else I consider spam. I'll keep an eye on it more carefully to better describe exactly what's happening. I was surprised that this one was sitting in my inbox and made it past both the server's spam assassin and your Bayesian filter. I don't think I experienced this issue before upgrading to 11, but maybe. I worked back and forth with the hosts people, and they weren't able to figure it out.
Post Reply