IMPORTANT ANNOUNCEMENT REGARDING PMs!
 

As many of you may know by now, we're getting ready to upgrade the site and forum software. In our testing of the import process to the new software we discovered that the PM database is completely out of control. We have never put official limits around PM boxes, but it seems some people are abusing this feature and never deleting anything. This slows down the import process significantly and takes up a lot of room on the server.

Therefore, the new forums will have PM limits installed that will not allow you to have more than 100 total PMs (in and out box combined) at one time or you won't be able to receive additional PMs. Before everyone starts yelling, also you should know that in the new software there are simple buttons where you can download your PMs to a CSV, XML or Text file. So in the future, staying under the limit should not be a problem for anyone. Moderators and Administrators are getting slightly larger boxes based on the fact they have to do corrective communication with users and if you fit into one of those categories, please see the moderator forum for more information.

However, the problem we now have is that we need to get everyone in compliance with this new limit before we do the site and software change. So starting now, everyone needs to work on their PM box and get the total messages under 100 PMs and as quickly as possible so this issue does not keep interfering with our testing process for the site conversion. If your PMs are not cleaned up before the site conversion, we will delete everything but your latest 100 PMs to make sure everyone is in compliance.

For most people, these limits won't even affect them. Less than 2% of our registered members have over 100 PMs, but those 2% make up over 2/3rds of our PM database, making their total number of PMs to be about 69,000 PMs! The highest person has over 3,000 PMs which is just freakin ridicuous. So it's time to clean it up folks, or we'll be cleaning it up for you.

Please contact me if you have any questions.

Re: IMPORTANT ANNOUNCEMENT REGARDING PMs!
 

Wow, that's a lot of PMs. I like to think I'm reasonably popular around here, and I don't think I've gotten even 100 PMs from the forumites. (plenty of notifications from admin and such, but not conversational PMs)

"[less than 100] or you won't be able to receive additional PMs."
Wouldn't it be better to have the oldest ones be deleted at that point, rather than (effectively) the newest ones being deleted?

Re: IMPORTANT ANNOUNCEMENT REGARDING PMs!
 

3000 PMs?? Yikes! Good luck trying to pick 100 to save out of that, whomever you are!

Re: IMPORTANT ANNOUNCEMENT REGARDING PMs!
 

I think this is more of a problem in the Dominions community because of the amount of multiplayer conversation going on in PMs. But I wanted to make sure you guys were also aware of this change.

I was even guilty of keeping way to much history stuff for moderation purposes, but I just cut and pasted the ones I needed to keep into a word document. Then again, I had a lot of admin alerts that were easy to delete, so I had no were near the numbers our top ten have in legitimate messages.

Re: IMPORTANT ANNOUNCEMENT REGARDING PMs!
 

SJ: Actually it's caused by the forum software, I think it simply refuses to let you have more unless you delete some, but I will look into it to see if there is any other settings.

Re: IMPORTANT ANNOUNCEMENT REGARDING PMs!
 

Fair; I'm off to check my PMs.

132 total. Hmm...

There. 100 total.

Re: IMPORTANT ANNOUNCEMENT REGARDING PMs!
 

62 in my inbox, 2 in outbox...

The oldest from May of 2004... ok it may not be over 100, but most of these can go!

Re: IMPORTANT ANNOUNCEMENT REGARDING PMs!
 

I cleared my PM bank. sorry about not doing that sooner, I didn't realize that I had over 200. Yikes!

Re: IMPORTANT ANNOUNCEMENT REGARDING PMs!
 

LOL, well we're only cracking down on those with totals over 100, but any messages you can get rid of will speed up the final import, which means less down time......so by all means purge whatever you don't want! http://forum.shrapnelgames.com/images/smilies/laugh.gif

Re: IMPORTANT ANNOUNCEMENT REGARDING PMs!
 

91 in, 6 out... phew, just made it! http://forum.shrapnelgames.com/images/smilies/wink.gif

Though I suppose I should start deleting some in case someone decides to send me four more! Some of my PM's date back to 2003 http://forum.shrapnelgames.com/images/smilies/eek.gif

Out of curiousity, what forum software are you upgrading to? A new version of UBB, or Something Completely Different? And what's happening to the rest of the site?

edit: P.S. in the new site, are the Shrapnel logos on the page header going to be links back to the main site? That would be handy... http://forum.shrapnelgames.com/images/smilies/wink.gif

Re: IMPORTANT ANNOUNCEMENT REGARDING PMs!
 

Going to something completely different. UBB has been going downhill for years now and parts of our version are just plain broken and it's not even worth upgrading to stay on software that is behind the times. We're moving to what is basically the standard among forum software right now which is vbulletin. I know I will hear some ahem complaining (to put it nicely) about the change in software as I know there are some who love phpbb since it's a free product and feel we should move to that, but we went with what we felt will offer us the most as far as flexibility, support and active modding community.

This should bring a lot of cool changes for the users and for the company. The whole site is getting a redesign as well courtesy of Tim. In addition, press releases are being 'automated' to a certain extent so that the news page is not being manually updated (you may have already noticed the new press release forums I have been putting up recently).

The only thing staying the same is the store.

Re: IMPORTANT ANNOUNCEMENT REGARDING PMs!
 

So long as it isn't UBB, we are happy. http://forum.shrapnelgames.com/images/smilies/wink.gif

Re: IMPORTANT ANNOUNCEMENT REGARDING PMs!
 

But Fyron, how are you ever going to get to 18,494 posts again? http://forum.shrapnelgames.com/image...ies/tongue.gif

Re: IMPORTANT ANNOUNCEMENT REGARDING PMs!
 

We haven't seen any issues with post counts during the import.

Re: IMPORTANT ANNOUNCEMENT REGARDING PMs!
 

well i'm safe http://forum.shrapnelgames.com/images/smilies/happy.gif

Message Box Message Box
Send a PM Total Unread
Received Private Messages 0 0
Sent Private Messages 0

Re: IMPORTANT ANNOUNCEMENT REGARDING PMs!
 

lol.. yeah, I think I've sent and received a total of 20 in the last 5 years (but then, I lurk a lot).

Re: IMPORTANT ANNOUNCEMENT REGARDING PMs!
 

* Looks at post count.
* Looks at join date.

...Ya think, TT? http://forum.shrapnelgames.com/images/smilies/laugh.gif

Re: IMPORTANT ANNOUNCEMENT REGARDING PMs!
 

I had one from AT http://forum.shrapnelgames.com/image...es/redface.gif a couple of years ago. BUT I deleted it! http://forum.shrapnelgames.com/images/smilies/eek.gif Sorry AT! http://forum.shrapnelgames.com/images/smilies/Sick.gif

Re: IMPORTANT ANNOUNCEMENT REGARDING PMs!
 

/me points Dominions community to a new and amazing thing called IRC

Re: IMPORTANT ANNOUNCEMENT REGARDING PMs!
 

I wonder if it's possible to limit PM's to a certain amount. like 150 characters per message...

Re: IMPORTANT ANNOUNCEMENT REGARDING PMs!
 

To tell you the truth, I don't think it's the size of the PMs themselves that is the issue, it's more about the total number of entries in the database that is the issue. They are a relational databse and therefore has to link various different tables and entries together for each PM and that slows down the import significantly.

I mean in theory we could have a more PMs than we have currently if every user had the limit I am imposing, but in reality it was less than 200 people who had more than 100 PMs with the top 10 people on the list taking up almost 20% of the PM database alone. Some people like to save every single PM including things like read receipts and the like at if you're going to be that much of a PM hoarder, you need to copy and paste them to a word document and save them on your own machine.

I was even guilty of saving too many moderation PMs for history's sake and although they are good to keep (so I know the history we've had with certain users), they don't have to be kept in their current form, so I saved them in a document and deleted them from the forum software.

Re: IMPORTANT ANNOUNCEMENT REGARDING PMs!
 

I have no actual knowledge of how this particular database is set up, but that doesn't make much sense to me unless the database is poorly designed or certain kinds of redundant checking are involved. Is there, by any chance, a constraint that all PMs have to be unique to prevent accidental double-sending or some such? I could see that making an import take forever by forcing the database to compare each PM to all previous PMs before inserting it. If such a constraint does exist, removing it temporarily for the import should speed things up dramatically.

Re: IMPORTANT ANNOUNCEMENT REGARDING PMs!
 

I don't understand the full way that it is set up, I only know tid bits of what Richard has explained to me and I know UBB has a lot of backwards way of doing things so who knows. All I know is the import started out taking the better part of a whole day, Richard has done a lot of research and found a way to speed up the majority of the process, but the PMs database still takes 4-5 hours (as much as all of the other parts of the import combined) and last time it failed totally towards the end of the PM import.

Frankly we should have put some contraints around PMs before now, but it is what it is and we're dealing with it now.

Re: IMPORTANT ANNOUNCEMENT REGARDING PMs!
 

I notice you mentioned in the Dom 3 forum that about 100000 PMs were taking longer than everything else put together. Considering there are over 5 times that many posts alone, plus the user list and various other data on top of that, I can't think of any reasonable explanation for it other than an index or constraint requiring excessive redundant checking.

My first guess would be that the PM content column has a unique constraint but no index, which would force a string comparison of each and every PM with each and every other PM. This isn't that big a deal in normal use when you only have to worry about one new PM at a time, but doing 100k at once takes forever. Removing the constraint in this case would drop the import time immensely. Creating an index for it instead would greatly reduce the number of comparisons required, but the index itself would take some extra resources to maintain. Assuming my guess is correct, I would first try creating an index for the column and see how much that helps. If it still takes too long, then remove the constraint entirely before importing and then see if it can be added back in a reasonable amount of time.

Edit: Then again, I don't know that much about how most databases handle indexes on strings specifically, so it's possible that it does have an index on PM content and that removing that index could help. I doubt this is the case, but it might be possible.

Re: IMPORTANT ANNOUNCEMENT REGARDING PMs!
 

Like I said, I don't know all the finer points of this sort of thing. I do know what Richard did to speed up the main part of the import was to take off the indexing, but I don't think the PMs work the same way and we still haven't figured out what the total time will be once we put the index back on at the end because we haven't had a complete, successful import to try it on yet since the PM part failed last time.

I don't really like talking about all of this because it's not my area of expertise and I only know a little bit about everything Richard is doing. He's the back end, I'm the one putting in modifications, tweaking settings, importing smilies, etc.

Re: IMPORTANT ANNOUNCEMENT REGARDING PMs!
 

For someone who's still has one foot in the 20th century, what the heck is a PM?

Re: IMPORTANT ANNOUNCEMENT REGARDING PMs!
 

Private message http://forum.shrapnelgames.com/images/smilies/happy.gif

Re: IMPORTANT ANNOUNCEMENT REGARDING PMs!
 

"Private Messages" existed for much of the latter decade of the 20th century. http://forum.shrapnelgames.com/images/smilies/wink.gif

Re: IMPORTANT ANNOUNCEMENT REGARDING PMs!
 

...So less than 10%?

Re: IMPORTANT ANNOUNCEMENT REGARDING PMs!
 

Oh, you are all very very wrong.

Private Messages have existed for as long as there has been children passing private messages in class. This of course was always accompanied by the Department of Homeland Security - formerly known as Teachers - whom would intercept and read them randomly and to your embarrassment.

Re: IMPORTANT ANNOUNCEMENT REGARDING PMs!
 

Didn't say they didn't exist before the 90s! They were just horribly inefficient, dependent upon the sneakernet.

Re: IMPORTANT ANNOUNCEMENT REGARDING PMs!
 

Give the users an option to have their oldest PMs drop off the bottom of the list. Then a new PM replaces the oldest PM, once they have 100.

It removes the need for maintenance labor from users (which is virtually guaranteed not to happen 100% of the time). And it could be set as a default for new accounts to minimize the problem.

Two options like so:
+ Replace oldest PM when inbox is full (100 PMs)
+ Refuse delivery when inbox is full (100 PMs)

Some users would prefer to maintain their own, though, so it's probably good to allow personal control of trashing for those who want to.

Re: IMPORTANT ANNOUNCEMENT REGARDING PMs!
 

How many times can you read a PM? For goodness sake, copy the thing to a txt file and save it on your own computer if you want to keep it for prosperity's sake!!! All PM's should be dropped when a certain amount is given (say 100 PM's) and new ones come in. The oldest is deleted and all new ones take their place. It's a simple concept, "Move it, OR lose it" http://forum.shrapnelgames.com/images/smilies/cool.gif

That's my story and I'm sticking to it!!!

Re: IMPORTANT ANNOUNCEMENT REGARDING PMs!
 

how about get rid of it completely and just have a form that would directly email users and get it off the server completely.

Re: IMPORTANT ANNOUNCEMENT REGARDING PMs!
 

Too many e-mails already.

Re: IMPORTANT ANNOUNCEMENT REGARDING PMs!
 

Limiting the size of a PM box on a forum is a good thing.