Re: LlamaServer 'sploded (though things will be okay)
 

When the servers go back online, make sure you give all the games extensions if they're close to hosting. Many people were unable to submit turns.

Jazzepi

Re: LlamaServer 'sploded (though things will be okay)
 

Right, back online. In the end I had to throw away the backlog of e-mails. It was 35000 strong due to a morass of error e-mails and despite my best efforts for some reason I couldn't get rid of the things.

All games are set to host at (now + hosting interval), i.e. in 24 hours for a 24-hour-hosting game. You should all hopefully have got an e-mail explaining things anyway.

Thanks again all for your patience and kindness!

On the mysteries of the LlamaServer 'splosion
 

This post is no more than technical ramblings about what happened to the LlamaServer this week.

Okay, so on Monday the mail server, which is run by hostgator.com, apparently went down due to Ike. On coming back online the mailbox had 35000 error messages in it.

The LlamaServer code was written, I guess, slightly foolishly, in that every minute another instance of it would start up, and try to download the mail (I've now fixed this so it couldn't happen again). Normally that wouldn't be a problem, but being as the massive mailbox took essentially forever to download, I guess none of these processes could finish. So I imagine perhaps CPU load climbed to 100%, and then the computer overheated and died.

But the weird thing is how I fixed it. The computer wouldn't boot at all - not a single thing appeared on the screen. So I systematically disconnected all the components and tested if it would boot after each one (I'd given up hope to be honest, and already started researching costs for a new machine on the internet, but I thought I might as well check). I got down to just having the RAM left. There are two 512MB sticks.

I took the front one out. It still wouldn't boot. I took the back one out too. Wouldn't boot (unsurprisingly). I put the one from the back in the front slot. It booted! Miracle! (I can't tell you how happy I was!) Then I put the other one (which at this point I had concluded must be faulty) in the back slot. Still worked! Then I put everything else back in, and it remains fine. So the only net change is that I swapped the RAM sticks over.

Anyone have any thoughts on this oddness? Perhaps it will just remain a mystery. I'd assume a loose connection, but I wouldn't have thought you could get a loose connection just from overheating, and also RAM sticks are very much not loose.

LlamaServer fixed
 

Anyone playing on the LlamaServer should have received the contents of this post as an e-mail. But I thought I would post it here as well, just to make sure people see it.

---
Hello all,

So, on Monday, something funny happened to the LlamaServer. Let's say, for the sake of melodrama, that it exploded (it didn't explode). In any case it stopped working. I suspect the fault was connected with Hurricane Ike, which apparently took out the mail server for a bit. Maybe the LlamaServer's frantic efforts to contact the mail server made it overheat (I think I will post more about this on the forum, for any who are interested).

Anyway, I managed to mysteriously fix it by taking it apart and putting it back together again, exactly as I found it. So everything is now running smoothly again.

Please note that all e-mails sent to the server during the downtime have been lost. So you should send your turns in again if you didn't get a receipt e-mail. However, I've reset the timers for all the games, so you have a bit of time (e.g. for a game on 24-hour quickhost, it'll host about 24 hours from now). Game admins take note: this will have overridden any other delays you may have put in place.

Thanks for your patience over the last few days. Everyone's been very patient and very helpful, which I really appreciate.

Now, back to the games!

llamabeast.
---

Re: LlamaServer 'sploded (though things will be okay)
 

Awesome!

A chorus of :clap: for llama :)

Many thanks for your efforts. I wish there was some secure online service by which one could express more than just words of thanks but also contribute $$$ to the cause.

BTW, MP game Victoria hosted shortly, I wonder if all players did manage to get their turns on time.

Re: LlamaServer 'sploded (though things will be okay)
 

Victoria hosted due to all turns coming in. You can see if anyone staled by going Admin options->Show staling data. I realise this is a bit unintuitive, I'm thinking of rearranging the admin options for which you don't need an admin password onto a different and more obvious page.

In response to a number of startlingly generous requests, I think I'm going to look into adding a Donate button to the page. I feel a little uncertain though... there is certainly no need for any donations, although I guess I certainly wouldn't object to people offering a bit towards the initial cost of the server, or even just buying me a beer, so long as they emphatically don't feel obliged.

Re: LlamaServer 'sploded (though things will be okay)
 

OK, the hotfix works perfectly.
Thanks for it, llama :banana:

Re: On the mysteries of the LlamaServer 'splosion
 

It's remotely possible that the extreme heat caused a small amount of stretching of the material - very slight. However, because RAM slots are so snug, when it cooled down, the connectors were still just slightly off, and shorting out, preventing boot. When you released them from their captivity, they were finally able to flex back to their original shape.

Re: LlamaServer 'sploded (though things will be okay)
 

Obliged? No, it'd be a privelege - of course, this means we have to converge in Limeyland for the official dominions 3 drinking contest. As per the relevant international treaties, round 1 is an alternation between shots of Soju, Irish Car Bombs, shots of Jaeger, shots of Tequila, and local entry #1 (which would be something with gin in it, I suppose.)

Be sure that your life can continue for 7 days without you - and keep in mind that the jet lag from a weeklong bender ending when you wake up in Johanesburg wearing portions of a women's IDF uniform (her phone # is written on the small of your back in permanent marker) to cover your new Triad tattoos can be fierce.

Re: LlamaServer 'sploded (though things will be okay)
 

Thanks llama! You should definitely add a donate button. Even if you don't have any costs running the server, you certainly put a lot of work into it and continue to do so. It would be a good way for people to show their appreciation.

I see that most games resumed normally, but I think there may some problem with FallenKingdoms. Before the incident turn 12 was due and four people have already sent their turns in. Now it is still turn 12 but no turns are in. This would be no problem if it weren't for the fact that the admin options show that everybody staled turn 11 (which hosted normally before the hurricane). I tried resending my turn to see whether turn 11 is valid, but I received no mail. Could you look into this please. (Turn resend is not broken, it worked for another game).

Re: LlamaServer 'sploded (though things will be okay)
 

llamabeast:

You should list all beers you can buy in your shop so we can choose what beer to buy for you :)

Re: On the mysteries of the LlamaServer 'splosion
 

Self healing machinery is the only possible solution. Smile, Nod and slowly back away..

Re: On the mysteries of the LlamaServer 'splosion
 

your computer is like Christine the car, and can repair itself. It would go on murderous rampages, except it has no wheels.

Re: On the mysteries of the LlamaServer 'splosion
 

Did you try percussive maintenance before disassembling your box?
http://en.wikipedia.org/wiki/Percussive_maintenance

Re: On the mysteries of the LlamaServer 'splosion
 

In the military we called them Lockheed Fixes. Sometimes just banging it, or taking it apart and putting it back together, would fix the problem. But I agree it was most likely the connections.

However IF flipping the RAM was the fix then the front stick might have problems. I would run a mem check.

Gandalf Parker

Re: LlamaServer 'sploded (though things will be okay)
 

Please do! I'd donate.

Jazzepi

Re: On the mysteries of the LlamaServer 'splosion
 

My guess is... you were lucky. Resetting everything, including the RAM, set the computer back to operational. I would make a bet that this machine is set to implode again at some point though. It just sounds too fishy.

Re: On the mysteries of the LlamaServer 'splosion
 

Nano technology, self repairing machines, its here, now!

Re: llamaserver and e-mail problems
 

"llamaserver and e-mail problems",
"LlamaServer 'sploded (though things will be okay)",
"LlamaServer fixed",
"On the mysteries of the LlamaServer 'splosion",
and now even a "Llama server turn time" thread on first page is a little much for my taste. I'm merging all those threads sans the last one.

Re: LlamaServer fixed
 

I have sent 3 emails to llamaserver over 3 hours using 2 different email sources.

I get no response. As it is getting close to deadline, I am going to delay the game, World in Crisis. Llama, if the turn doesn't show in a timely fashion, could you look into it?

If it does, you can force host, if so inclined.

Thanks

Re: LlamaServer fixed
 

I think it's for all games. Llamaserver did not update for an hour.

Re: LlamaServer fixed
 

I just changed the message to players successfully, so Llama server is running. Someone want to force a hosting? Kickabout is on hold due to calem going AI.

Re: LlamaServer fixed
 

What I'd like to add to this discussion is that I hate power cuts, especially when you get 4 in one day. That's not meant to happen in England.

Re: LlamaServer fixed
 

Thanks llama....

The weird thing was that I could ping / traceroute to your server. I could go to the admin pages. I just could *not* get the turn submitted.

We all know email is best effort....

But, perhaps I could make one small suggestion to the llamaserver.

Have a Env Variable - Hostable?.
Have a keep alive email send - say every 5 min or so.
If the llamaserver doesn't receive its own keepalive email it disables hosting (postpones all games) until 30 minutes after email reception returns.

This would prevent games from hosting in at least some instances where email couldn't be received.

Alternately, you could do keep alive pinging. Choose 2-3 well known ip addy's such as 4.2.2.2. Test for the return. If you can't receive ICMP returns - delay hosting.

My .02

Chris

Re: LlamaServer fixed
 

I've one and only one complaint about Llama server, I tried to donate and the damn thing wants pounds, What the hell, I deal with US dollars. I don't want to do the conversion to whatever you guys use. I bought a gift card to do it and I'm miffed that theres no user friendly conversion to your money. :D

Re: LlamaServer fixed
 

Thanks Bwaha! Apparently each pound is 1.73 dollars.

Chris - the reason you could still ping it is that the web server is a completely different machine to the LlamaServer itself. There's some explanation in the FAQ I think, although it might be outdated.

Anyway, a system similar to what you describe is already in place. The server won't host for as long as it's unable to check its email.

If contact with the mail server goes down, the website is updated intelligently to explain what's happening and hosting times are postponed. Unfortunately if the LlamaServer itself goes down most of the intelligence is lost, so the website just sits there dumbly pretending things are fine (which is what happened yesterday). Happily, power cuts here are pretty rare.

Re: LlamaServer fixed
 

Pretty Slick llama