Sign in with
Sign up | Sign in
Your question

2012 r2 server shuts down every thursday

Tags:
  • Servers
  • Business Computing
  • Software
Last response: in Business Computing
Share
August 28, 2014 10:59:14 AM

Hello,
I recently took over responsibility for a company's Tech related issues and one issue that has plagued me is that, one of the property's servers shuts itself off every Thursday. I have looked to see if it was scheduled and it was not. Looked threw any of the programs to see if they had done something to make it shut off or even reboot, found nothing. Finally i have searched the event logs and its not even showing up that it is shutting down or rebooting. Normally this would not be an issue but this server has our main reservations software on it and when it goes down no one can make reservations or check people in. The server is running win server 2012 r2. It has all it's updates and it has been doing this from what i hear sense it was first installed. I have searched the Microsoft forums for anything related to this and haven't found anything yet. I even went as far to contact the company that installed it, no surprise they didn't have a fix ether. Any help would be greatly appreciated.

More about : 2012 server shuts thursday

August 28, 2014 11:35:20 AM

Is there something like a generator test every Thursday affecting power. My boss one time set up a script(run from a script server) that rebooted a couple of servers(by IP). He replaced a server, and reused one of these IP's and it took hims several months to figure out what was causing the new server to reboot.
m
0
l
August 28, 2014 12:11:02 PM

ss202sl said:
Is there something like a generator test every Thursday affecting power. My boss one time set up a script(run from a script server) that rebooted a couple of servers(by IP). He replaced a server, and reused one of these IP's and it took hims several months to figure out what was causing the new server to reboot.


I just looked threw the server for any script files. The only ones that i found were the .wsf and there is no generator for this property. I also looked threw the ups that the server is connected to and it hasn't been set to do anything with the power ports, in fact the ups is set to its factory default.
m
0
l
Related resources
August 28, 2014 3:45:31 PM

Is it the same time every Thu or just x number of hours after it was brought up on the previous Friday? Is there a power schedule in the BIOS or IPMI that could be shutting it down?
m
0
l
August 29, 2014 9:34:04 AM

kanewolf said:
Is it the same time every Thu or just x number of hours after it was brought up on the previous Friday? Is there a power schedule in the BIOS or IPMI that could be shutting it down?


It looks like its about the same time every Thursday morning. I have went and looked for anything that could be a IPMI and found nothing. the server is basically just bare min on software. I will look at the bios when the next slow day hits.

m
0
l
August 29, 2014 12:56:36 PM

I had an issue similar to this but it turned out to be the UPS was running a weekly test and failing causing the server to lose power. Does the server have redundant power supplies?
m
0
l
August 29, 2014 3:36:04 PM

@Jeff-J -- I have also had that problem. To see if that REALLY is the problem, during a non-critical time, login to the UPS and manually run the self-test. Mine was instantaneous. If that is the case you need to order new batteries immediately and disable the self-test until the new batteries are installed.

If the time on the UPS isn't synced to NTP, then the schedule it "thinks" it is running might not match to wall clock time and the schedule might say "Tuesday" but the clock is that far off....
m
0
l
August 29, 2014 10:35:33 PM

Alright guys i will test this out on the next slow day and let you know if that fixes the issue I really hope its something that simple.
m
0
l
September 3, 2014 7:24:41 AM

Ok, i looked into the UPS config and did not find anything that would restart it. Also found out that the server does have redundant power supplies. As for the bios....haven't been able to take the server down yet.
m
0
l
September 3, 2014 7:47:45 AM

So the UPS doesn't have a periodically scheduled self test?
m
0
l
September 3, 2014 8:22:48 AM

kanewolf said:
So the UPS doesn't have a periodically scheduled self test?


It has an option for one but, its not enabled.
m
0
l
September 3, 2014 10:42:40 AM

Are both power supplies plugged into the UPS?
m
0
l
September 3, 2014 11:44:39 AM

I agree with @Jeff-J, if BOTH power supplies are on the UPS, put one on commercial power and one on the UPS. That will eliminate the UPS as the culprit. You will still be protected from a commercial power outage, but you will also be protected from a UPS outage ...
m
0
l
September 3, 2014 12:01:38 PM

Alright i will do that. and give ya an update tomorrow
m
0
l
September 4, 2014 9:05:01 AM

Ok so i moved 1 over the power cords to a power port that is not on the ups. And it still rebooted.
m
0
l
September 4, 2014 10:22:34 AM

Have you checked scheduled tasks?

On the server, open a command prompt as administrator and type:

at <return>

This will either return a list of scheduled tasks or a message that there are no scheduled tasks.


Alternatively you can run the at command from a remote machine:

at \\<computername>

m
0
l
September 4, 2014 10:55:41 AM

ex_bubblehead said:
Have you checked scheduled tasks?

On the server, open a command prompt as administrator and type:

at <return>

This will either return a list of scheduled tasks or a message that there are no scheduled tasks.


Alternatively you can run the at command from a remote machine:

at \\<computername>



It looks like it is POSSIBLE to have a group policy control power off, but it seems like it would leave a scheduled task. Not sure about the specific mechanics on this one....
m
0
l
September 4, 2014 11:13:12 AM

ex_bubblehead said:
Have you checked scheduled tasks?

On the server, open a command prompt as administrator and type:

at <return>

This will either return a list of scheduled tasks or a message that there are no scheduled tasks.


Alternatively you can run the at command from a remote machine:

at \\<computername>



Yes, I have looked threw the scheduled tasks to see if there were any scheduled task. there are none that will restart the computer. about the only thing in scheduled task is for it to look for updates. And even then updates are set to download but not install and thus far that part has been working as expected.
m
0
l
September 4, 2014 11:26:20 AM

kanewolf said:
ex_bubblehead said:
Have you checked scheduled tasks?

On the server, open a command prompt as administrator and type:

at <return>

This will either return a list of scheduled tasks or a message that there are no scheduled tasks.


Alternatively you can run the at command from a remote machine:

at \\<computername>



It looks like it is POSSIBLE to have a group policy control power off, but it seems like it would leave a scheduled task. Not sure about the specific mechanics on this one....


Group policy is installed but not setup. The server is part of a loosely connected group of computers. And the only rights that any of the computers that are on the same network as the Server, have rights to the program via the programs interface. Other than that the server is isolated. And the only ones that have access to the server itself are myself, and the owner. Basically the server has only what it needs to run a program called RoomMaster. And as for you remark above, it is possible to have gp reboot the server without a scheduled task, as scripts do not need to be connected to scheduled task to run. And with gp you can just point it to the scripts to get a specific action. Atleast thats how i got it to work on my test server at home. Also if i am not providing enough info, please feel free to ask and I will be more than willing to get the info you need or want :D 
m
0
l
September 4, 2014 11:48:14 AM

Have you had a chance to look in the BIOS for a timed event? And if you could list the server specs it could help, there could be a known issue with a component.

Is there anything that happens in or around the office on Thursday mornings? Janitorial work, landscaping, garbage trucks, warehouse deliveries, etc?

If the reboot happened even with diversified power:

Are both power supplies rated to handle the server by themselves?
Are both power supplies good?
Is the UPS battery still good?
How long does it last on battery power?
Is that longer than any time the server's cut off from power?
Is the UPS on the same circuit as the server?
m
0
l
September 4, 2014 12:41:04 PM

I would also check and see if any memory dump files have been created in %SystemRoot%\MEMORY.DMP, also go to system properties, and on the advanced tab select startup and recovery, and uncheck automatically restart for system failure. This way if for some reason the server is BSOD this will stop it from automatically rebooting and you can see the error on the screen.
m
0
l
September 4, 2014 1:26:05 PM

Saberus said:
Have you had a chance to look in the BIOS for a timed event? And if you could list the server specs it could help, there could be a known issue with a component.

Is there anything that happens in or around the office on Thursday mornings? Janitorial work, landscaping, garbage trucks, warehouse deliveries, etc?

If the reboot happened even with diversified power:

Are both power supplies rated to handle the server by themselves?
Are both power supplies good?
Is the UPS battery still good?
How long does it last on battery power?
Is that longer than any time the server's cut off from power?
Is the UPS on the same circuit as the server?


]Have you had a chance to look in the BIOS for a timed event?

Ok i managed to get a little bit of time just a few min ago to take the server down and get into the bios. The only settings that were telling it to do anything involving, rebooting or time out "aka powering back on" were all turned off. It is set to just turn back on after a power loss.

"And if you could list the server specs it could help, there could be a known issue with a component."

I was unable to keep it down long enough to get a full list, tho i was able to get some of the components, they are as follows,
2 Intel Xeon E5606 x4 cpu, Win server 2012 r2 64bit os, 2x Intel MegaSR SCSI hdd, 1x WD 3200AAK external hdd, 12gb ram, 2x 1000w psu

"Is there anything that happens in or around the office on Thursday mornings? Janitorial work, landscaping, garbage trucks, warehouse deliveries, etc?"

Not unless you count the occasional gest that leaves at the crack of dawn.

"Are both power supplies rated to handle the server by themselves?
Are both power supplies good?
Is the UPS battery still good?
How long does it last on battery power?
Is that longer than any time the server's cut off from power?
Is the UPS on the same circuit as the server?"

I would say they are sence it doesnt take even 1 1000kw psu to run it.
Yes both psus are good and in working order
Yes the UPS is still good, last time I was able to spend more than a few min with the server I had it powering the server for about 30min.
I do not know, I only get calls when they notice that its down, and they don't notice that till after 2am.
Yes, infact the server is plugged into the battery backup ports on the UPS.
I will try to get more info on the server, but we are hitting a busy time for us, so that may be a while. i hope this info helps.
m
0
l
September 4, 2014 1:28:01 PM

jeff-j said:
I would also check and see if any memory dump files have been created in %SystemRoot%\MEMORY.DMP, also go to system properties, and on the advanced tab select startup and recovery, and uncheck automatically restart for system failure. This way if for some reason the server is BSOD this will stop it from automatically rebooting and you can see the error on the screen.


No memory dump files have been created. And i have done as you suggested and turned off the auto reboot for system failure.
m
0
l
September 4, 2014 1:28:57 PM

Another question is what is the make and model of your server? Is it an HP or Dell and how old is the server?
m
0
l
September 4, 2014 1:42:13 PM

Have you also looked externally?

Are there any users with credentials enough to issue a remote shutdown command?
m
0
l
September 4, 2014 2:05:04 PM

jeff-j said:
Another question is what is the make and model of your server? Is it an HP or Dell and how old is the server?


Certera P2400v2
Certera is the maker and P2400v2 is the model
m
0
l
September 4, 2014 2:10:08 PM

ex_bubblehead said:
Have you also looked externally?

Are there any users with credentials enough to issue a remote shutdown command?


No, only myself.
m
0
l
September 4, 2014 2:12:29 PM

There does not seem to be any technical info available about that server. It seems to be a house brand for "PCsForEveryone" ... No docs, and only 3 Google references all to the same website.
m
0
l
September 4, 2014 2:15:37 PM

Yea I was afraid of that. At least it is proving to be a good challenge lols.
m
0
l
September 4, 2014 3:35:24 PM

You might need to talk to someone at PCforEveryone, then. If you can get a hold of anyone. You might have firmware that's buggy and they could have a new version you can push to fix this.
m
0
l
September 4, 2014 9:51:16 PM

Are there any non-power related scheduled tasks that run around that time on Thursday? I am way out of my league with IT Pro servers, but I've had desktop PCs schedule maintenance and then the maintenance caused a BSOD, leading to it rebooting and getting stuck in a cycle when the "updated" drivers failed to load. It won't hurt to look anyway. Also have you looked at the event logs to see if any kernel errors or other critical errors occurred around the same time? (I assume Windows Server has this just like desktop windows).
m
0
l
September 5, 2014 7:23:51 AM

I checked out the server and it uses a standard Intel S2400SC2 server board. You can download the latest firmware update from Intel's website for the motherboard https://downloadcenter.intel.com/Detail_Desc.aspx?Dwnld.... Is there any RAID cards installed? But I would go with Saberus comment first and give PCforEveryone a call, and hope your server is still under warranty. Also just by the off chance what software are you using for backup?
m
0
l
September 8, 2014 9:11:36 AM

Saberus said:
You might need to talk to someone at PCforEveryone, then. If you can get a hold of anyone. You might have firmware that's buggy and they could have a new version you can push to fix this.


Well at the time of posting this, i have been on the phone with PCforEveryone for 2hrs now, been disconnected 4 times, and been basicaly told that there is nuthing that they can "or will do". I know one company that i will avoid in the future. lols
m
1
l
September 8, 2014 9:16:49 AM

jeff-j said:
I checked out the server and it uses a standard Intel S2400SC2 server board. You can download the latest firmware update from Intel's website for the motherboard https://downloadcenter.intel.com/Detail_Desc.aspx?Dwnld.... Is there any RAID cards installed? But I would go with Saberus comment first and give PCforEveryone a call, and hope your server is still under warranty. Also just by the off chance what software are you using for backup?


Yes there is 1 raid card installed. Had some time last thursday to look further into it when i was in the bios. And sadly contacting PCforEveryone was a flop. They basically told me that since it was a beta model at first that they are not supporting it. XD I had many rages at that point lols
m
0
l
September 8, 2014 9:24:13 AM

Snowolf said:
....They basically told me that since it was a beta model at first that they are not supporting it. XD I had many rages at that point lols

I can only imagine the new words that came out of your mouth :) 

Who the H.E. double Hockey Sticks puts Beta hardware into a production environment? I hope that individual is long gone, or is on the way out. If possible it might be time to cut your losses and replace that POS.
m
0
l
September 8, 2014 10:08:28 AM

+1 for telling all this to whoever is in charge of budgets, and say "We need to replace this server! Make sure it is in next year's budget.... Unless you have money that needs to be spent this fiscal year."
m
0
l
September 8, 2014 11:03:14 AM

Lols, I didn't chew them out or say any unpleasant words to them, even tho I was wanting to. I have been on the receiving end of an upset customer too many times, especially when there wasn't anything that i was allowed to do. So that is what keeps me from doing that to another guy who is most likely in the same situation. But I do think that if I take the pos apart and find the models of all the exact parts that that will help me with my next attempt to fix it. Worst case scenario, I just tell my boss that its a flaw with the model.
m
0
l
!