Service Crashing

[Skirge]

I've been having this issue for a while now, but I'm hoping to get it resolved. On a seemingly random interval, the HDSentinel service just crashes. Generally, it's after about a week or so. The most recent incident happened after 10 days. Looking through event viewer, I see the following errors occurring around the same time:

Faulting application name: storectrl.dll, version: 4.17.0.0, time stamp: 0x5492ec1b
Faulting module name: RPCRT4.dll, version: 6.1.7601.18532, time stamp: 0x53c3352a
Exception code: 0xc0000005
Fault offset: 0x000b05b1
Faulting process id: 0x19b8
Faulting application start time: 0x01d061c34f849e60
Faulting application path: C:\Program Files (x86)\Hard Disk Sentinel\storectrl.dll
Faulting module path: C:\Windows\syswow64\RPCRT4.dll
Report Id: 8eb8a1f4-cdb6-11e4-9f09-000c299f1e6b

Faulting application name: storectrl.dll, version: 4.17.0.0, time stamp: 0x5492ec1b
Faulting module name: ole32.dll, version: 6.1.7601.17514, time stamp: 0x4ce7b96f
Exception code: 0xc0000005
Fault offset: 0x0013ad59
Faulting process id: 0x2848
Faulting application start time: 0x01d061c3216ffc05
Faulting application path: C:\Program Files (x86)\Hard Disk Sentinel\storectrl.dll
Faulting module path: C:\Windows\syswow64\ole32.dll
Report Id: 6085c8c8-cdb6-11e4-9f09-000c299f1e6b
(Note that this event was previously reported and investigated by you with no resolution. At the time, there didn't appear to be any issues with HDS otherwise. I'm not sure if the other events were also happening then.)

The HardDiskSentinelService service failed to start due to the following error:
The service did not respond to the start or control request in a timely fashion.

A timeout was reached (30000 milliseconds) while waiting for the HardDiskSentinelService service to connect.
(This one happened 3 times, which I assume is from Windows trying to restart the service as many times as it is configured to.)

[hdsentinel]

Thanks for your message and excuse me for the troubles.

However, I can confirm that the issue is not really related to Hard Disk Sentinel, at least its main service and the application in general.

The crashing file storectrl.dll is related to the RAID controller you are using: it contains the special detection code which is required to access hard disk details on LSI / Intel / IBM / Dell RAID controllers.
It seems that after some time, the RAID controller does not respond as should (or respond very-very slowly): and if this happens, Hard Disk Sentinel need to wait (even forever).
In application mode, then the software would seems frozen (waiting for the response of the controller) - and in service mode it may also stops (or even can't start) as the external module is not responding, indicating that there is an issue with the RAID controller or its current configuration.

I'd suggest to use Report menu -> Send test report to developer option in Hard Disk Sentinel (when it's running in application mode) so then it is possible to check the actual situation, the RAID controller itself and verify what may cause the slower response of that.
Also I'd suggest to check if there is an updated firmware and/or driver available for the RAID controller, as it may help.

And if you use older version, please update Hard Disk Sentinel to latest 4.60, as new versions always have improved compatibility for various RAID controllers.

[Skirge]

Thanks for the info. I sent a report as you requested. I am already running the latest HDS software, but I believe there is updated firmware for the controllers I use. I will need to schedule some downtime to perform that upgrade.

[hdsentinel]

Thanks for the report!
I checked and yes, now I see things are fine: the status could be detected.
To be honest, not sure if the issue is related to the firmware or VMWare itself, but it worth to try the update when possible.

Also if you ever experience troubles next time, I'd suggest to try manually starting HDSentinel.exe (to start in application mode, not in service mode).
Then we can see if the detection of this controller (and the drives) can be completed (just may require very-very long time, more than what Windows allows for the service to respond) - or if the controller seems completely not responding.
And if Hard Disk Sentinel starts then (even after longer time, eg. minutes) please use Report menu -> Send test report to developer option again, I can compare the differences with the current status, verify what may cause the delay (eg. if there is a drive where timeout occors or so).

[Skirge]

It usually stops responding after about a week, so I'm sure I'll get another chance. I can tell you that when it does happen, I can start the application itself without issue. From there, I can go to Integration, opt to use HDS as a service again and it starts working once more. In fact, I'm pretty sure the service was crashed when I went to send you the report, so I started up the UI and sent the report, followed by telling HDS to go back to service mode. I've never had to restart the computer to get HDS working again, which is probably why I've been willing to live with it this long.

I'm attempting to flash the cards this week, but I need to refresh my memory on the process due to just how convoluted it is.

[Skirge]

I was able to get the time to upgrade the flash on both RAID cards to the latest version (P20) this evening. It's too soon for the service to crash, but I am still receiving the 2 Windows Errors in Event Viewer every 30 minutes, as mentioned in the first post of this thread. I'll post again if/when the HDS service crashes. Due to the frequent crashing, I have daily status emails sent so that I'll know it crashed if I do not receive one.

If you'd like, I can send another report so you have a baseline with the current firmware prior to the service crashing and then I'll send another one once it crashes again.

[hdsentinel]

Thanks for the news, I see.

I suspect the slower response of the RAID controller is not really related to the firmware, but there may be (even minor) difference.
So yes, please send new report: each new reports can help - as they may give ideas, different results.

The problem is that it seems your controller(s) (or one or more devices on that) usually respond well - but from time to time a very long time passed with no response at all, causing the Windows event log entries.

If possible, I recommend to try

1) use Hard Disk Sentinel in application mode, not in service mode.
In that case, even if the RAID controller responds slower than expected and the software need to wait for that, Windows may not record it as problem and then Hard Disk Sentinel can continue running and monitoring. So we can see if there is a difference in that situation.

2) specify less frequent detection cycles: on the Configuration -> Advanced options page, move the "Detection Frequency" slider to right (eg. to detect once per every 2-4 hours only).
Then Hard Disk Sentinel will generally ask the RAID controller less frequently - and we can also see if there is any difference.

[Skirge]

Thanks for the ideas. I sent you another report and made the 2 suggested changes. I didn't want to push the frequency to 4 hours, just in case a drive starts going bad, so I upped it to 2 hours and left the application running. If it continues running without issue for a few weeks, I'll drop the frequency back down, but keep the app running and see if anything bad happens.

MINOR UPDATE: I failed to point out in an earlier reply that when I performed the firmware update I noticed that the two RAID cards were on different versions. The one with only 2 drives (the one with the issue) had firmware from 2012, while the other card had firmware from 2011. They're both now on firmware from 2014 and I'm somewhat optimistic that the firmware update might resolve the issue.

UPDATE #2: The two faulting applications appearing in Event Viewer are still there, but have slowed to the frequency HDS was pushed to (2 hours). What I don't understand is what determines which application is going to fault. For example, here are the most recent ones:

Code: Select all

RPCRT4.dll	Error	3/26/2015 12:10:28	Application Error	1000	(100)
ole32.dll	Error	3/26/2015 10:08:44	Application Error	1000	(100)
ole32.dll	Error	3/26/2015 08:00:02	Application Error	1000	(100)
ole32.dll	Error	3/26/2015 06:05:14	Application Error	1000	(100)
ole32.dll	Error	3/26/2015 04:03:30	Application Error	1000	(100)
ole32.dll	Error	3/26/2015 02:01:46	Application Error	1000	(100)
ole32.dll	Error	3/26/2015 00:00:02	Application Error	1000	(100)
ole32.dll	Error	3/25/2015 23:17:38	Application Error	1000	(100)
RPCRT4.dll	Error	3/25/2015 21:15:52	Application Error	1000	(100)
RPCRT4.dll	Error	3/25/2015 19:14:03	Application Error	1000	(100)
ole32.dll	Error	3/25/2015 17:12:10	Application Error	1000	(100)
RPCRT4.dll	Error	3/25/2015 15:09:40	Application Error	1000	(100)
RPCRT4.dll	Error	3/25/2015 13:07:56	Application Error	1000	(100)
RPCRT4.dll	Error	3/25/2015 11:06:12	Application Error	1000	(100)
ole32.dll	Error	3/25/2015 09:04:28	Application Error	1000	(100)
ole32.dll	Error	3/25/2015 08:00:02	Application Error	1000	(100)
RPCRT4.dll	Error	3/25/2015 07:02:38	Application Error	1000	(100)
RPCRT4.dll	Error	3/25/2015 05:00:52	Application Error	1000	(100)

With the limited knowledge I have about how HDS works, the faulting dll seems random to my eyes. Does the frequency of each mean anything at all? Is it worth tracking them like this?

Also, here are my drives:

Image 010.png (9.57 KiB) Viewed 38272 times

Disks 12 and 13 are the only drives on the 2nd RAID card and those are the ones which I believe are failing to respond. As you can see, they are two completely different brands, but I also have identical or similar models on the other card. In the past, I also tried swapping out their cable with a new one with no change in behavior. They are also physically located in 2 different drive cages in the server, each fully populated with 4 other drives. When the service crashes, disks 12 and 13 disappear from the HDS status image above.

When the next issue arises or the next time I take the server down, I'll move the cable to the other port on the RAID card to see if that does anything.

Just trying to provide as much detail as possible to hopefully figure out what the cause is.

[Skirge]

Update #3: Update frequency is back down to 5 minutes (default) and the program has not crashed for about a week so far. So, it would appear that the only time it crashes is when running as a service.

[hdsentinel]

Thanks for the update !

Generally, when service mode used, the detection code (including the code which detects the hard disk status of the RAID controller) runs in the context of the SYSTEM account. Not sure if the controller driver or VMWare itself, but it seems something does not like this method when running for longer time - that's why things may be diffrent in application mode.

[Skirge]

Everything is back to the way it was, except that it's running as an application using my user account (which is an admin) and the application still has not crashed. I know that when you choose integration via the GUI, it gets set up to run under the system account, as you alluded to. I see that the service runs the standard .exe file. Is there a command or parameter to set it to run as a service when it starts? I'd like to try setting HDS to automatically run as a service using an admin account to see if that that resolves the issue.

[hdsentinel]

Generally, NT services (including Hard Disk Sentinel when runs in service mode) run in the context of the system account. Services can't run in the context of any other (real user) account, even if that account is administrator (or has admin user rights).

Yes, the same EXE runs in both application and service mode - the difference is the user (and the startup mode as system account does not "log in"). So when you switch to service mode - then it runs in the context of the system account.

I'm sorry, but I know no methods to configure any service to run in the context of a different user.

[Skirge]

Not being a programmer, I must be misunderstanding what you're attempting to explain because I have numerous services installed under Windows using accounts I specifically created for those services and they are admin accounts. In most cases, I did this so that the service would have access to network shares. For programs which didn't have the option to install as a service, I originally used a program called SRVANY to create the service, but now I use NSSM (Non-sucking Service Manager) to do so.

[hdsentinel]

Thanks for the info.

I did not know these tools, not sure how they work, but this sounds interesting.

If I understand correctly, these tools can help to start any application as service - and configure the user for that.
So what happens if you try to configure HDSentinel.exe with them (completely ignoring the "use as service" mode of Hard Disk Sentinel)?

[Skirge]

My pleasure! You understand perfectly. I went ahead and tried to install this using NSSM under an admin account and it worked, just as well as the integration part does. By that, I mean that I see the same oddity with NSSM as I've seen with the integration, which is that HDS only seems to show disks 4 through 11 on the statuswin.bmp graphic and skips disks 12 and 13. I wonder if this is indicative of what's causing the built-in service to fail, as well. When the full GUI is running, disks 12 and 13 show up as expected in the statuswin.bmp graphic.

I did some further testing and enabled disks 1 through 3 to display (they're VMware Virtual Disks, so I don't monitor them), just to determine if the graphic was limited to 8 disks when running as either service or GUI. Disk 1 through 11 displayed just fine on the graphic in all scenarios, but 12 and 13 were missing when HDS was running via NSSM or via the integration.

I also removed disk 4 from monitoring in both the statuswin.bmp and from the left pane of the GUI to see if disk 12 might show up when running as a service, but it did not.

Finally, I mentioned this in a previous post and just verified that disk 12 and 13 are the only two disk on my second RAID controller. I haven't taken the server down recently to try the 2nd port on the 2nd controller or to swap the cable entirely. But, I don't know why the GUI would be able to see the drives and the service wouldn't. I'm open to additional ideas/suggestions.

[hdsentinel]

> just to determine if the graphic was limited to 8 disks when running as either service or GUI.

No, I can confirm that there is no such limit at all, all detected disks should be displayed.


> Finally, I mentioned this in a previous post and just verified that disk 12 and 13 are the only two disk on my second RAID controller.
> I haven't taken the server down recently to try the 2nd port on the 2nd controller or to swap the cable entirely.
> But, I don't know why the GUI would be able to see the drives and the service wouldn't.

Sorry for the confusion, but there is no special "GUI".
There is an "application" mode - when Hard Disk Sentinel runs as any standard application, showing full user interface and with all functions, features.
And there is a "service" mode - where Hard Disk Sentinel runs as a NT service.

Generally the difference is how it starts and the context where the software runs (in the context of the user OR the system account).
The difference is also that all detection codes (including the external modules which designed to access the RAID controllers and drives connected to them) run in the context where Hard Disk Sentinel runs.

So I suspect the problem is with the code which access your RAID controllers (including the 2nd controller where you0 see no status in service mode) when we run in the context of the system account.

Just not 100% sure if the issue is related
- to the code itself
- VMWare which may cause issue in this case
- limitation of the RAID controller or its firmware
- limitation of the Windows driver of the RAID controller

So personally yes, I suspect it may be a good idea to try running the service in the context of the user account - just I never tried it personally (as I thought it is not possible at all).

[Skirge]

Thanks for the info. I've been running HDS in application mode using my primary admin account for some time now and that's the way I've managed to prevent it from crashing for over a month. Then, I installed the service using NSSM and had it use the same primary admin account. Using this method, HDS does run in service mode using the primary admin account, but disks 12 and 13 do not appear in the statuswin.bmp graphic, just as when it's running under the system account using the integration part of HDS. Given the similarities, I suspect that the service will begin crashing if I leave it running like this, but I'll give it a try.

Any other ideas on how we might be able to track down where the issue lies? I'd even be willing to set up a Teamviewer session so you could check out the production system, if you thought that might help.

[hdsentinel]

To be honest, currently I also have no idea about the difference between the detection code of that particular RAID controller.
In theory, there should be no difference if the software runs in application mode and service mode (and performs the RAID controller access in these modes) but maybe Windows, VMWare, a driver or something causes a difference.

I'd suggest to
- use Report menu -> Send test report to developer optin now in application mode (when things seem fine)
- send the file LSIInfo.TXT from the folder of Hard Disk Sentinel to info@hdsentinel.com
- then when you use in service mode and it starts, send the updated LSIInfo.txt file again

as then it is possible to check the differences between the information detected about these RAID controllers (especially about your 2nd controller).
If we see the differences and possible errors reported by the controller, it may give ideas.

[Skirge]

I was away for a little bit, but I did send the information a couple days ago.

[hdsentinel]

Thanks, yes, I can confirm the reports and the information received, still examining and checking, will let you know any results soon.