System, AeXSVC.exe and SQL running 100%

Mike · 05-02-2006, 04:09 PM

I thought I was good at fixing this problem, but the cause here is a new one by me:

I am troubleshooting a Notification Server that is running Veeeeeery Sloooooow.

Symptoms are that it is @ 100% cpu pretty much all of the time. The processes using the most CPU are

AeXSvc
System
Sql

typically in that order of impact. I certainly find this weird as I usually find SQL owning the CPU.

The configuration of this server is:

Win2k
Sql2k sp3

about 1800 managed nodes. Dual Proc Dell 2650 w/ 4gb of RAM.

There are no NSE's in que except for a continuous feed of large NSE's that show up in the EVTQueue Slow directory. The ones that end up in the "Bad" folder display the following contents:

<row c1="-2147467259" c2="Rule not found in agent rule cache" c3="{04e9d5f8-3f18-04ed-c001-0502c0010502}" />

...over and over again for hundreds of lines. I have no doubt that this is a continuous feed into the server and is the cause of the trouble. The header of the nse is titled 'CE_NfySvrRemoteEvent'. What could be causing this? Any ideas?

Nick · 05-02-2006, 04:25 PM

Are they running Monitor Solution Metrics by any chance?

Mike · 05-02-2006, 04:29 PM

I'm not. Jim Harings from the Altiris forum believes it's patch management related. I wonder if the our "lord or the patch" has anything to say about it?

Mike · 05-02-2006, 04:52 PM

Check this out!

I turned off the dispatcher service and within 10 minutes I have over 623 NSEs in the QueFast folder! 1/3rd of this organizations machines have sent events within 10 minutes? This is crazy! The Altiris Agent is configured to check in every 6 hours!

The majority of the NSEs in the quefast folder look like this:

<?xml version='1.0' ?>
<XmlData>
<WbemEventObject class='CE_NfySvrRemoteEvent'>
<MessageID></MessageID>
<Resource guid='{9C0DBFA5-D36E-40D7-8C89-ED45980EB904}' typeGuid='493435F7-3B17-4C4C-B07F-C23E7AB7781F'><Attribute name='Domain' value='value'/><Attribute name='Name' value='computername'/><Attribute name='Altkey1' value=''/><Attribute name='Altkey2' value='44-45-53-54-42-00'/></Resource>
<ScenarioGUID>{06D80F00-9D63-4714-ACD7-CE980E8C139A}</ScenarioGUID>
<RemoteEvent>
<inventory>
<dataClass guid="{27a4854e-b26a-465e-b0ca-78cb079a954a}">
<data>
<resource> <row c1="{75DA0CCA-0095-44E7-8FBE-960FA0D72075}" c2="1115" c3="107" c4="1008" c5="0" c6="232" />
<row c1="{9D326BB4-75D4-4DB9-9946-59141BD1D44E}" c2="1107" c3="89" c4="1018" c5="0" c6="232" />
</resource>
</data>
</dataClass> </inventory>
</RemoteEvent><TimeGenerated>20060502164902.403000+2 40</TimeGenerated>
<MessageBody></MessageBody>
<MessageLabel></MessageLabel>
<TransportFlag></TransportFlag>
<TransactionCookie></TransactionCookie>
</WbemEventObject></XmlData>

This is the most interesting Altiris installation I've ever encountered.

Mike · 05-02-2006, 05:38 PM

One more note and interesting item. I believe there is something Patch Management related with the issue as I am being bombed by NSE files in my EVTQueueFast and they all look something like this:

<?xml version='1.0' ?>
<XmlData>
<WbemEventObject class='CE_NfySvrRemoteEvent'>
<MessageID></MessageID>
<Resource guid='{A86740AE-7DC0-4417-B915-402BF9B554A1}' typeGuid='493435F7-3B17-4C4C-B07F-C23E7AB7781F'><Attribute name='Domain' value='Value'/><Attribute name='Name' value='ComputerName'/><Attribute name='Altkey1' value=''/><Attribute name='Altkey2' value='<clientmacaddress>'/></Resource>
<ScenarioGUID>{BEF25806-58DC-43B4-ABF7-F0B0C0F898B0}</ScenarioGUID>
<RemoteEvent>
<inventory>
<dataClass guid="{27a4854e-b26a-465e-b0ca-78cb079a954a}">
<data>
<resource> <row c1="{1AC7A644-5AB5-4434-B24E-B3C27BA17DED}" c2="40" c3="2" c4="38" c5="0" c6="8" />
<row c1="{31967BBA-7A64-495E-A61E-48018592D068}" c2="66" c3="2" c4="64" c5="0" c6="14" />
</resource>
</data>
</dataClass> </inventory>
</RemoteEvent><TimeGenerated>20060502172319.060000+2 40</TimeGenerated>
<MessageBody></MessageBody>
<MessageLabel></MessageLabel>
<TransportFlag></TransportFlag>
<TransactionCookie></TransactionCookie>
</WbemEventObject></XmlData>

If you look up the "data class guid" you learn that this is owned by Altiris Inventory Rule Management.

So then I look up the "Scenario Guid" and the first one I examine shows "default windows software release inventory policy".

All PM inventory policies carey the default setting of "send every 4 hours, only if changed". Why are we getting killed by this and how do we fix it?

I'm hoping our Golden State Goucho has something to say about it.

LordofthePatch · 05-02-2006, 11:13 PM

How large is your InventoryRuleCache.iad file? I would try stopping the Altiris Agent service, deleting this file, and then restarting the service. By default it is located at C:\Program Files\Altiris\Altiris Agent\Agents\InventoryRuleAgent.

Mike · 05-03-2006, 06:47 AM

Thanks Andrew, I'll check that out. So these messages in the EvtQueFast are abnormal then, right?

I'm getting so many of them from so many different systems in this environment that I think I'll do that to all the compy's system wide. I'll let you know if it fixes the issue.

Nick · 05-03-2006, 09:22 AM

You definitely have me interested on this one. Keep us in the loop on what is going on.

LordofthePatch · 05-03-2006, 09:48 AM

I wouldn't call this "abnormal" per se. Normal Inventory Rule evaluation does not cause this many NSE files. However, if you have the Send Inventory Summary option enabled on 3 of the 4 policies, then this kind of traffic would be normal (hence why it is recommended to turn them off). Go to Configuration tab > Solutions Settings > Software Management > Patch Management > Windows > Software Inventory and under the Global and Microsoft folders check out each policy and make sure the Send Inventory Summary box is unchecked.

Mike · 05-04-2006, 06:42 AM

We've deleted everyone's .iad file with a job like this from Deployment Server:

Code:

Rem look for .iad file
if not exist "%programfiles%\altiris\altiris agent\agents\inventoryruleagent\inventoryrulecache.iad" goto end
 
REM stop Altiris Agent
net stop "altiris agent"
 
REM Delete .iad file(s)
cd\
cd "%programfiles%\altiris\altiris agent\agents\inventoryruleagent"
del *.iad
 
REM start altiris agent
net start "Altiris Agent"
 
:end

and turned off the "summary inventory" an we are rocking. We were getting around 2000 "bad" items in the EvtQueueFast\Bad\exception directory and 3 in the EvtQueueSlow\bad\exception every hour and now we are down to the normal parameters of just a couple of failed inventory processings a day. The server is running great again. We are still running 6.0 sp2 hf18 and the older version of patch. When we get on Sp3 and I tune up the machine.config / coresettings.config we'll be kickin @$$.

Still not sure what created it. I walked into this problem as I was brought in to help fix it.

Thanks for your help,

M

LordofthePatch · 05-04-2006, 09:29 AM

I'm sure that batch file will come in handy for someone else. Also, I have a .vbs version if anyone would prefer that. Glad to hear things are looking better Mike. Just make sure to send us a free T-Shirt or somethin'...

kschroeder · 05-04-2006, 04:21 PM

Quote:

Originally Posted by deus_1978

When we get on Sp3 and I tune up the machine.config / coresettings.config we'll be kickin @$$.

So what sorts of nifty tweaks can be done to machine.config? I read about some on BWG (they had a PPT/PDF with a bunch of tweaks for maxthreadsomething, etc, but they killed my NS when I tried them and I had to back it out). I have a real beast of a NS here (4X 3.0GHz Xeons, 8GB RAM) and it seems awfully slow to respond in the console (and the CPUs will be running at 20% at most).

Nick · 05-04-2006, 05:21 PM

Do you have the /3GB and /PAE switches going on the bad boy?

kschroeder · 05-04-2006, 05:30 PM

Yup, on Server 2003 SP1. I've got SQL 2000 SP4 set to use a hard 5GB RAM also and set it to "reserve physical memory for SQL server".

jperry933 · 12-17-2006, 08:39 PM

I know this thread is a bit old, but I have been having the issue described. Is it a good idea to delete the iad files on a maintenace schedule? I am getting a bunch of bad event files from this and the inv_inventory_agent_error table has tons of records. 8119269 for 819 distinct machines, to be exact. Thanks for any suggestions.

dylan · 01-07-2007, 11:08 PM

I have just turned off the send inventory summary also but my issue really is the size of the IAD file (64mb) and that my users get 100% Cpu ALLOT !!

JoBro · 03-25-2008, 10:19 PM

I have roughly 3,500 workstations & 300 servers reporting to my NS 6.0 SP3 R7 Server which is a IBM Quad core Quad xeon 3.0ghz with 16gig ram and plenty of scsi/san disks.

When i run the Install vs. Used Report, my box uses 100% of all cpu cores for about 10 minutes before returning a result for my report.

I am running windows server 2003 sp2 enterprise 32bit with sql 2005 sp1 32bit enterprise edition with AWE enabled and the /pae switch in my boot.ini, sql is configured to use all 16 cores if needed.

Does anyone have any idea's how i can reduce the load on the cpu and speed up the report generation time?

Nick · 03-25-2008, 10:37 PM

Might want to check the size of your Inv_AeX_SW_Audit_Software_Spt table. If it is large it can take the query a long time to process.

Mike · 03-25-2008, 11:00 PM

also check the size of your AeX_AM_Inventory table. In the Notification Server Infrastructure report group there's a db tablesize report. It doesn't matter how beefy your server is, if you have a 10GB table you're hosed.

Defrag your indexes too

GeoMac · 03-26-2008, 09:36 AM

Quote:

Originally Posted by LordofthePatch

I'm sure that batch file will come in handy for someone else. Also, I have a .vbs version if anyone would prefer that.

Hey Andrew, wouldn't the executable attached to this KB article:
https://kb.altiris.com/article.asp?article=22750&p=1
do the same thing?

GeoMac · 04-14-2008, 11:51 AM

Ah, I see the KB article has been updated (Last Updated: 4/1/2008) with Andrew's VBS file.

LordofthePatch · 04-14-2008, 02:39 PM

Hey, it make take we awhile, but I eventually do everything I'm told...

gvirdi · 01-13-2009, 06:54 PM

I faced low space issue with Microsoft update patches when I forced the pmimport (for 64 bit).

As a result I disabled ALL the bulletins and moved all downloaded updates shared folders to some other location as I wanted to create the fresh tasks.

After that I ran Inventory rule clear cache script.

Now I am restaging all the bulletins one by one and creating taks after staging them. Ever since I started this, CPU utilization is always 100% mostly by AeXSVC.exe .
What to do now?

kschroeder · 01-14-2009, 01:21 PM

The problem is that now all your clients have invalid packages since you deleted all the packages. They are trying to check for status for all the patch packages, and it is failing since the packages no longer exist. I would disable any existing Patch tasks you have and leave them disabled for a day or so and see if it doesn't improve.

gvirdi · 01-14-2009, 01:29 PM

I already disabled all the patching tasks before moving them. And even as of now, none is the task is active.

I have noticed that when I disabled the Altiris Client message dispatcher sevice, Aexsvc is back to normal.

How long can I keep that service in stop state and what needs to be done now so as things are steady and back to normal?