Shibd process consumes more memory and leads to CPU spike

[pradeep kumar]

Hi,

We have been running our application with Shibboleth 2.3 for last one year
with around 4k user load. Now we have increased our user base up to 20k and
all in a sudden we have observed CPU spike due to shibd process consumes
more memory. The interesting part here is, this happens every day in
between 4-5PM EST. The shibd process alone consumes around 1GB memory when
the CPU spike happens.

P.S. Our application has been used as a intranet hence the user load is
consistent everyday.

Has this been noticed already anywhere? We feel it could be due to
Shibboleth memory leak. Any pointers will be highly appreciated since it
occurs in our production environment.

Here are the system details

OS: Red Hat Enterprise Linux Server release 5.8 (Tikanga)
CPU: Intel(R) Xeon(R) CPU E5530 @ 2.40GHz
RAM size:16GB

Thanks
Pradeep

[Cantor, Scott]

Post by pradeep kumar
We have been running our application with Shibboleth 2.3 for last one
year with around 4k user load. Now we have increased our user base up to
20k and all in a sudden we have observed CPU spike due to shibd process
consumes more memory. The interesting part
here is, this happens every day in between 4-5PM EST. The shibd process
alone consumes around 1GB memory when the CPU spike happens.

Probably because of a prefork Apache, and using a default thread stack
size. This is discussed a number of times in the list archives and is
discussed a bit in the documentation now (NativeSPApacheConfig).

If you're running in worker mode, then something else is going on, but I
would check that first and you can follow up if not.

Post by pradeep kumar
Has this been noticed already anywhere? We feel it could be due to
Shibboleth memory leak.

There is no leak of that magnitude, and even if there were, you're running
an unsupported version and would have no recourse anyway but to upgrade.

I would also politely suggest that you need to take a look at the security
advisories.

-- Scott


--
To unsubscribe from this list send an email to users-unsubscribe-***@public.gmane.org

[pradeepkumar]

Hi Scott,

We are already running in apache worker mode with four nodes configured in
it.

Post by Cantor, Scott
I would also politely suggest that you need to take a look at the security
advisories

Any pointers/suggestion related to security concern would be greatly
appreciated.

Pradeep



--
View this message in context: http://shibboleth.1660669.n2.nabble.com/Shibd-process-consumes-more-memory-and-leads-to-CPU-spike-tp7582635p7582685.html
Sent from the Shibboleth - Users mailing list archive at Nabble.com.
--
To unsubscribe from this list send an email to users-unsubscribe-***@public.gmane.org

[pradeepkumar]

Also to mention the stack size is not default, it is somewhere around 250K.



--
View this message in context: http://shibboleth.1660669.n2.nabble.com/Shibd-process-consumes-more-memory-and-leads-to-CPU-spike-tp7582635p7582686.html
Sent from the Shibboleth - Users mailing list archive at Nabble.com.
--
To unsubscribe from this list send an email to users-unsubscribe-***@public.gmane.org

[Cantor, Scott]

Post by pradeepkumar
Also to mention the stack size is not default, it is somewhere around 250K.

I would shrink it a bit more than that if you're still having issues, but in short, there's no way you'd get any significant spike then, so I don't know what your issue is. I can load test at much higher capacity than you would get in any kind of load increase and there's no issue.

You are not on a supported release, so that is your next step.

-- Scott


--
To unsubscribe from this list send an email to users-unsubscribe-***@public.gmane.org

[pradeepkumar]

Thanks for your prompt response. We are already in the process of upgrading
to 2.5.

The main problem over here is, this force us to restart the webserver
everyday in advance before the issue occurs (As I mentioned earlier it
occurs in between 4-5 PM, hence we restart at 2PM).

Also my main concerns is your previous statement.

*There is no leak of that magnitude*, and even if there were, you're

running

an unsupported version and would have no recourse anyway but to upgrade.
*I would also politely suggest that you need to take a look at the security

advisories*.

I am curious to dig deeper into that. Any pointers/suggestion related to
that would be of much help.

Pradeep



--
View this message in context: http://shibboleth.1660669.n2.nabble.com/Shibd-process-consumes-more-memory-and-leads-to-CPU-spike-tp7582635p7582693.html
Sent from the Shibboleth - Users mailing list archive at Nabble.com.
--
To unsubscribe from this list send an email to users-unsubscribe-***@public.gmane.org

[Cantor, Scott]

Post by pradeepkumar
The main problem over here is, this force us to restart the webserver
everyday in advance before the issue occurs (As I mentioned earlier it
occurs in between 4-5 PM, hence we restart at 2PM).

Restarting the web server isn't going to help you if the SP is involved, shibd is a separate process.

There are relatively small leaks in versions prior to 2.5 but to be affected by them on a daily basis you'd have to be doing hundreds of thousands of sessions (not requests, actual unique logins). And they would not occur because of load spikes.

I think you should double check that you're actually in worker mode.

Post by pradeepkumar
Also my main concerns is your previous statement.
I am curious to dig deeper into that. Any pointers/suggestion related to
that would be of much help.

https://wiki.shibboleth.net/confluence/display/SHIB2/SecurityAdvisories

-- Scott


--
To unsubscribe from this list send an email to users-unsubscribe-***@public.gmane.org