Shibboleth SP 2.1 shibd Shutdowns under hight load

[Mike Jennings]

We are doing some load testing of our servers and under high loads we
are seeing these following error messages, then the shibd automatically
shuts down. I am assuming that this is because the server is running
out of memory, but they have 8gb of memory available and only 1.5 is
being uses.

2009-06-25 14:32:11 ERROR XMLTooling.Threads : pthread_create error
(12): Cannot allocate memory
2009-06-25 14:32:12 CRIT Shibboleth.Listener : error starting new server
thread to service incoming request
2009-06-25 14:32:12 INFO Shibboleth.Listener : listener service shutting
down

Does anyone have any thoughts on this?

Mike Jennings

[Scott Cantor]

Post by Mike Jennings
We are doing some load testing of our servers and under high loads we
are seeing these following error messages, then the shibd automatically
shuts down. I am assuming that this is because the server is running
out of memory, but they have 8gb of memory available and only 1.5 is
being uses.

I can't tell you anything other than that the error returned by
pthread_create is a memory allocation failure based on the call to strerror.

However, I believe the shutdown behavior is corrected in 2.2. I can't find
the bug report right this second, but the error message is different, and
the normal exception handler is not setting the shutdown flag that tears
down the service.

I would suspect that the memory failure might just be a thread exhaustion
condition, perhaps. I have yet to implement any thread limiting, it just
hasn't come up in normal conditions because it's bounded by what Apache
allows anyway.

-- Scott

[André Cruz]

Post by Mike Jennings
We are doing some load testing of our servers and under high loads
we are seeing these following error messages, then the shibd
automatically shuts down. I am assuming that this is because the
server is running out of memory, but they have 8gb of memory
available and only 1.5 is being uses.
2009-06-25 14:32:11 ERROR XMLTooling.Threads : pthread_create error
(12): Cannot allocate memory
2009-06-25 14:32:12 CRIT Shibboleth.Listener : error starting new
server thread to service incoming request
2009-06-25 14:32:12 INFO Shibboleth.Listener : listener service
shutting down
Does anyone have any thoughts on this?

Maybe you have process limits in place?

You can try running a "ulimit -a" as the user running shibd.

André

[Mike Jennings]

I am going to upgrade the SP to be 2.2 to fix the issue and then I wlil
run a ulimit -a on the user to see what happens under high load.


==============================================================================
Mike Jennings
Identity Management Developer
University of North Carolina at Chapel Hill

Office: [phone]
Cell: [phone]

Post by André Cruz

Post by Mike Jennings
We are doing some load testing of our servers and under high loads we
are seeing these following error messages, then the shibd
automatically shuts down. I am assuming that this is because the
server is running out of memory, but they have 8gb of memory available
and only 1.5 is being uses.
2009-06-25 14:32:11 ERROR XMLTooling.Threads : pthread_create error
(12): Cannot allocate memory
2009-06-25 14:32:12 CRIT Shibboleth.Listener : error starting new
server thread to service incoming request
2009-06-25 14:32:12 INFO Shibboleth.Listener : listener service
shutting down
Does anyone have any thoughts on this?

Maybe you have process limits in place?
You can try running a "ulimit -a" as the user running shibd.
André

[gviczai]

Experiencing the same problem with version 2.5.

Under heavy load shibd logs:

"2012-09-04 16:34:38 ERROR XMLTooling.Threads : pthread_create error (12):
Cannot allocate memory"

(more than 10G of system memory is free), than
watch "pmap 24890 | grep 10240 | wc -l; ls /proc/24890/task | wc -l" reports
increase in threads, and after about 200 thread shibd exits.




--
View this message in context: http://shibboleth.1660669.n2.nabble.com/Shibboleth-SP-2-1-shibd-Shutdowns-under-hight-load-tp3176891p7581621.html
Sent from the Shibboleth - Users mailing list archive at Nabble.com.
--
To unsubscribe from this list send an email to users-unsubscribe-***@public.gmane.org

[Cantor, Scott]

Post by gviczai
Experiencing the same problem with version 2.5.
Cannot allocate memory"
more than 10G of system memory is free), than
watch "pmap 24890 | grep 10240 | wc -l; ls /proc/24890/task | wc -l" reports
increase in threads, and after about 200 thread shibd exits.

Stop using Apache prefork, or alternatively make your stack size something
reasonable. The Linux default is huge.

See
https://wiki.shibboleth.net/confluence/display/SHIB2/NativeSPUnixListener
or set it via ulimit.

-- Scott


--
To unsubscribe from this list send an email to users-unsubscribe-***@public.gmane.org