From ccalfred@ust.hk Mon Feb 14 04:03:59 2011 Received: from mx3.ust.hk (mx3.ust.hk [143.89.13.11]) by underdog.stansell.org (8.14.4/8.14.4) with ESMTP id p1E43qZc005312 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Mon, 14 Feb 2011 04:03:58 GMT Received: from [143.89.141.77] (dy141-077.ust.hk [143.89.141.77]) by mx3.ust.hk (8.13.8/8.13.8) with ESMTP id p1E43o1r079581 for ; Mon, 14 Feb 2011 12:03:51 +0800 (HKT) Message-ID: <4D58A9A6.1050300@ust.hk> Date: Mon, 14 Feb 2011 12:03:50 +0800 From: Alfred Chau User-Agent: Mozilla Thunderbird 1.0.6 (Windows/20050716) X-Accept-Language: zh-tw, en-us, en MIME-Version: 1.0 To: users@conserver.com Subject: Consever crash and teminated with "out of memory" in conserver log Content-Type: text/plain; charset=Big5 Content-Transfer-Encoding: 7bit X-Spam-Score: 3.555 (***) BAYES_50,FH_DATE_PAST_20XX X-Scanned-By: MIMEDefang 2.67 on 209.182.219.30 X-BeenThere: users@conserver.com X-Mailman-Version: 2.1.12 Precedence: list List-Id: Conserver Users List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Feb 2011 04:03:59 -0000 Dear conserver users, I'm using CentOS release 5 (Final) with 4G physical memory. I have been using Consever for two years. Recenty, Conserver crashed frequency with message at /var/log/conserver as conserver: out of memory [Thu Feb 10 15:53:08 2011] conserver (31728): child pid 31736: exit(69), shutting down [Thu Feb 10 15:53:08 2011] conserver (31728): terminated I checked the usage of memory and found it's still more than required still no in use. Anything I can overcome or work around the problem? Thanks a lot Alfred Chau ITSC HKUST From wernli@in2p3.fr Mon Feb 14 09:15:42 2011 Received: from ccsrelay02.in2p3.fr (ccsrelay02.in2p3.fr [134.158.66.52]) by underdog.stansell.org (8.14.4/8.14.4) with ESMTP id p1E9FXSa028296 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Mon, 14 Feb 2011 09:15:42 GMT Received: from ccswiss.in2p3.fr (ccswissrp.in2p3.fr [134.158.71.221]) (authenticated bits=0) by ccsrelay02.in2p3.fr (8.14.2/8.14.2/IN2P3) with ESMTP id p1E9FWkh024613 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Mon, 14 Feb 2011 10:15:32 +0100 X-Virus-Status: Clean X-Virus-Scanned: clamav-milter 0.96.5 at ccsrelay02 Received: by ccswiss.in2p3.fr (Postfix, from userid 3942) id A52DCC131; Mon, 14 Feb 2011 10:15:31 +0100 (CET) Date: Mon, 14 Feb 2011 10:15:31 +0100 From: Fabien Wernli To: users@conserver.com Subject: Re: Consever crash and teminated with "out of memory" in conserver log Message-ID: <20110214091531.GA3560@ccswiss.in2p3.fr> Mail-Followup-To: users@conserver.com References: <4D58A9A6.1050300@ust.hk> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4D58A9A6.1050300@ust.hk> Organization: CC-IN2P3 (CNRS) User-Agent: Mutt/1.5.20 (2009-06-14) X-Spam-Score: 3.555 (***) BAYES_50,FH_DATE_PAST_20XX X-Scanned-By: MIMEDefang 2.67 on 209.182.219.30 X-BeenThere: users@conserver.com X-Mailman-Version: 2.1.12 Precedence: list Reply-To: wernli@in2p3.fr List-Id: Conserver Users List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Feb 2011 09:15:43 -0000 Hi, On Mon, Feb 14, 2011 at 12:03:50PM +0800, Alfred Chau wrote: > I'm using CentOS release 5 (Final) with 4G physical memory. I have been > using Consever for two years. Recenty, Conserver crashed frequency with > message at > /var/log/conserver as > conserver: out of memory How many consoles does your server handle, and of what type? In the past I've had memory leaks on a conserver handling many (hundreds) "exec" consoles. The issue seemed to be related to ipmitool, and has disappeared after upgrading. From glance@acc.umu.se Mon Feb 14 10:20:28 2011 Received: from mail.acc.umu.se (mail.acc.umu.se [130.239.18.156]) by underdog.stansell.org (8.14.4/8.14.4) with ESMTP id p1EAKMdg029998 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL) for ; Mon, 14 Feb 2011 10:20:28 GMT Received: from localhost (localhost [127.0.0.1]) by amavisd-new (Postfix) with ESMTP id 16D17D27 for ; Mon, 14 Feb 2011 11:20:21 +0100 (MET) X-Virus-Scanned: amavisd-new at acc.umu.se Received: from kennedy.acc.umu.se (kennedy.acc.umu.se [130.239.18.157]) by mail.acc.umu.se (Postfix) with ESMTP id EA10DD26 for ; Mon, 14 Feb 2011 11:20:19 +0100 (MET) Received: by kennedy.acc.umu.se (Postfix, from userid 24471) id D610C9B4; Mon, 14 Feb 2011 11:20:19 +0100 (MET) Date: Mon, 14 Feb 2011 11:20:19 +0100 From: Anton Lundin To: users@conserver.com Subject: Re: Consever crash and teminated with "out of memory" in conserver log Message-ID: <20110214102019.GX18254@kennedy.acc.umu.se> References: <4D58A9A6.1050300@ust.hk> <20110214091531.GA3560@ccswiss.in2p3.fr> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20110214091531.GA3560@ccswiss.in2p3.fr> User-Agent: Mutt/1.5.18 (2008-05-17) X-Spam-Score: 3.555 (***) BAYES_50,FH_DATE_PAST_20XX X-Scanned-By: MIMEDefang 2.67 on 209.182.219.30 X-BeenThere: users@conserver.com X-Mailman-Version: 2.1.12 Precedence: list List-Id: Conserver Users List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Feb 2011 10:20:29 -0000 On 14 February, 2011 - Fabien Wernli wrote: > Hi, > > On Mon, Feb 14, 2011 at 12:03:50PM +0800, Alfred Chau wrote: > > I'm using CentOS release 5 (Final) with 4G physical memory. I have been > > using Consever for two years. Recenty, Conserver crashed frequency with > > message at > > /var/log/conserver as > > conserver: out of memory > > How many consoles does your server handle, and of what type? > > In the past I've had memory leaks on a conserver handling many (hundreds) > "exec" consoles. The issue seemed to be related to ipmitool, and has > disappeared after upgrading. > We run currently 985 various consoles connected to conserver. All exec consoles get a memory overhead of about 0.5 mb for the dash-shell wrapping the exec. Would be about twice if it were bash. SSH-based consoles ( ilo, ras, ilom, cyclades .. ) is about 2.5 mb each for buffers etc. ipmitool-based is a bit more trickey. I have done some tweeking with ipmitool to get rid of some redicolus memory leaks[1], but there are some left. The one's that doesn't leak is between 1.6 mb to 1.8 mb, but the ones that still has leaks is between 14-16 mb after 12 days running. All this ends up with a memory footprint of about 3.2 gb. Have you checked what your memory footprint is when it runns? do you have any memory limits applied to your conserver? OutOfMem() is may get called on a number of places in the code, so without a stacktrace its impossible to diagnose this any further. 1. http://www.mail-archive.com/ipmitool-devel@lists.sourceforge.net/msg01471.html -- Anton Lundin +46702-161604 From glance@acc.umu.se Mon Feb 21 09:08:16 2011 Received: from mail.acc.umu.se (mail.acc.umu.se [130.239.18.156]) by underdog.stansell.org (8.14.4/8.14.4) with ESMTP id p1L988Ld026127 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL) for ; Mon, 21 Feb 2011 09:08:15 GMT Received: from localhost (localhost [127.0.0.1]) by amavisd-new (Postfix) with ESMTP id E99355A2 for ; Mon, 21 Feb 2011 10:08:06 +0100 (MET) X-Virus-Scanned: amavisd-new at acc.umu.se Received: from kennedy.acc.umu.se (kennedy.acc.umu.se [130.239.18.157]) by mail.acc.umu.se (Postfix) with ESMTP id C40CD5A1 for ; Mon, 21 Feb 2011 10:08:05 +0100 (MET) Received: by kennedy.acc.umu.se (Postfix, from userid 24471) id A31F39DA; Mon, 21 Feb 2011 10:08:05 +0100 (MET) Date: Mon, 21 Feb 2011 10:08:05 +0100 From: Anton Lundin To: users@conserver.com Subject: Re: [PATCH] Power control Message-ID: <20110221090805.GN18254@kennedy.acc.umu.se> References: <20101029134359.GU4570@kennedy.acc.umu.se> <20101102092908.GD19794@ccswiss.in2p3.fr> <4CD015BC.2070807@redhat.com> <20101102154357.GT4570@kennedy.acc.umu.se> <20101104090128.GA8019@ccswiss.in2p3.fr> <8A3999E8-4114-4D17-9BEA-E42CE7C2B9D2@conserver.com> <20101230190351.GC8896@kennedy.acc.umu.se> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: <20101230190351.GC8896@kennedy.acc.umu.se> User-Agent: Mutt/1.5.18 (2008-05-17) X-Spam-Score: 1.242 (*) BAYES_00,FH_DATE_PAST_20XX X-Scanned-By: MIMEDefang 2.67 on 209.182.219.30 X-BeenThere: users@conserver.com X-Mailman-Version: 2.1.12 Precedence: list List-Id: Conserver Users List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Feb 2011 09:08:16 -0000 Allmost 2 months and zero feedback. Thus its perfect? =) //Anton On 30 December, 2010 - Anton Lundin wrote: > Second generation of this patch, done. > > http://www.acc.umu.se/~glance/conserver/02-conserver-8.1.18-power-control.patch > > Fixed the "multiplexing" so that the powercmd can't hang the whole > Kiddie()-process. Its pretty much the same as its done for initcmd's. > > There is probably still some fiddeling to be done before its perfect > but its god enuff to take a look at. > > Things on the todo: > * processgroup for the powercmd like its done for initcmd's? > * stdin for powercmd? currently its just close()'d > * put the output from the powercmd in the log? > * kill hanging powercmd's after $foo seconds? > * maybee rename everything to ServerSite-kommands? > > //Anton - Who is heading home to end this year with a big bang... > > On 04 November, 2010 - Bryan Stansell wrote: > > > There is certainly going to be a need to either redirect stdin/stdout of the command to the console or not (some "interactive" flag). If not, stdin would probably be /dev/null and stdout could be anyone connected to the console (so everyone sees the output). Maybe it should just be the console or nothing at all. Something to think about... > > > > Bryan > > > > On Nov 4, 2010, at 2:01 AM, Fabien Wernli wrote: > > > > > On Wed, Nov 03, 2010 at 02:43:46PM -0700, Bryan Stansell wrote: > > >> And I'm all for making it more generic as well...perhaps by making the 'k' command programmable - in the sense that you can associate a command with any character ([a-z0-9]) after 'k' and give it a label so you know what the command is supposed to do. > > >> > > >> So, 'k' for 'k'ommand...or perhaps even '!' to invoke a command? Harder to type, but consistent with vi, for example. > > > > > > Great. I vote for '!'. > > > Also, maybe it would be interesting to have a distinction between commands > > > that output something or not on std(err|out). > > > > > > _______________________________________________ > > > users mailing list > > > users@conserver.com > > > https://www.conserver.com/mailman/listinfo/users > > > > > > _______________________________________________ > > users mailing list > > users@conserver.com > > https://www.conserver.com/mailman/listinfo/users > > -- > Anton Lundin +46702-161604 > _______________________________________________ > users mailing list > users@conserver.com > https://www.conserver.com/mailman/listinfo/users -- Anton Lundin +46702-161604