From ronaldo.mercado@diamond.ac.uk Thu Jan 15 15:56:06 2015 Received: from smtp-out4.electric.net (smtp-out4.electric.net [192.162.216.183]) by underdog.stansell.org (8.14.9/8.14.9) with ESMTP id t0FFu3rW009866 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Thu, 15 Jan 2015 15:56:06 GMT Received: from 1YBmm5-0001UN-V2 by out4a.electric.net with emc1-ok (Exim 4.84) (envelope-from ) id 1YBmm5-0001VU-W6 for users@conserver.com; Thu, 15 Jan 2015 07:55:57 -0800 Received: by emcmailer; Thu, 15 Jan 2015 07:55:57 -0800 Received: from [130.246.236.9] (helo=exchsmtp.stfc.ac.uk) by out4a.electric.net with esmtps (TLSv1:AES128-SHA:128) (Exim 4.84) (envelope-from ) id 1YBmm5-0001UN-V2 for users@conserver.com; Thu, 15 Jan 2015 07:55:57 -0800 Received: from EXCHMBX01.fed.cclrc.ac.uk ([169.254.3.158]) by EXCHHUB01.fed.cclrc.ac.uk ([130.246.236.9]) with mapi id 14.03.0174.001; Thu, 15 Jan 2015 15:55:57 +0000 From: To: Subject: Question about limiting rate of conserver output Thread-Topic: Question about limiting rate of conserver output Thread-Index: AdAw27/FTcOH9LPOT0Ck42KLIflryg== Date: Thu, 15 Jan 2015 15:55:56 +0000 Message-ID: <3D65E04168FA09429B823CA3FE25C4478C563A9C@EXCHMBX01.fed.cclrc.ac.uk> Accept-Language: en-GB, en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [172.23.4.59] X-Outbound-IP: 130.246.236.9 X-Env-From: ronaldo.mercado@diamond.ac.uk X-PolicySMART: 3590380, 3595258 MIME-Version: 1.0 Content-Language: en-US Content-Type: text/plain; charset=us-ascii X-Virus-Status: Scanned by VirusSMART (c) X-Virus-Status: Scanned by VirusSMART (s) X-Spam-Score: -1.5 () BAYES_00 X-Scanned-By: MIMEDefang 2.72 on 198.151.248.21 Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by underdog.stansell.org id t0FFu3rW009866 X-BeenThere: users@conserver.com X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Conserver Users List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Jan 2015 15:56:06 -0000 Hello, I am writing because I would like to limit the output conserver writes to disk to a configurable amount per second, say like a serial line. My application captures messages from devices that can send output as fast as possible over a network and the resulting log files are larger than the storage available. I don't think a feature to limit conserver logs this is in the current code, correct? Could you suggest what part of the code to change? Even better, has somebody already solved my problem? Thanks! -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom From john@stoffel.org Thu Jan 15 16:38:02 2015 Received: from mailrelay4.lanline.com (mailrelay4.lanline.com [216.187.10.25]) by underdog.stansell.org (8.14.9/8.14.9) with ESMTP id t0FGc0PL011564 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Thu, 15 Jan 2015 16:38:02 GMT Received: from mycroft.westnet.com (Mycroft.westnet.com [216.187.52.7]) by mailrelay4.lanline.com (8.14.5/8.14.5) with ESMTP id t0FGbvem082255; Thu, 15 Jan 2015 11:37:58 -0500 (EST) (envelope-from john@stoffel.org) Received: from quad.stoffel.org (66-189-26-192.dhcp.oxfr.ma.charter.com [66.189.26.192]) (authenticated bits=0) by mycroft.westnet.com (8.14.4/8.14.4) with ESMTP id t0FGXxJo023656 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 15 Jan 2015 11:34:00 -0500 (EST) Received: by quad.stoffel.org (Postfix, from userid 1000) id ABE54A6F8A; Thu, 15 Jan 2015 11:37:56 -0500 (EST) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <21687.60644.509852.70941@quad.stoffel.home> Date: Thu, 15 Jan 2015 11:37:56 -0500 From: "John Stoffel" To: Subject: Re: Question about limiting rate of conserver output In-Reply-To: <3D65E04168FA09429B823CA3FE25C4478C563A9C@EXCHMBX01.fed.cclrc.ac.uk> References: <3D65E04168FA09429B823CA3FE25C4478C563A9C@EXCHMBX01.fed.cclrc.ac.uk> X-Mailer: VM 8.2.0b under 23.4.1 (x86_64-pc-linux-gnu) X-Virus-Scanned: clamav-milter 0.98.1 at mycroft X-Virus-Status: Clean X-Spam-Score: -1.5 () BAYES_00 X-Scanned-By: MIMEDefang 2.72 on 198.151.248.21 Cc: users@conserver.com X-BeenThere: users@conserver.com X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Conserver Users List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Jan 2015 16:38:03 -0000 > I am writing because I would like to limit the output conserver > writes to disk to a configurable amount per second, say like a > serial line. So are you wanting to discard output and just write every third line or something? Just slowing down the output isn't going to address the problem here, which is too small space for logs. > My application captures messages from devices that can send output > as fast as possible over a network and the resulting log files are > larger than the storage available. The problem is the storage size, not the devices. How will slowing down those devices make things work better? > I don't think a feature to limit conserver logs this is in the > current code, correct? > Could you suggest what part of the code to change? Even better, has > somebody already solved my problem? How about using the 'logfilemax' parameter from conserver.cf (see the man page) and then have a cronjob which copies old log files over the network to another system with more disk space. Also, look at the 'timestamp' option, since you probably want to turn that off as well. John From ronaldo.mercado@diamond.ac.uk Fri Jan 16 09:15:10 2015 Received: from smtp-out4.electric.net (smtp-out4.electric.net [192.162.216.181]) by underdog.stansell.org (8.14.9/8.14.9) with ESMTP id t0G9F6V8000514 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Fri, 16 Jan 2015 09:15:09 GMT Received: from 1YC2gt-0005sK-Vm by out4b.electric.net with emc1-ok (Exim 4.84) (envelope-from ) id 1YC2gu-0005tP-U7 for users@conserver.com; Fri, 16 Jan 2015 00:55:40 -0800 Received: by emcmailer; Fri, 16 Jan 2015 00:55:40 -0800 Received: from [130.246.236.11] (helo=exchsmtp.stfc.ac.uk) by out4b.electric.net with esmtps (TLSv1:AES128-SHA:128) (Exim 4.84) (envelope-from ) id 1YC2gt-0005sK-Vm; Fri, 16 Jan 2015 00:55:39 -0800 Received: from EXCHMBX01.fed.cclrc.ac.uk ([169.254.3.158]) by EXCHHUB03.fed.cclrc.ac.uk ([130.246.236.11]) with mapi id 14.03.0174.001; Fri, 16 Jan 2015 08:55:39 +0000 From: To: Subject: RE: Question about limiting rate of conserver output Thread-Topic: Question about limiting rate of conserver output Thread-Index: AdAw27/FTcOH9LPOT0Ck42KLIflrygABd2cAACDQfMA= Date: Fri, 16 Jan 2015 08:55:39 +0000 Message-ID: <3D65E04168FA09429B823CA3FE25C4478C56413E@EXCHMBX01.fed.cclrc.ac.uk> References: <3D65E04168FA09429B823CA3FE25C4478C563A9C@EXCHMBX01.fed.cclrc.ac.uk> <21687.60644.509852.70941@quad.stoffel.home> In-Reply-To: <21687.60644.509852.70941@quad.stoffel.home> Accept-Language: en-GB, en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [172.23.4.59] X-Outbound-IP: 130.246.236.11 X-Env-From: ronaldo.mercado@diamond.ac.uk X-PolicySMART: 3590380, 3595258 MIME-Version: 1.0 Content-Language: en-US Content-Type: text/plain; charset=us-ascii X-Spam-Score: -1.5 () BAYES_00 X-Scanned-By: MIMEDefang 2.72 on 198.151.248.21 Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by underdog.stansell.org id t0G9F6V8000514 Cc: users@conserver.com X-BeenThere: users@conserver.com X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Conserver Users List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Jan 2015 09:15:10 -0000 Hi John, Thanks for replying. > So are you wanting to discard output and just write every third line or > something? Just slowing down the output isn't going to address the > problem here, which is too small space for logs. The underlying problem is the service writing out too many errors. I am the guy in charge of fixing it. A sustained message storm broke past a few protection mechanisms of log rotating and disk space monitoring over several days finally caused problems in other systems. Limiting the rate could offer additional protection. > How about using the 'logfilemax' parameter from conserver.cf (see the man > page) and then have a cronjob which copies old log files over the network > to another system with more disk space. Effectively I want "logfilemax" that throws away messages in case of a sustained message storm. I prefer throwing away messages because they tend to be repeated anyway. The system will be fixed eventually and the service restarted. Logging can continue from where it left off. There is a cronjob rotating log files at midnight already. I did not realise there are similar facilities in conserver. Ronaldo -- This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail. Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd. Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message. Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom