From netllama@gmail.com Thu Jul 23 17:59:55 2015 Received: from mail-ob0-f179.google.com (mail-ob0-f179.google.com [209.85.214.179]) by underdog.stansell.org (8.15.2/8.15.2) with ESMTPS id t6NHxrpI008249 (version=TLSv1.2 cipher=AES128-GCM-SHA256 bits=128 verify=FAIL) for ; Thu, 23 Jul 2015 17:59:55 GMT Received: by obre1 with SMTP id e1so160323669obr.1 for ; Thu, 23 Jul 2015 10:59:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=6DiAS9RUP7jaVB9K9eOuPfMyWRbXGXCvQ6Yg/0EO9m0=; b=M1FNATWrGnIHmr/98YdVJA/UGSGxggiOIP7Y5CAsuO7pTTLsSkO4JF8tgmF9HAYE88 i+35aABKwLrBWD92he9ek+bSeXLfKd7Q0QrTfFGvHOSzG75q8Q4FNOuB0IRVNVQtPLPu zTULGAcBBlDwF2SZm5RZdSXQ0FY9MSqAWPVYivIjwBeqcp7xXwy9/SxoXZjtaZi1l1d1 Cgkn4LrTUvk4e4gfTfVoOZwkOt7MBbJXA6QmXFvxgVigHx5zAIbp5BxvlWfUb81QvGVo UPUCP5Qkvc9J16wjftdQhvA3WzXiCoPjNaCmMHFJGkuGZiLfpdV//JSyGUk1xLcBAnuH kbMw== MIME-Version: 1.0 X-Received: by 10.182.71.49 with SMTP id r17mr10117756obu.77.1437674390712; Thu, 23 Jul 2015 10:59:50 -0700 (PDT) Received: by 10.50.224.148 with HTTP; Thu, 23 Jul 2015 10:59:50 -0700 (PDT) Date: Thu, 23 Jul 2015 10:59:50 -0700 Message-ID: Subject: down consoles fail to reconnect automatically From: Lonni J Friedman To: users@conserver.com Content-Type: text/plain; charset=UTF-8 X-Spam-Score: -1.489 () BAYES_00,FREEMAIL_FROM,T_DKIM_INVALID X-Scanned-By: MIMEDefang 2.72 on 198.151.248.21 X-BeenThere: users@conserver.com X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Conserver Users List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Jul 2015 17:59:55 -0000 Greetings, I'm running conserver-8.2.1 on an Ubuntu-14.04.2 server, with several thousand clients, connected over IPMI. Most of the time it works fine, however occasionally we lose a VPN concentrator that maintains a VPN tunnel between remote sites and the console server, and we a large number of console sessions go into the 'down' state. Usually when the tunnel comes back up, the console sessions come back up on their own, however there are times when they do not come back up for hours, or not at all for no obvious reason. In nearly 100% of those cases, if someone manually runs 'console $consoleName' (where $consoleName is the name of the console session that is listed as 'down'), it will immediately come back up. According to the 'console' man page ( http://www.conserver.com/docs/console.man.html ), if I invoke 'console' with: -z bringup it should "Try to connect all consoles marked as down (this is equivalent to sending the server a SIGUSR1)". I've tried that: #### $ console -v -z bringup console: interface address 127.0.0.1 (lo) console: interface address 10.200.53.130 (eth0) 127.0.0.1: ok -- bringing up consoles #### However it doesn't seem to do anything at all. None of the down consoles come up ever. Yet I can still force them up manually if I connect to them one at a time. I'm unclear whether I'm misunderstanding how the 'bringup' command is intended to work, or if there's a bug somewhere. Can someone comment? thanks! From bryan@conserver.com Fri Jul 24 07:23:31 2015 Received: from [192.168.2.9] (host-90-232-179-79.mobileonline.telia.com [90.232.179.79]) (authenticated bits=0) by underdog.stansell.org (8.15.2/8.15.2) with ESMTPSA id t6O7NH7W009074 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Fri, 24 Jul 2015 07:23:30 GMT Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2102\)) Subject: Re: down consoles fail to reconnect automatically From: Bryan Stansell In-Reply-To: Date: Fri, 24 Jul 2015 00:23:16 -0700 Message-Id: <2FF2E6C2-410C-47D6-8406-83B6E2CA369B@conserver.com> References: To: users@conserver.com X-Mailer: Apple Mail (2.2102) X-Spam-Score: 0.163 () BAYES_00,RDNS_DYNAMIC X-Scanned-By: MIMEDefang 2.72 on 198.151.248.21 Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by underdog.stansell.org id t6O7NH7W009074 X-BeenThere: users@conserver.com X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Conserver Users List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Jul 2015 07:23:31 -0000 What you’re doing sounds all correct, as are your expectations (it should attempt to bring up any downed consoles). My simple test setup shows that it works for me, but with lots of consoles, there could be a bug or some side-effect that happens with more. Or possibly some config settings that aren’t playing well together. Do you have any “interesting” messages in the conserver log file that appear when you run the command? Bryan > On Jul 23, 2015, at 10:59 AM, Lonni J Friedman wrote: > > Greetings, > I'm running conserver-8.2.1 on an Ubuntu-14.04.2 server, with several > thousand clients, connected over IPMI. Most of the time it works > fine, however occasionally we lose a VPN concentrator that maintains a > VPN tunnel between remote sites and the console server, and we a large > number of console sessions go into the 'down' state. Usually when the > tunnel comes back up, the console sessions come back up on their own, > however there are times when they do not come back up for hours, or > not at all for no obvious reason. In nearly 100% of those cases, if > someone manually runs 'console $consoleName' (where $consoleName is > the name of the console session that is listed as 'down'), it will > immediately come back up. > > According to the 'console' man page ( > http://www.conserver.com/docs/console.man.html ), if I invoke > 'console' with: > -z bringup > > it should "Try to connect all consoles marked as down (this is > equivalent to sending the server a SIGUSR1)". I've tried that: > #### > $ console -v -z bringup > console: interface address 127.0.0.1 (lo) > console: interface address 10.200.53.130 (eth0) > 127.0.0.1: ok -- bringing up consoles > #### > > However it doesn't seem to do anything at all. None of the down > consoles come up ever. Yet I can still force them up manually if I > connect to them one at a time. > > I'm unclear whether I'm misunderstanding how the 'bringup' command is > intended to work, or if there's a bug somewhere. > > Can someone comment? > > thanks! > _______________________________________________ > users mailing list > users@conserver.com > https://www.conserver.com/mailman/listinfo/users From netllama@gmail.com Fri Jul 24 17:36:40 2015 Received: from mail-ig0-f179.google.com (mail-ig0-f179.google.com [209.85.213.179]) by underdog.stansell.org (8.15.2/8.15.2) with ESMTPS id t6OHab3V006470 (version=TLSv1.2 cipher=AES128-GCM-SHA256 bits=128 verify=FAIL); Fri, 24 Jul 2015 17:36:39 GMT Received: by iggf3 with SMTP id f3so22249582igg.1; Fri, 24 Jul 2015 10:36:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=/3c9B2GAVrnxSU2Xinbz92O9lRDoByTlGSNUkODleCI=; b=w/eELhqOrmLizuY2jIsJmMQhx9bEYdIF3CGGVrGv2gyx76ozITDOBVaWtalfmOW4Dk 6WO2+612/LpXrnYicpjkezjCdNL47bMYjt3MFMQsduokF0AJWH1hGvSgigd7fCELBLMj 3C9eAX53GfozFhlh7YjX5obAaFtaYiCcx096bDzjoOxq9hwicXiNrcbblpFliuMDL0wC fsv0LEyPYs5CB5T2e0vPqUqG2wrA/GoJ8ERXPnI8EL19t3YU52yY+nBCAVMAr252xXna nIXqUigfqgAx4RJn4OgRjHDqyyxTBk+UiQN8bPwcITnHYVq/CF+IjX6RcN21u1i/8Ctw +WTg== MIME-Version: 1.0 X-Received: by 10.50.79.169 with SMTP id k9mr8643138igx.44.1437759394988; Fri, 24 Jul 2015 10:36:34 -0700 (PDT) Received: by 10.50.224.148 with HTTP; Fri, 24 Jul 2015 10:36:34 -0700 (PDT) In-Reply-To: <2FF2E6C2-410C-47D6-8406-83B6E2CA369B@conserver.com> References: <2FF2E6C2-410C-47D6-8406-83B6E2CA369B@conserver.com> Date: Fri, 24 Jul 2015 10:36:34 -0700 Message-ID: Subject: Re: down consoles fail to reconnect automatically From: Lonni J Friedman To: Bryan Stansell Content-Type: text/plain; charset=UTF-8 X-Spam-Score: -1.489 () BAYES_00,FREEMAIL_FROM,T_DKIM_INVALID X-Scanned-By: MIMEDefang 2.72 on 198.151.248.21 Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by underdog.stansell.org id t6OHab3V006470 Cc: users@conserver.com X-BeenThere: users@conserver.com X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Conserver Users List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Jul 2015 17:36:40 -0000 Hi Bryan, When I run "console -v -z bringup", I see a lot of "console initializing" for every session that is currently down. Then 10 seconds later, I see: connect timeout: forcing down for every console that was previously listed as initializing. For a console which was down (c042.ytr001.ix), and where I manually connected and brought it up immediately, I see: [Thu Jul 23 14:05:02 2015] conserver (13867): [c042.ytr001.ix] automatic reinitialization [Thu Jul 23 14:05:02 2015] conserver (13867): [c042.ytr001.ix] console initializing [Thu Jul 23 14:05:12 2015] conserver (13867): ERROR: [c042.ytr001.ix] connect timeout: forcing down [Thu Jul 23 14:05:23 2015] conserver (13867): [c042.ytr001.ix] login ncconserverprod@localhost [Thu Jul 23 14:05:23 2015] conserver (13867): [c042.ytr001.ix] console initializing [Thu Jul 23 14:05:26 2015] conserver (13867): [c042.ytr001.ix] console up Unfortunately, I don't currently have any consoles in the weird state of failing to re-initialize automatically, yet coming up immediately with a manual console session, so I can only look at what was logged yesterday. Let me know if you need any other info. thanks On Fri, Jul 24, 2015 at 12:23 AM, Bryan Stansell wrote: > What you’re doing sounds all correct, as are your expectations (it should attempt to bring up any downed consoles). My simple test setup shows that it works for me, but with lots of consoles, there could be a bug or some side-effect that happens with more. Or possibly some config settings that aren’t playing well together. Do you have any “interesting” messages in the conserver log file that appear when you run the command? > > Bryan > >> On Jul 23, 2015, at 10:59 AM, Lonni J Friedman wrote: >> >> Greetings, >> I'm running conserver-8.2.1 on an Ubuntu-14.04.2 server, with several >> thousand clients, connected over IPMI. Most of the time it works >> fine, however occasionally we lose a VPN concentrator that maintains a >> VPN tunnel between remote sites and the console server, and we a large >> number of console sessions go into the 'down' state. Usually when the >> tunnel comes back up, the console sessions come back up on their own, >> however there are times when they do not come back up for hours, or >> not at all for no obvious reason. In nearly 100% of those cases, if >> someone manually runs 'console $consoleName' (where $consoleName is >> the name of the console session that is listed as 'down'), it will >> immediately come back up. >> >> According to the 'console' man page ( >> http://www.conserver.com/docs/console.man.html ), if I invoke >> 'console' with: >> -z bringup >> >> it should "Try to connect all consoles marked as down (this is >> equivalent to sending the server a SIGUSR1)". I've tried that: >> #### >> $ console -v -z bringup >> console: interface address 127.0.0.1 (lo) >> console: interface address 10.200.53.130 (eth0) >> 127.0.0.1: ok -- bringing up consoles >> #### >> >> However it doesn't seem to do anything at all. None of the down >> consoles come up ever. Yet I can still force them up manually if I >> connect to them one at a time. >> >> I'm unclear whether I'm misunderstanding how the 'bringup' command is >> intended to work, or if there's a bug somewhere. >> >> Can someone comment? >> >> thanks! From consoleteam@gmail.com Sat Jul 25 18:27:14 2015 Received: from mail-pd0-f182.google.com (mail-pd0-f182.google.com [209.85.192.182]) by underdog.stansell.org (8.15.2/8.15.2) with ESMTPS id t6PIRCj5003718 (version=TLSv1.2 cipher=AES128-GCM-SHA256 bits=128 verify=FAIL); Sat, 25 Jul 2015 18:27:14 GMT Received: by pdbnt7 with SMTP id nt7so29403932pdb.0; Sat, 25 Jul 2015 11:27:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=references:mime-version:in-reply-to:content-type :content-transfer-encoding:message-id:cc:from:subject:date:to; bh=ir69xiJH2porebfq08HxbabWq1CEBNQRIullmbR7bEA=; b=TsXbQdB2hroWCJz/8EQdBHo+KKevOZPbao7Lo9C1t348uU3GQVty4ZzJGFokPOOMz5 o35aEC+7AcS13xpmHrmmABtOI0fDEibDmO1RIxahB3VM0j4Pog7jf/zDTz9WYeZZgyvM sO9fcR7S2DiKqys64K6Cb2fQbF2Wj+wuQfga9Jj6T246whAWEYnamaetuHgjTq8/HOuc kr630NF7qiu6Dq+YHdQK63HNEY/JfNgUWeGKjJUEAo3G+1RM+zSoeftuSba8UHS3G7/S YWtgTn/hPIu19kBSVOA5HPJnPznGg05accttgc6fw+K1TWFMKjTsHGBlEK8N6t4G05Il 0AkQ== X-Received: by 10.70.64.162 with SMTP id p2mr47079554pds.54.1437848830431; Sat, 25 Jul 2015 11:27:10 -0700 (PDT) Received: from [10.51.118.31] (mobile-166-171-249-145.mycingular.net. [166.171.249.145]) by smtp.gmail.com with ESMTPSA id b4sm12247834pdn.42.2015.07.25.11.27.08 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sat, 25 Jul 2015 11:27:08 -0700 (PDT) References: <2FF2E6C2-410C-47D6-8406-83B6E2CA369B@conserver.com> Mime-Version: 1.0 (1.0) In-Reply-To: Content-Type: text/plain; charset=utf-8 Message-Id: <6E88C5A2-346C-4B3E-A827-F9ADCCC69D65@gmail.com> X-Mailer: iPhone Mail (10B350) From: Zonker Subject: Re: down consoles fail to reconnect automatically Date: Sat, 25 Jul 2015 11:27:06 -0700 To: Lonni J Friedman X-Spam-Score: -1.489 () BAYES_00,FREEMAIL_FROM,T_DKIM_INVALID X-Scanned-By: MIMEDefang 2.72 on 198.151.248.21 Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by underdog.stansell.org id t6PIRCj5003718 Cc: "users@conserver.com" , Bryan Stansell X-BeenThere: users@conserver.com X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Conserver Users List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Jul 2015 18:27:14 -0000 What is the state of the VPN when they are reporting down? (Does the VPN recognize the conserver traffic, such that the traffic will trigger a downed VPN to re-initialize the VPN path?) Best regards, -Z- On Jul 24, 2015, at 10:36 AM, Lonni J Friedman wrote: > Hi Bryan, > When I run "console -v -z bringup", I see a lot of "console > initializing" for every session that is currently down. Then 10 > seconds later, I see: > connect timeout: forcing down > > for every console that was previously listed as initializing. > > For a console which was down (c042.ytr001.ix), and where I manually > connected and brought it up immediately, I see: > > [Thu Jul 23 14:05:02 2015] conserver (13867): [c042.ytr001.ix] > automatic reinitialization > [Thu Jul 23 14:05:02 2015] conserver (13867): [c042.ytr001.ix] console > initializing > [Thu Jul 23 14:05:12 2015] conserver (13867): ERROR: [c042.ytr001.ix] > connect timeout: forcing down > [Thu Jul 23 14:05:23 2015] conserver (13867): [c042.ytr001.ix] login > ncconserverprod@localhost > [Thu Jul 23 14:05:23 2015] conserver (13867): [c042.ytr001.ix] console > initializing > [Thu Jul 23 14:05:26 2015] conserver (13867): [c042.ytr001.ix] console up > > Unfortunately, I don't currently have any consoles in the weird state > of failing to re-initialize automatically, yet coming up immediately > with a manual console session, so I can only look at what was logged > yesterday. > > Let me know if you need any other info. > > thanks > > On Fri, Jul 24, 2015 at 12:23 AM, Bryan Stansell wrote: >> What you’re doing sounds all correct, as are your expectations (it should attempt to bring up any downed consoles). My simple test setup shows that it works for me, but with lots of consoles, there could be a bug or some side-effect that happens with more. Or possibly some config settings that aren’t playing well together. Do you have any “interesting” messages in the conserver log file that appear when you run the command? >> >> Bryan >> >>> On Jul 23, 2015, at 10:59 AM, Lonni J Friedman wrote: >>> >>> Greetings, >>> I'm running conserver-8.2.1 on an Ubuntu-14.04.2 server, with several >>> thousand clients, connected over IPMI. Most of the time it works >>> fine, however occasionally we lose a VPN concentrator that maintains a >>> VPN tunnel between remote sites and the console server, and we a large >>> number of console sessions go into the 'down' state. Usually when the >>> tunnel comes back up, the console sessions come back up on their own, >>> however there are times when they do not come back up for hours, or >>> not at all for no obvious reason. In nearly 100% of those cases, if >>> someone manually runs 'console $consoleName' (where $consoleName is >>> the name of the console session that is listed as 'down'), it will >>> immediately come back up. >>> >>> According to the 'console' man page ( >>> http://www.conserver.com/docs/console.man.html ), if I invoke >>> 'console' with: >>> -z bringup >>> >>> it should "Try to connect all consoles marked as down (this is >>> equivalent to sending the server a SIGUSR1)". I've tried that: >>> #### >>> $ console -v -z bringup >>> console: interface address 127.0.0.1 (lo) >>> console: interface address 10.200.53.130 (eth0) >>> 127.0.0.1: ok -- bringing up consoles >>> #### >>> >>> However it doesn't seem to do anything at all. None of the down >>> consoles come up ever. Yet I can still force them up manually if I >>> connect to them one at a time. >>> >>> I'm unclear whether I'm misunderstanding how the 'bringup' command is >>> intended to work, or if there's a bug somewhere. >>> >>> Can someone comment? >>> >>> thanks! > > _______________________________________________ > users mailing list > users@conserver.com > https://www.conserver.com/mailman/listinfo/users From bryan@conserver.com Sun Jul 26 09:43:40 2015 Received: from [192.168.2.2] (host-90-232-45-63.mobileonline.telia.com [90.232.45.63]) (authenticated bits=0) by underdog.stansell.org (8.15.2/8.15.2) with ESMTPSA id t6Q9hRxs029324 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Sun, 26 Jul 2015 09:43:39 GMT Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2102\)) Subject: Re: down consoles fail to reconnect automatically From: Bryan Stansell In-Reply-To: Date: Sun, 26 Jul 2015 02:43:24 -0700 Message-Id: <6341AEFE-34A6-43A4-B3DC-AB87847FE4C2@conserver.com> References: <2FF2E6C2-410C-47D6-8406-83B6E2CA369B@conserver.com> To: users@conserver.com X-Mailer: Apple Mail (2.2102) X-Spam-Score: 0.163 () BAYES_00,RDNS_DYNAMIC X-Scanned-By: MIMEDefang 2.72 on 198.151.248.21 Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by underdog.stansell.org id t6Q9hRxs029324 X-BeenThere: users@conserver.com X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Conserver Users List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Jul 2015 09:43:40 -0000 I keep looking at code and thinking about this, and the only thing that makes sense is a bug somewhere. When a SIGUSR1 or '-z bringup' happens, it just walks the consoles and performs a ConsInit() on them. The exact same thing happens when you connect to a console and "open" it (with some extra stuff for feedback to the client). So, my only explanation is that it's a bug somewhere. And just to clarify, if you run 'console -z bringup' multiple times, they continue to get "connect timeout: forcing down" messages? But as soon as you connect to one, it'll come up on the first try? I just want to make sure the situation is correct so I can, hopefully, think about how a bug might produce the situation and try and find a fix. Right now, though, I'm just scratching my head. Bryan > On Jul 24, 2015, at 10:36 AM, Lonni J Friedman wrote: > > Hi Bryan, > When I run "console -v -z bringup", I see a lot of "console > initializing" for every session that is currently down. Then 10 > seconds later, I see: > connect timeout: forcing down > > for every console that was previously listed as initializing. > > For a console which was down (c042.ytr001.ix), and where I manually > connected and brought it up immediately, I see: > > [Thu Jul 23 14:05:02 2015] conserver (13867): [c042.ytr001.ix] > automatic reinitialization > [Thu Jul 23 14:05:02 2015] conserver (13867): [c042.ytr001.ix] console > initializing > [Thu Jul 23 14:05:12 2015] conserver (13867): ERROR: [c042.ytr001.ix] > connect timeout: forcing down > [Thu Jul 23 14:05:23 2015] conserver (13867): [c042.ytr001.ix] login > ncconserverprod@localhost > [Thu Jul 23 14:05:23 2015] conserver (13867): [c042.ytr001.ix] console > initializing > [Thu Jul 23 14:05:26 2015] conserver (13867): [c042.ytr001.ix] console up > > Unfortunately, I don't currently have any consoles in the weird state > of failing to re-initialize automatically, yet coming up immediately > with a manual console session, so I can only look at what was logged > yesterday. > > Let me know if you need any other info. > > thanks > > On Fri, Jul 24, 2015 at 12:23 AM, Bryan Stansell wrote: >> What you’re doing sounds all correct, as are your expectations (it should attempt to bring up any downed consoles). My simple test setup shows that it works for me, but with lots of consoles, there could be a bug or some side-effect that happens with more. Or possibly some config settings that aren’t playing well together. Do you have any “interesting” messages in the conserver log file that appear when you run the command? >> >> Bryan >> >>> On Jul 23, 2015, at 10:59 AM, Lonni J Friedman wrote: >>> >>> Greetings, >>> I'm running conserver-8.2.1 on an Ubuntu-14.04.2 server, with several >>> thousand clients, connected over IPMI. Most of the time it works >>> fine, however occasionally we lose a VPN concentrator that maintains a >>> VPN tunnel between remote sites and the console server, and we a large >>> number of console sessions go into the 'down' state. Usually when the >>> tunnel comes back up, the console sessions come back up on their own, >>> however there are times when they do not come back up for hours, or >>> not at all for no obvious reason. In nearly 100% of those cases, if >>> someone manually runs 'console $consoleName' (where $consoleName is >>> the name of the console session that is listed as 'down'), it will >>> immediately come back up. >>> >>> According to the 'console' man page ( >>> http://www.conserver.com/docs/console.man.html ), if I invoke >>> 'console' with: >>> -z bringup >>> >>> it should "Try to connect all consoles marked as down (this is >>> equivalent to sending the server a SIGUSR1)". I've tried that: >>> #### >>> $ console -v -z bringup >>> console: interface address 127.0.0.1 (lo) >>> console: interface address 10.200.53.130 (eth0) >>> 127.0.0.1: ok -- bringing up consoles >>> #### >>> >>> However it doesn't seem to do anything at all. None of the down >>> consoles come up ever. Yet I can still force them up manually if I >>> connect to them one at a time. >>> >>> I'm unclear whether I'm misunderstanding how the 'bringup' command is >>> intended to work, or if there's a bug somewhere. >>> >>> Can someone comment? >>> >>> thanks! > > _______________________________________________ > users mailing list > users@conserver.com > https://www.conserver.com/mailman/listinfo/users From netllama@gmail.com Sun Jul 26 16:35:14 2015 Received: from mail-ig0-f175.google.com (mail-ig0-f175.google.com [209.85.213.175]) by underdog.stansell.org (8.15.2/8.15.2) with ESMTPS id t6QGZBk9020574 (version=TLSv1.2 cipher=AES128-GCM-SHA256 bits=128 verify=FAIL); Sun, 26 Jul 2015 16:35:13 GMT Received: by igbij6 with SMTP id ij6so46274286igb.1; Sun, 26 Jul 2015 09:35:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=14vFyVB5NOn7ucFF3nWkauDIzqy6iuuuMdiYsg79MJU=; b=ejL5FeRX7EsgwdWxd9VAxk2R7vWBmGbYSq6/hCG3yyfKbiySFXUxT86L3yH+t5g5GT 8TvFNVwczadS3hoQMyv8q8M99vhen/JglO2yQqrm8N46o7tawlUXCWWMVUBKlu9C1M4E ZDzS/nHDF467/evFbTZ9miryO98YFZkMoPbV0qqgO78krSu4LISFCg2HgtOA7x1toVH4 rDHzErgZ3Ehb8GdKqV0OSG5YzilSOeUWKpX4+wvn+bRhHthedxqvW7lWJugyJ652h7BD WNtEsk1R/BYfRJOSrizhpmCrc9rzZYW26ZJzMiIxQr4ypCRQNLFQdvHBrviiku0LLkPY HQXA== MIME-Version: 1.0 X-Received: by 10.50.7.68 with SMTP id h4mr11996853iga.40.1437928509941; Sun, 26 Jul 2015 09:35:09 -0700 (PDT) Received: by 10.50.224.148 with HTTP; Sun, 26 Jul 2015 09:35:09 -0700 (PDT) In-Reply-To: <6E88C5A2-346C-4B3E-A827-F9ADCCC69D65@gmail.com> References: <2FF2E6C2-410C-47D6-8406-83B6E2CA369B@conserver.com> <6E88C5A2-346C-4B3E-A827-F9ADCCC69D65@gmail.com> Date: Sun, 26 Jul 2015 09:35:09 -0700 Message-ID: Subject: Re: down consoles fail to reconnect automatically From: Lonni J Friedman To: Zonker Content-Type: text/plain; charset=UTF-8 X-Spam-Score: -1.489 () BAYES_00,FREEMAIL_FROM,T_DKIM_INVALID X-Scanned-By: MIMEDefang 2.72 on 198.151.248.21 Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by underdog.stansell.org id t6QGZBk9020574 Cc: "users@conserver.com" , Bryan Stansell X-BeenThere: users@conserver.com X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Conserver Users List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Jul 2015 16:35:14 -0000 The VPN is up & fully functional at the time when manually starting a console session brings up the 'down' console session. On Sat, Jul 25, 2015 at 11:27 AM, Zonker wrote: > What is the state of the VPN when they are reporting down? (Does the VPN recognize the conserver traffic, such that the traffic will trigger a downed VPN to re-initialize the VPN path?) > > Best regards, > > -Z- > > > On Jul 24, 2015, at 10:36 AM, Lonni J Friedman wrote: > >> Hi Bryan, >> When I run "console -v -z bringup", I see a lot of "console >> initializing" for every session that is currently down. Then 10 >> seconds later, I see: >> connect timeout: forcing down >> >> for every console that was previously listed as initializing. >> >> For a console which was down (c042.ytr001.ix), and where I manually >> connected and brought it up immediately, I see: >> >> [Thu Jul 23 14:05:02 2015] conserver (13867): [c042.ytr001.ix] >> automatic reinitialization >> [Thu Jul 23 14:05:02 2015] conserver (13867): [c042.ytr001.ix] console >> initializing >> [Thu Jul 23 14:05:12 2015] conserver (13867): ERROR: [c042.ytr001.ix] >> connect timeout: forcing down >> [Thu Jul 23 14:05:23 2015] conserver (13867): [c042.ytr001.ix] login >> ncconserverprod@localhost >> [Thu Jul 23 14:05:23 2015] conserver (13867): [c042.ytr001.ix] console >> initializing >> [Thu Jul 23 14:05:26 2015] conserver (13867): [c042.ytr001.ix] console up >> >> Unfortunately, I don't currently have any consoles in the weird state >> of failing to re-initialize automatically, yet coming up immediately >> with a manual console session, so I can only look at what was logged >> yesterday. >> >> Let me know if you need any other info. >> >> thanks >> >> On Fri, Jul 24, 2015 at 12:23 AM, Bryan Stansell wrote: >>> What you’re doing sounds all correct, as are your expectations (it should attempt to bring up any downed consoles). My simple test setup shows that it works for me, but with lots of consoles, there could be a bug or some side-effect that happens with more. Or possibly some config settings that aren’t playing well together. Do you have any “interesting” messages in the conserver log file that appear when you run the command? >>> >>> Bryan >>> >>>> On Jul 23, 2015, at 10:59 AM, Lonni J Friedman wrote: >>>> >>>> Greetings, >>>> I'm running conserver-8.2.1 on an Ubuntu-14.04.2 server, with several >>>> thousand clients, connected over IPMI. Most of the time it works >>>> fine, however occasionally we lose a VPN concentrator that maintains a >>>> VPN tunnel between remote sites and the console server, and we a large >>>> number of console sessions go into the 'down' state. Usually when the >>>> tunnel comes back up, the console sessions come back up on their own, >>>> however there are times when they do not come back up for hours, or >>>> not at all for no obvious reason. In nearly 100% of those cases, if >>>> someone manually runs 'console $consoleName' (where $consoleName is >>>> the name of the console session that is listed as 'down'), it will >>>> immediately come back up. >>>> >>>> According to the 'console' man page ( >>>> http://www.conserver.com/docs/console.man.html ), if I invoke >>>> 'console' with: >>>> -z bringup >>>> >>>> it should "Try to connect all consoles marked as down (this is >>>> equivalent to sending the server a SIGUSR1)". I've tried that: >>>> #### >>>> $ console -v -z bringup >>>> console: interface address 127.0.0.1 (lo) >>>> console: interface address 10.200.53.130 (eth0) >>>> 127.0.0.1: ok -- bringing up consoles >>>> #### >>>> >>>> However it doesn't seem to do anything at all. None of the down >>>> consoles come up ever. Yet I can still force them up manually if I >>>> connect to them one at a time. >>>> >>>> I'm unclear whether I'm misunderstanding how the 'bringup' command is >>>> intended to work, or if there's a bug somewhere. >>>> >>>> Can someone comment? >>>> >>>> thanks! From netllama@gmail.com Tue Jul 28 14:02:38 2015 Received: from mail-ig0-f178.google.com (mail-ig0-f178.google.com [209.85.213.178]) by underdog.stansell.org (8.15.2/8.15.2) with ESMTPS id t6SE2aRx006950 (version=TLSv1.2 cipher=AES128-GCM-SHA256 bits=128 verify=FAIL); Tue, 28 Jul 2015 14:02:38 GMT Received: by igk11 with SMTP id 11so107744956igk.1; Tue, 28 Jul 2015 07:02:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=GjVNj8K7OexM3+S9uWw9gruPMSFUGVaqk6gGeZ0+8ho=; b=cTgztUy61NefvNs+ICd4Llpm13QGpNi22A+vM2dhsuScC7gML0GNbn+mFOKqgVfWuR 5KSZGsB8b9HO/VSHcBN4BGvch5dpi2HLhqu3bqTb8x/FOXz+GWiSQ1IoQ7UQg0nDwkHG Z0FERLdrKX8liZBAptGv0zPD/DXZs/7DIjWnWNQ2NG4Z0ir+B6EarLI4Q2hGK+HHLiPc HdGIrKhdNauR5SyIm1pL2dySScR+HR/Ux6gVl2R3jeFXis8eHZsill6Nv3R2boS930Vy cY0JgY4j2pki9FqzEd4TpOnDMFMExIEtqb6CLROtAKw5Ibjwl7OFeJDtw2B7HHRdJdTI brZA== MIME-Version: 1.0 X-Received: by 10.107.161.71 with SMTP id k68mr54490010ioe.118.1438092154852; Tue, 28 Jul 2015 07:02:34 -0700 (PDT) Received: by 10.50.224.148 with HTTP; Tue, 28 Jul 2015 07:02:34 -0700 (PDT) In-Reply-To: <6341AEFE-34A6-43A4-B3DC-AB87847FE4C2@conserver.com> References: <2FF2E6C2-410C-47D6-8406-83B6E2CA369B@conserver.com> <6341AEFE-34A6-43A4-B3DC-AB87847FE4C2@conserver.com> Date: Tue, 28 Jul 2015 07:02:34 -0700 Message-ID: Subject: Re: down consoles fail to reconnect automatically From: Lonni J Friedman To: Bryan Stansell Content-Type: text/plain; charset=UTF-8 X-Spam-Score: -1.489 () BAYES_00,FREEMAIL_FROM,T_DKIM_INVALID X-Scanned-By: MIMEDefang 2.72 on 198.151.248.21 Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by underdog.stansell.org id t6SE2aRx006950 Cc: users@conserver.com X-BeenThere: users@conserver.com X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Conserver Users List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Jul 2015 14:02:39 -0000 On Sun, Jul 26, 2015 at 2:43 AM, Bryan Stansell wrote: > I keep looking at code and thinking about this, and the only thing that makes sense is a bug somewhere. When a SIGUSR1 or '-z bringup' happens, it just walks the consoles and performs a ConsInit() on them. The exact same thing happens when you connect to a console and "open" it (with some extra stuff for feedback to the client). So, my only explanation is that it's a bug somewhere. > > And just to clarify, if you run 'console -z bringup' multiple times, they continue to get "connect timeout: forcing down" messages? But as soon as you connect to one, it'll come up on the first try? I just want to make sure the situation is correct so I can, hopefully, think about how a bug might produce the situation and try and find a fix. Right now, though, I'm just scratching my head. Yes, that's exactly the behavior I've seen. > > Bryan > >> On Jul 24, 2015, at 10:36 AM, Lonni J Friedman wrote: >> >> Hi Bryan, >> When I run "console -v -z bringup", I see a lot of "console >> initializing" for every session that is currently down. Then 10 >> seconds later, I see: >> connect timeout: forcing down >> >> for every console that was previously listed as initializing. >> >> For a console which was down (c042.ytr001.ix), and where I manually >> connected and brought it up immediately, I see: >> >> [Thu Jul 23 14:05:02 2015] conserver (13867): [c042.ytr001.ix] >> automatic reinitialization >> [Thu Jul 23 14:05:02 2015] conserver (13867): [c042.ytr001.ix] console >> initializing >> [Thu Jul 23 14:05:12 2015] conserver (13867): ERROR: [c042.ytr001.ix] >> connect timeout: forcing down >> [Thu Jul 23 14:05:23 2015] conserver (13867): [c042.ytr001.ix] login >> ncconserverprod@localhost >> [Thu Jul 23 14:05:23 2015] conserver (13867): [c042.ytr001.ix] console >> initializing >> [Thu Jul 23 14:05:26 2015] conserver (13867): [c042.ytr001.ix] console up >> >> Unfortunately, I don't currently have any consoles in the weird state >> of failing to re-initialize automatically, yet coming up immediately >> with a manual console session, so I can only look at what was logged >> yesterday. >> >> Let me know if you need any other info. >> >> thanks >> >> On Fri, Jul 24, 2015 at 12:23 AM, Bryan Stansell wrote: >>> What you’re doing sounds all correct, as are your expectations (it should attempt to bring up any downed consoles). My simple test setup shows that it works for me, but with lots of consoles, there could be a bug or some side-effect that happens with more. Or possibly some config settings that aren’t playing well together. Do you have any “interesting” messages in the conserver log file that appear when you run the command? >>> >>> Bryan >>> >>>> On Jul 23, 2015, at 10:59 AM, Lonni J Friedman wrote: >>>> >>>> Greetings, >>>> I'm running conserver-8.2.1 on an Ubuntu-14.04.2 server, with several >>>> thousand clients, connected over IPMI. Most of the time it works >>>> fine, however occasionally we lose a VPN concentrator that maintains a >>>> VPN tunnel between remote sites and the console server, and we a large >>>> number of console sessions go into the 'down' state. Usually when the >>>> tunnel comes back up, the console sessions come back up on their own, >>>> however there are times when they do not come back up for hours, or >>>> not at all for no obvious reason. In nearly 100% of those cases, if >>>> someone manually runs 'console $consoleName' (where $consoleName is >>>> the name of the console session that is listed as 'down'), it will >>>> immediately come back up. >>>> >>>> According to the 'console' man page ( >>>> http://www.conserver.com/docs/console.man.html ), if I invoke >>>> 'console' with: >>>> -z bringup >>>> >>>> it should "Try to connect all consoles marked as down (this is >>>> equivalent to sending the server a SIGUSR1)". I've tried that: >>>> #### >>>> $ console -v -z bringup >>>> console: interface address 127.0.0.1 (lo) >>>> console: interface address 10.200.53.130 (eth0) >>>> 127.0.0.1: ok -- bringing up consoles >>>> #### >>>> >>>> However it doesn't seem to do anything at all. None of the down >>>> consoles come up ever. Yet I can still force them up manually if I >>>> connect to them one at a time. >>>> >>>> I'm unclear whether I'm misunderstanding how the 'bringup' command is >>>> intended to work, or if there's a bug somewhere. >>>> >>>> Can someone comment? >>>> >>>> thanks! From netllama@gmail.com Fri Jul 31 17:41:55 2015 Received: from mail-ig0-f171.google.com (mail-ig0-f171.google.com [209.85.213.171]) by underdog.stansell.org (8.15.2/8.15.2) with ESMTPS id t6VHfqj5025002 (version=TLSv1.2 cipher=AES128-GCM-SHA256 bits=128 verify=FAIL); Fri, 31 Jul 2015 17:41:54 GMT Received: by iggf3 with SMTP id f3so21530096igg.1; Fri, 31 Jul 2015 10:41:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=ntqJCvZWbZkjihe3k+93NxACtMY9hANQWmZgTw+R0kk=; b=f+wwRe8Of7C5V/DSWghXGa6CPGQfjgMySdO5V7+zdP8q5AW/rld50GTGlFFrdYGDey pfBmuxkJJnMEHZJczQY7bouUbFToRpcSxUl+B5zTC9/kIO1LV0erpEe8v8WLfhg3wz5w Arh2ecqY/NobiRG423Aw3+YgOtvKdhY1xx7Yvjq1o7uS6/FNjJnxhTY+uLdU8bb74b1d U4O4MV35i27HEPYn4cWPzC33F7i8E7BOoOURrlgBNUtJOdP0ezN0ZLSEnUJfrQE06mzn Uo689VHgVfZT5+uOhYE4eXhdwV8pLGMaB31nx3ZY5OpR7JgQMpo99Xkdl9Us/7Tl4ybm A+Iw== MIME-Version: 1.0 X-Received: by 10.50.56.10 with SMTP id w10mr7781135igp.3.1438364510931; Fri, 31 Jul 2015 10:41:50 -0700 (PDT) Received: by 10.50.224.148 with HTTP; Fri, 31 Jul 2015 10:41:50 -0700 (PDT) In-Reply-To: References: <2FF2E6C2-410C-47D6-8406-83B6E2CA369B@conserver.com> <6341AEFE-34A6-43A4-B3DC-AB87847FE4C2@conserver.com> Date: Fri, 31 Jul 2015 10:41:50 -0700 Message-ID: Subject: Re: down consoles fail to reconnect automatically From: Lonni J Friedman To: Bryan Stansell Content-Type: text/plain; charset=UTF-8 X-Spam-Score: -1.489 () BAYES_00,FREEMAIL_FROM,T_DKIM_INVALID X-Scanned-By: MIMEDefang 2.72 on 198.151.248.21 Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by underdog.stansell.org id t6VHfqj5025002 Cc: users@conserver.com X-BeenThere: users@conserver.com X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Conserver Users List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 31 Jul 2015 17:41:56 -0000 Hi Bryan, I was wondering if you had any ideas about this issue? Or id you needed any more info from me to investigate further? thanks On Tue, Jul 28, 2015 at 7:02 AM, Lonni J Friedman wrote: > On Sun, Jul 26, 2015 at 2:43 AM, Bryan Stansell wrote: >> I keep looking at code and thinking about this, and the only thing that makes sense is a bug somewhere. When a SIGUSR1 or '-z bringup' happens, it just walks the consoles and performs a ConsInit() on them. The exact same thing happens when you connect to a console and "open" it (with some extra stuff for feedback to the client). So, my only explanation is that it's a bug somewhere. >> >> And just to clarify, if you run 'console -z bringup' multiple times, they continue to get "connect timeout: forcing down" messages? But as soon as you connect to one, it'll come up on the first try? I just want to make sure the situation is correct so I can, hopefully, think about how a bug might produce the situation and try and find a fix. Right now, though, I'm just scratching my head. > > Yes, that's exactly the behavior I've seen. > >> >> Bryan >> >>> On Jul 24, 2015, at 10:36 AM, Lonni J Friedman wrote: >>> >>> Hi Bryan, >>> When I run "console -v -z bringup", I see a lot of "console >>> initializing" for every session that is currently down. Then 10 >>> seconds later, I see: >>> connect timeout: forcing down >>> >>> for every console that was previously listed as initializing. >>> >>> For a console which was down (c042.ytr001.ix), and where I manually >>> connected and brought it up immediately, I see: >>> >>> [Thu Jul 23 14:05:02 2015] conserver (13867): [c042.ytr001.ix] >>> automatic reinitialization >>> [Thu Jul 23 14:05:02 2015] conserver (13867): [c042.ytr001.ix] console >>> initializing >>> [Thu Jul 23 14:05:12 2015] conserver (13867): ERROR: [c042.ytr001.ix] >>> connect timeout: forcing down >>> [Thu Jul 23 14:05:23 2015] conserver (13867): [c042.ytr001.ix] login >>> ncconserverprod@localhost >>> [Thu Jul 23 14:05:23 2015] conserver (13867): [c042.ytr001.ix] console >>> initializing >>> [Thu Jul 23 14:05:26 2015] conserver (13867): [c042.ytr001.ix] console up >>> >>> Unfortunately, I don't currently have any consoles in the weird state >>> of failing to re-initialize automatically, yet coming up immediately >>> with a manual console session, so I can only look at what was logged >>> yesterday. >>> >>> Let me know if you need any other info. >>> >>> thanks >>> >>> On Fri, Jul 24, 2015 at 12:23 AM, Bryan Stansell wrote: >>>> What you’re doing sounds all correct, as are your expectations (it should attempt to bring up any downed consoles). My simple test setup shows that it works for me, but with lots of consoles, there could be a bug or some side-effect that happens with more. Or possibly some config settings that aren’t playing well together. Do you have any “interesting” messages in the conserver log file that appear when you run the command? >>>> >>>> Bryan >>>> >>>>> On Jul 23, 2015, at 10:59 AM, Lonni J Friedman wrote: >>>>> >>>>> Greetings, >>>>> I'm running conserver-8.2.1 on an Ubuntu-14.04.2 server, with several >>>>> thousand clients, connected over IPMI. Most of the time it works >>>>> fine, however occasionally we lose a VPN concentrator that maintains a >>>>> VPN tunnel between remote sites and the console server, and we a large >>>>> number of console sessions go into the 'down' state. Usually when the >>>>> tunnel comes back up, the console sessions come back up on their own, >>>>> however there are times when they do not come back up for hours, or >>>>> not at all for no obvious reason. In nearly 100% of those cases, if >>>>> someone manually runs 'console $consoleName' (where $consoleName is >>>>> the name of the console session that is listed as 'down'), it will >>>>> immediately come back up. >>>>> >>>>> According to the 'console' man page ( >>>>> http://www.conserver.com/docs/console.man.html ), if I invoke >>>>> 'console' with: >>>>> -z bringup >>>>> >>>>> it should "Try to connect all consoles marked as down (this is >>>>> equivalent to sending the server a SIGUSR1)". I've tried that: >>>>> #### >>>>> $ console -v -z bringup >>>>> console: interface address 127.0.0.1 (lo) >>>>> console: interface address 10.200.53.130 (eth0) >>>>> 127.0.0.1: ok -- bringing up consoles >>>>> #### >>>>> >>>>> However it doesn't seem to do anything at all. None of the down >>>>> consoles come up ever. Yet I can still force them up manually if I >>>>> connect to them one at a time. >>>>> >>>>> I'm unclear whether I'm misunderstanding how the 'bringup' command is >>>>> intended to work, or if there's a bug somewhere. >>>>> >>>>> Can someone comment? >>>>> >>>>> thanks! From bryan@conserver.com Fri Jul 31 19:17:00 2015 Received: from [192.168.2.2] ([95.202.143.243]) (authenticated bits=0) by underdog.stansell.org (8.15.2/8.15.2) with ESMTPSA id t6VJGlgZ006867 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Fri, 31 Jul 2015 19:16:59 GMT Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2102\)) Subject: Re: down consoles fail to reconnect automatically From: Bryan Stansell In-Reply-To: Date: Fri, 31 Jul 2015 12:16:46 -0700 Message-Id: <673D4FD8-0B41-4289-A908-F734593DC638@conserver.com> References: <2FF2E6C2-410C-47D6-8406-83B6E2CA369B@conserver.com> <6341AEFE-34A6-43A4-B3DC-AB87847FE4C2@conserver.com> To: users@conserver.com X-Mailer: Apple Mail (2.2102) X-Spam-Score: -0.024 () BAYES_00,HELO_MISC_IP,RDNS_NONE X-Scanned-By: MIMEDefang 2.72 on 198.151.248.21 Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by underdog.stansell.org id t6VJGlgZ006867 X-BeenThere: users@conserver.com X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Conserver Users List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 31 Jul 2015 19:17:00 -0000 I'm still at a bit of a loss as to what's up. I'm not even sure what info would help yet. The extreme measure would be to run conserver in debug mode so that you get copious logs and reproduce the issue. But, even with that, I'm not sure the existing logging would help. And with several thousand connections, that's a lot of stuff. I can only look at this stuff in my free time, but I'm still investigating. We could/should probably email directly until we have some sort of result that can be shared with the entire list. Bryan > On Jul 31, 2015, at 10:41 AM, Lonni J Friedman wrote: > > Hi Bryan, > I was wondering if you had any ideas about this issue? Or id you > needed any more info from me to investigate further? > > thanks > > On Tue, Jul 28, 2015 at 7:02 AM, Lonni J Friedman wrote: >> On Sun, Jul 26, 2015 at 2:43 AM, Bryan Stansell wrote: >>> I keep looking at code and thinking about this, and the only thing that makes sense is a bug somewhere. When a SIGUSR1 or '-z bringup' happens, it just walks the consoles and performs a ConsInit() on them. The exact same thing happens when you connect to a console and "open" it (with some extra stuff for feedback to the client). So, my only explanation is that it's a bug somewhere. >>> >>> And just to clarify, if you run 'console -z bringup' multiple times, they continue to get "connect timeout: forcing down" messages? But as soon as you connect to one, it'll come up on the first try? I just want to make sure the situation is correct so I can, hopefully, think about how a bug might produce the situation and try and find a fix. Right now, though, I'm just scratching my head. >> >> Yes, that's exactly the behavior I've seen. >> >>> >>> Bryan >>> >>>> On Jul 24, 2015, at 10:36 AM, Lonni J Friedman wrote: >>>> >>>> Hi Bryan, >>>> When I run "console -v -z bringup", I see a lot of "console >>>> initializing" for every session that is currently down. Then 10 >>>> seconds later, I see: >>>> connect timeout: forcing down >>>> >>>> for every console that was previously listed as initializing. >>>> >>>> For a console which was down (c042.ytr001.ix), and where I manually >>>> connected and brought it up immediately, I see: >>>> >>>> [Thu Jul 23 14:05:02 2015] conserver (13867): [c042.ytr001.ix] >>>> automatic reinitialization >>>> [Thu Jul 23 14:05:02 2015] conserver (13867): [c042.ytr001.ix] console >>>> initializing >>>> [Thu Jul 23 14:05:12 2015] conserver (13867): ERROR: [c042.ytr001.ix] >>>> connect timeout: forcing down >>>> [Thu Jul 23 14:05:23 2015] conserver (13867): [c042.ytr001.ix] login >>>> ncconserverprod@localhost >>>> [Thu Jul 23 14:05:23 2015] conserver (13867): [c042.ytr001.ix] console >>>> initializing >>>> [Thu Jul 23 14:05:26 2015] conserver (13867): [c042.ytr001.ix] console up >>>> >>>> Unfortunately, I don't currently have any consoles in the weird state >>>> of failing to re-initialize automatically, yet coming up immediately >>>> with a manual console session, so I can only look at what was logged >>>> yesterday. >>>> >>>> Let me know if you need any other info. >>>> >>>> thanks >>>> >>>> On Fri, Jul 24, 2015 at 12:23 AM, Bryan Stansell wrote: >>>>> What you’re doing sounds all correct, as are your expectations (it should attempt to bring up any downed consoles). My simple test setup shows that it works for me, but with lots of consoles, there could be a bug or some side-effect that happens with more. Or possibly some config settings that aren’t playing well together. Do you have any “interesting” messages in the conserver log file that appear when you run the command? >>>>> >>>>> Bryan >>>>> >>>>>> On Jul 23, 2015, at 10:59 AM, Lonni J Friedman wrote: >>>>>> >>>>>> Greetings, >>>>>> I'm running conserver-8.2.1 on an Ubuntu-14.04.2 server, with several >>>>>> thousand clients, connected over IPMI. Most of the time it works >>>>>> fine, however occasionally we lose a VPN concentrator that maintains a >>>>>> VPN tunnel between remote sites and the console server, and we a large >>>>>> number of console sessions go into the 'down' state. Usually when the >>>>>> tunnel comes back up, the console sessions come back up on their own, >>>>>> however there are times when they do not come back up for hours, or >>>>>> not at all for no obvious reason. In nearly 100% of those cases, if >>>>>> someone manually runs 'console $consoleName' (where $consoleName is >>>>>> the name of the console session that is listed as 'down'), it will >>>>>> immediately come back up. >>>>>> >>>>>> According to the 'console' man page ( >>>>>> http://www.conserver.com/docs/console.man.html ), if I invoke >>>>>> 'console' with: >>>>>> -z bringup >>>>>> >>>>>> it should "Try to connect all consoles marked as down (this is >>>>>> equivalent to sending the server a SIGUSR1)". I've tried that: >>>>>> #### >>>>>> $ console -v -z bringup >>>>>> console: interface address 127.0.0.1 (lo) >>>>>> console: interface address 10.200.53.130 (eth0) >>>>>> 127.0.0.1: ok -- bringing up consoles >>>>>> #### >>>>>> >>>>>> However it doesn't seem to do anything at all. None of the down >>>>>> consoles come up ever. Yet I can still force them up manually if I >>>>>> connect to them one at a time. >>>>>> >>>>>> I'm unclear whether I'm misunderstanding how the 'bringup' command is >>>>>> intended to work, or if there's a bug somewhere. >>>>>> >>>>>> Can someone comment? >>>>>> >>>>>> thanks!