| Guest19 | anyone have a working guest session with Beowulf 3.1 and LXQT? | 00:18 |
|---|---|---|
| rwp | Guest19, What is a "guest session"? Is that where someone is logged into X on DISPLAY=:0 and then they lock the screen and from the locked screen someone else can open another X session logged in with DISPLAY=:1 is that it? | 00:25 |
| Guest19 | Simpler than that, lightdm has option to enable temporary Guest user without password that uses tmpfs for homedir, then gets wiped out on logout | 00:29 |
| rwp | lightdm has that? I have used lightdm but I never noticed. /me goes off to look... | 00:33 |
| Guest19 | yeah, it's been disabled by default for a few years in lightdm.conf, but its there | 00:34 |
| rwp | Oh, then that's why I don't see it just now when I went looking. I see in /etc/lightdm/lightdm.conf a "allow-guest = True" commented out entry. | 00:37 |
| Guest19 | yeah, that's one of a few needed. I found it was missing some scripts needed for working guest session but package arctica-greeter-guest-session provides it all back, but even with that, the greeter does show guest session option, but it kicks out immediately after mouse cursor and black/blank desktop appears for a sec, then logs back out to | 00:40 |
| Guest19 | greeter. | 00:40 |
| Guest19 | and not seeing anything obvious in /var/log/lightdm.log, comparing to a working user. The guest accounts are cleaned up, so nothing remains after the login/logout cycle, but /var/log/wtmp does show the new guest-xxxx user existed briefly, as well as in lightdm.log | 00:43 |
| ted-ious | Guest19: Is there a reason to use beowulf for something that is concerned with security? | 00:43 |
| Guest19 | yeah, older laptop hardware, its not used for anything important, just some basic browsing and youtube | 00:44 |
| Guest19 | actually I can try Chimaera, but the newer browsers are a bit heavier. | 00:46 |
| rwp | JFTR but I have no idea. Good luck and good hunting! | 00:54 |
| Guest19 | no prob, just wondered if people running guest sessions were common, I've got a few suspects for the failure and will try Chimaera in the next few days | 01:00 |
| ted-ious | Guest19: Browsers are the one set of apps I think it's insane to not upgrade. | 01:44 |
| ted-ious | You're going to get hacked sooner than later if you are running an old browser. | 01:45 |
| ted-ious | If you need to save ram you can try something like surf with javascript disabled. | 01:45 |
| ted-ious | Or lynx. :) | 01:46 |
| n4dir | netsurf was not too bad last time i checked | 01:47 |
| n4dir | falkon doesn't help that much, but a little bit | 01:48 |
| Guest19 | nah, not worried about getting hacked on this basically single-purpose device; they can hack the guest account all they want, it gets wiped out on boot. Browser bloat is a problem for me though. But thanks for the browser suggestions guys, I'll check those out. | 02:03 |
| n4dir | Guest19: netsurf is really very basic, but at least not a command-line-webbrowser. falkon, last time i checked, is a modern browser and works on all sites, but you don't really save a lot of ressources. Worth a try though | 02:05 |
| systemdlete | gnarface (and anyone else): I'm seeing intermittent problems with "apt update" from apt-cacher-ng server: e.g, W: Failed to fetch http://security.debian.org/debian[...] Connection failed [IP: 172.XX.YY.ZZ 3142] | 02:16 |
| systemdlete | I have a log at https://dpaste.com/9GGU7RQRR | 02:18 |
| systemdlete | I swear on a stack of Linux Manuals that I have not mucked (much) with the acng.conf file. I followed the directions gnarface offered me, and consulted the man pages (and some web pages also) | 02:19 |
| systemdlete | https://unix.stackexchange.com/questions/623174/apt-cacher-ng-random-download-failures-with-apt-update-acgn | 02:19 |
| systemdlete | I disabled ipv6 EVERYWHERE on my home network, but it did not help. (I was just following their suggestion) | 02:19 |
| onefang | I use falkon when all my firefox-esr protective plugins means some crappy web site wont work. | 02:20 |
| systemdlete | Apparently, I am not the only one having this issue. It is intermittent, which makes it hard to figure out. | 02:20 |
| systemdlete | I think my network is OK, since I DO get successful apt-get's | 02:21 |
| systemdlete | And no other programs or applications are complaining about network issues (recently I mean). | 02:22 |
| gnarface | systemdlete: mine hasn't made a peep lately | 02:22 |
| systemdlete | gnarface, this seems to be most frequent when hitting a lot of packages at once, such as the dist-upgrade I am attempting on one system | 02:23 |
| systemdlete | The first one (apt-get upgrade, after repointing the sources file to chimaera from beowulf) was 640 packages, the second one over 1000. | 02:24 |
| systemdlete | But I've also seen it, a bit less often maybe, with small updates | 02:24 |
| gnarface | hmm, i also haven't updated that much at once for a while, i suspect | 02:24 |
| systemdlete | Overall, I am VERY pleased with the cacher! | 02:24 |
| gnarface | i wonder if it's a problem where the cache can invalidate due to age while the update is still in progress? | 02:25 |
| systemdlete | I'm wondering if I should try to throttle apt-cacher-ng somehow | 02:25 |
| systemdlete | hmmm. | 02:25 |
| systemdlete | how often are packages released to the repos? | 02:25 |
| systemdlete | I mean, this happens several times within, say, an hour | 02:25 |
| gnarface | i could only speculate... doesn't seem like quite the error i ever got, the only transient errors i ever got it was clearly complaining about some checksum mismatch, and it'd always just go away if i waited until the next cache invalidation | 02:26 |
| gnarface | (or if i logged into the control panel and flushed it manually if i was feeling impatient) | 02:26 |
| systemdlete | I have seen a lot of web pages re your issue (checksum mismatch), but I really haven't noticed that one myself here. | 02:26 |
| gnarface | odd | 02:27 |
| gnarface | some other hidden environmental cause we must be missing | 02:27 |
| systemdlete | It's really strange, because I was in the middle of the upgrade and dist-upgrade steps, and the process would start getting hit with these errors. Then, I simply restarted it as suggested on devuan's upgrade instructions page. Then, no problem with same packages, only moments later. But then others would/might fail, and rinse and repeat... | 02:28 |
| gnarface | so, i wonder if it could just be something caused by the repos updating during your downloads | 02:28 |
| gnarface | you said it was in the security section after all, which is what i'd expect to be changing most | 02:29 |
| systemdlete | as you were saying, above | 02:29 |
| systemdlete | no, that was an "e.g." ^^^^ | 02:29 |
| gnarface | oh | 02:29 |
| systemdlete | there are many different errors; that was just the general format | 02:29 |
| systemdlete | (I'm trying hard to be precise as possible here...) | 02:30 |
| gnarface | is there any firewall between you and the apt-cacher-ng server? i wonder if it could just be an issue with it failing to hold enough connection states? | 02:30 |
| systemdlete | I'm afraid that the actual messages I was seeing rolled off the bottom of my copy-paste buffer | 02:30 |
| systemdlete | yes, there are a few | 02:31 |
| systemdlete | but! | 02:31 |
| systemdlete | this happens (sometimes) even on the cacher server itself!!! | 02:31 |
| systemdlete | That server DOES have a firewall running, but I don't think that will matter for local connections, right? | 02:31 |
| gnarface | depends on how you set it up | 02:32 |
| systemdlete | that's why I am not very suspicious of the network, at least not for the cacher clients | 02:32 |
| systemdlete | It's a basic gufw generated fw | 02:32 |
| gnarface | but i know that if you're tracking state of inbound or outbound connections, there's a theoretical possibility of running up against a kernel memory limit | 02:33 |
| onefang | We tell package mirrors to update every 30 minutes, but some updated less often. | 02:33 |
| systemdlete | I've added a few pinholes for ports like the cacher's | 02:33 |
| gnarface | like, if you end up tracking too many states at once you could start losing connections randomly | 02:33 |
| gnarface | usually this is not a problem with default kernel builds unless you're doing something weird that is causing connections to stick around too long though | 02:34 |
| systemdlete | ok, so maybe bumping up the limit might help? | 02:34 |
| gnarface | sure, in theory | 02:34 |
| gnarface | although often needing to is the sign of something else wrong | 02:35 |
| systemdlete | I mean, this is overall rather minor. It's mainly a nuisance. | 02:35 |
| systemdlete | just restarting the upgrade has always been enough to get it working again (so far) | 02:35 |
| systemdlete | gnarface, did you look at the paste I put up? | 02:36 |
| gnarface | systemdlete: no, and i doubt it will give me any insight but i will if you use paste.debian.net or just /msg it to me | 02:37 |
| systemdlete | you don't trust dpaste.bin? | 02:37 |
| gnarface | i just stopped adding new domains to my personal "trust" list years ago | 02:37 |
| gnarface | it's not about any particular domain, it's about reducing attack surface | 02:37 |
| gnarface | not just for me, but also for them | 02:38 |
| gnarface | (i'm a mafia target and sometimes that gets innocents involved unnecessarily) | 02:38 |
| systemdlete | https://paste.debian.net/hidden/779bf2a3/ | 02:38 |
| systemdlete | (sorry to hear that bad news) | 02:38 |
| gnarface | hmm, this is a error i certainly never have seen | 02:39 |
| gnarface | failed to move stale item out of the way... no such file or directory | 02:39 |
| systemdlete | hmmm, let me look at inodes | 02:39 |
| gnarface | scattered across several hours | 02:40 |
| gnarface | so it can't move it out of the way because... it's gone already? | 02:40 |
| gnarface | you don't have two apt-cacher-ng instances running on the same cache directory, do you? | 02:41 |
| systemdlete | sneaky devils. | 02:41 |
| systemdlete | uh, I don't think so... | 02:41 |
| systemdlete | no, just one | 02:41 |
| gnarface | double check, but this seems like quite a puzzle | 02:41 |
| systemdlete | I have over 1.4M available inodes on /var | 02:41 |
| gnarface | do you think disk I/O is really high at those points? | 02:42 |
| systemdlete | good point... | 02:42 |
| gnarface | i wonder if it could be some sort of race condition caused by disk bottlenecknig | 02:42 |
| gnarface | just guessing | 02:42 |
| systemdlete | maybe, but wouldn't that be a problem on the upstream side also? | 02:42 |
| gnarface | dunno | 02:42 |
| gnarface | never seen this before but my cache server is not under load | 02:42 |
| systemdlete | I mean, I'm only seeing the problem from the client's POV. On the other hand, that log file seems to indicate :I: and :O: which I am guessing is input and output | 02:43 |
| gnarface | not only is it not under load but i actually just recently upgraded it to a hardware raid with two SSDs | 02:43 |
| systemdlete | I'm still on mech disks. They are still a good value for the money here. | 02:43 |
| systemdlete | but anyway | 02:44 |
| gnarface | i ran the old one right into the ground, it died making grinding noises :D | 02:44 |
| gnarface | i figured it was a good idea to have drives that were less sensitive to vibration | 02:45 |
| gnarface | (my downstairs neighbors bang on the walls a lot) | 02:45 |
| systemdlete | Some of these drives (2TB Seagate and 2TB WD) are up to 3 years old, maybe older. Since I started leaving my PCs on 24x7, the frequency of hardware replacements has gone waaaaay down (MB, disks, etc). So I've become a believer. | 02:46 |
| systemdlete | well, that can't be the only annoying thing arising from that problem | 02:46 |
| gnarface | well, if it were the drives you'd see i/o errors in dmesg around the same time | 02:48 |
| gnarface | but i'm suspecting this is not hardware related | 02:48 |
| systemdlete | most of the clients are VMs | 02:49 |
| gnarface | it looks like a race condition but i can't imagine why it'd be happening | 02:49 |
| systemdlete | and I have a thruk installation here so I will find out fairly quickly if there are problems. | 02:49 |
| systemdlete | and I also check, periodically, the host log files for hw errors | 02:50 |
| gnarface | how many VMs you usually run the updates with at a time? | 02:53 |
| gnarface | i'm rarely updating more than 2 things at once, i wonder if you're seeing something i'm not seeing just because i'm not throwing 1000 clients at it at once | 02:54 |
| systemdlete | only one running apt upgrade at a time, otherwise the whole network crawls. However, there can be more than one running apt update at a time | 02:54 |
| gnarface | hmm | 02:54 |
| systemdlete | well, as I said, even casual upgrades of just a few packages can see this happen... iirc | 02:55 |
| systemdlete | when I upgrade from chimaera to daedalus, I will try to monitor the caecher server with htop | 02:57 |
| systemdlete | (I'm upgrading beowulf -> chimaera -> daedalus because skip upgrades are not officially supported) | 02:58 |
| systemdlete | the chimaera upgrade is almost complete. It is unpacking and installing the dist-upgrade part right now. | 02:59 |
| systemdlete | So if all goes well, and the resulting chimaera doesn't have problems for a day or so, I'll do the daedalus upgrade. | 02:59 |
| systemdlete | gnarface, how do I msg you a file? I have it gzip'd down to about .5MB (I cannot paste it to deb paste bin) | 03:13 |
| systemdlete | this file is the actual log, whereas what I pasted was the error log | 03:14 |
| gnarface | systemdlete: oh, i just meant literally paste it and just wait for the flood protect throttling to let it finish going through | 03:15 |
| systemdlete | oh... | 03:15 |
| systemdlete | it's 8MB | 03:15 |
| gnarface | is all 8MB relevant? | 03:15 |
| gnarface | i mean, large pastes will take a while but it also will all get here, but i'm not sure if 8MB will actually fit in your clipboard | 03:16 |
| systemdlete | of course not | 03:16 |
| * systemdlete duh | 03:16 | |
| systemdlete | I can snip it | 03:16 |
| systemdlete | last 1000 lines? | 03:19 |
| gnarface | sure | 03:19 |
| systemdlete | it's about 100k | 03:19 |
| gnarface | i think it'll work, just say something when it's done | 03:19 |
| systemdlete | how can I paste it? | 03:20 |
| systemdlete | it might end up here in this channel! | 03:20 |
| systemdlete | wait | 03:20 |
| systemdlete | let me try pasting this smaller file | 03:20 |
| gnarface | i just sent you 1 line, do you see that open in a new tab on your IRC client? | 03:20 |
| gnarface | it would just be: /msg gnarface [paste] | 03:21 |
| gnarface | but if your client lets you reply to the message i just sent, that might be easier | 03:21 |
| systemdlete | here: https://paste.debian.net/hidden/e7acf4f1/ | 03:21 |
| systemdlete | (yes, I see the tab, but I think the paste is easier) | 03:22 |
| gnarface | hmmm | 03:22 |
| gnarface | 1713568491|E|108923|192.168.40.41|devrep/merged/pool/DEBIAN/main/p/python-pip/python-pip-whl_20.3.4-4+deb11u1_all.deb [HTTP error, code: 502] | 03:22 |
| systemdlete | yeah. Lots of errors | 03:22 |
| gnarface | 502 though | 03:22 |
| gnarface | lemme double check, but i think 502 means remote server replied "aak, i'm misconfigured!" | 03:23 |
| systemdlete | I just looke dit up | 03:23 |
| gnarface | i only see a couple 502 errors... the rest looks completely normal | 03:23 |
| systemdlete | it means gateway interference, which is hard to believe when I'm getting the errors on the cacher server itself! | 03:24 |
| gnarface | bad gateway yea | 03:24 |
| systemdlete | unless | 03:24 |
| gnarface | wait, but is that the cache server? or is that actually the debian mirror responding? | 03:24 |
| systemdlete | unless it is on the upstream side? | 03:24 |
| gnarface | that's what i was thinking | 03:24 |
| systemdlete | yeah | 03:25 |
| gnarface | might be a problem with a mirror that's just rippling down into your cache | 03:25 |
| systemdlete | now, between the cacher and the Internet, there be some gateways, routers, firewalls... | 03:25 |
| gnarface | hmm, but i think 502 means it had to actually get some sort of reply though... | 03:26 |
| systemdlete | but as I mentioned, I don't have many network issues here (other than stupid things I caused, which are now corrected, to the best of my knowledge) | 03:26 |
| onefang | If it's a mirror, please try to identify which one. | 03:26 |
| gnarface | yea, see if it always happens while apt-cacher-ng is hitting the same remote mirror | 03:26 |
| gnarface | it might be all their fault | 03:26 |
| systemdlete | how can I tell? | 03:27 |
| gnarface | first thing that comes to mind is tail the log file while watching the raw network traffic | 03:27 |
| onefang | And if it's all their fault, I might have to do something. | 03:27 |
| systemdlete | the log file (I just pasted) doesn't give that info | 03:27 |
| systemdlete | onefang | 03:27 |
| onefang | Find the IP of that mirror. | 03:27 |
| systemdlete | onefang: don't fret, this happens to a lot of people outside devuan | 03:27 |
| gnarface | yea, you'd just have to be tailing it and watching the network traffic at the same time, then see which remote ip is in the tcpdump at the same time that error hits the logs | 03:27 |
| systemdlete | tail which log file? | 03:28 |
| gnarface | the one you just pasted | 03:28 |
| systemdlete | (i have a wide assortement of them) | 03:28 |
| systemdlete | but the log doesn't show the upstream, does it? Maybe I misse dthat) | 03:28 |
| gnarface | no, but my guess is the error will show up in that log file within milliseconds of the actual remote connection being made, so if you're just running tcpdump at the same time you'll see the corresponding IP | 03:29 |
| systemdlete | ah | 03:29 |
| systemdlete | tcpdump | 03:29 |
| gnarface | in fact, with the right filter you would be able to even see the 502 error in the response | 03:29 |
| gnarface | then you'd be sure which side of the network it was coming from | 03:29 |
| systemdlete | drat. | 03:29 |
| gnarface | actually, what am i thinking? you could certainly filter tcpdump output just for http errors | 03:30 |
| systemdlete | I forgot to obfuscate my network addresses in that paste | 03:30 |
| systemdlete | but probalby not an issue since my network is behind a residential firewall anyway | 03:30 |
| gnarface | i noticed, but they're all private anyway | 03:30 |
| systemdlete | right | 03:30 |
| systemdlete | now, why would it be that apt is able to round-robin the repo servers, but apt-cacher has a problem doing the same--don't they both use the same code, ultimately? | 03:31 |
| gnarface | uh, i don't know that they do | 03:32 |
| gnarface | and apt may just be coded to retry without complaint | 03:32 |
| onefang | Maybe not. I know debootstrap doesn't use apt, but mdebstrap does. | 03:32 |
| systemdlete | ok | 03:32 |
| gnarface | but with tcpdump you should be able with some fiddling to isolate the source of these 502 errors explicitly | 03:33 |
| gnarface | there might be a simpler way that i'm not thinking of, but tcpdump is definitely up to this task | 03:33 |
| systemdlete | right. I will do that | 03:33 |
| onefang | And I definitely know that apt-panopticon doesn't use apt directly, but that's a different case. It's testing every step of the apt process on the package mirrors. | 03:34 |
| gnarface | if you find it, the payload might even actually make the cause obvious | 03:34 |
| systemdlete | tcpdump port https | 03:35 |
| systemdlete | (and src host...) | 03:36 |
| systemdlete | I wonder if wireshark might be easier in order to easily examine the packets | 03:37 |
| systemdlete | but either way, it looks like it should be easy enough to gather the stream | 03:37 |
| gnarface | wireshark might be easier but i have less familiarity with it | 03:42 |
| gnarface | i'm not exactly a tcpdump pro but when i learned it there weren't alternatives | 03:42 |
| gnarface | all you should have to do is filter for http 502 error headers, https might sabotage that though | 03:43 |
| systemdlete | I just realized something. When I looked at https://unix.stackexchange.com/questions/623174/apt-cacher-ng-random-download-failures-with-apt-update-acgn, I failed to distinguish what they meant by disabling ipv6. | 03:43 |
| systemdlete | They meant, in the acng.conf file! Not the actual network interfaces. (though no harm there) | 03:44 |
| gnarface | hmm, i do also have ipv6 disabled everywhere afaik, though i don't see any particular indication that's what's causing this problem | 03:45 |
| systemdlete | the cacher can be configured to only listen to ipv4 (or ipv6) as desired. I missed that, and I think I will try that before starting the next step of the upgrade to daedalus | 03:45 |
| systemdlete | Yeah, I agree. And besides, it really OUGHT to work for ipv6-enabled networks. | 03:46 |
| systemdlete | It's reallly sad if those networks users are deprived of this functionality | 03:47 |
| gnarface | yea, but there could be an issue where just one mirror in the round-robin is missing the ipv6 dns entries or something weird like that | 03:47 |
| gnarface | we've seen stuff like that cause problems before | 03:47 |
| systemdlete | gnarface, some of the devuan mirrors are supported by users like us, right? I mean, some might be on big hardware in a DC somewhere, but maybe not all, and some might not be correctly configured for ipv6? | 03:48 |
| gnarface | yes | 03:48 |
| systemdlete | ok | 03:48 |
| gnarface | it has come up before | 03:48 |
| gnarface | as have https issues | 03:48 |
| systemdlete | not meaning to repeat you, just trying to get clear on your meaning | 03:48 |
| gnarface | sure | 03:49 |
| onefang | That's why if there's some mirror issue, I want to know the IP of the errant mirror. Especially if it's something apt-panopticon isn't finding. | 03:49 |
| systemdlete | onefang, the "errant mirror"-- do you mean a debian mirror, or only devuan? | 03:49 |
| gnarface | i think in theory it could be either | 03:50 |
| systemdlete | because debian users are hitting this too, if my survey of ddg hits on this topic is an indicator | 03:50 |
| onefang | Could be either, but I'm only in charge of Devuan package mirrors, though if it's a Debian mirror problem, that's good to know as well. | 03:50 |
| gnarface | though obviously scrutiny will be on the ones we can do something about first... | 03:50 |
| systemdlete | as well as people using the cacher for openwrt (which I do as well) | 03:50 |
| gnarface | is openwrt also a deb-based distro? | 03:51 |
| systemdlete | no! | 03:51 |
| gnarface | hmm | 03:51 |
| systemdlete | I think it is freebsd, but I always forget. It's one of the BSDs | 03:51 |
| onefang | There's at least one package mirror running from someones home server. I even once had an offer for a home based mirror server running over that PsaceX satellite network. lol | 03:52 |
| systemdlete | I know they will be moving to pkg in the future; they're already migrating some of their tools that way. But for now at least, apt-cacher-ng works for openwrt | 03:52 |
| onefang | How did I manage to typo SpaceX twice in a row, once while trying to fix the first typo. lol | 03:53 |
| systemdlete | you must have borrowed my fingers for a moment... | 03:54 |
| onefang | You can have them back. Keep 'em. | 03:54 |
| systemdlete | not your fault. You expected mine to work correctly I think. | 03:54 |
| systemdlete | lol | 03:54 |
| systemdlete | I've typo'd here about 4 or 5 times today already | 03:55 |
| systemdlete | gnarface: yeah, I followed the directions at openwrt.org wiki to set up the cacher for openwrt packages. | 03:56 |
| systemdlete | It's great with openwrt. I have several routers, and by upgrading them in the order so that my gateway is last, by that time, all the packages are cached for that upgrade, and I don't have to do any special pre-downloading (gateway needs some firmware and other stuff not available in the release iso) | 03:57 |
| systemdlete | so I'm really indebted to you for having me set this up. Even with this annoying issue... | 03:58 |
| onefang | gnarface is always very helpful, we should all thank them. | 03:59 |
| systemdlete | So onefang and gnarface, I will do my utmost to track down that repo for you. | 03:59 |
| systemdlete | (yes, they are!!!) | 03:59 |
| systemdlete | and you are too, and so are the rest of the folks here | 03:59 |
| systemdlete | The 502 Bad Gateway error is an HTTP status code that occurs when a server acting as a gateway or proxy receives an invalid or faulty response from another server in the communication chain. This error indicates a problem with the communication between the involved servers and can result in disruption of internet services. Wikipedia | 04:08 |
| systemdlete | That sounds exactly like what we are thinking here. | 04:08 |
| systemdlete | So it is almost definitely an upstream-side issue | 04:08 |
| gnarface | systemdlete: no problem, i hope you can identify it | 04:27 |
| gnarface | i don't know if apt-cacher-ng can be made to add the remote server IPs to the log files or not | 04:28 |
| gnarface | might be worth looking into, but packet sniffing will work, albeit probably quite tediously | 04:29 |
| systemdlete | I just made that change to only listen on ipv4. But I don't see why that will make any difference, that is, if the problem is almost certainly on the upstream side. | 04:29 |
| systemdlete | But no one can say I didn't at least try it. | 04:30 |
| gnarface | time will tell | 04:30 |
| systemdlete | this upgrade is taking all day, literally. | 04:30 |
| gnarface | heh, yea that's expected | 04:30 |
| gnarface | i did upgrade a full kde desktop, beowulf->chimaera->daedalus recently, i think something like 18 hours? | 04:30 |
| gnarface | i might have been sleeping for part of that, but it took a long time either way | 04:31 |
| onefang | My upgrade to daedalus has been ongoing since last year. This time I want to write configs from scratch instead of just copying them across, and I'm doing a shit load of testing. Once it's done testing, I'll roll it out to all my Linux computers. | 04:31 |
| onefang | I'm also using my own script for the system building. | 04:32 |
| systemdlete | same here | 04:33 |
| systemdlete | still in development, just the same way. I've used it a few times, but it needs work... | 04:34 |
| onefang | That's the other reason mines taking so long, I'm doing major surgery to the scripts. Not to mention my crappy life keeping me busy this last year. lol | 04:35 |
| systemdlete | It would have been more expeditious, I think, to simply clone a new daedalus VM from a template VM I created months ago and just update/upgrade to current package levels and do restores to the home areas | 04:35 |
| systemdlete | and add in all the programs and configuration from restores... ay uh | 04:35 |
| systemdlete | maybe not better, idk | 04:36 |
| systemdlete | I really have done a lot of customization on this system I'm upgrading. So maybe this will be worth it. | 04:36 |
| systemdlete | https://askubuntu.com/questions/119298/apt-get-using-apt-cacher-ng-fails-to-fetch-packages-with-hash-sum-mismatch#answer-431764 | 04:46 |
| systemdlete | (of course, that is a 10-year-old answer) | 04:47 |
| AlexLikeRock | hahahahhahah | 04:47 |
| AlexLikeRock | nice nick " systemdlete " | 04:47 |
| AlexLikeRock | hahhahahaha | 04:47 |
| systemdlete | def systemdlete: "systemd: delete from all systems, immediately!" | 04:48 |
| systemdlete | (i.e., nothing to do with "lete" or "l3t3" or whatever meme it is...) | 04:49 |
| AlexLikeRock | yeh!! | 04:49 |
| AlexLikeRock | i now | 04:49 |
| AlexLikeRock | jhahahah | 04:49 |
| systemdlete | thank you AlexLikeRock | 04:49 |
| AlexLikeRock | nice | 04:49 |
| systemdlete | gnarface, onefang: oopsies. Looks like some hard drive errors on host as it turns out. I hadn't been alerted about them, and I could have sworn I had thruk set up to alert me upon hardware problems. | 07:27 |
| systemdlete | So I might have just wasted your time, but I am not 100% sure about that. None of the articles I read about the cacher problem indicated a hard disk error. | 07:28 |
| onefang | No problem for me, I'm in weekend mode, so I was mostly letting you and gnarface work on it. Was waiting to see if you had found an actual broken mirror for me to sort out. | 07:48 |
| systemdlete | onefang, that could take some time. I am still trying to finish the upgrade to chimaera and that has taken ALL DAY. So it may be some time before I can explore that further. For one thing, I need to get my kernel logging configured to make thruk alert me for hard disk problems (which I thought I had already done...) | 07:50 |
| systemdlete | I'll be focusing on that to avoid more problems going forward. | 07:50 |
| systemdlete | But I will definitely be back here to let you know what I find out. | 07:50 |
| onefang | Fair enough. | 07:50 |
| systemdlete | I promise you. | 07:50 |
| systemdlete | I'm sort of hoping, though, that these hard disk errors are the smoking gun. The first of the errors seems to have begun around April 14 (PST time), and that was about the time I began to notice the errors, but I did not enter them into my journal here... | 07:51 |
| systemdlete | I'm suspicous of them but not really sure. | 07:52 |
| systemdlete | *suspicious | 07:52 |
| CueXXIII | systemdlete: anything in smartclt of that harddisk? otherwise it might be bad cabling producing those errors | 08:00 |
| systemdlete | CueXXIII, not sure yet. I'm juggling a few things... | 08:43 |
| systemdlete | I did look at the cell values on that harddisk and did not see anything that looked like hard errors. I only see evidence in the kern.log | 08:49 |
| systemdlete | I will take the system down in a while, just as soon as my disks resync (RAID1). The system has been up 46 straight days, so maybe it is "tired"... | 08:51 |
| systemdlete | I'll remove the bad disk and test it on a test box. I have some spares (good thinking, systemdlete, for once). | 08:51 |
| systemdlete | I usually run badblocks for a day or two to see if it errors. Since the test box has its own cables, that will help to eliminate that possiblity. | 08:52 |
| gnarface | systemdlete: which filesystem are you using on these? | 09:46 |
| systemdlete | ext4, everywhere | 09:50 |
| systemdlete | ext4 on VMs and hosts | 09:51 |
| systemdlete | I've tried some of the more exciting stuff, I think xfs? or something like that | 09:51 |
| systemdlete | but I had problems with it, but years ago, so I should prob try again | 09:52 |
| systemdlete | oh, no. it was btrfs | 09:52 |
| systemdlete | not xfs. I don't think I've ever tried that one | 09:53 |
| gnarface | hmm, well being not btrfs or anything similarly experimental, i dunno, but ext4 did have one bad corruption bug that was affecting upgrades a few releases back, only i thought that was before beowulf | 09:55 |
| gnarface | (and it did manifest itself exactly like a physical drive failure) | 09:55 |
| systemdlete | right | 09:56 |
| systemdlete | I vaguely recall that... | 09:56 |
| gnarface | the issue was to do with using older e2fsprogs with newer kernels or something like that | 09:56 |
| systemdlete | yeah, I think that's right | 09:57 |
| systemdlete | I am pretty sure I stumbled into that one at some point. | 09:57 |
| systemdlete | Thing is, before I started this upgrade, I cloned the VM, and upon boot I have all filesystems set up to fsck every time (I don't reboot any systems much). | 09:58 |
| systemdlete | So I believe I have a clean file system to start with. But still, if the actual hardware backing the virtual FS has actual hard errors, then maybe there is still issues. | 09:59 |
| systemdlete | I've been carefully checkking the kern.log's on both VM and host for any suspicious errors | 09:59 |
| systemdlete | since I noticed it earlier | 09:59 |
| systemdlete | ok now what am I doing wrong? I tried upgrading rsyslog on a beowulf system with the backports so I could get a more recent version. But there is no /etc/init.d/rsyslog | 11:43 |
| systemdlete | I checked the package; it says it is included | 11:43 |
| systemdlete | maybe a trigger is not running? I forget the details of how that happens... | 11:44 |
| fsmithred | what does 'apt policy rsyslog' tell you? | 11:52 |
| systemdlete | I removed the upgrade and tried re-installing the package from the regular repo, but same thing happens | 11:53 |
| fsmithred | I just tried to install rsyslog from beowulf-backports and apt tells me that i already have the newest version | 11:53 |
| fsmithred | but apt policy tells me that version is in beowulf. No rsyslog in backports. | 11:53 |
| fsmithred | want mine? | 11:53 |
| systemdlete | apt policy shows the correct versions, and the one I have installed has a star | 11:54 |
| systemdlete | sorry, I meant chimera backports, not beowulf | 11:55 |
| systemdlete | (I'm exhausted from all of this today...) | 11:55 |
| onefang | Go and rest. | 11:55 |
| systemdlete | but it doesn't matter; the rsyslog script does not get installed | 11:55 |
| systemdlete | either version | 11:56 |
| systemdlete | trouble is, I now have NO rsyslog running on that system! | 11:56 |
| systemdlete | I can launch it manually | 11:56 |
| systemdlete | but this is sick | 11:56 |
| rrq | I haven't read backlog, but which script are you talking about? | 11:59 |
| gnarface | hmm, did you debootstrap? i seem to recall a problem with rsyslog and debootstrap some point | 11:59 |
| gnarface | i think it was in ascii though, and maybe only on arm64 | 12:00 |
| systemdlete | no, nothing to do with installing the system. The version of rsyslog I had was 2102, and I wanted the 23.02 version. | 12:01 |
| gnarface | yea, according to my notes i had to exclude rsyslog and udev and include syslog-ng instead to successfully debootstrap ascii on arm64 | 12:01 |
| systemdlete | ascii... that's long time ago. | 12:01 |
| systemdlete | I don't have any ascii systems here | 12:01 |
| gnarface | never had any other problems with rsyslog that i can recall | 12:01 |
| systemdlete | (or jessie) | 12:01 |
| gnarface | does it work if you just reinstall it? | 12:02 |
| systemdlete | I removed all rsyslog versions from the apt-cacher and I'll see if this works | 12:02 |
| systemdlete | yeah, tried re-installing | 12:02 |
| systemdlete | I'm going to see if maybe it will cache a fresh copy from the repos | 12:02 |
| systemdlete | well reboot did not help, even shutting it down completely and restarting it "cold" (and swapping out the bad drive for a new one) | 13:14 |
| systemdlete | then I tried to apt remove rsyslog and apt install rsyslog, but for whatever reason, it does not install the /etc/init.d/rsyslog script | 13:15 |
| systemdlete | I also note that the archive does not contain the deb file for rsyslog | 13:16 |
| systemdlete | (I am using the cacher, but I didn't figure that would affect the client as far as grabbing a copy of the deb file) | 13:16 |
| systemdlete | (but what do I know, really?) | 13:16 |
| gnarface | wait maybe that's because it's in beowulf-security now? or chimaera-security or whichever you're on? | 13:17 |
| gnarface | make sure your sources.list is complete | 13:17 |
| gnarface | if you have to, you can always switch to syslog-ng but i was pretty sure rsyslog worked... | 13:17 |
| gnarface | i would be tempted to examine the preinst/postinst scripts | 13:18 |
| gnarface | but rsyslog is definitely working on my beowulf systems | 13:19 |
| systemdlete | chimaera here | 13:19 |
| systemdlete | this is a different sea of trouble than the one we were dealing with before. | 13:19 |
| systemdlete | and rsyslog was installed just fine previously. What happened was that I was running into some odd error messages from rsyslogd and I thought maybe upgrading to a more recent version could correct that (very hopeful thinking here) | 13:22 |
| systemdlete | but after doing the upgrade to chimaera-backports | 13:23 |
| systemdlete | I noticed that the script was missing (and maybe other stuff, idk) | 13:23 |
| systemdlete | this is what happens when you got up early the day before your birthday, ran into technical problems, stayed up all night trying to fix them, and kept running into more problems... | 13:24 |
| systemdlete | so I am totally exhausted, but I won't be able to sleep wondering about this. | 13:25 |
| systemdlete | gnarface, should I expect to see a deb file for rsyslog in the client's cache directory? | 13:30 |
| systemdlete | or does using the cacher cause different behavior? | 13:31 |
| systemdlete | (I never thought to check this) | 13:31 |
| systemdlete | I don't have the https_proxy set in my env???!!! | 13:34 |
| systemdlete | or maybe that was only for openwrt clients of the cacher... | 13:40 |
| gnarface | systemdlete: yes, it should still use /var/cache/apt/archives normally, if that's what you're asking | 14:22 |
| gnarface | apt-cacher-ng shouldn't affect that | 14:22 |
| gnarface | i'm setting the proxy in /etc/apt/apt.conf.d/ | 14:24 |
| gnarface | only using it for apt | 14:24 |
| systemdlete | but I'm NOT seeing it there | 14:46 |
| CueXXIII | oh dang, american holiday today… it's 4 20… | 16:36 |
| CueXXIII | ah, wrong tab -.- | 16:36 |
| Harzilein | hi :) | 21:33 |
| Harzilein | jaromil: grats for getting a new dyne release out of the door. i think that should showcase devuan very nicely. | 21:36 |
Generated by irclog2html.py 2.17.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!