| junalio | reinstalling with chimaera - same problem. now expert install with skipping apt as you suggested | 00:10 |
|---|---|---|
| junalio | which steps after configuring the package manager (which i will leave out) are mandatory, i..e where should i continue? | 00:14 |
| gnarface | uh, you'll have to refresh my memory | 00:14 |
| gnarface | what's next on the list? | 00:14 |
| gnarface | just gimme them one at a time and i'll tell you what to skip | 00:15 |
| gnarface | oh, and just another side question about what might have changed... you don't happen to have a USB drive left plugged in that wasn't there when this last worked, do you? | 00:16 |
| junalio | configure the package manager / select and install sofware / select an init system / install the grub boot loader / continue without boot loader / finish the installation | 00:16 |
| junalio | i am using another usb drive now then before, because the other one is at work still | 00:16 |
| junalio | *than | 00:17 |
| gnarface | i wonder if that could be the mystery variable | 00:17 |
| junalio | hmm maybe i should try another one then. i have some in the basement which i could try tomorrow | 00:17 |
| gnarface | well, that aside, try to do all the rest of the steps, but for "select and install software" just uncheck all the boxes - select nothing | 00:17 |
| gnarface | if it refuses to let you even do that without configuring apt first, then just skip both those steps | 00:18 |
| gnarface | you need the bootloader, and you need the init system, so you can't skip those | 00:18 |
| junalio | ah yes it refuses that | 00:18 |
| gnarface | ok, just skip right to init then | 00:18 |
| gnarface | use sysvinit for now just so there's less questions | 00:19 |
| gnarface | as long as you can get through installing grub it should boot | 00:19 |
| junalio | but tbh i want to find out whether it is due to the usb thumb. i remember that about 1 or 2 years ago i also had issues getting a linux iso to work with some usb keys (not this one though i think). so before i have to manually reconfigure everything tomorrow i will just try another usb key tomorrow | 00:19 |
| gnarface | i don't blame you, i'd want to know too | 00:20 |
| junalio | i will let you know tomorrow whether it worked with another usb key. thanks a lot for your help tonight! | 00:21 |
| gnarface | no problem | 00:21 |
| paculino | Once I get around to setting up a wm, I think I will try to set up something with sysvinit to automate things I do manually (all are bash one-liners but manually timed). Would it be beneficial to combine two or three things together instead of making them separate? | 00:29 |
| gnarface | eh, you just trade one type of complexity for another | 00:30 |
| gnarface | if it's only you maintaining these things, what matters is you're happy with it | 00:31 |
| gnarface | the right way would be to give each service its own init script, but if you never run them separately and you're the only one who even sees them, then what matters most is that it makes sense to you | 00:32 |
| gnarface | for that matter, you can just bundle a bunch of commands into /etc/rc.local and not even write init scripts for them, that's the "quick+dirty" way | 00:32 |
| gnarface | but most the init scripts are boilerplate though for simple tasks, so maybe it's not such a big deal to give each one their own | 00:33 |
| gnarface | the real thing is, if you bundle them all together then the task gets more complex somewhere down the road, you might find yourself painted into a corner | 00:34 |
| gnarface | giving them each their own init script might seem like unnecessary complexity at first, but later on if their needs diverge you might be glad you set it up that way to begin with, rather than having to "untangle" them all after the fact | 00:35 |
| gnarface | i've done it both ways for different things; when it's for personal use only, in the end you just gotta use your own judgment | 00:36 |
| paculino | I guess I'll make each one separate, since they are never done at the same time. Ntp is whenever I think of it, but ideally it would be at regular intervals, and adjusting r/b gamma is manually but the correct time is trivially calculated | 00:46 |
| onefang | gnarface: I may have mentioned this before, instead of putting things in /etc/rc.local, put tiny individual scripts (mine are mostly one liner shell scripts) in /etc/boot.d which gets run by rc.local by default these days. Then you keep your rc.local pristine, ready for the next upgrade to it. | 01:16 |
| gnarface | ah yes, i always forget about that | 01:25 |
| gnarface | that's a nice middle ground maybe | 01:25 |
| onefang | That's why I use it. Often after looking in orphan scripts and not finding anything, I'll poke at any systemd unit that might have been installed, typically they are a one liner. Sure nothing fancy like start / status / stop or logging, but for somethings as you said "a nice middle ground". | 01:28 |
| onefang | Works For Me. (tm) | 01:29 |
| rwp | I always trip over that /etc/boot.d/ is not /etc/rc.local.d/ and am annoyed that they did not follow the typical naming convention. | 01:57 |
| rwp | They missed it by *this* much. | 01:57 |
| onefang | At least they didn't call it /etc/sandal.d/ B-) | 01:59 |
| rwp | I am sure you are referring to that Red Hat did call their boot daemon "pumpd" because a pump is a type of shoe. | 02:01 |
| onefang | Didn't know the first one, did know the second. lol | 02:02 |
| onefang | But off topic. | 02:02 |
| rrq | I guess using @boot lines in crontab is too old school :) | 02:09 |
| rrq | (@reboot) | 02:11 |
| djph | rrq: yep. you should be using @systemd-startup-blahblah now :P | 02:11 |
| rwp | rrq, For non-root boottime start of my own personal user stuff, such as a tmux session running my irssi irc client, I use @reboot and it works very well. | 02:57 |
| gnarface | has anyone successfully used hardware video decoding in nouveau lately? | 04:25 |
| gnarface | (or ever?) | 04:27 |
| rkta | I regularly get an mail from cron because some apt process could not acquire a lock. I see that there is a hanging apt process started by cron, but I have no idea where this process is coming from. Something is calling '/usr/bin/apt-get -qq update'. I would like to debug this further. Any ideas how to find what started this? grep -r 'apt-get' /etc/cron* shows nothing. | 07:04 |
| gnarface | rkta: maybe something to do with /etc/cron.daily/apt-compat | 07:07 |
| rkta | yep, grepping for -qq update didn't show anything, but I see now, that they construct some options during runtime containing -qq. I'll try to enable some verbose stuff. | 07:15 |
| CueXXIII | look for the parent process of that apt-get, ps -faux should display a tree of processes (or pstree, seperate package) | 07:19 |
| rkta | it's a shell invoked by apticron | 07:20 |
| CueXXIII | yeah, that should not hang, though… | 07:24 |
| rkta | I tried setting verbose output for this apt-compat stuff, let's wait and see. | 07:30 |
| gnarface | rkta: you don't have anything weird like a read-only /run/ or /var/ do you? | 07:32 |
| gnarface | and uh, which devuan release is it? | 07:35 |
| rkta | gnarface: no and it's daedalus | 07:36 |
| gnarface | and no partitions are out of space, right? | 07:37 |
| rkta | no, df -h shows no Use above 86% | 07:38 |
| gnarface | should be alright, cutting it close though... i'm pretty sure ext4 reserves 10% by default | 07:38 |
| gnarface | should be enough for a lockfile though... | 07:39 |
| gnarface | do you have any flavor of automated updates enabled? | 07:39 |
| rkta | It's /boot and /home, / has 8.4GB left | 07:40 |
| rkta | no, no autoupdates. I do get mail for pending upgrades, though. | 07:41 |
| onefang | That 10% thing annoys me. My biggest partition has 1.2 TB free, but coz that's less than 10% of the size .... | 07:41 |
| gnarface | you can change it | 07:41 |
| gnarface | i usually set it to 1% | 07:42 |
| onefang | I haven't bothered to track it down. Can it be changed to some fixed size rather than a useless percentage? | 07:42 |
| gnarface | well, no, i usually use xfs instead, but when i'm using ext4 i typically set it to 1% | 07:42 |
| gnarface | hmm, good question, not sure | 07:42 |
| gnarface | been a while since i actually did it | 07:42 |
| onefang | 10% of a small boot SSD is a whole different matter to 10% of a huge spinning rust storage monster. | 07:43 |
| adhoc | tune2fs uses, -m reserved-blocks-percentage | 07:44 |
| adhoc | "Normally, the default percentage of reserved blocks is 5%." | 07:44 |
| adhoc | this is not the man page on devuan, however I suspect it is the same ... | 07:45 |
| adhoc | yeah, same text in man page for chimaera | 07:46 |
| adhoc | so when you run, say; tune2fs -l /dev/sda4 | 07:49 |
| adhoc | you get; | 07:49 |
| adhoc | Reserved block count: 5811852 | 07:49 |
| adhoc | not a percentage... | 07:49 |
| onefang | I guess if you can set it per partition, then you can calculate for yourself how much free space percentage you think works for that one. | 07:55 |
| adhoc | yeah, do not set / or /var much lower than 10% | 07:56 |
| adhoc | /home ... what ever | 07:57 |
| onefang | Never been annoying enough to make me sort it out. "What do you mean that 1.2 TB isn't enough space for a few hundred GB?" gets the response "Meh, I'll copy it as root." | 07:59 |
| adhoc | sounds like foot guns in the making. | 08:08 |
| CueXXIII | adhoc, onefang: tune2fs -r reserved-blocks-count | 09:06 |
| onefang | Cool, thanks. | 09:06 |
| CueXXIII | obviously only works for ext2/3/4, but other filesystems should have similar tools | 09:07 |
| adhoc | CueXXIII: ah. well spotted. | 09:31 |
| adhoc | ext4 is what I have gone back to for any large voluem these days. | 09:31 |
| adhoc | considering how many JFS and XFS filesystems I have had fail, I no longer bother with them. | 09:32 |
| adhoc | interesting;Reserved blocks uid: 0 (user root) | 09:35 |
| adhoc | Reserved blocks gid: 0 (group root) | 09:35 |
| CueXXIII | yeah, the reserved blocks have 2 purposes, i think | 09:35 |
| CueXXIII | one to allow root to write some emergency data when the space runs out, and two to keep fragmentation low | 09:36 |
| adhoc | you can change the user by uid or name | 09:36 |
| adhoc | and the group by gid or group name | 09:36 |
| CueXXIII | ext2 starts to fragment faster over 95% usage, afaik, but that's not that big speed issue with ssd anymore | 09:36 |
| CueXXIII | ah, the named values are probably converted to id's by tune2fs | 09:37 |
| adhoc | do folks still use ext2 ? | 09:39 |
| CueXXIII | maybe in places where the log does not matter, like /boot | 09:40 |
| adhoc | seems its file size and volume size limits are finally an issue now; | 09:41 |
| adhoc | https://en.wikipedia.org/wiki/Ext2#File-system_limits | 09:41 |
| adhoc | when I first used ext2 volume sizes were still less than 1GB | 09:42 |
| adhoc | is xiafs still even in the kernel ? | 09:42 |
| CueXXIII | not since 2.1.21 acc. to its wikipedia article | 09:44 |
| adhoc | thanks CueXXIII | 10:16 |
| systemdlete | apt-cacher-ng issues again. This time, I was doing updates on my testbox. The host and one VM is devuan D, and the other VM is devuan C. Both VMs had no problem with apt update. But the host complained about a bad key for daedalus-backports | 20:53 |
| systemdlete | I commented out the apt-cacher in /etc/apt/conf.d/proxyfile for a moment, ran the update (hitting the web, not my cacher), which updated fine. Then I switched back to the cacher by uncommenting that line in the apt config. | 20:54 |
| systemdlete | All is normal again. | 20:54 |
| systemdlete | ??? | 20:54 |
| systemdlete | This sounds like some kind of problem where a cacher client is not getting updated keys. But I don't know the guts of it all. | 20:54 |
| CueXXIII | or did the update include the package devuan-keyrings? | 20:58 |
| mason | What praytell is "access" vs "std" in the minimal/live image? | 21:00 |
| systemdlete | sorry for the bounce. Comcast decided to have a fit suddenly. All fixed now. | 21:01 |
| mason | Ah, I'm betting it's "accessible". But if I'm wrong, please tell me. | 21:01 |
| systemdlete | ah. I did not check that time is sync'd. That can create problems... | 21:01 |
| systemdlete | time is off by .021ms, so it's good. But it IS a testbox, so I don't assume anything | 21:02 |
| systemdlete | CueXXIII, this was an update for thunderbird. But the problem occurs during apt update, not apt upgrade. | 21:03 |
| CueXXIII | yeah, during apt update the signatures of the package lists are checked | 21:04 |
| systemdlete | so whether the update included keyrings would not be relevant, would it? | 21:05 |
| systemdlete | (again, I have little to no understanding of apt's guts) | 21:06 |
| systemdlete | but CueXXIII that gives me a notion: Can I force re-install the keyrings? | 21:07 |
| systemdlete | And, more importantly, would that fix anything? | 21:07 |
| CueXXIII | sure, apt-get --reinstall install devuan-keyring | 21:07 |
| systemdlete | but why is it that this only occurs, and only sometimes, with the cacher? As I described the most current inccident, it seems that "leaving" the cacher environment for a moment, doing the update, and switching back heals the problem. | 21:09 |
| CueXXIII | yeah, seems to be morea problem with the cacher | 21:10 |
| systemdlete | ok, if so, then why did the other 2 systems have a clean update with no problems? | 21:10 |
| systemdlete | if the problem were with the cacher, I'd think they would all have problems, or at least the VM running devuan D, since the host is running D and had the problem | 21:11 |
| systemdlete | It's so inconsistent, that it is hard to find a pattern to narrow things down | 21:11 |
| CueXXIII | is /etc/apt/sources.list identical on all systems? | 21:11 |
| systemdlete | should be | 21:12 |
| systemdlete | I don't mess with those too much, but I'll check. | 21:12 |
| gnarface | i think if you start pulling the mirrors out of the list you will deductively find that only some of them cause the issue | 21:12 |
| systemdlete | gnarface, ? | 21:12 |
| gnarface | i don't think the mirrors are identical | 21:12 |
| systemdlete | ok, but | 21:12 |
| systemdlete | there was only one package getting an update: tbird | 21:12 |
| systemdlete | same for all 3 systems | 21:13 |
| gnarface | yes but there's also some sort of timestamping, no? | 21:13 |
| systemdlete | by the time I tried/retried to upgrade the host, the package would have been cached by then, right? | 21:13 |
| systemdlete | any keys and other paraphenalia should have also been up to date with it, I'd think | 21:14 |
| systemdlete | not saying you are wrong, but I don't see the connection | 21:14 |
| systemdlete | (in this instance; other times the mirrors could well be a culprit) | 21:15 |
| gnarface | it's just a suspicion based on a vague foggy memory of someone else looking into this at length and then eventually washing their hands of it shortly after finding out the truth would require enforcing certain behaviors that were currently variable at the mirror level | 21:16 |
| gnarface | my thought based on that is that if you could just skip certain mirrors maybe it would stop happening | 21:17 |
| gnarface | maybe not even skipping them all the time, maybe just skipping them when they're between certain steps of updating, or something like that | 21:18 |
| systemdlete | it was consistently breaking on the one client (host running D), but not the other 2. I ran "apt update" several times on all 3 just to make sure I wasn't going crazy | 21:19 |
| systemdlete | the failures/successes were perfectly consistent | 21:19 |
| systemdlete | (over, say, 3 or 4 attempts on each) | 21:19 |
| systemdlete | I think CueXXIII might be on to something though. I will explore that first. | 21:20 |
| gnarface | i just think that as a test you should try to have the cacher only point to one particular mirror for a while, see if that changes the behavior | 21:20 |
| gnarface | then if it does change it favorably, start adding in mirrors one at a time, at spaced out intervals so you can tell which are associated with some weird race conditions | 21:21 |
| systemdlete | so I copied all the sources.list files to a common directory (naming them source.host, source.vmC, source.vmD) | 21:35 |
| systemdlete | diff source.host with source.vmD absolutely no differnces | 21:35 |
| systemdlete | the VM running C is a bit different, and uses pkgmaster instead of the ones used by the other two | 21:35 |
| systemdlete | but the VM running Chimaera wasn't the one having the problem! But, still, I think it was a commendable idea per CueXXIII | 21:36 |
| systemdlete | (thanks) | 21:36 |
| gnarface | was it the one that was hit right before the one that was causing problems? | 21:37 |
| systemdlete | gnarface, back to your idea: Are you saying that each time the cache is hit, the cacher goes and downloads files anyway? | 21:37 |
| systemdlete | (no, in fact, host was the second to be tried for update) | 21:37 |
| systemdlete | if so, then the cacher would not be very efficient | 21:38 |
| systemdlete | now, maybe every time it gets hit, perhaps the cacher gets *some* info from the mirrors, idk. | 21:39 |
| gnarface | i think, from my vague memory of this, that it does the exact same thing every time, having no local state to control the order of operations, and that set of operations includes some sort of timestamp or cache check, yes, and some transient difference between the mirror responses occasionally trips it up, that's the hypothesis, in a nutshell | 21:39 |
| systemdlete | but why it would consistently hit the same bad mirrors for specific clients doesn't make sense to me. | 21:39 |
| gnarface | difference in the dns round-robin randomization in the versions of whatever library that controls it are installed there, would be the obvious plausible explanation for that | 21:40 |
| systemdlete | but these "mirror responses"--are those performed every time between client hits? | 21:40 |
| gnarface | i think so, at least when the network connection is available, so they're probably not actually always necessary, but maybe even that could cause issues | 21:40 |
| systemdlete | I don't think I'd design a cacher that way... | 21:41 |
| gnarface | the thing is, this is happening a lot more to you than me too, so there might be a local variable, but have you tried just connecting it to only one mirror for a while to see? | 21:41 |
| gnarface | i also don't use mine as much, so maybe that's it | 21:42 |
| systemdlete | no, I haven't tried that | 21:42 |
| gnarface | but i think basically, it does something that assumes it's always hitting the same host with the same contents, the dns round-robin sabotages that assumption when... something on the mirror is out of sync from the last one it hit, but it can't tell that from the HTTP request so it can't react right | 21:43 |
| gnarface | i just remember the takeaway from that other guys' fact-finding mission on the matter resulted in the conclusion that patches would have to be made to the mirrors, to completely fix the issue | 21:44 |
| gnarface | and it was something that apt itself didn't care about, so ... obviously that would have gone nowhere | 21:44 |
| systemdlete | (I recall that part of the story also, as related by you to me in the past) | 21:45 |
| gnarface | it amounts to hearsay but that's all i've got here | 21:45 |
| systemdlete | hear, hear! | 21:45 |
| systemdlete | I think that a thorough, deep-dive analysis is what is needed. And maybe a complete re-write or refactoring of the cacher code. | 21:46 |
| systemdlete | But there is one other thing that pops into my mind | 21:46 |
| systemdlete | SSH keys | 21:46 |
| systemdlete | sorry, I mean GPG keys | 21:46 |
| gnarface | the first thing i would check is see if the HTTP request includes some sort of timestamps from the server that apt-cacher-ng is parsing, and then look for a divergent one | 21:47 |
| systemdlete | at times, I know that some of these systems have gotten off a bit from NTP. | 21:47 |
| systemdlete | I just checked the host (which had the problem) and its time was only off .021ms | 21:47 |
| gnarface | yea, but we don't know if all the mirrors even use NTP, do we? | 21:48 |
| systemdlete | but what if... what if at some time in the past, that same host had been off timewise? | 21:48 |
| gnarface | what if the mirror is off? | 21:48 |
| systemdlete | (good point, I wouldn't know) | 21:48 |
| gnarface | if apt-cacher-ng is deciding on cache hits based on some timestamps returned from the server, and in between runs it switches servers to one that has a clock off by a few minutes because it's not even running NTP, could that be triggering the issue...? seems plausible based on what little we know right now | 21:49 |
| systemdlete | I don't know enough about GPG to go farther with this sketchy idea, but I am thinking that perhaps that client got a key that was not kosher due to the client being off timewise | 21:49 |
| systemdlete | So it had the key, perhaps, but believed it to be "bad" | 21:49 |
| systemdlete | the issue seems to be between the cacher and its clients, not between the cacher and the mirrors, at least in the scenario that began this thread today | 21:50 |
| gnarface | yea, for example hypothetically, say one mirror is in the middle of updating and the other isn't, so apt-cacher-ng gets two responses from different servers at two different phases of the update process but thinks they're the same server and doesn't have the logic to handle the discrepancy right...? | 21:50 |
| gnarface | it might just look like a problem between the cacher and the clients because that's where the last checks are done | 21:51 |
| gnarface | or i dunno, maybe just something much simpler and utterly mundane with regards to timestamp discrepancy from two different servers | 21:53 |
| gnarface | seems worth checking | 21:53 |
| gnarface | you might just hit this more often than me because you have way more things to update | 21:53 |
| systemdlete | I wonder if our mirror suport folk could have a tool that periodically goes out and checks the NTP time on each server and compares them? | 21:54 |
| gnarface | worth asking | 21:54 |
| gnarface | i forget the panopticon url | 21:54 |
| * systemdlete wonders, with what little understanding of black hat computing themself, if there is a supply chain attack being set up as we speak... hmmm. | 21:56 | |
| systemdlete | those guys spend months setting up attacks before they pop | 21:56 |
| systemdlete | maybe not likely, but still | 21:57 |
| gnarface | well, nothing would surprise me at this point, this issue has been a complaint with apt-cacher-ng since at least Debian Squeeze | 21:58 |
| gnarface | *but this issue... | 22:01 |
| systemdlete | I get it. The issue is ancient, and there has been no attention to it | 22:01 |
| systemdlete | The trouble with volunteer armies. | 22:01 |
| gnarface | well, it's not even that there's been no attention to it, it's that it seems everyone who finds the source of the issue immediately washes their hands of the matter | 22:02 |
| gnarface | and i think if you find that 1/3rd of the mirrors not only aren't running NTP but are maintained by people who refuse to run NTP on their mirror, that might be the smoking gun | 22:02 |
| gnarface | not sure that's the issue, just a working hypothesis, but it might lead to the real issue, like something else they're doing differently, like handing back files or HTTP headers in a different order or something | 22:05 |
| systemdlete | ok, so you might have seen me bouncing in and out because I was upgrading one of my openwrt routers--this one being conneted directly to the Internet. I have my openwrt routers configured to use apt-cacher-ng also. | 22:33 |
| systemdlete | Guess what happened? | 22:33 |
| systemdlete | So I did a full cleaning of the apt-cacher and then all was good. | 22:34 |
| systemdlete | Now, I just upgraded 2 routers yesterday, and a couple more a day or two before that. | 22:34 |
| systemdlete | So This is odd. | 22:34 |
| systemdlete | If the problem is the RR mirrors, then how do we explain the issue with openwrt also? And this is not the first time I've seen problems with the cacher in re openwrt | 22:35 |
| systemdlete | It could be a completely diff issue, idk. | 22:35 |
| systemdlete | But I thought I'd report it just to add more fuel to this fire | 22:36 |
| gnarface | openwrt doesn't also use a DNS-RR? | 22:40 |
| systemdlete | It probably does | 22:40 |
| gnarface | i'm not using it here, so i'm not familiar with their practices | 22:40 |
| gnarface | but the lack of a DNS-RR involved doesn't preclude some weird timestamp discrepancy issue | 22:41 |
| gnarface | were you updating them in parallel or one at a time? | 22:41 |
| systemdlete | one at a time | 22:42 |
| gnarface | have you noticed any difference in incident count depending on whether you're updating stuff in parallel or series? | 22:43 |
| systemdlete | good question. I'd say that I am occasionally doing parallel updates. Usually those don't result in any calamities. | 22:45 |
| systemdlete | I wouldn't say any noticeable increase in problems. | 22:46 |
| gnarface | it might not be a factor, but if it turns out to be, that might be a important clue | 22:46 |
| systemdlete | In fact, if anything, it seems like those are the times that the fewest issues arise | 22:46 |
| * systemdlete tries to recall if they did updates to tbird on those test systems in parallel... | 22:47 | |
| * systemdlete goes to check the apt logs on those systems... | 22:47 | |
| gnarface | hypothetically, if the issue were related to server time drift, there'd be intervals where they were close enough in temporal proximity and intervals where they weren't, depending on some periodic alignments of something like a 5-minute time window | 22:48 |
| gnarface | so then, further hypothetically, a bunch of updates done in parallel while they were all within the cutoff window would always all work, while a bunch of the same updates spaced out over a longer time frame would be more likely statistically to hit a unhandled discrepancy | 22:50 |
| systemdlete | no, all different times. In fact, I had completely forgotten that I rebooted the VM running C so I am just now upgrading tbird there | 22:51 |
| systemdlete | tea time, bbs | 22:57 |
| * onefang wakes up, reads the backlog, adds a "have apt-panopticon check mirror server time" TODO item. | 23:15 | |
| * systemdlete notices onefang still has only one eye open, and is not sure what the other one is looking at | 23:16 | |
| onefang | Well I have two monitors, one dedicated to IM systems, sometimes there's an eye on each. | 23:17 |
| systemdlete | is your vision blurred like mine is before first cup? | 23:18 |
Generated by irclog2html.py 2.17.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!