Jul 07 20:13:03 * SashaGrey (972909da@gateway/web/freenode/ip.151.41.9.218) has joined #ubuntu
Jul 07 20:14:18 <SashaGrey> Hi i've got a problem with my ubuntu server that is on the datacenter, after the apt-get update + upgrade i was unable to start my machine, because it was unablel to mount the /home. Now i'm in with the rescue mode but i don't know what to do. I've tried to mess around with fstab with no success. Please someone help.
Jul 07 20:15:07 <daftykins> SashaGrey: compare fstab with "sudo blkid" which shows the UUIDs of your disk partitions. are you using encryption?
Jul 07 20:16:06 <SashaGrey> daftykins i'm not encrypting the /home folder. Can you please provide step by step instruction to do so?
Jul 07 20:16:27 <SashaGrey> daftykins maybe we can go on a pvt stream to not bother the channel if you want
Jul 07 20:16:58 <daftykins> !pm | SashaGrey No, we cannot
Jul 07 20:16:59 <ubottu> SashaGrey No, we cannot: Please ask your questions in the channel so that other people can help you, benefit from your questions and answers, and ensure that you're not getting bad advice.
Jul 07 20:17:11 <SashaGrey> ok wathever
Jul 07 20:17:16 <SashaGrey> what do i have to do
Jul 07 20:17:18 <daftykins> SashaGrey: running 'sudo blkid' then viewing /etc/fstab - all should be clear, UUIDs match up with partitions
Jul 07 20:18:01 <SashaGrey> aftykins, you may want to understand that i'm not booted on my system and i'm on a sideloaded usb key on the datacenter
Jul 07 20:18:29 <daftykins> SashaGrey: i see... so you're not capable of the above?
Jul 07 20:18:53 <SashaGrey> that sounds better sir
Jul 07 20:22:11 <SashaGrey> daftykins: i got 2 fstab sir one normal and fstab.d
Jul 07 20:22:38 <SashaGrey> ok
Jul 07 20:22:41 <SashaGrey> buddy
Jul 07 20:26:02 <SashaGrey> daftykins: fstab http://pastebin.com/sAsxvbFz
Jul 07 20:27:07 <ioria> SashaGrey, which ubuntu are you on ?
Jul 07 20:28:25 <SashaGrey> ioria: it has to be server 14.04 IF i remember well. apt-get update + upgrade and gg everything messed up
Jul 07 20:30:45 <daftykins> SashaGrey: right so those mounts are by device (sda#) instead of UUID, so this doesn't really look like proper ubuntu server - now is the time to say if this is really something else.
Jul 07 20:33:45 <SashaGrey> daftykins: it IS ubuntu server!
Jul 07 20:34:32 <daftykins> SashaGrey: right, so try mounting /home since you said that was the problematic one and see what errors (if any) you get
Jul 07 20:34:43 <daftykins> SashaGrey: how is your command line skill?
Jul 07 20:35:07 <SashaGrey> daftykins really bad
Jul 07 20:35:31 <SashaGrey> daftykins not "that" bad
Jul 07 20:35:39 <SashaGrey> daftykins not pro
Jul 07 20:36:04 <daftykins> SashaGrey: so "sudo mount /dev/sda3 /mnt" from the live session, what does it say? errors? return to prompt with no output?
Jul 07 20:36:48 <SashaGrey> daftykins: no errors
Jul 07 20:37:00 <daftykins> SashaGrey: ls /mnt - is the data there?
Jul 07 20:37:33 <SashaGrey> yes there are 3 of my folders yes daftykins
Jul 07 20:37:58 <daftykins> SashaGrey: each usernames?
Jul 07 20:38:19 <SashaGrey> daftykins: yes there's me and an account i've made
Jul 07 20:39:06 <daftykins> SashaGrey: alright, you're going to need to boot the actual install and view the boot process from the console... then share whatever error it actually displays when failing
Jul 07 20:39:19 <daftykins> SashaGrey: safely unmount that partition first with "sudo umount /dev/sda3"
Jul 07 20:54:44 <SashaGrey> daftykins: the technician must report the error to me but most likely will be the same as this morning: "The disk drive for /home is not ready yet or not present. keys: Continue to wait, or Press S to skip mounting or M for manual recovery"
Jul 07 20:55:36 <daftykins> SashaGrey: hmm, no console access huh?
Jul 07 20:55:55 <SashaGrey> daftykins: if it doesn't boot no for sure
Jul 07 20:56:02 <SashaGrey> it's stuck with this message
Jul 07 20:56:30 <SashaGrey> daftykins: ot's stuck at the boot with this message
Jul 07 20:56:39 <daftykins> SashaGrey: ah, lots of servers have like an IP KVM so you can see what a physically connected display would show
Jul 07 20:57:04 <daftykins> SashaGrey: bit weird you don't use sda1 really
Jul 07 20:57:41 <SashaGrey> daftykins i don't know what happened. everything worked out well so far
Jul 07 20:58:40 <TJ-> SashaGrey: Does pressing "M" drop to a busybox shell in the initrd ?
Jul 07 20:59:12 <SashaGrey> TJ- i can't because i'm not phisically on the server, they report the error from the datacenter
Jul 07 20:59:29 <SashaGrey> and they allow me to boot to a rescue usb key
Jul 07 21:00:33 <TJ-> SashaGrey: no remote KVM?
Jul 07 21:00:49 <SashaGrey> TJ- i don't even know what it is
Jul 07 21:00:55 <SashaGrey> i'm on Kimsufi
Jul 07 21:01:29 <TJ-> SashaGrey: Keyboard Video Mouse ... a datacenter without that isn't really a datacenter!
Jul 07 21:02:35 <SashaGrey> TJ- i've got some boot options maybe i'm missing it
Jul 07 21:02:57 <TJ-> SashaGrey: "The disk drive for /home is not ready yet" tells you the file-system needs manually repairing with 'fsck' - it must have failed the fsck scsan done by mountall. If you know the device that hosts /home/ then the solution is to press M (for manual) at that prompt, then do "fsck /dev/XXXX"
Jul 07 21:04:41 <SashaGrey> TJ- i can't do this with the live session?
Jul 07 21:04:57 <SashaGrey> TJ- i can't do this with the live session rescue mode?
Jul 07 21:05:03 Sabotender Sachiru Sander^work SashaGrey Saturn812 SaucisseCocktail Savemech Saviq
Jul 07 21:05:24 <TJ-> daftykins: SashaGrey OK, I see now ... had to grep from a long time back to see all the commentary
Jul 07 21:05:44 <SashaGrey> TJ- NEWS: they got at the machine and pressed yes
Jul 07 21:05:52 <SashaGrey> TJ- just sent me an email
Jul 07 21:06:41 <TJ-> SashaGrey: Not sure what "yes" is since that prompt is either "S" for skip the mount, "M" for manual recovery, or "C" for continue to wait
Jul 07 21:07:03 <SashaGrey> TJ- here is the email http://pastebin.com/yeDZvnuu
Jul 07 21:07:10 <TJ-> SashaGrey: hopefully your previous efforts fixed the file-system and now it is just slow on booting
Jul 07 21:08:45 <TJ-> SashaGrey: OK, so they pressed "S" so the mount of /home/ was skipped. You might have problems connecting to it if you rely on SSH since your /home/$USER/ directory won't be there. Hopefully you can get KVM or console root access
Jul 07 21:09:55 <SashaGrey> TJ- goddamn them, i've disabled root by ssh and i don't have this KVM... now what? i'm locked out?
Jul 07 21:10:31 <TJ-> SashaGrey: This is *why* we have KVM or IPMI on remote headless servers, so we can get in at the firmware/BIOS level when the OS goes wrong
Jul 07 21:12:54 <SashaGrey> TJ- there's no way to do it via the live usb ?
Jul 07 21:13:29 <histo> SashaGrey: yes fsck /dev/sda3 from live usb
Jul 07 21:14:13 <SashaGrey> histo i'm gonna try this again
Jul 07 21:14:17 <TJ-> SashaGrey: if the mount is still failing when the system boots you have no way to work with it; live isn't going to help here. These situations need console access, as I said
Jul 07 21:14:49 <histo> SashaGrey: what data center are you using?
Jul 07 21:14:59 <SashaGrey> Kimsufi
Jul 07 21:15:10 <SashaGrey> histo: Kimsufi
Jul 07 21:16:09 <SashaGrey> TJ- histo i'm confused one says it can be done the other no
Jul 07 21:17:15 <TJ-> SashaGrey: daftykins said you'd already checked and mounted the /home/ file-system using Live earlier, in which case the file-system is OK, so if it still isn't mounting you need to go deeper, which requires you be able to interact with the initrd shell
Jul 07 21:17:16 <histo> SashaGrey: can't you see the console in their 'control panel' ?
Jul 07 21:18:03 <TJ-> SashaGrey: is this a Virtual Machine or a Dedicated Server, and is it owned by you or by the hoster?
Jul 07 21:18:18 <SashaGrey> TJ- it's a dedicated server
Jul 07 21:18:48 <TJ-> SashaGrey: I'd suspect failing hardware... are the disks RAIDed ?
Jul 07 21:19:46 <SashaGrey> TJ- i suspect failing hardware too, the disk is one single 2 Tb disk. I've done all their tests and everything seems normal.
Jul 07 21:20:16 <TJ-> SashaGrey: I give up! server in a remote datacenter and there's no KVM/IPMI console and disks aren't RAIDed !?!
Jul 07 21:20:35 <SashaGrey> TJ- come on buddy please you can't just leave me
Jul 07 21:20:44 <SashaGrey> TJ- there are months of work on that server
Jul 07 21:21:05 <TJ-> SashaGrey: You have good backups then obviously
Jul 07 21:21:19 <SashaGrey> TJ- sort of
Jul 07 21:22:23 <SashaGrey> TJ- wait a minute. we can login on the live and enable on the ssh config file the root access get in fix the problem and restore the old config
Jul 07 21:30:37 <SashaGrey> TJ- ???
Jul 10 09:12:09 * SashaGrey (97292949@gateway/web/freenode/ip.151.41.41.73) has joined #ubuntu
Jul 10 09:12:18 <SashaGrey> Hey TJ- u there?
Jul 10 09:14:00 <SashaGrey> Nevermind guys my servers gets stuck at the start, can't mount /home. I don't have KVM but i've managed to skip the mount and get in via ssh. Anyone willing to help?
Jul 10 09:14:34 <histo> SashaGrey: is /home on a separate partition?
Jul 10 09:14:50 <SashaGrey> yes it is... i don't even know why
Jul 10 09:14:55 <SashaGrey> jisto
Jul 10 09:14:57 <SashaGrey> histo
Jul 10 09:15:18 <histo> SashaGrey: any error while mounting it?