<plars> elopio: if you have a moment today, can you help me sort out how to run the snappy selftests now? I was running a snapshot of the older adt stuff in lp:~snappy-dev/selftests but I understand it's all in trunk now
<elopio> plars: yes! I wanted to meet with you after we got all the tests running in bbb, but we are close to that now.
<elopio> plars: take a look here: http://bazaar.launchpad.net/~snappy-dev/snappy/snappy/view/head:/_integration-tests/README.md
<plars> elopio: I have a fully automated local setup still for automated provisioning and testing
<plars> elopio: just landed some more code to support that in a more flexible way, so you can specify how the device gets a hard reset, boot mode selection, and stuff like that through just a config change
<plars> elopio: that way once the hardware lands in the lab, we can easily support that
<elopio> plars: nice. And are we going to be able to choose the release, channel and image that gets flashed?
<plars> elopio: yeah, I looked at that, but I couldn't get it to work, getting stuff like this:
<plars> https://www.irccloud.com/pastebin/jqCr0J8s/
<plars> elopio: well, it's assuming you provide an image, so that's external to this process
<elopio> plars: you need to set up the go tree
<elopio> and export the env vars.
<plars> elopio: for now, I need to build the image outside and provide a url to download it from (preferably gzipped - downloading a 4GB image to bbb is *really* slow)
<elopio> that's explained on the main REAMDE.
<plars> elopio: yeah, I thought I had all that set up though... I'll look again, maybe I missed one
<elopio> plars: that sounds fine. We are generating the image, we would just need to uplaod it.
<plars> elopio: does this not run easily through adt anymore?
<plars> elopio: it seems like there should be an easy way to run this all from a script
<elopio> plars: it does run through adt. Easily, that's relative.
<elopio> plars: your go tree should look like /tmp/scratch/go/src/launchpad.net/snappy.
<plars> elopio: how would you run it through adt today? I didn't see a debian/tests or anything
<elopio> plars: once you get it set up as the root README says, and have autopkgtest installed, you can ./run-checks.
<elopio> plars: we generate the adt-run control file on-the-fly. And pass the path as  an argument.
<elopio> plars: we can't really put the tests in debian/tests as they access the internet, and they require an snappy image. If we put them in debian/tests they will run in proposed migration, making a mess.
<elopio> plars: once we are able to specify this requirements in the control test-class, we could move them to debian/tests. Make a default control file that will have everything we want proposed migration to run.
<elopio> plars: anyway, we can meet whenever you want. I would prefer fgimenez to be around, but that would have to wait until friday.
<elopio> if you prefer to do it earlier, we can just fill him in by mail.
<plars> elopio: normally, I would use the ssh provider or similar for adt and run something like "adt-run --built-tree /tmp/scratch/selftests --output /tmp/results-0123 --- ssh -d -l ubuntu -P ubuntu -H 192.168.1.147
<elopio> plars: yeah, no, we had to wrap that in our own script because adt-run works fine for running, but not for provisioning.
<plars> elopio: the recent changes seem to have made this work very differently from running other tests
<plars> elopio: oh, I provision it ahead of time
<plars> elopio: by provisioning you mean installing the snappy image?
<elopio> plars: go run _integration-tests/main.go --ip  192.168.1.147 would give you that.
<plars> elopio: right, but that's what requires all the aditional go setup, and not just an adt-run
<elopio> plars: if you pass the ip, it does only adt-run, not building the image.
<elopio> plars: but we still need to generate the tests, generate snappy if you want to validate trunk, and some other stuff.
<plars> elopio: I'm not seeing much about go/path setup in http://bazaar.launchpad.net/~snappy-dev/snappy/snappy/view/head:/_integration-tests/README.md
<elopio> plars: take a look at that main.go. Is now nice and readable thanks to fgimenez' refactor.
<plars> elopio: I'm just interested in running the selftests on a device that has already been provisioned
<elopio> plars: http://bazaar.launchpad.net/~snappy-dev/snappy/snappy/view/head:/README.md
<elopio> plars: yes, you still have to copy the tests to that provisioned device.
<elopio> maybe I should add a pointer to the other README.
<plars> ah, I see some of the integration tests are in go now... I didn't look one level deeper
<plars> I was just seeing the shell ones
<elopio> plars: hopefully, all of them will be go by the end of this week.
<plars> doing things like this means a lot more requirements on the host side need to be supported, but if they have to be in go, I'm not sure there's a great way to do it
<plars> I can see the problem
<elopio> plars: what kind of requirements do you think we need?
<plars> elopio: just all the go stuff to get it building on the host, paths set up, etc... not as simple as the adt-run we did previously
<elopio> plars: right, that's because previously you were running the tests on the published image.
<elopio> we need to run the tests before an image is published. So we have to build snappy and get it into the image.
<plars> elopio: I got as far as go build working, but the test run bit doesn't work
<plars> it just drops a binary for snappy itself in /tmp
<elopio> plars: can you paste the output?
<plars> elopio: I did, it was the previous pastebin
<elopio> plars: on the previous pastebin you didn't have the proper go path, that's why it doesn't find the gocheck module.
<sergiusens> plars: godeps -u dependencies.tsv
<plars> $ echo $GOPATH
<plars> /home/plars/go/work
<plars> elopio: under that, it seems to have downloaded all the necessary bits
<plars> elopio: but I'd rather it didn't, don't we want it to pull from the branch checked out so it gets what's there?
<elopio> plars: so cd /home/plars/go/work/src/launchpad.net/snappy and ./run-checks in there.
<plars> sergiusens: I don't seem to have godeps
<elopio> plars: not sure what you meant with the last statement.
<plars> elopio: that's using a downloaded copy from launchpad.net right? how do you tell it to just use what you have checked out (and possibly changed locally)
<plars> elopio: or do you have to download it with 'go get' and modify the copy in your $GOPATH/src/...
<elopio> plars: with go get you only download the dependencies.
<elopio> you get snappy from the bzr branch.
<elopio> not yet sure if that's what you are asking though :)
<elopio> fgimenez: https://code.launchpad.net/~elopio/snappy/test_go_info/+merge/264790 is again ready for review.
<fgimenez> elopio, ok i'll take a look
<elopio> not sure if it's already late for you. If so, don't worry, we'll get it on friday.
<plars> elopio: run-checks seems to be trying to generate the image, which is precisely what I *don't* want to do
<elopio> plars: I know.
<elopio> that's just to check that you have everything in place.
<plars> elopio: I suppose I would also need to build these for arm right?
<elopio> no, instead of ./run-checks, do
<elopio> go run _integration-tests/main.go --ip {remote-device-ip}
<elopio> plars: ah, right
<elopio> go run _integration-tests/main.go --ip {remote-device-ip} --arch arm
<plars> adt-virt-ssh: error: unrecognized arguments: --reboot
<plars> elopio: I guess I need a different version of adt-run?
<elopio> plars: you need the latest one.
<plars> I just have 3.13 from vivid
<elopio> on the _integration-tests/REAMDE you'll find the link.
<plars> ah, I see
<elopio> plars: the one for wily is ok, but sometimes pitti fixes something for us so even that one can be outdated.
<plars> yeah
<elopio> plars: https://trello.com/c/KHFqlCce/157-make-sure-that-the-machines-running-the-tests-have-the-latest-adt-run
<plars> elopio: ok, it's getting farther... I'll see what I can do with this. Thanks a lot for the help!
<plars> elopio: which test beds is that aimed at? something you have set up somewhere?
<elopio> plars: just in general. Everything we use as a test host needs the latest adt-run.
<elopio> we are currently playing with a canonistack machine that will deploy other canonistack machines as testbed. And locally from our dev machines.
<plars> ah, ok. so a pre-run check
<elopio> plars: yes.
<elopio> plars: so, when you get this bbbs running from the lab, we will extend main.go to run the tests in there in addition to in kvms.
<elopio> we'll give you an image path, and you'll return us an ip. Everything will then just work (tm).
<plars> elopio: well, the image really needs to feed into it before you call main.go
<elopio> plars: why is that?
<plars> elopio: so we can just call ubuntu-device-flash somewhere else, build the image and push it to a download site, then the provisioner makes sure the device is stable and flashes it, boots into the image, then the script to run the test is called
<elopio> plars: I think we need to start that whole process from main.go.
<plars> elopio: it's already a very weird process from everything else, but that's the only sane thing I can think of to do
<plars> elopio: that's going to be a problem I'm afraid
<plars> elopio: the provisioning is handling a lot more than building an image, and in fact doesn't even build the image
<plars> it needs the image handed to it
<plars> elopio: it also handles detecting if the device is even available, or if it needs to be recovered to some working state
<elopio> plars: ok, we can call main.go to an already generated and provisioned device.
<elopio> plars: but I'd like two things:
<elopio> 1. tarmac notices a new merge proposal is ready for review, so it triggers the whole test suite in kvm and bbb.
<plars> hah, watching adt-run complain about apparmor rules and autopilot introspection in this context is a bit funny
<elopio> 2. something notices that there's a new image in the channel and triggers the whole suite in kvm and bbb.
<elopio> I think you are thinking about giving us 2, which is useful. But 1 it's more important atm.
<plars> elopio: that part is on your side like all other teams, we just provide the hardware :)
<plars> elopio: you really should sync up with the ci (ols) team and start taking a look at what they are working on
<sergiusens> plars: that ci is not the thing we need; I don't think they are doing source level ci which is what elopio wants; in any case, once we have webhooks and we can talk to this hardware somehow, it won't be a problem
<plars> sergiusens: I'd like to understand how that works, I would imagine that someone wants to test on hardware?
<beuno> elopio, plars, sergiusens, CI is *only* focusing on product testing. That is, testing an image with one or many snaps, and their test suite however that's defined
<beuno> unit tests on branches it out of scope for the CI work being done, or planned to be done
<beuno> the entry point is a built snap
<elopio> beuno: yes, this is not unit testing. This is testing snappy as-installed.
<elopio> now, if snappy is a deb, not a snap. So, that's out of your scope?
<plars> beuno: right, and this may be very different from the tarmac stuff they want to do right now, but if anyone wants to use the selftests as we have done before for testing snappy images, it would need to fit into that process somehow
<beuno> elopio, we start off from a pre-built image, and can put snaps on top. CI won't care how that image is put together
<beuno> elopio, once it's all snaps, it will be able to compose images on-the-fly
<beuno> plars, yeah, we have uses cases for testing snaps before putting them in the store (or publishing them to any channel). Is that what you had in mind?
<beuno> it's still a snap as a starting point
<elopio> ok, so everything will just work. We just need to plug everything.
<beuno> elopio, correct
<elopio> once I know who to get a bbb provisioned, I'll add it.
<elopio> and once I know how to define a test as part of a snapp, I'll add it.
<beuno> elopio, the first MVP will be able to take a URL for an image, trigger test runs in jenkinses controlled by you and in hardware controlled by plars
<beuno> report back
<elopio> and once I know how to generate a image on the fly, I'll add it. 
<beuno> second MVP, that, but also install a snap
<elopio> sounds easy :)
<beuno> elopio, I expect plars to sort out bbb provisioning for you
<beuno> he owns the lab
<beuno> and you just focus on where and how to run tests
<beuno> you slap on a small agent to the jenkinses and CI takes over in driving tests and reporting back
<elopio> beuno: yes, all good.
<elopio> however, I expect we will start having problems when the boundaries between runner, provisioning and test are blurry.
<elopio> for example, we'll run the whole suite on a given image, that's the easy case.
<plars> beuno: as I understand it, a test request can be made with or without a new snap. It just needs to be provided with a image to install and a test payload
<elopio> then we will want to specify image 15.04 alpha -1. Update it, reboot and then run more tests.
<beuno> plars, correct
<elopio> that's harder. But we'll get to it after solving the first case.
<plars> beuno: BBB provisioning is sorted out locally, and as soon as the hardware we've ordered arrives, it will be sorted in the lab
<beuno> elopio, indeed. So provisioning will need to support that, and there's plenty of places to collaborate
<beuno> plars, awesome. The good thing about our current model is that we could run it on your bbb and elopio's until we do  :)
<plars> beuno: my concern here was that provisioning needs an image, not a go binary to try to handle everything fo rit
<elopio> plars: so, for now, we just need an ip from you.
<elopio> we will adt-run into that ip.
<plars> beuno: indeed! I'm sitting next to a breadboard with relays and a BBB fully instrumented right now :)
<plars> elopio: but you don't know how to drive those relays, the control mechanism, hard power on/off, etc
<plars> elopio: that's why I need to have the local provisioning agent handle that
<elopio> plars: what you need is make an adt-run controller that knows how to handle that.
<plars> elopio: it's not that your scripts can't be taught to do that, but it's configured locally per device, so there's no way to know how to do that in a generic way for any device in any environment
<elopio> but in our current tests, we don't need hard power. Not yet.
<plars> elopio: I do, for provisioning and recovery
<elopio> that's only needed on your side, after a test ends.
<plars> elopio: that's why the hard poweroff is needed
<plars> elopio: no
<plars> elopio: for provisioning, the bbb needs to boot either into emmc or sd
<elopio> plars: but that's on your side too.
<plars> elopio: you can never assume that the image on there before was ok to boot from
<plars> elopio: exactly, that's why you *should* want me to handle provisioning, not your tests, and definitely not adt
<elopio> we will give you an image, you provision however you want and we'll wait.
<plars> that's way too late in the process
<elopio> once we get the ip, we run the tetsts.
<plars> and there's no way to make it generic enough for those to handle it
<plars> it's a local device problem to solve
<plars> that way you can just say "here's the image, you sort out how to get it on the device"
<elopio> plars: you are saying just what I'm saying, I don't understand why you say it's too late.
<elopio> we will give you the image, somehow.
<plars> elopio: then that's fine
<elopio> you put it in a device, somehow.
<elopio> when that's ready, you give us an ip, and we adt-run into that ip.
<plars> sort of...
<plars> you would tell us where the test payload lives, which we need to figure out
<elopio> plars: adt-run takes care of setting up the tests, through ssh.
<plars> then a local script would probably have to detect that it's this selftest thing, not a regular adt test, and set up the go environment, call your main.go with the ip of $device
<elopio> you don't need to know anything about the tests.
<plars> then save off the results somewhere you can get to them
<plars> elopio: well, sadly I do now
<plars> elopio: because I can't treat it as a regular adt-run
<elopio> not yet, at least. Once we want to test hardware power-off, then it's harder. But again, we'll figure it out later.
<elopio> plars: adt-run takes care of collecting the results, through ssh again.
<plars> off the device... then what?
<plars> elopio: you don't have access to the local setup, and all that is going to get destroyed locally, not kept forever
<elopio> why can't you treat them as regular adt-run?
<plars> elopio: you said I couldn't
<elopio> plars: yes, but you destroy it after the test finished. After adt-run collected everything.
<plars> elopio: you said I had to set up the go build environment and build it first
<elopio> plars: oh, I meant not as regular dep8 tests.
<plars> right :)
<elopio> these are not normal debian package tests, because they can't run in any debian, they need to run in a snappy.
<plars> elopio: so at that point the tests are sitting on a host system half-way around the world... I assume you'd like to get at them somehow?
<elopio> plars: so, the host is my machine, and the test bed is a bbb in the lab.
<plars> elopio: sure, the previous selftest could run directly from adt-run though, and didn't require building and setup ahead of time, that's what I meant
<plars> elopio: ah, that's where the confusion is
<elopio> once I have the ip of the bbb, from my machine I use adt-run to get the tests  into the bbb, to run the tests, and to collect back the results.
<plars> elopio: your local machine may be the one requesting the test, but it's definitely not the host machine from which adt-run gets called
<plars> elopio: assume  lab, private network, etc
<elopio> plars: it has to be either my machine, or the tarmac machine, or the jenkins machine.
<plars> elopio: otherwise, if you want to instrument beaglebones and run them all locally, that's an option
<plars> elopio: why?
<plars> elopio: this is not that different to the problem on phones... you need something that manages that resource, connects to it locally, etc
<plars> if it's in a lab, it needs to be something there, providing a service that lets you request provisioning, testing, etc. and feeds you back the results
<plars> elopio: otherwise there's no way to stop you from stomping on stuff others are doing, or stop them from messing up your stuff
<plars> elopio: now if it's something sitting locally on your desk, that's not a shared device and you can locally control access to it
<plars> elopio: but the automated provisioning has to be handled locally, or else done manually
<elopio> plars: yes, so you will have in the lab a tarmac machine that will ./run-checks from a snappy branch. And that will request a bbb and run tests in there.
<elopio> that tarmac machine will take care of collecting the results, and  posting them to launchpad.