cloud-init networking allows networking configuration to be defined
by the:
  user or image builder: /etc/cloud/cloud.cfg or /etc/cloud/cloud.cfg.d/*.cfg
  datasource: datasource discovery and returned in DataSource.network_config
  kernel command line / initramfs: ip= on the command line
  fallback:

Part of cloud-init netorking support is to allow the datasource
or the image creator to name network devices consistently.

Previously, ubuntu images have had the following in 
/etc/network/interfaces:
  auto eth0
  iface eth0 inet dhcp

/etc/network/interfaces will be referred to as ENI for brevity.

This was less than ideal for a number of reasons.
 * 'eth0' is not a stable device name.
   The ethX namespace is owned by the kernel, and thus users
   can't really rely on names in that namespace.
 * in order to make that ENI file right, we have to boot
   with systemd naming disabled. (kernel command line net.ifnames=0)

In order for 'cloud-init-local' to search and apply network configuration
before the system brings up the network, we'll employ the following behavior
on ubuntu.

1.) cloud-init udev rule 79-cloud-init-net-setup-link.rules
    this uses IMPORT to execute 
       /lib/udev/cloud-init-wait cloud-init-local
    udev will thus stop processing this event until cloud-init-wait
    is finished.

2.) cloud-init-wait service [timeout]
    This program blocks until the provided service has finished.
    if cloud-init is disabled, it will exit immediately.
    if timeout is reached it will exit non-zero.

3.) cloud-init-local will will populate files systemd.link files and ENI

    systemd.link files will be written with name template
    50-cloud-init-*.link with rules to accomplish the desired device
    names.  the '*' is the device name.  this should be sufficient
    as cloud-init will only render .link files for naming reasons.

    cloud-init-local should delete any 50-* link files that it did not
    author.

    an alternative to '*' would be with 
    'mac_address'.lower().replace(":", "")

    At the point when cloud-init-local exits,
    /etc/network/interfaces.d/50-cloud-init.cfg
    will have been populated with the content provided.


== control of network configuration ==
All automatic network configuration in cloud-init can be disabled system
configuration in /etc/cloud/cloud.cfg or /etc/cloud/cloud.cfg.d/ via:
  network:
    config: disabled

Simple example:
  echo 'network: {config: disabled}"' > /etc/cloud/cloud.cfg.d/go-away.cfg

== Upgrades ==
upgrades that find a working local datasource (or one with a quick search)
would be fine.  No networking would be written by 'cloud-init local'
as it would not be found to be a new instance id.

Upgrades from older versions of cloud-init should mark that they are
upgrade, via /var/lib/cloud/data/upgraded-network.  
This file being present has the same affect as
  network: config: disabled
An upgrade from a network-enabled version of cloud-init to a new version will
not need to write this file.

When testing, where we insert a cloud-init and want to boot with network
config, we will just need to ensure that file is removed after package
upgrade.


== Precedence ==
network configuration is consulted in the following order, with
subsequent definition taking precedence.
 TODO: is it right to have kernel command line override datasource?
       alternative would be to make the datasource able to easily
       consume the kernel command line, and leave that up to the ds.

 * fallback
   this will dynamically generate a network config
   based entirely on hueristics.  it will essentially
   scan the system and say "dhcp on everything" and assign
   'en*' namespace for nic names
   a system with one nic would end up with fallback config like:
     config:
       - type: physical
         name: en0
         mac_address: "c0:d6:9f:2c:e8:80"
         auto: true
         subnets:
          - type: dhcp4

   if networking was configured in the initramfs (ip= on cmdline)
   it will write config for each device that was configured in initramfs.

 * image /etc/cloud/cloud.cfg or /etc/cloud/cloud.cfg.d/*.cfg
   this allows the image author (or CPC) to provide network
   config like:
     | network:
     |  config:
     |    - type: physical
     |      name: en0
     |      match:
     |         Path: pci-0000:00:1a.0-*
   if something more dynamic than this is needed, then we should
   improve the datasource to dynamically determine this config
   or the datasource should actually find the configuration from
   a declaritive source.

 * datasource
 * kernel command line (ip=)

== Use Cases ==
1.) EC2
   characteristics:
    - no local source of network information
    - network device needed to dhcp to reach metadata service
   behavior:
    - determine which device is to be dhcp'd on for MD access
    - author ENI for "dhcp on eth0"

2.) openstack config drive
   characteristics:
    - local declarative network configuration
    - network configuration provides 'id's for devices, but
      not intended names. id is to be used as the device name.
    - quick check for "is new instance id"
    - network configuration contains mac addresses
   behavior:
    - author ENI on first boot
    - render network device rules that make 'id' == 'name'

3.) cloud-localds
   characteristics:
    - local declarative network configuration
    - network configuration provides device names
    - quick check for 'seed'ed data, but not for attached disk
   behavior:
    - by default do *not* move devices out of 'eth' namespace

3.) maas iscsi root
   characteristics:
    - ip= on kernel command line
   behavior:
    - need to leave in place devices brought up by ip=

4.) iscsi root bigstep
   characteristics:
    - ip= or iscsi ibft configuration 
    - datasource 'local' mode might be tough to determine
      explicitly that this is bigstep. (but ds=bigstep might be an option)

5.) maas install via curtin
    curtin writes 70-persistent-net.rules and correct
    /etc/network/interfaces.  So here we either have to 
     a.) have cloud-init *NOT* muck with ENI or 
         if cloud-init only writes systemd.rules and to 50-cloud-init.cfg
         then situation as it is right now cloud-init's changes
         would have no affect at all, and end result the same.

         alternatively, curtin install could disable network config.
     b.) curtin need to change to just writing its network_config
         to where cloud-init would read it and apply on boot

    'a' is how this worked in the past.  'b' is more like we'd
    be working on other first boot clouds.

6.) cloud instance disabled cloud-init
    the user boots a cloud instance and then is done with cloud-init
    and wants to avoid magic happening on reboot.  They thus touch
    /etc/cloud/cloud-init.disabled.
    The next reboot is expected to function fine, but not have
    cloud-init affecting it in any way.

    The fallout of this use case is that cloud-init needs to write
    its systemd.link files to a persistent directory.
    thus /etc/systemd/network/ is chosen over /run/systend/network

7.) lxd
    characteristics:
      - stable device names in 'eth0' range

== cloud-init local ==
 * cloud-init local
   a.) add 'quick check' for data sources
       if there is a previous datasource in
       /var/lib/cloud/data/previous-datasource then ask that datasource
       if the instance is the same.

       For openstack and azure, simply read dmi data and compare
       to previous.  If this is the same, exit quickly.
       /sys/class/dmi/id/product_uuid on azure

       This would essentially behave like 'manual_cache_clean: true'
       in this case.

   b.) if local datasource is found:
       if new instance:
          apply networking config per Precedence above
       else:
          no network changes made

   c.) if no local datasource is found 
       if manual_cache_clean:
          remove link
       apply fallback network config
       

general:
 * search any previously found datasource first
   the most likely case is that you've snapshotted on the same cloud.

   This will generally improve boot speed.

   If user is snappshotting with intent to move to another cloud, they can
   rm /var/lib/cloud/data/datasource

 * [brainstorm] it seems maybe that all all datasources could have a local datasource
   portion that searched and said 'yes' or 'no', and provided network
   config.   Then, should be able to
   search for network config.  Ie, even EC2 or MAAS would 
   
 * [brainstorm] All datasources run in network mode

== Code items ==
 * cloudinit/net: support rendering ENI to a specific file
   instead of rendering to /e/n/i, we need to render to 
   /e/n/i.d/50-cloud-init.cfg

 * cloudinit/net: support rendering systemd.link rules
   Note, we want to render systemd.link files rather than
   70-persistent-net.rules because systemd.link files are consumed
   more dynamically [although this statement is untested].
   cloud-init-local can write these files while the ADD event
   is being run (and blocked). Where as rules files are not
   dynamically re-read during the procesing of an event.

 * cloudinit/net: support 'match' attribute on an interface
   network config like:
     |  config:
     |    - type: physical
     |      name: en0
     |      match:
     |         Path: pci-0000:00:1a.0-*

   should render into a systemd.link file named 50-cloudinit-en0.link
      | [Match]
      |  MACAddress=12:34:56:78:9a:bc
      |  Driver=brcmsmac
      |  Path=pci-0000:02:00.0-*
      |
      |  [Link]
      |  Name=en0

 * bin/cloud-init: quick check support

 * bin/cloud-init: change search to prefer previous datasource first

 * cloudinit/net: read 'ip=' / initramfs code
   started https://code.launchpad.net/~wesley-wiedenmeier/cloud-init/trunk.net1-cmdline-ip
   Wesley has support for merging there, which is nice.

 * [CPC] stop writing /etc/network/interfaces.d/eth0.cfg:
   cloud-init will start writing the equivalent.

 * cloudinit/stages (or possibly cloudinit/net): apply networking generically
   networking configuration was previously applied in the datasource
   instead we need to implement that here.  Still using the distro
   but the Init after it finds the datasource it should apply the config.

 * udev/cloud-init-wait: 
   this takes the name of a service to wait for.  It waits until
   it has finished (pass or fail) and then exits.
   We may have to employ 'sleep' loop to do this.  One simple
   solution is just to have a loop do
      while [ ! -f /some/file ]; do sleep 1; done'
   and then have cloud-init-local touch that file.

 * datasource rework
   currently datasources are either "Network" or "Local"
   They ones that describe themselves as local are called by cloud-init
   in cloud-init-local.  And those network are called at network time.
   
   This needs changing, to instead allow for the datasource to be
   invoked when local is present and still claim itself.

   the general path is present in both the DataSourceNoCloud and
   the DataSourceOpenstack , where they use 'dsmode' to by default
   have the local datasource just return not found so that it is
   then consumed later in network mode.

   this should be generalized, so that for example Azure's "Local"
   datasource would claim 'found'.  This way other datasources
   dont have to get searched, the search is over, but
   cloud-init [network] would still load the datasource.

 * cloudinit/sources/DataSourceNoCloud.py
   Add 
   change make 'ds=nocloud' read data in /var/lib/cloud/data/seed/nocloud-net
   it should read this if /var/lib/cloud/ata/seed/nocloud is not populated.
   this is desireable as then the local
   

== Notes ==
 * when writing rules, cloud-init will not move device names out of
   the kernel's internel namespaces (ethX).
   Ideally, the network config provider should provide us names not in
   that namespace.

   We could add a cloud-config variable:
      network_config_replace_namespace:
         eth: foo
         
   that would then replace 'eth0' with 'foo0'.

 * cloud-init will render networking into 
   /etc/network/interfaces.d/50-cloud-init.cfg
   If /etc/network/interfaces in the image does not include
   source /etc/network/interfaces.d/*.cfg
   Then it will not be read.
   cloud-init should probably WARN if its writing that file
   and ENI has no 'source' line that would seem to match

== Not Addressed ==
 * user puts rules (either udev 70-persistent-net.rules or 
   systemd.link files */systemd/network/*.link) that conflict
   with cloud-init's name.
   ie, user puts a file in 10-local.link that says match Path=*
   name=mynic0.  Cloud-init would author rules that named it 'en0'
   but the nic's name would be 'mynic0'.  ENI would contain 'en0'
   and would sit around waiting for that device to appear.

   if we wanted to address this, we'd have two options:
     a.) cloud-init implement its own rules and naming function
     b.) cloud-init have to parse */systemd/network/*.link for

== Trusty / 14.04 ==
On 14.04 we will not have 'net_setup_link' in udev or the ability
to use systemd.link.  There, we would have to have cloud-initg

== Things to think about ==
 * iscsi root. bigstep and maas use iscsi root.
   initramfs naming will not have the benefit of knowledge in cloud-init
   (except for on updated initramfs).
   Because the root's network device comes up, it cannot be subsequently
   renamed.  cloud-initramfs-dyn-netconf and bigstep links below
   have a solution for writing ENI for this device.


Links:
 * https://public.etherpad-mozilla.org/p/cloud-init-networking
 * https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1549403
 * bigstep udev rules
   https://git.launchpad.net/~utlemming/+git/bigstep-initramfs/tree/sbin/cloud-bigstep-eth.sh
   https://git.launchpad.net/~utlemming/+git/bigstep-initramfs/tree/lib/udev/rules.d/74-bigstep.rules
 * smoser random work towards this from http://paste.ubuntu.com/15419087/
   bzr+ssh://bazaar.launchpad.net/~smoser/cloud-init/trunk.net1/