Ansible for SAN Automation: Part 2 Cisco MDS

The most difficult hurdle I see in automating a SAN infrastructure is with network equipment like Fiber Channel switches. Typically these tend to be devices that the network team dont want to administer so this equipment management usually falls to the storage team, and rightly so. A problem I see a lot is lack of education, often storage admins and engineers come with platform (linux, unix, windows) experience but very few are converted network admins with prior switch management experience. But even if they did have prior Cisco ios/nxos experience its very rare that I meet a network admin (current or prior) with knowledge of Fiber Channel protocol. Which is strange, I would assume they would get some with Nexus training, but I digress.

Perhaps the most irksome aspect of fiber channel switch configuration is zoning. It is a straight forward enough concept, and maybe I will find time to create a more in depth post on the topic, but essentially zoning a switch is the act of allowing two or more fiber channel nodes to communicate with each other. So when you are configuring a switch from greenfield you are often creating hundreds or maybe even thousands of zones in your config. It can be very time consuming and subject to human errors if manually done. Anytime you get any new fiber channel connected equipment, well you have to make new zones so it can communicate with all your existing equipment. My point is, if we can automate this action we can save a lot of time. Arguably most important, we can keep cleaner switch configs and ensure our naming conventions and our zones are laid out consistently between fabrics.

In this article I will lay out the current methods and setup I am using to automate Cisco MDS configurations. Specifically, we will use Ansible playbooks ran from an Ansible control node to automate initial MDS switch setup from greenfield. We will create playbooks to automate vsan creation, bulk and singular alias creation, bulk zone creation, zoneset creation, zoneset activation, and finally off-device configuration backups. Lets get started..

Inventory:

Your inventory of your control node should have entires that appear as follows for your Cisco MDS switches. Where you keep your vars is a bit of a personal preference, in the example below I have placed them in my inventory file which is a sin to some. Place them where you prefer just bear in mind Ansible's variable precedence.

[cisco_mds]
a-9710-mds.example.net fabricid=A vsanid=2 vsanname=production switch_name=a-9710-mds zs_name=zs_vsan2_A
b-9710-mds.example.net fabricid=B vsanid=3 vsanname=production switch_name=b-9710-mds zs_name=zs_vsan3_B

[cisco_mds:vars]
ansible_connection=ansible.netcommon.network_cli
ansible_network_os=cisco.nxos.nxos
ansible_user=admin
ansible_password=!vault...
ansible_become=yes
ansible_become_method=enable
ansible_become_password=!vault...
ansible_ssh_common_args='-o ProxyCommand="ssh -W %h:%p -q bastion01"'

[all:vars]
domain_name=example.net

Switch Setup:

This is a simple initial device setup playbook, to be clear this should be ran after running the initial setup command via the console. For ansible to be able to connect to the switch it requires an existing user, password, and IP address. Either SSH or HTTP/S must be enabled. This play book enables features and does some extended configurations like NTP, email alerts, DNS servers etc. It makes a number of assumptions on options that are preferred for my environment but yours maybe different so please customize it for your environment before running.

#!/bin/ansible-playbook
########################

# DESCRIPTION
# Performs initial setup of Cisco MDS

# DIRECTIONS
# Usage to setup a new Cisco MDS switch or to ensure settings are reapplied to an existing switch run:
# Run ./setup-switch.yml --limit <your_switch_inventory_name> -v

---

- name: "##### Performs initial setup of Cisco MDS #####"
  gather_facts: no
  hosts: '{{target|default("cisco-mds")}}'
  vars:
    creds:
      host: "{{ inventory_hostname }}"
      username: "{{ ansible_user }}"
      password: "{{ ansible_password }}"
      transport: cli #nxapi

  tasks:

    - name: Configure hostname and domain-name
      cisco.nxos.nxos_system:
        hostname: "{{ switch_name }}"
        domain_name: "{{ domain_name }}"
        provider: "{{ creds }}"

    - name: Configure DNS servers
      cisco.nxos.nxos_system:
        name_servers: "{{ dns_servers }}"
        provider: "{{ creds }}"

    - name: Configure NTP Services
      cisco.nxos.nxos_ntp:
        server: "{{ ntp_servers }}"
        key_id: 32
        prefer: enabled
        provider: "{{ creds }}"

    - name: Remove the current motd banner
      cisco.nxos.nxos_banner:
        banner: motd
        state: absent
        provider: "{{ creds }}"

    - name: Configure the motd banner
      cisco.nxos.nxos_banner:
        banner: motd
        text: "{{ banner_text }}"
        state: present
        provider: "{{ creds }}"

    - name: Create new ansible user
      cisco.nxos.nxos_user:
        name: ansible
        sshkey: "{{ lookup('file', '~/.ssh/id_rsa.pub') }}"
        state: present
        provider: "{{ creds }}"

    - name: Set ansible users role
      cisco.nxos.nxos_user:
        aggregate:
        - name: ansible
        role: network-admin
      state: present
        provider: "{{ creds }}"

    - name: Ensure npiv feature is enabled
      cisco.nxos.nxos_feature:
        feature: npiv
        state: enabled
        provider: "{{ creds }}"

    - name: Ensure fport-channel-trunk feature is enabled
      cisco.nxos.nxos_feature:
        feature: fport-channel-trunk
        state: enabled
        provider: "{{ creds }}"

    - name: run multiple commands on remote nodes
      cisco.nxos.nxos_command:
        commands:
        - configure terminal
        - callhome
        - enable
        - email-contact "{{ groupmailbox }}"
        - switch-priority 0
        - transport email reply-to "{{ groupmailbox }}"
        - transport email smtp-server "{{ smtp_server }}"
        - exit
        - callhome test
        provider: "{{ creds }}"

    - name: Ensure the required vsans exist
      cisco.nxos.nxos_vsan:
        vsan:
        - id: "{{ vsanid }}"
          name: "{{ vsanname }}"
          remove: false
          suspend: false
        provider: "{{ creds }}"

Create Aliases:

Now that our switches are connected, running, and have a basic configuration to function properly in our environment, I first like to identify all my connected flogi devices. My preferred method is to assign all WWPNs a device-alias. There are two types of aliases in Cisco speak, device-alias and fcalias, device-alias is the newer and recommended alias type to use if your fabric is homogeneously cisco. One nice thing that makes device-aliases superior to fcalias is they are not bound to a single vSAN.

I have made the following playbook for you to run against the [cisco_mds] ansible group. This playbook creates device-alias entries either adhoc or in bulk using a device-alias.csv file. See the # DIRECTIONS section in the playbook for usage instructions. I run this playbook against both my A and B fabric switches. Yes, I define all aliases on both A and B fabrics, it's true the same WWPN wont be plugged into both A and B but in the event it does get plugged into the wrong fabric switch I like to be able to easily identify it.

#!/bin/ansible-playbook
########################

# DESCRIPTION
# Creates and Removes Client Device Alias on Cisco MDS

# DIRECTIONS
# Usage to create a single device-alias
# Run ./create-device-alias.yml --tags create-adhoc -e "alias_name=esx-server01,wwpn=56:2:22:11:22:88:11:67" -v

# Usage to remove a single device-alias (note, wwpn value not required to remove)
# Run ./create-device-alias.yml --tags remove-adhoc -e "alias_name=esx-server01" -v

# Usage to create multiple device-alias using device-alias.csv file: 
#     1. create a file called device-alias.csv in the same directory as this playbook.
#     2. Make 2 columns titled "Alias_Name" and "WWPN" (case sensitive)
#     3. fill in the values in all columns for all the hosts you want to create.
#
# Run ./create-device-alias.yml --tags create-csv -v

# Usage to remove multiple device-alias using device-alias.csv file:
# Run ./create-device-alias.yml --tags remove-csv -v
---
- name: "##### Creates and Removes Client Device Alias on Cisco MDS #####"
  gather_facts: no
  hosts: '{{target|default("cisco_mds")}}'
  vars:
    creds:
      host: "{{ inventory_hostname }}"
      username: "{{ ansible_user }}"
      password: "{{ ansible_password }}"
      transport: cli
  tasks:

    - name: Reading the csv file
      read_csv:
        path: device-alias.csv
      register: alias_list
      delegate_to: localhost      
      tags:
      - create-csv
      - remove-csv    

    - name: Creates Cisco MDS device-alias from device-alias.csv
      cisco.nxos.nxos_devicealias:
          distribute: yes
          mode: enhanced
          da:
              - { name: '{{ item.Alias_Name }}', pwwn: '{{ item.WWPN }}'}
             # - { name: '{{ item.Alias_Name }}', remove: True}
          provider: "{{ creds }}"          
      loop: "{{ alias_list.list }}"
      register: result
      tags:
      - create-csv
    - debug: var=result

    - name: Removes Cisco MDS device alias from device-alias.csv
      cisco.nxos.nxos_devicealias:
          distribute: yes
          mode: enhanced
          da:
             # - { name: '{{ item.Alias_Name }}', pwwn: '{{ item.WWPN }}'}
              - { name: '{{ item.Alias_Name }}', remove: True}
          provider: "{{ creds }}"          
      loop: "{{ alias_list.list }}"
      register: result
      tags:
      - remove-csv
    - debug: var=result

    - name: Creates a single device alias on Cisco MDS
      cisco.nxos.nxos_devicealias:
          distribute: yes
          mode: enhanced
          da:
              - { name: '{{ alias_name }}', pwwn: '{{ wwpn }}'}
             # - { name: '{{ alias_name }}', remove: True}
          provider: "{{ creds }}"          
      loop: "{{ alias_list.list }}"
      register: result
      tags:
      - create-adhoc
    - debug: var=result 

    - name: Removes a single device alias on Cisco MDS
      cisco.nxos.nxos_devicealias:
          distribute: yes
          mode: enhanced
          da:
             # - { name: '{{ alias_name }}', pwwn: '{{ wwpn }}'}
              - { name: '{{ alias_name }}', remove: True}
          provider: "{{ creds }}"          
      loop: "{{ alias_list.list }}"
      register: result
      tags:
      - remove-adhoc
    - debug: var=result  
    

After you have ran the above playbook against both fabric A and B you should be able to run the command show flogi data and easily identify all the connected devices by their alias.

Create Zones

Now that we have all our devices defined as device-alias entries zoning and future administration will be much simpler. I have developed the following playbook to allow you to create zones with ansible automation. Unlike the create-device-alias.yml playbook above, this doesn't not accept a tag to create a single zone, seemed to me the juice wasnt worth the squeeze. Instead it only accepts zoning information from a A_zones.csv or B_zones.csv file for A and B fabrics respectively.

When you run the playbook it will create multi-target Cisco Smart Zones named Host_Alias_Name_Arrayname where Host_Alias_Name is the initiator and Array_Alias_CT0 + Array_Alias_CT1 are the targets (one for each storage controller). For each zone created it will place it in the zoneset named zs_vsanA or zs_vsanB and set that as the active zoneset configuration.

#!/bin/ansible-playbook
########################

# DESCRIPTION
# Creates and Removes Zones on Cisco MDS

# DIRECTIONS
# Usage to create multiple fabric zones using A_zones.csv or B_zones.csv file: 
#     1. create a file called A_zones.csv and B_zones.csv in the same directory as this playbook.
#     2. Make 4 columns titled "Host_Alias_Name", "Arrayname", "Array_Alias_CT0", "Array_Alias_CT1"
#         a. "Host_Alias_Name" is the alias of the host initiator.
#         b. "Arrayname" is the short name for the storage array, not the individual controller names.  Does not need to correspond to an alias.
#         c. "Array_Alias_CT0" and "Array_Alias_CT1" respectively are the alias names on each controller (CT0=Controller 0, CT1=Controller 1.  Some array manufactures call these "SP" for storage processors or "SC" for storage controller ) for the ports you want to make the targets.  In a highly available setup you should have one port on each controller per zone. This depends upon your cabling strategy, in general I recommend a simple strategy where even FC ports connect to fabric A and odd FC ports connect to fabric B.
#     3. fill in the values in all columns for all the hosts you want to create.
#     4. Note: Your switches should have defined vars for "vsanid" and "fabricid".

# Run ./create-zones.yml --tags create-csv -v

# Usage to remove multiple zones using A_zones.csv or B_zones.csv file:
# Run ./create-zones.yml --tags remove-csv -v

---

- name: "##### Creates zones and adds to zoneset on Cisco MDS #####"
  gather_facts: no
  hosts: '{{target|default("cisco_mds")}}'
  vars:
    creds:
      host: "{{ inventory_hostname }}"
      username: "{{ ansible_user }}"
      password: "{{ ansible_password }}"
      transport: cli #nxapi

  tasks:

    - name: Reading the A_zones.csv file
      read_csv:
        path: A_zones.csv
      register: A_zones_list
      delegate_to: localhost
      when: fabricid == "A"    
      tags:
      - create-csv
      - remove-csv

    - name: Reading the B_zones.csv file
      read_csv:
        path: B_zones.csv
      register: B_zones_list
      delegate_to: localhost
      when: fabricid == "B"    
      tags:
      - create-csv
      - remove-csv  

    - name: Ensure the required vsans exist
      cisco.nxos.nxos_vsan:
        vsan:
        - id: "{{ vsanid }}"
          name: "{{ vsanname }}"
          remove: false
          suspend: false
        provider: "{{ creds }}"
      tags:
      - create-csv

    - name: Create zones from csv file
      cisco.nxos.nxos_zone_zoneset:
        provider: "{{ creds }}"
        zone_zoneset_details:
        - vsan: "{{ vsanid }}"
          mode: enhanced
          smart_zoning: yes
          zone:
          - members:
            - devtype: initiator
              device_alias: "{{ item.Host_Alias_Name }}"
            - devtype: target
              device_alias: "{{ item.Array_Alias_CT0 }}"
            - devtype: target
              device_alias: "{{ item.Array_Alias_CT1 }}"
            name: '{{ item.Host_Alias_Name }}_{{ item.Arrayname }}'
            remove: no
      when: fabricid == "A"
      loop: "{{ A_zones_list.list }}"
      tags:
      - create-csv

    - name: Add zones to active zoneset
      cisco.nxos.nxos_zone_zoneset:
        provider: "{{ creds }}"
        zone_zoneset_details:
        - vsan: "{{ vsanid }}"
          mode: enhanced
          smart_zoning: yes
          zoneset:
          - action: activate
            members:
            - name: '{{ item.Host_Alias_Name }}_{{ item.Arrayname }}'
            name: "{{ zs_name }}"
            remove: no
      when: fabricid == "A"
      loop: "{{ A_zones_list.list }}"
      tags:
      - create-csv

    - name: Create zones from csv file
      cisco.nxos.nxos_zone_zoneset:
        provider: "{{ creds }}"
        zone_zoneset_details:
        - vsan: "{{ vsanid }}"
          mode: enhanced
          smart_zoning: yes
          zone:
          - members:
            - devtype: initiator
              device_alias: "{{ item.Host_Alias_Name }}"
            - devtype: target
              device_alias: "{{ item.Array_Alias_CT0 }}"
            - devtype: target
              device_alias: "{{ item.Array_Alias_CT1 }}"
            name: '{{ item.Host_Alias_Name }}_{{ item.Arrayname }}'
            remove: no
      when: fabricid == "B"
      loop: "{{ B_zones_list.list }}"
      tags:
      - create-csv

    - name: Add zones to active zoneset
      cisco.nxos.nxos_zone_zoneset:
        provider: "{{ creds }}"
        zone_zoneset_details:
        - vsan: "{{ vsanid }}"
          mode: enhanced
          smart_zoning: yes
          zoneset:
          - action: activate
            members:
            - name: '{{ item.Host_Alias_Name }}_{{ item.Arrayname }}'
            name: "{{ zs_name }}"
            remove: no
      when: fabricid == "B"
      loop: "{{ B_zones_list.list }}"
      tags:
      - create-csv

    - name: Removes zones from the active zoneset
      cisco.nxos.nxos_zone_zoneset:
        provider: "{{ creds }}"
        zone_zoneset_details:
        - vsan: "{{ vsanid }}"
          mode: enhanced
          smart_zoning: yes
          zoneset:
          - action: activate
            members:
            - name: '{{ item.Host_Alias_Name }}_{{ item.Arrayname }}'
              remove: True            
            name: "{{ zs_name }}"
            remove: no
      when: fabricid == "A"
      loop: "{{ A_zones_list.list }}"
      tags:
      - remove-csv

    - name: Deletes zones found in zones.csv file
      cisco.nxos.nxos_zone_zoneset:
        provider: "{{ creds }}"
        zone_zoneset_details:
        - vsan: "{{ vsanid }}"
          mode: enhanced
          smart_zoning: yes
          zone:
          - members:
            - device_alias: "{{ item.Host_Alias_Name }}"
            - device_alias: "{{ item.Array_Alias_CT0 }}"
            - device_alias: "{{ item.Array_Alias_CT1 }}"
              remove: True
            name: '{{ item.Host_Alias_Name }}_{{ item.Arrayname }}'
            remove: True
      when: fabricid == "A"
      loop: "{{ A_zones_list.list }}"
      tags:
      - remove-csv

    - name: Removes zones from the active zoneset
      cisco.nxos.nxos_zone_zoneset:
        provider: "{{ creds }}"
        zone_zoneset_details:
        - vsan: "{{ vsanid }}"
          mode: enhanced
          smart_zoning: yes
          zoneset:
          - action: activate
            members:
            - name: '{{ item.Host_Alias_Name }}_{{ item.Arrayname }}'
              remove: True            
            name: "{{ zs_name }}"
            remove: no
      when: fabricid == "B"
      loop: "{{ B_zones_list.list }}"
      tags:
      - remove-csv

    - name: Deletes zones found in zones.csv file
      cisco.nxos.nxos_zone_zoneset:
        provider: "{{ creds }}"
        zone_zoneset_details:
        - vsan: "{{ vsanid }}"
          mode: enhanced
          smart_zoning: yes
          zone:
          - members:
            - device_alias: "{{ item.Host_Alias_Name }}"
            - device_alias: "{{ item.Array_Alias_CT0 }}"
            - device_alias: "{{ item.Array_Alias_CT1 }}"
              remove: True
            name: '{{ item.Host_Alias_Name }}_{{ item.Arrayname }}'
            remove: True
      when: fabricid == "B"
      loop: "{{ B_zones_list.list }}"
      tags:
      - remove-csv
    

Configuration Backups

Thats it, now your switches have a working configuration with vsans, device aliases, and zones. All your devices should see the storage array and vis-versa, you are almost ready to register all the WWPNs of the clients to the storage array but first I recommend now that you have that great working configuration you setup an automatic backup job to protect it. Now if you are working in an open environment where you dont have software restrictions I would alternatively suggest you look at a product like Oxidized to automate your configuration backups as it supports a lot of different equipment from a lot of different vendors but I dont have that open network luxury and I will save that setup and configuration for another future blog article.

Instead today we are going to use a simple Ansible playbook for configuration backups. Cisco MDS is able to backup with a variety of protocols to external hosts; FTP, TFTP, SFTP, SCP. Seems like a lot of Cisco admins prefer TFTP for ease of use and quick setup but I always find that my ansible control node is very handy as it already has built in SFTP server. I wouldn't recommend using SCP in your playbook because it was recently announced to be depreciated so its only a matter of time before it will cease to function.

In the past I have shown you how to manually configure your switch to push backups to an SFTP repository, but this is an example of how to do something similar with Ansible. My playbook below will log into the Cisco MDS switch, perform a backup of the startup-config, and place that backup in a path of your choice on your ansible control node. For this method I would recommend using an NFS or SMB mount to an external NAS, preferably one with capacity reduction techniques like inline compression and data deduplication.

To get started simply edit the backup_path and retention to values you prefer then schedule to playbook to run at a specific time via crontab on your ansible server. I use a simple crontab line like 0 6 * * * ansible-playbook /data/ansible/playbooks/cisco/backup-config.yml -v | mailx -v -s "Cisco MDS Backup Report" -S smtp=smtp://smtp.example.net:25 -r ansible-control01.example.net [email protected], this will email me the log of the playbook running on all my switches everyday at 06:00 am. Additionally you could - import_playbook this playbook in any of your other Cisco MDS playbooks to have a backup taken before or after a change such as zoning. Cool huh.

#!/bin/ansible-playbook
########################

# DESCRIPTION
# Creates backup of configuration of Cisco MDS or Nexus Switches

# DIRECTIONS
# Usage to backup a switch or switches, first filter vars below for "backup_path" and "retention" to values you prefer.
# Run ./backup-config.yml

# RECOMMENDATION
# Schedule this script to run via crontab
---
- name: "##### Backups up Configuration information for Cisco MDS Switches #####"
  hosts: '{{target|default("cisco_mds")}}'
  gather_facts: true

  vars:
    backup_path: /data/backup
    retention: 181d 

  tasks:

    - name: "Display Ansible date_time fact and register"
      setup:
        filter: "ansible_date_time"
        gather_subset: "!all"

    - name: Backup Cisco MDS Configuration
      cisco.nxos.nxos_config:
        backup: yes
        backup_options:
          filename: "{{ inventory_hostname }}_backup_{{ ansible_date_time.date }}_{{ ansible_date_time.time }}.cfg"
          dir_path: "{{ backup_path }}/{{ inventory_hostname }}/"

    - name: Find old config backups to prune
      delegate_to: localhost
      find:
        paths: "{{ backup_path }}"
        patterns: "*.cfg"
        age: "{{ retention }}"
        recurse: yes
      register: files_to_delete

    - name: Prune old config backups
      delegate_to: localhost
      file:
        path: "{{ backup_path }}"
        state: absent
      with_items: "{{ files_to_delete.files }}"
      

In my next entry to the Ansible for SAN Automation series I will cover automating an entire Pure Storage Flash Array deployment, stay tuned