Automating ASR9K Software Upgrades with Cisco CSM
Cisco Software Manager (CSM) is an orchestration and management platform for Cisco ASR9K IOS-XR devices that provides a framework for the automated scheduled deployments of SMUs (Software Maintenance Upgrades) and SPs (Service Packs). The freely downloadable software, written in Python and deployed with Docker, has built-in CCO integration for downloading and deploying patches, service-packs and version upgrades to your Cisco inventory.
We decided to trial CSM as a platform early this month, and here I would like to document our findings.
Installation
Now the CSM application is written in Python, and is deployed in the form of 3 containers (csm-server, csm-database and csm-supervisor).
For installation, we used a script provided by Cisco:
export https_proxy=http://10.100.20.9:8080/
cd /opt
curl -L https://devhub.cisco.com/artifactory/software-manager-install-group/install.sh -O
chmod +x install.sh
./install.sh
Once installed, you should have be able to see your containers running:
Once there, go ahead and browse to your host on port 5000!
Setup
Jump Host
If you restrict remote access to your network hosts by source IP (as you should), you will likely already have your designated jump host that you use to access equipment. CSM will need SSH access to this host in order to establish a connection to your inventory. Setup the jump host from the dashboard:
Repositories
While your CSM server is able to download software directly from CCO, your network hosts will ultimately be the ones pulling these files. We typically use a designated FTP server for this, though CSM supports SCP, SFTP, TFTP and Local (already stored on the device).
Set up your repository accordingly:
CCO
In order to download software from CCO, your Cisco ID will need to have the appropriate access and contract ID's associated with your Partner Profile Account. I'm not going into this, it's a mess.
Plop in your Cisco account credentials into your CSM user preferences:
Inventory
I'm not sure about "Inventory", and don't see it's purpose. Nevertheless I populated all of our ASR9Ks as follows:
Managed Hosts
Now this is what I initially thought "Inventory" was. Import your network hosts with the appropriate access credentials:
I think we are about there now.
Our First Upgrade
Well, this was a huge success! We chose to test CSM with the following tests, in order:
1) Hitless SMU (Standard bug fix - no reload required)
2) Impactful SMU (Needs reload)
3) Service Pack (Needs reload)
4) Version upgrade
And you know what...all tests passed with ease :) Here's how they work:
The Upgrade...
1) Select your Managed Host from the Dashboard, and look for this beautiful button:
We will want to schedule an installation here, and have an option of how far we trust CSM to go. In my case, I am choosing "ALL" which gives CSM the directive to automate the entire upgrade, from initial Pre-Check show commands to the FTP-ing and activating of the pies / rpms. For caution, you may want to try "Add" which will limit the process to only downloading the software and copying over to the device:
2) The following screen will now show you all available SMUs and Service Packs for the base firmware version the device is running. Select all software that you wish to push to the host:
Assuming the software cannot be found on your designated repository, it will be downloaded from CCO to the repository as the first step:
You can check on the status of your CCO downloads from the CCO download dashboard.
3) You should now see the pending jobs running:
If any issues occur, you can see the full logs for every process involved. Having faced teething problems setting CSM up, I was incredibly impressed with the level of logging available. Cisco logs, Python exceptions, Linux permission errors, device storage warnings....the logs obviously depend on the specific process, but I found troubleshooting all issues we encountered a relative breeze:
4) Now if you are incredibly fortunate, you will soon see these green boiiis:
...and that's just awesome.
Successful commits indicate that the SMU / Service Packs / Firware were downloaded, FTP-ed to the ASR, activated, verified and commited. If your specific software required post-activation reloads, CSM will automate this as part of the commit step. You can now see your software on the device:
5) If you now have inactive packages that you want to clean up to save space, CSM can help with that from out familiar dropdown:
Voila!
We are still very new to CSM, and there is so much more that it can help automate! Golden ISOs, 32-bit to 64-bit migrations, patching conformance, etc. I hope to dig into these features in the near future.
Thank you for reading :)