Opinion – It’s not CCIE or Automation – It’s Both!

Posted by

Many people are pinning the CCIE vs going into network automation like it’s a crossroads where you have to make a “once in a lifetime” decision and then live with the consequences. One path will lead to misery and the other to people rejoicing and singing kumbaya around a camp fire with a million dollars in the bank.

After explaining my view of this “debate” maybe someone will go: “It’s easy for Daniel to have an opinion like this. He is already is certified at the expert level and doesn’t have to make a choice between the two in his career”. Well thank you. The view is astounding up here in the ivory tower. We still have a few rooms to spare… Let me explain why it’s not a choice between the two.

Knowledge still matters

You can’t automate what you don’t know. I’m not suggesting that everyone needs to be a CCIE or that the only path to expert level knowledge is the CCIE but you can’t automate what you don’t know. If you don’t know how the protocols work, how can you verify? How can you know when you have found an anomaly? How do you handle the exceptions? It makes sense to start with the low hanging fruit and automate all of the simple things. That might get you 80% of the way. It’s the remaining 20% that you have to decide if it’s worth automating or not depending on how complex it is.

So let’s say that you have two BGP peerings with two different providers. Someone with CCNA level skills could probably get the peerings up an running, perhaps with a little help from a colleague or by just googling some stuff. The peerings are up and running just fine. All traffic is flowing as expected through Provider 1.

At a later stage, one of the peers starts flapping. Where is the problem coming from? Physical layer? BGP? Maybe the providers router was updated with a new software and it’s sending a malformed update or the TCP authentication is off so the peering can’t come up. Maybe the provider starts sending more or less prefixes than normal. How would you detect this? Maybe traffic shifts to the secondary provider. Would you notice? Maybe you did maintenance on your end and shifted traffic to the secondary provider but after the maintenance the traffic won’t shift back. What happened? (google BGP wedgie).

The point here is that automating the configuration is easy but the person creating the templates still needs knowledge of BGP. Someone still needs knowledge of how BGP operates. Someone needs to understand BGP communities, attributes and traffic engineering. This does not go away with automation.

Certifications are a learning path

Most people overestimate how much of the knowledge that is vendor specific in the CCIE. Most of the knowledge can be used on any vendor equipment. Sure, you spend a lot of time learning the CLI and knobs and behavior of that vendors equipment but if you think that’s all you learn then you either haven’t tried the CCIE or you took the wrong approach while studying for it (sorry). The CCIE is about developing an expert level of protocols and how to implement these. Knowing the CLI is just a byproduct of this. Personally I made sure I spent a lot of time studying TCP/IP, the history of Ethernet, messing around with STP and how BPDUs get formatted depending on what type of link in use etc. I followed the blueprint but I went deeper where I felt necessary and I allowed myself to “mess around” outside of the blueprint to learn things that I expected someone at the expert level should know. Yes, it took my a lot longer than some people to pass the thing but in the end, who cares? I got the job done and learned a ton by doing so. This knowledge helps me in everything new I study as I have learned how to study efficiently and I have a good understanding of protocols and algorithms.

Automation today vs automation tomorrow

Many people argue that the “normal” networking knowledge is becoming obsolete. They argue about vendor CLI and implementations but at the same time have no problems on putting all of their eggs in the baskets of Ansible, Chef, Puppet etc. Going open source is not the same as not having any lock-in. You always have some form of lock-in when you learn a product and develop all of your tooling around that product. I don’t expect that Python and Ansible will be the main automation tools forever. That doesn’t mean that the knowledge is lost. Just like knowledge from learning older networking protocols was not a waste. In the end we will see more and more vendor solutions that are the “total package”. Think Cisco SDA. Think Apstra. Where you can have a “turn key” solution where you don’t need to have Python and Ansible skills to run the products. That doesn’t mean that you can’t leverage that knowledge to extend these products but it’s not going to be required to have those skills to operate these products. That’s why I don’t believe in the saying “every neteng must become a coder”. It’s still useful knowledge. It’s just not how I see the market turning out.

Let experts do what they do best

If you read the interview I did with Ivan recently, you will know that he doesn’t think that every neteng has to become a coder. Quoting Ivan: “there are other people out there that are way better programmers than you are, so focus on what you’re doing best (= networking) and let other experts do what they do best.”. This doesn’t mean that you can’t help developing pseudo code or writing proof on concepts or writing some scripts. However when you put things into production, if you wrote it, you are responsible for it. That means that you have to take the responsibility for bugs and developing the code, adding new features etc. Not everyone enjoys this part of coding, “the grind”. They are more into creating things and the amazement you get when you see something running and it works. It’s an entirely different story to be responsible for the quality of the code, testing it and having to live with the code for a long time (code never dies). So in some cases it’s best to leave the coding to the experts. It all depends on the size and structure of your organization.

Blast radius

You thought deleting the wrong VLAN on one switch stack was bad? How about doing it on 100 switches at the same time? Could be disastrous and a resume generating event. When automating things the blast radius is much larger because it’s so much easier and faster to deploy things than when you entered all of the commands manually or copy/pasting them in. When you do it manually you can often notice when something goes bad, maybe the TTY hangs or you catch something in the logs. When deploying the change en masse you aren’t logged in yourself so you might not catch what’s on the TTY or in the logs. Of course you should do testing but how many have a testing environment that matches up with their production? A typo or the wrong command can have a huge impact when the blast radius is so large. For this reason it’s even more important to have someone knowledgable that understand what needs to be done in a change, the risk of implementing the change and how the change can be recovered if needed to. The person also needs to understand how errors can be detected, what data needs to be gathered in that case and recovering gracefully (if possible). This requires expertise in both protocols, implementation and possibly CLI (if not using APIs). This all comes back to the point above regarding “you can’t automate what you don’t know”. Even if there isn’t an exact testing replica of production, large changes should be deployed in smaller pockets of the network first. These should be selected based on having the least impact on the organization if something goes wrong.

Summary

It’s not CCIE or automation. It’s CCIE and automation. I’m not saying everyone needs expert level knowledge. Someone with CCNA or CCNP level knowledge can become great netengs and combined with automation skills they will become really attractive and successful in the market. There’s still going to be a demand for experts though. Maybe not as many as today because implementation will not be the main factor, knowledge will be. You have to decide if you want to be the person that goes super deep and enjoys knowing the protocols inside out or if you are satisfied with knowing enough to work more with automation and looking up details as you go. Having a CCIE and automation skills will of course make you super attractive in the market. We also have to remember that not all organizations can or are willing to automate at this time for various reasons such as organizational structure, costs or fear of what’s new. It’s not wrong enjoying to work on the CLI. If that’s what you like, then do so. It might not last you a life time but hey, people are still doing COBOL… So don’t think that knowledge doesn’t matter. It does. More than ever. There are new job roles though and the market is changing (as it always does) so go for the thing that is most rewarding and interesting to YOU. Don’t forget about the fundamentals though… Or you might end up repeating mistakes of the past and blasting yourself out of orbit 🙂 Don’t be afraid to go for an expert level certification if you want to and if you don’t want to, that’s fine… Good luck!

14 comments

  1. Great post!

    I agree the value of expert networking knowledge is not being supplanted by automation. I think you are spot on in that estimation.

    I disagree that the CCIE means expert networking knowledge. If you work at a Cisco partner that requires CCIE’s, it’s great and lucrative. If you’re not at a partner, it’s not worth the time, IMHO. Instead, take all of the time you would spend labbing and typing commands for obscure features as fast as humanly possible and read case studies, RFC’s, or other high quality technical resources. Become a real network expert.

    1. The ROI is individual and based on your career path and the area you live in etc. See Phil’s comment above. For me personally it’s been very rewarding in every way.

      I read case studies, blogs, books, RFCs and many resources while studying for the CCIE. It took me around two years to get my CCIE which probably says a lot about how I approached it. I studied for probably around 1500h or more. If people only focus on typing fast and knowing the bare minimum to pass, I’m not sure you can fault a certification for that. People generally go through the path of least resistance to pass something, this holds true for certifications, degrees and most things in life.

      Like I’ve said many times. Degrees, certifications etc. are only tools and it’s how you study for them and how you use them that makes the difference.

      I do agree though that the need for CCIEs in pure numbers will probably go down but not go away.

      1. The ROI is individual and based on your career path and the area you live in etc. See Phil’s comment above. For me personally it’s been very rewarding in every way.

        Maybe you work for a Cisco partner? They require CCIE’s for status, I believe. This may make it lucrative financially and rewarding emotionally, as the partner will pay well to keep you onboard and you would be critical to the larger organization.

        I read case studies, blogs, books, RFCs and many resources while studying for the CCIE. It took me around two years to get my CCIE which probably says a lot about how I approached it. I studied for probably around 1500h or more. If people only focus on typing fast and knowing the bare minimum to pass, I’m not sure you can fault a certification for that. People generally go through the path of least resistance to pass something, this holds true for certifications, degrees and most things in life.

        Maybe you read case studies, blogs, books, RFCs, and many resources in the pursuit of being a networking expert, but I’m not sure you studied these for the CCIE. As you say, you can pass the exam with less, but you probably can’t be a networking expert with less.

        I don’t fault the certification for being what it is. I fault people for assigning false value to it. Take for instance the title of your post, “Opinion – It’s not CCIE or Automation – It’s Both!” You compare a certification to a concept. But in the body of your post, you compare being a networking expert to configuration management, scripting, and intent/policy network configuration. You’re assigning networking expertise to the certification, which I do not think it deserves.

        But I get it. It’s easier to categorize in our minds that a cert = expertise. It makes our lives easier and we don’t have to think too hard.

        Like I’ve said many times. Degrees, certifications etc. are only tools and it’s how you study for them and how you use them that makes the difference.

        On this, we agree.

        I do agree though that the need for CCIEs in pure numbers will probably go down but not go away.

        If the certification in this sentence was replaced with network expert, do you still agree?

        I agree that comparing tools and methods to a body of knowledge is silly. You’re right when you say this isn’t a choice between learning networking and learning to automate networking. This isn’t even a thought for systems engineers. Automation tools are critical to systems’ lifecycle.

        1. You do make valid points.

          I do work for a Cisco partner but we have enough certified people that if my CCIE isn’t really critical in keeping us a gold partner. It is however important for marketing towards customers and to give me credibility in my role.

          In my world a CCIE is a networking expert because my own path was to become an expert and as a byproduct of that pass the lab. The people I talk to the most and my friends that are CCIEs are also true experts so my view is biased based on that. I interact with people daily that are experts, and certified. I know this is certainly not true across the industry and there are sadly a lot of paper CCIEs out there (and other certs).

          We don’t really have anything better to measure people by. It’s all subjective in the end. Many of the people I have interviewed here I would consider real experts, regardless of certifications or not. For example Ivan let his CCIE lapse but he’s certainly not less of an expert because of that.

          I think the number of people with only expertise in networking will decrease, yes. As management tools improve and more things become abstracted we will probably see more people just doing “next, next, finish” like in the systems administrations world. These people will have a basic understanding but not really know what’s going on under the covers. The experts will still be needed for advanced troubleshooting etc. when the pretty GUI fails but they won’t be needed as much for pure implementation. But I think that even as a pure CLI jockey you probably still have at least 5-10 years to go before you would have any trouble finding a job.

  2. I agree with your perspective, Daniel. Learn all of the things! It’s not an all or nothing proposition when it comes to expert level networking skill or automation skill – it’s both. My only caveat is the crossroads we face as the reality of job opportunities begin to change as our careers progress in the context of our local job markets.

    The crossroads in my case isn’t necessarily about choosing between a CCIE or learning automation, but between recognizing what particular skills the job opportunities in my area, my sphere of influence, and in my life require. So for me, the crossroads isn’t between two paths, per se, but in redefining what it means to be a network engineer. That means I’m still working on CCIE labs almost daily because I agree with you about the importance of the knowledge it provides, but the potential opportunities out there also value the soft skills and day 2 sysadmin skills I can bring to the table.

    However, for the last 10 years I’ve completely ignored those soft skills and other skills (such as programming, for example) in favor of staying laser-focused on networking and earning (mostly Cisco) certifications. The crossroads for me is a personal re-defining of how I approach my career considering my local job market, my strengths and weaknesses, and my age.

    For example, I may want to go into management at some point (not likely), and in that instance I would have to re-think how I approach my career, which skills I choose to develop, and how I spend my professional development effort. For some this is easy. They can just go with the flow and shift to accommodate new roles.

    For me, this is difficult because pursuing the next certification and being laser-focused on networking has been both foundational in my life and something I’m passionate about. And therein lies the dilemma. It’s not between paths or between a CCIE or focusing only on something else such as automation, but on the very deep and personal adjustment we make in terms of our identity and our career strategy.

    So when I think of a crossroads in my own personal career, it’s a matter of changing the way I approach being a network engineer for the next stage of my life. Maybe this is difficult because I’m such a creature of habit, or maybe this is difficult for me because it introduces an element of the unknown into my life which is scary – I really don’t know. What I do know, is that settling that within ones-self is definitely a process, a matter of wrestling with ideas and fears, a matter of working it out over time.

    1. Great insights, Phil.

      Choosing a path consists of many inputs like your current knowledge level, current certifications, current job role, desired job role, location, family situation and a lot of things. We have to weigh all of this to come to what is best for forming us and our own career path. So we can certainly be at personal cross roads and I question my own path often. Do I go broader or do I go deep in an adjacent architecture such as DC? Sometimes I'm leaning one way and sometimes another way. It's only human to think about this and question our choices.

      Some people relocate to find other options but you are probably very rooted with your family and the house so I understand that's not an option for you. We all make our choices and work is only such a big part of life, we also need to have balance in life and time for the family and live in an area we are comfortable in.

      I think you know enough and are senior enough to still be really useful if you stray more towards automation than going down the CCIE path. You don't have to give up your passion for networking. Stay passionate and keep learning but put the IE on halt if you need to. The certification will not go away. If your situation changes you can always dive back into it. It's much easier to get on the horse if you have at least kept some studies going though. So to have both options open I wouldn't abandon the CCIE completely which I don't think you were planning to do.

      Good luck and you can always ping me for a chat if you want to.

    2. I think Phil hit it on the head in the last two paragraphs. I think for a lot of folks this goes deeper than the technology itself, and perhaps pangs more on a personal level. When I started in this field I was in my early 20’s, single and no obligations. Now I’m in my mid 30’s, two kids, and a single income family (my wife has the hard job). Change now is different than change 15 years ago.

      The market is beginning to move in some ways, and a lot of us have spent a significant amount of time and effort honing our networking skills, perhaps even abandoning other things due to the amount of focus it required. The networking field itself shifts and grows at an almost exhausting pace, and trying to digest the notion of needing to learn a whole new set of skills on top of it can be tough.

      To say it’s a major change and redefinition of myself in my career is accurate. Being completely candid, since my skills in automation are significantly weaker than my skills in networking, this shift pressures me to prioritize the former to some degree to round myself out. In the end, we’ve all been weathered the storms of the past and come this far, so I think we’ll be fine as long as we embrace change and continue to move forward… but as Phil notes, it will be a process, and it will take time to find true north.

      1. Great insights.

        We all have to evaluate our current skillset, something people often forget. Based on that and where we want to go we need to make up an adequate plan on how to become more well rounded. People make up 3-5 year business plans but for personal development they don’t even know what the next day will look like. I don’t have a strict plan that I follow but I have a plan in my head for the next 3-5 years on how I want to evolve and where I want to go.

        There are of course like the two of you mentioned a lot of more factors that goes into the decision process. I live in a small place so local options are poor but I work for a company in a large (by our standards) city and I can do 99% of the work remote and I travel to visit clients etc. In some disciplines of IT this is perfectly valid so that may or not be options for you as well.

        Looking back I wish I had more of a background in Linux and systems, scripting etc. On the other hand I think we often tend to see someone elses path as better because they are more skilled than us in a discipline that is becoming popular. The systems guy might have the same view of you “the grass is always greener” etc.

        We can only work with what we have now and go from there. The main point I’m trying to make is that studying certifications is a learning path and a tool and it’s never a waste to do so. Knowledge is never wasteful. If the ROI is there or not depends on your personal situation. And I don’t believe in the “all netengs must become coders”. Knowing to code is certainly useful but not a requirement to survive. What is needed is an understanding of application stacks, traffic flows, how systems tie into each other, security, networking, virtualization etc. Noone can be an expert in it all but you need to have the big picture.

  3. I have a completely different view. CCIE is about knobs, knobs and knobs. It’s not about knowledge. Sure, CCIE teaches you basic knowledge but its end goal is to teach you all the cisco knobs possible so they can lock you in. Or have you forgotten what your exam was about? Maybe configuring OSPF in an NBMA network tweaking some timers while configuring EIGRP with some obscure knobs and redistributing prefixes between a few BGP prefixes and OSPF and then to EIGRP, does that ring a bell? It certainly rings a bell to me.

    CCIE knowledge is the antithesis of automation and it’s the very reason why some networks are so hard to automate. Who do you think designed and configured all those snowflakes everybody keeps talking about? The brave new world we live in today is about scale and numbers, it’s not about quality and specialization.

    Also, CCIE knowledge drags down most engineers as they hardly see anything outside it; “you can’t do X, that’s not how it works”. Turns out BGP is code and code can do anything you want regardless of what your instructor told you 😉

    Note that I am not talking about CCIEs, I am talking about “CCIE knowledge”. If you got a CCIE I am sure you are capable but I don’t need/want on my network 90% of the knowledge you got while studying CCIE.

    If you want proper theory and understand how things work, don’t let a vendor pick the topics for you, go to the IETF, read RFCs and/or read papers and forget about whitepapers and vendor certifications. That’s where the real theory is.

    In summary, get your CCIE knowledge off my lawn and let’s talk about this new hot RFC.

    1. We have a very different view on things, and that’s OK. No, the CCIE for me was not about knobs. Sure, I had to learn a lot of strange things and I questioned a lot of them while studying. Later though I learned that some of these features and knobs were heavily in use at customers. Multiple times when studying I thought to myself “Who in their right mind would ever configure something like this?!” to later find that a customer (or whoever configured for them) had done exactly that.

      What version of the lab did you take? I took the RSv4 which was way too heavy on services and had too much hidden bombs etc. The newer version is better. I still stand by my viewpoint that certifications are a learning path, they are a tool, they are as useful as you make them be. Having “CCIE knowledge” or “real knowledge” is not mutually exclusive. For me they overlap almost completely but knowing a vendor and their implementation is still valuable.

      As you probably know I’m an architect and a CCDE so I know a lot about network design. I don’t design networks like a CCIE without design knowledge would. I have probably a similar view as you on snow flake networks. The truth is though that sometimes we need to do things in a special way because the business requirements demand it. It sounds like you are used to working in very standardized DCs or perhaps you even work for a web scaler? Believe me, I would like to design networks like that but when you don’t own the application stack it becomes much more complex, L2 needs to be stretched, and so on. We can have a discussion on this if you want to.

      I do all of the things you mention (except go to IETF) and I still learn vendor technologies because I have use for it in my role. That doesn’t mean it’s all wasted knowledge. Cisco SDA for example is based on VXLAN like all of the overlay technologies. If you understand normal L2/L3 forwarding, VXLAN etc. you have most of the knowledge required for such a solution and you don’t have to focus on only the vendor specific parts.

    2. I know I probably shouldn’t be responding to this, but its hard not to.

      Thankfully, your reply is just a viewpoint. Let me clarify that a bit.

      The basic knowledge statement is simply incorrect. In your studies for RS for example, you learn protocol theory down to the packet level. Anyone who has ever read Routing TCP/IP vol. 1 & 2 knows this. Yes, its taken from a Cisco point of view, but unless you are really off track, 95% of the protocol implementations are directly taken from the RFC’s which you in your last statement says are so important.

      Scale and numbers statement.. True, especially if you are Facebook, LinkedIN, Google etc. Im not arguing against automation, but unless you know what the hell you are automating, it makes no sense. The networks we have today, which enables you to write this comment, was done by the best technology and people available at the time (including the Code running it).

      Sure, you can do X with BGP, you can even modify it yourself and render it completely non-interoperable unless you can get the vendors on board. Is that what you want? probably Yes, if you are the above mentioned companies and have the resources to do so. 99% of companies do not have those kind of resources available, so they rely on vendors and their partners to help them out.

      So you dont want 90% of the stuff thats on a specific CCIE/JNCIE/etc. track? Thats fine, do you want the remaining 10% to work in an optimal fashion or do you just want your automation script to install static routes to everything? Quality matters in the real world, period.

      I dont know if you have any CCIE/JNCIE. etc, but almost from ground zero (CCNA level), you get to read RFC’s to solidify your understanding of protocol theory. Especially so in the expert level certifications.
      In my SP studies I spent at least as much time on white papers and RFC’s as I did on any Cisco Press book.

      One point I will agree with is that its important to read details on particular topics from multiple sources. This is no different than any other academia thats worth mentioning. However, if you want to learn the additional 5% of protocol implementation by Juniper or Cisco or whoever, you need to goto those sources.

      I will happily step of your lawn of Code and talk about a new hot RFC (IPv6 these days).

      I look forward to seeing your real name on the up and coming RFC’s on total world automation domination.

      Ps. dont forget to use your real email address on those RFC submissions!

      /Kim

  4. I am agree with Denial never give up Basic Fundamental 😀😀but my question is why market demanding vedor based certifiation now days if i having knowledge that should be enough.

Leave a Reply

Your email address will not be published. Required fields are marked *

To create code blocks or other preformatted text, indent by four spaces:

    This will be displayed in a monospaced font. The first four 
    spaces will be stripped off, but all other whitespace
    will be preserved.
    
    Markdown is turned off in code blocks:
     [This is not a link](http://example.com)

To create not a block, but an inline code span, use backticks:

Here is some inline `code`.

For more help see http://daringfireball.net/projects/markdown/syntax