Don’t Miss the “Cloud Native Infographic” !

Everything is cloud Native from 5G core to RAN, transport and orchestration. Either you know about it or Nothing about Cloud. In this “FREE” one page infographic poster, I have made it a QUICK and EASY reference for Cloud Native main concepts which are otherwise very complex to understand. Plus get notified when important blogs are published.

Cloud Native Infographic

Don’t Miss the “Cloud Native Infographic” !

Everything is cloud Native from 5G core to RAN, transport and orchestration. Either you know about it or Nothing about Cloud. In this “FREE” one page infographic poster, I have made it a QUICK and EASY reference for Cloud Native main concepts which are otherwise very complex to understand. Plus get notified when important blogs are published.

The Marriage of IP with OTN/DWDM !

When revenue per bit starts falling and cost per bit starts increasing, it’s time to think about alternative strategies to expand Transmission/IP network.

As IP is predominant traffic in Transmission networks, therefore any expansion strategy on Transmission should be driven by IP demands. Indeed, both IP and Transmission departments need to work for a joint and synergetic expansion strategy, in this regard.

Hence, I intend to share with you some insights and tips that will enable you to plan the expansion of IP and Transmission network in a holistic and synergetic way. If you stay till the end, you will see how IP can live with lower layers of Transmission (DWDM, OTN, SDH, and L2) through a successful marriage.

Whether you belong to IP department (responsible for the IP/MPLS domain) or Transmission department (responsible to run SDH, DWDM, OTN and most recently Ethernet), there is some food for thought for you in this article. More specifically, I will show you how to put your traffic on the right layer that will reduce the TCO (Total Cost of Ownership) of your Core network.

So let’s see where is the problem?

Problem#1

The Problem#1 is that of treating both IP and Transmission as two different entities   in terms of any expansion strategy. There is a reason for that: classically, both IP and Transmission are run by separate departments.  Transmission (lower layer) is treated as a resource to serve IP layer (upper layer) without the need for Transmission to understand or help upper layer ( except connectivity).

I will give you one example:

Have you heard of the “Dumb Pipe” approach?

An approach, in which IP uses Transmission as transparent pipe. In other words, shifting complete aggregation, protection and intelligence to IP layer, thus keeping Transmission as dumb and non-intelligent pipe.

For obvious reasons, this strategy is simple and easy to implement. As Transmission does not need to provide any other service except connectivity, therefore there is little or no interaction needed between IP and Transmission folks.

But there is another side of this approach:

More CAPEX and OPEX spending!

How?

Such operators need to consistently build up two layers in parallel: the IP layer and the Transmission layer.

I call it “Grow-Routers-Fast, Grow-Lower-Layers-Fast” (GRF-GLF) approach.

In GRF-GLF approach, Transmission is participating neither in aggregation, nor in protection or any intelligence for IP packets. Therefore, IP layer takes complete burden to achieve these objectives. More specifically, the bits, after entering into IP core, need to touch every core router/IP layer on its way to destination. All this because, IP is the only layer that can aggregate, protect or route the traffic.

As, all bits have to pass through the transit routers as well, so we end up building interfaces for three layers:

  1.  Originating Core Router
  2. Terminating Core Router
  3. All Intermediate/Transit Core Routers

Isn’t it unfortunate that the intermediate router has to take the burden of the bits that are not dropped or added by it?

Worst still:

With every scale of routers, there would be a need to corresponding scale the Transmission resource. So indeed, we are constantly building routers and building Transmission with fast pace- a GRF-GLF Strategy!

Problem# 2

Secondly and in contrast to above, there is another approach that calls for “Router Offload”. The term “Router Offload” has been abused a lot by opponents and proponents on both sides. “Router offload” in its simplest definition mean that any transit traffic at any intermediate core router should be offloaded to lower layer like SDH/OTN/ DWDM.

Therefore, core routers at any intermediate site do not need to be scaled for transit traffic. This will result in huge cost savings.

However, terms like “Router offload” and “IP offload” are pitched in a sense that gives out a massage that core routers are not needed at all. They go to the extent of even calling it “hollow core” designs with no core router in a network at all. Indeed, a wrong perception. Later on, I can show you that “Router offload” for some traffic does make sense but not for all kind of traffic.

It will also become apparent that Router offload is part of a solution but not a complete solution, if applied to all traffic.

What is the solution? How can IP and Transmission live through a successful marriage?

Having presented the problem, let’s see how we can solve these issues.

I would call upon some statistics that will support me to solve these problems.

Some stats form Cisco VNI-2013: (Annual report on future global data; following excerpts are from the updated report of June 2014)

  1. Fixed internet data will rise at CAGR of 20% annually up to 2018
  2.  Mobile data (includes Mobile data and Internet traffic) will rise at CAGR of 61% annually up to 2018
  3. By 2018,Consumer Internet will represent two thirds of all IP traffic followed by managed IP at 20%

The key message here is that internet data will rise considerably (at CAGR 20%) more than any other data (e.g. enterprise data) in networks. And by 2018, consumer Internet will be two thirds of all data carried over networks.

I repeat the key message is, “Internet data” ( Also called HSI:” High Speed Internet”) is/will be the pre-dominant traffic in networks”.

The direct corollary is; “The nature of “Internet data/ HSI data” should drive any plan for network expansion”.

So what is the nature of the Internet data? The Internet data flows to fixed destination in network: towards internet gateways from originating routers. In fact this is a “point to point” data .This very nature is quite different than any other layer 3 VPN data  that has to follow any to any/ mesh connectivity and needs to stop at every core router.

In other words, for the Internet data, the destination is fixed and known to an operator.

This nature of Internet data, as probably you have guessed, is similar to the point to point behavior of lower layers of Transmission. Wouldn’t it be appropriate to push this date to lower layers and take advantage of the lower layers?

This works on the principle that ““Cost to carry data on the lower layers is less than carrying it over IP layer”

By intelligently designing core network, routers can take help of the lower layers (L2, SDH, OTN or DWDM) to carry the traffic to the Internet Gateway site.

This means, the data does not need to touch IP layer at every intermediate core router because of transit router bypass.

Wouldn’t it result in slow down of core routers build up?

In fact, core routers do not need to grow as fast as lower layers need to, thus bringing significant cost savings for Operators. This strategy would result in “Grow-Routers-Slowly, Grow-Lower-Layers-Fast” (let’s call it GRS-GLF strategy). For Obvious reasons this will bring the TCO of the network lower.

Going to the earlier debate I can now say that GRS-GLF strategy neither makes the Transmission dumb as lower layers are helping the routers in aggregation (in addition to protection also but that is a subject for another day), neither makes a complete offload of core routers (as we are talking about the Internet data only) as the proponents of ” Router Offload” would sometimes want you to believe.

Having explained why the GRS-GLF strategy makes sense, let me now share some guidelines on how this can be implemented in networks.

(PS: While the discussion would refer to “OTN”. The data applies equally to SDH. Note that OTN here means OTN switch, not the OTN mapper in DWDM)

TIP #1 Know the nature of your Traffic very well

Meaning you should know what kind of traffic is carried over routers. What is the mix of layer 3 VPN versus Internet data? In other words, what is the ratio of traffic for enterprise/business connectivity versus internet data? Perhaps the biggest mistake the planners do is designing the network for one pattern and one type of traffic.

Let’s say the Internet data is 60% (which is a decent estimate) compared to other data. This means you can start zooming on this 60% of data to be offloaded to lower layers.

TIP #2  Know the Origin and Destination of your Internet Data

An easy way to look is the IP destination of the traffic. The IP destination will tell you whether the traffic is destined for Internet Gateways or not. The best point would be the aggregation points in your network where you are collecting the Internet data.

Here you can ask this question. Can I send this traffic directly to the Internet Gateway bypassing the intermediate core routers?

If the answer is yes, you need to answer the question in TIP 3.

TIP #3  Select Suitable Lower Layer to take the traffic to Destination (DWDM or OTN or L2 Switch?)

You need to take important decision here. Which layer to offload to?

If the up-link pipe of router is congested and already filled with internet data and up-link rate is same as DWDM lambda rate, the best would be to go to DWDM layer directly as OTN is not adding any benefit of its aggregation capabilities. Let the DWDM take the traffic directly to Internet Gateway.

If the uplink pipe of the router is partially filled, there would be a need for a lower layer that has aggregation/statistical multiplexing capabilities, both low layers like Layer 2 Switch and OTN can help here. (See next point also)

Interestingly these days, layer 2 switch and OTN are packaged in the same switch thus operators can take advantage of such switches for grooming capabilities. Remember we are talking about GRS-GLF strategy, so we are building lower layers fast but not the IP layer at same pace.

TIP #4  Keep Router Up-Link rate to OTN, lower compared to DWDM lambda rate!

(I am assuming that you already own OTN switch, if not you can still offload traffic to lower layers like DWDM and L2 Switch as TIP 3 tells).

Why would use an OTN switch ( in between router and DWDM) and then keep the up-link of the router at same rate as DWDM rate. This is in fact more CAPEX. We are now growing higher layers and lower layers at the same time.

If you are using an OTN switch, it is recommended to keep the uplink of the routers low compared to DWDM lambda rate. For example if you are using 100G lambdas, recommended uplink rate for router to OTN should be 10G. This would enable you to use the grooming functionality of the OTN in the intermediate sites. One of the mistakes I see is that some service providers keep both rates same. This would deprive the operator of using the grooming functionality of the OTN layer.

TIP #5 Use Color/DWDM Interfaces on OTN

If you are using OTN switch and DWDM from same vendor, it would make sense to explore the use of color DWDM interfaces that can give traffic directly over the DWDM instead of using a lot of back to back grey interfaces between OTN and DWDM. This will result in a lot of CAPEX/ OPEX saving.

If you are using OTN switch and DWDM system from different vendors, you would need to run interoperability tests to use the color interface of the OTN of one vendor over the DWDM system of the other.

TIP #6  The IP and Transmission Departments need to Handshake

Last but not the least!

If the IP and Transmission departments are different ( as usually in Tier 1 Operators) , it would need a good coordination between the two in order understand from each other on how best to expand both IP and Transmission layer that can bring total cost savings for operators. They would need to plan together on how to use all layers intelligently bringing the Transmission cost per bit down.

If there is one department that plans both the above networks, this is ideal as there will be combined strategy for IP and Transmission expansion.

Conclusion:

As you can see that all the above tips apply to using lower layers of Transmission to support efficient and cheapest transport of IP data. These apply in particular to Internet/HSI data and I repeat by using selective offloading of IP traffic, an operator can get considerable cost benefits.

I can bet that by exploring these out of box approaches, the IP and OTN/DWDM can live happily together through a successful marriage.

While this article is mainly looking at how Transmission pipes can help the upper layers in terms of cheap transport of data by using aggregation intelligently, I can write at same later time on how the lower layers protection options can further help the MPLS layer in bringing down the cost of protection at upper layers.

Now it’s your turn!

I would love to listen to  what do you think about the arguments and Tips in this blog and further,how you are coping with the tremendous expansion in the IP data in your network.

 

40 thoughts on “The Marriage of IP with OTN/DWDM !”

  1. Faisal,

    An excellent article and valid arguements in support of augmenting lower layers to support exponential growth in Internet/HSI traffic. The IP folk may raise some eyebrows; however, the fact is that lowers (DWDM, OTN, L2) are always considered dumb and impotent to make intelligent decisions at their own. Although WSON/GMPLS and ASON/GMPLS have added intelligence to the lower layers; however, the advantages related to aggregation, grooming and statistical multiplexing of the OTN switches have been largely ignored by the operators/carriers.

    Your point of view is convincingly valid and shall be well taken by the readers regarding treatment of Internet/HSI traffic and your proposed marriage between layer-2 and layer-3 network planners can reduce double investments in these layers to the benefits both the network owners and users.

    Your blog has cleared many misconceptions about the router offloading and and your tips has provided a way forward for the network designers, network managers and decision makers put the dollars where they are needed, ultimately avoiding double investment, reducing TCO and getting the best benefit of the available technology and assets.

    Will be anxious to read the 2nd part of this blog where you will discuss how the lower layers protection mechanism can augment and add value to upper layers.

    Shaheen Qamar
    Executive Manager, Telecom

    1. Thanks Shaheen,

      You got the message very well.

      Your comments are very valid. Largely we see that the designers are planning networks in their own domain, paying less attention to how the the network should be looked at from a broader perspective. As I said one of the key take away from this blog would be that transmission and IP folks need to do handshake to develop a joint strategy.

      thanks once again for commenting.

      Faisal

  2. Syed Muhammad Ahsan

    Dear Faisal,

    I started reading this article with the intention of going through a text briefing the outlined concept of IP and OTN/DWDM as a single platform. However, this write-up ended up explaining the very details of how progressively the technology evolution and reducing guard bands in-between IP stack layers have made the choice of terminal equipment, access methods and ultimately traffic engineering, something to ponder upon more than ever needed before.

    Additionally, this has led operators to generalize the Data and Transmission departments which alternatively means a merger, not yet an acceptable term in Telecommunication world.

    This article on it’s own is one complete analysis and guideline for the vendors to present technology business to operators resulting in a lot of interest for appropriate business sales and for operators to understand their exact customer needs and propose best solutions for them.

    The reference to revenue calculations in this article considering the targeted customer needs can be extremely helpful for solution developers and network planners to utilize the investments in a confident manner resulting in boosted revenue generation.

    Excellent blog. In future, I’d surely be interested in probing more about operational side of OTN/DWDM networks merging with IP plane.

    Syed Muhammad Ahsan
    Technology Consultant, Telecom,
    Middle East.

    1. Thanks Ahsan for nice comments. I am taking your inputs and will be looking at the operational side of technologies in future also.

      Faisal

  3. Dear Faisal
    I enjoyed the article and believe that you have a lot to say regarding the handshake, strategic planning and offload decisions (possibilities and challenges) . this is also applied to Shaheen (greetings to him).

    We’ve seen a soled model of this via merger rather than handshake or coordination in major technology leaders like Cisco when they introduced “HERO” High End Routing and Optical Group! as a new organization to refine the product/s roadmap and production plans.

    I conclude here by saying that the action is for the operators top management towards complete merger in order for planners and designers to speak one language.

    ALI ALGHAMDI

    1. Thanks Ali Al Ghamdi,

      Glad to know that you enjoyed it.

      You brought up important point. Yes vendors are going through the phase of merging IP and Optics into one domain but the operators are still lagging behind. So in the absence of such merger, there need to be a close handshake 🙂

      faisal

  4. wellington guta

    interesting and informative……Does this mean that the world of transmission is going back to layer 2? What technology will best suit this type of marriage…MPLS?

    1. Thanks,

      No, this does not mean that the world is going back to L2. 🙂

      This means that use the right traffic at the right layer. IP has its own space as well as the lower layers.

  5. Congratulations Faisal,

    With many technologies development going on (as the growth of carrier ethernet, MPLS-TP, SDN.. etc)

    You’ve just pointed that there’s no “Best Technology”

    But the best way to treat your network traffic as it is needed.

    Great tips, great article!

  6. This is a excellent article, introduce the current problem, GRF-GLF, and the advantage of use different lower aggregating technology (DWDM/OTN/) for different scenario.
    But I think the fight between IP and TxM never stop in past, current and future, even the two departments were merged, one equipment could support L0/L1/L2/L3. As operator, firstly, need understand his traffic totally (you mentioned in TIP#1), include IP and TXM, select suitable technology for different traffic (FIX, Mobile, Enterprise, DATA Center…), make a strategy as a guidance and policy to planning and design traffic. Like VDF group has made a blueprint to regulate their subnet to plan traffic and establish network in IP and TXM, each subnet group need follow this policy.
    I think the synergy IP and Optical is a big topic, I am anxious read next part article.

    Thanks you for your share.

    Eric.

    1. Excellent Analysis Eric.

      As you mentioned. The issue is that we are building all parts of the layers without a concern on which layer should be grown fast compared to the other.

      thanks,

      Faisal

  7. Dear Mr.Faishal Khan,
    Your articles are excellent and written with clarity. This shows your indepth practical knowledge. I have been working on a project of establishing a OTN integrated DWDM network with the IP overlay network of MPLS. I have understood the value of segregating the Internet traffic and Enterprise traffic for traffic engineering at L3 and L2 level. I am looking forward to your articles.

    With best regards,

    Janakiraman

    1. RajeshKumar Babu

      Hello Mr.Jankiraman

      I would be interested to have the working model of your “OTN integrated DWDM network with the IP overlay network of MPLS”. Can you share some of the technical specs and logical diagram.

      Regards
      Rajesh

  8. Fine essay as always from Faisal. I stand for your side for one point which we share same comments. That is the efficiency of bearer network. Considering the expensive invement for Router 40G/100G interfaces, it is really the last choice to peel off the traffic at most of Router node and then dress on again without any dropping or adding. It is not wise to pay for the curiosity. Statiscally, the majority of traffic should be pre-set to bypass through router for a point- point journey ultimately.

    1. Thanks Braun,

      Repeating again, right traffic at right layer ! Thats the message of the article

      Faisal

  9. Hi,
    Nice to read your blog.
    I always had wish foresee a time when we will have OTU\OMU\ODU type cards get embedded into routers. Its not far though one step is already taken by manufacturers …e.g. Cisco CSR and Juniper TX Series and other vendors too! calling it IPoDWDM which eliminated the need of separate DWDM transponder unit and only requires Mux\Demux + Amplification. I hope this gets integrated soon. And we will have a king device of bearer networks.

    Regards,
    Tech Manager ZTE – Packet Transport Networks

    1. Thanks Qasim for your comments. IPoDWDM is carrying IP directly over DWDM. This is another extreme by the router vendors compared to transport vendors. Every technology has Pros and Cons. It is for the operator to decide how to use a technology judiciously and look at the traffic demands to decide on what is best for the traffic and network.

  10. Raymond Macharia

    An excellent article. You have articulated very well thoughts I have had on how capacity should be addressed on networks of the future.
    Thank you Faisal

    Raymond Macharia

  11. Hi Faisal

    A well presented article with thought provoking tips & analysis for future IP & Transport network planners.

    I do have one query though, when you say that HSI traffic will be the prevalant traffic in future (2018) networks, and then you talk of its nature being P2P and deterministic, with the analysis of offloading the intermediate core routers (since their job is just to pass on the IP traffic without any L3 processing required, which mimics the lower layers boxes functionality to some extent), IP/MPLS planners will see this as lower layer invasion, and thus their remit will be limited purely to the core edge routers. This surely will be alarming for them. The only planning / expansion they would come across would limited to the edge routers.

    However, having said that, broadly speaking, a successful, and I repeat “successful” marriage between the IP & SDH/OTN/DWDM will definately bring massive cost savings and lower TCO for the operator as a whole. The decision makers will be bang on target in terms of putting their money in the right baskets and reaping the better ROI.

    A few notes….

    TIP#3

    There are many Tier-1 operators in WEU, including UK, where no OTN switch has been deployed, as they do not see the requirement of statistical multiplexing / grooming, since they did not experience the transition from SDH->NG-SDH->Hybrid (that mobily did), and thus phased out completely the circuit swtiched and brought upon the packet switched technologies for metro networks. However, in your case, if an uplink pipe of the router is not at the same rate as DWDM lambda, then OTN would be a best fit…what in your view will be the solution to operators in the west that have don’t have OTN available prior to DWDM.

    Ali

    1. Thanks Ali, for thought provoking comments.

      OTN switch is not absolutely mandatory, though it is best for sub lambda grooming. So if you don’t have the OTN switch, what you will do is to offload the traffic to either a layer 2 or layer zero ( lambda). Lets say you go with the lambda path, then you will determine the source and destination of such traffic , configure a point to point lambda and bypass all the transit routers for that lambda. The TCO of the transport of bytes will decrease in that way.

      Ali, all such initiatives need careful planning. I see that is lacking, today, in Tier 1 operators. I am expecting SDN will solve a lot of such issues, specifically regarding multi layer optimization from layer zero to layer three. I have discussed this in one of my another blog at the following. Recommend to visit that one also.

      https://telcocloudbridge.com/blog/things-transport-sdn-vendors-may-not-tell-you-but-you-must-ask-them/

      cheers,

      Faisal

  12. Dear Faisal,
    Cloud Computing, Caching server would change the nature of Costumer Internet Traffic, and we would need Router Intervention many intermediate locations to improve customer experience.
    What is your take on this…please reply

    1. Dear Dilip,

      You asked a good question.

      Cloud computing and Caching server would relieve the backbone bandwidth for sure. We are not saying that the intermediate router intervention is not needed at all. What we are saying is that intermediate router intervention is not needed for all traffic. Lets take caching server for example, even though the traffic is processed locally there would be some traffic that has to go to Gateway. Now does this traffic makes sense to stop at the intermediate site ? Can we offload it to DWDM or perhaps OTN/Ethernet layer. If we can, we should offload it. The intial CAPEX of the bypass of the core routers may be high compared to stopping at every core router but the long term TCO analysis show that bypass has better CAPEX/OPEX savings.

  13. Great Stuff and explanation is very clear on Marriage of IP with lower layers like DWDM/OTN/SDH. I completely agree that IP designers/Planners think on thier perspective and Transmission planners/designers think on thier own perspectives. Now it is time for merger of two departments and work towards the success of organization. When I see service providers, there are seperate departments for Transsmission Engineering and IP Engineering. I would say that, single department design the network . Stratagic planning apporach is the best by forseeing future requirements.

    1. Thanks Abdul Rauf for your comments, I agree there should be one department for both IP and transmission.

      Faisal

  14. Hi Mr. Faisal,

    Thanks very much for publishing such article which is blending two technologies IP & TXM plus enlightening hidden areas which always driven

    by two different departments/authorities like IP planning (Enterprise/DC +MPLS) & Transmission.

    It is found very interesting to see & start orchestrating bandwidth planning across these two Depts into one Box (Lamda & Gigabits)

    Is it possible if you can share lamdas XY/Gigabits XY compatibility units/limits in tabular format ? as best deploying practice for IP Design Engineer.

    Regards
    Haroon Yousaf

    1. Thanks Mr.Haroon, I was not able to understand what you need. I think we need to have a one to one session to explain you further !

  15. RajeshKumar Babu

    Hello Mr.Faisal

    Thanks for this article. I am in the process of designing a 40G network with 30 sites around the county. We are basically targeting to have the wholesale bandwidth model. Pls let me know if you have any RFPs for such project and what would be your proposal for this business model.

    Regards
    Rajesh

      1. RajeshKumar Babu

        Thanks Mr.Faisal for the update. What Model you would suggest for the wholesale bandwidth market. IPoDWDM or OTN?

        Rajesh

  16. Hi Faisal..you find that “Internet” traffic is very point to point in fashion therefore you can implement a lower layer technology to support..however it seems quite opposite to me with the substantial number of content end points and number of peering arrangements any one carrier may have. Isn’t IP a much better solution than treating this as point to point? Plus the end point of content on a network today may not be where it goes tomorrow.

    1. @joe, the example you gave is for the internet backbone running cross countries and geography which is true to have many internet peering points. The example in the blog is for a service provider running in one country that would have one or a couple of internet gateways and the local traffic generated is destined only to these two peering points.There, the transport on lower layers make sense as the destination is known. Beyond these local handover points, if the requirements become more meshy as in the example given by you then IP routing would make make more sense.

Leave a Comment

Your email address will not be published. Required fields are marked *