Tuesday, 3 December 2013

Configuring 2 static IPs with fibre PPPoE with Eclipse Internet : MTU size issue


This blog entry describes difficulties and solutions related to PPPoE, in particular issues related to MTU size and iptables configurations.

I've recently moved from BeThere to Eclipse Internet. It was a long marriage with Be, for 5 years, but I had to go away for technical reasons.

In a matter of a few days I've got a new Zyxel NBG4604 from Eclipse. The router is IPv6 capable, which was a surprise to me, since I've researched about the modem and I haven't found anything mentioning IPv6. Eclipse still do not offer IPv6, but you already have IPv6 even without knowing it, if you have static IPs.

IPv6 apart, I stumbled with a much simpler thing: an annoying issue which happens on certain websites, leading to sluggish performance. So, what's the point of migrating to fibre if navigation is seriously impacted?

But, what the issue is? and what causes it?

The sluggish fiber connection

The issue is that some websites fail to load properly into the browser. This is an example: github.com employs avatars from gravatar.com . It happens that github seems to "work fine", whilst gravatar "fails" to load. The browser keeps trying to load something from gravatar and stays there, trying and trying and the request is never completed. This jeopardizes navigation of github, which is a primary source of concern to me.

Long story short, the issue is related to PPPoE (PPP over Ethernet), which is basically the authentication layer which is employed by many ISPs, including Eclipse. Explaining in slightly deeper detail, it is necessary to adjust a parameter called MTU size when you have PPPoE due to technicalities you can find more details here.

I've opened a ticket with Eclipse Internet asking for the recommended MTU size, which they promptly responded. But there are some more details involved, as I explain below.

If I were using my Zyxel router like any regular end user, behind the fibre modem... I suppose I would not have any trouble. But I've decided to employ a Debian box as my main router / firewall. It basically means that I need to configure it properly, understand some technicalities I wouldn't care otherwise.

Path to solution

Long story short, I've configured these things:
  •  PPPoE
  •  /etc/network/interfaces with 2 static IPs (or multiple static IPs)
  •  MTU size
  •  iptables rules related to MTU size
I've connected the fibre modem to my NIC eth0 whilst the other NIC eth1 is connected to the LAN.

The NIC which faces the Internet, via the fibre modem has to acquire 2 IPs from the ISP. The first IP is acquired when the PPPoE layer authenticates and negotiates stuff with the ISP side. The additional IP address need to be configured after the first one, not requiring any special negotiation by the PPPoE layer.

OK. See below my /etc/networks/interfaces (with some fake addresses):

auto lo
iface lo inet loopback

iface eth0 inet manual

iface eth1 inet manual

auto dsl-provider
iface dsl-provider inet ppp
    pre-up    /sbin/ifconfig eth0 up
    post-down /sbin/ifconfig eth0 down
    provider  dsl-provider

    post-up   sleep 7 ; \
              gw=$( /sbin/ifconfig ppp | \

                    head -2 | tail -1 | \
                    sed -E 's/(.*P-t-P:)([0-9.]+)( .*)/\2/' ) ; \
              echo "Define default gateway $gw" ; \
              /sbin/route add default gw $gw ppp0 ; \
              echo Bringing up ppp0:1 ; \
              /sbin/ifconfig ppp0:1 netmask; \
              echo Bringing up eth1 ; \
              /sbin/ifconfig eth1 up; \
              echo Disable IP forward for security reasons; \
              echo 0 > /proc/sys/net/ipv4/ip_forward
    pre-down  echo Bringing down eth1 ; \
              /sbin/ifconfig eth1 down ; \
              echo Bringing down ppp0:1 ; \
              /sbin/ifconfig ppp0:1   down

The important bits are:

* eth0 must be left without any IP configuration because it will be employed by PPPoE in order to talk with the fibre modem.
* eth1 can be configured with a LAN IP, but you should not bring it up until you define the default gateway, which will be some IP address on the ISP side.
* you really don't know what the default gateway is, until the moment the connection is stablished with your ISP because this IP can change and will probably change every time you disconnect and connect again.
* interface ppp0, despite not configured by you, will be configured for you when dsl-provider brings up.

More details

Make a backup copy of /etc/network/interfaces as it is presented in the section above. Observe that, when you install PPPoE, it will change your configuration. But I already told you how it should be. So, make a backup copy!

$ cp /etc/network/interfaces /etc/network/interfaces.BACKUP
$ sudo apt-get install pppoe pppoeconf -y

During the installation, pppoeconf runs and tries to find your fibre modem. Make sure you connected eth0 with your modem.

It will ask your username and password, required to authenticate against the ISP. Eclipse sent me a letter with this stuff, but it is also available in the Connection Manager page.

When PPPoE runs, it changes your /etc/networks/interfaces. Have a look at it and see what happened. If you are following this recipe the way I describe, you will see that we had already configured everything which is needed in our version of /etc/networks/interfaces. Simply restore the backup copy.

$ cp /etc/networks/interfaces.BACKUP /etc/networks/interfaces

Just a reminder: make sure you put your 2 (or more) IPs as mentioned in the Connection Manager page into your /etc/networks/interfaces, in interfaces ppp0:1, ppp0:2, .... as much as you have static IPs, remembering that your first IP must go to ppp0 itself.

In other words, I've put the line below somewhere into my /etc/networks/interfaces:

  /sbin/ifconfig ppp0:1 netmask

If you have more that 2 static IPs, you will be interested on configuring additional virtual interfaces.

Now try to connect to your ISP:

$ sudo ifup dsl-provider

You should see something like this:

$ sudo ifconfig ppp0
ppp0      Link encap:Point-to-Point Protocol 
          inet addr: P-t-P:  Mask:
          RX packets:1200866 errors:0 dropped:0 overruns:0 frame:0
          TX packets:748261 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:3
          RX bytes:1633001082 (1.5 GiB)  TX bytes:59749711 (56.9 MiB)

Now verify if your second IP is connected:

$ ifconfig ppp0:1
ppp0:1    Link encap:Point-to-Point Protocol 
          inet addr:  P-t-P:  Mask:

There are 2 important aspects to be noted at this point:

1. You may see that a netmask is, which means our IP is connected one-on-one to an IP on the ISP side. Well, this is what point-to-point means, and it makes sense! But all netmasks should honour what we have configured in /etc/networks/interfaces, which does not seem to be the case. We will address this issue later.

2. The MTU size is 1478, which is a recommended value I've got from Eclipse. Chances are that you are seeing some other value. No worries, we will address this issue later.

Let's dive a bit into these aspects in the next sections.

Interface configuration

The interface ppp0 happens to be wired to "P-t-P:" in this case in particular. Actually, every time you connect you may potentially connect to a different IP on the ISP side. It means that you cannot assume that a certain IP in particular is your default gateway permanently. In our /etc/network/interfaces we find dynamically what is the default gateway we need to configure:

      gw=$( /sbin/ifconfig ppp | \
            head -2 | tail -1 | \
            sed -E 's/(.*P-t-P:)([0-9.]+)( .*)/\2/' ) ; \
      /sbin/route add default gw $gw ppp0 ; \


The netmask of ppp0 should be actually This is the value I said it should be in /etc/networks/interfaces, but it is stubborn and insists on It's possibly an issue I still need to fix on PPPoE configuration.
PENDING: I said before we would be addressing this issue. Well, not yet :( ... I still need to figure out how this can be done.

My ppp0:1 is a virtual interface which is configured with my second static IP address. Observe that it is connected with itself "P-t-P:", which does not look to be correct. It should be connected to some IP in the vicinity of "P-t-P:", which is the IP ppp0 is currently connected to.
PENDING: I still need to fix this!

The netmask of ppp0:1 is already, which honours the configuration I've put on /etc/networks/interfaces. This is good.

Note: despite of pending items, lots of things are working just fine here.

Static routes

Now have a look at the static routing table:

$ sudo route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface         UG    0      0        0 ppp0 U     0      0        0 ppp0 UH    0      0        0 ppp0   U     0      0        0 eth1

PENDING: I still need to fix the netmask marked in red.
PENDING: I still need to accomodate ppp0:1 in the routing table!

The important bits are:

1. Flags UG means that this route is the default gateway. This route must be associated to interface ppp0 and must be associated with the IP address given by the ISP at PPPoE negotiation time. This was already explained in the section above.

2. Flags UH means that a given route is a host route, i.e: a route to talk to a single host in particular. In this case, interface ppp0 is responsible for talking to the IP address given by the ISP at PPPoE negotiation time.These flags are set by ppp, since it creates a point-to-point connection to a specific host on the ISP side.

3. I still need to configure ppp0:1 and make it appear in the routing table. The way it is at the moment "works" and I can even ping this address from outside, but it actually routes via ppp0, which is an additional hop, which adds some latency.
PENDING: I still need to accomodate ppp0:1 in the routing table!

MTU size configuration

Long story short, it's necessary to configure the MTU size in order to accomodate some information with is hanging on each packet of data when you are using PPPoE. The actual value of MTU size may vary under different circumstances and may even depend on what you have on your side of the connection. But let's keep it simple at this point and simply stick to the value Eclipse informed me to employ, which is 1478.

Keeping it simple, all you have to do is edit your /etc/ppp/peers/dsl-provider and make sure you have a block like this:

    connect /bin/true
    mtu 1478

Then reconnect, making sure you release everything before connecting again:

$ sudo ifdown dsl-provider; sudo poff ; \
   sudo ifconfig eth1 down; sudo ifconfig eth0 down; \
   sudo ifup dsl-provider

Try to navigate to websites like http://github.com and see if the browser successfully retrieves everything, completing the request in a few seconds. If the icon keeps rolling and rolling in the browser's location bar... this is not a very good sign.

Note: Actually, chances are that this test will not work very well if you have a configuration similar to mine, I mean: you are using your Linux box as a router and/or firewall. This leads us to the next section.

iptables configuration

I have a firewall based on iptables running on my Debian box. I'm definitely not a network engineer and I'm not willing to become one, but I managed to configure my firewall relatively easily using a software called fwbuilder. It took me some time to get used to how things work... but, as I said, it can be done relatively easily if you know some basics of TCP/IP. No need to hire a network engineer ;-)

Long story short, if you put this below in the epilog script of your firewall configuration, you will be telling iptables to clamp MSS to MTU.

echo "Running epilog script"
# This is needed for NAT on ppp0
$IPTABLES --table nat --append POSTROUTING --out-interface ppp0 -j MASQUERADE

# This is needed for hosts in DMZ-10 to accept requests
$IPTABLES --append FORWARD --in-interface virbr1 -j ACCEPT

# http://adsl.cutw.net/mtu.html
# http://www.tldp.org/HOWTO/IP-Masquerade-HOWTO/mtu-issues.html
# http://www.cisco.com/en/US/tech/tk175/tk15/technologies_tech_note09186a0080093bc7.shtml

$IPTABLES -I FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
$IPTABLES -A OUTPUT  -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
$IPTABLES -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

# Now it's safe to enable IP forward
echo 1 > /proc/sys/net/ipv4/ip_forward
echo "Epilog script done"

At this point, you must be curious about what virbr1 is in the block above, which is something which appeared from nothing in this discussion. It's explained in the next section.

You may also be curious about what the hell clamping MSS to MTU means. Look for more information in section References below.

Bootstrapping virtual servers

My Debian box is at the same time the router, the firewall and it hosts several virtual machines, mimicking a typical DMZ scenario, with virtual machines facing the Internet and virtual machines facing the LAN.

When my server comes up, it first brings up the interface dsl-provider, because it's marked as auto in /etc/networks/interfaces:

      auto dsl-provider
      iface dsl-provider inet ppp


... then it starts the virtual machines, then it starts the firewall.

For virtual machines, I use virt-manager , which provides among other things, the service /etc/init.d/libvirt-guests, which is responsible for bringing up all virtual machines. This process also creates subnets for those virtual machines facing the Internet and for those machines facing the LAN.

When I start the firewall, all static routing is already defined, whatever if interfaces involved are physical, related to ppp or related to virt-manager. All that the firewall needs to do is enforce security on these routings and make sure that certain requests arriving from the Internet are properly routed to certain virtual servers sitting on the subnet which should face the Internet, which in my case is virbr1. It's also necessary to remember to enable ip_forward in the kernel.

But we still need to make sure that the firewall starts after services provided by virt-manager. The way to do this is to put something like this below in your firewall configuration, in the editor tab of your firewall definition.

#Provides:       firewall
#Required-Start: $network $remote_fs $syslog libvirt-bin libvirtd libvirt-guests
#Required-Stop:  $network $remote_fs $syslog
#Default-Start:  2 3 4 5
#Default-Stop:   0 1 6
#Description:    firewall rules

When fwbuilder deploys the firewall rules onto your server, it will create /etc/init.d/firewall with a header like shown below, which is is what you is needed in order to provide dependency information between services. Some spaces are stubbornly added by fwbuilder, but it works without need to edit anything by hand, which is great.

# Provides:       firewall
# Required-Start: $network $remote_fs $syslog libvirt-bin libvirtd libvirt-guests
# Required-Stop:  $network $remote_fs $syslog
# Default-Start:  2 3 4 5
# Default-Stop:   0 1 6
# Description:    firewall rules


If you followed this article, you will be probably able to connect to Eclipse Internet without any router between your Debian box and your modem. You will also be able to tackle problems related to slowness which happens due to misconfigured parameters related to ppp and iptables.

I hope this article is useful and please let me know if you find it incomplete, misleading or wrong. If you have suggestions or something to complement, please let me know :)




If you found this article useful, please consider sharing it or linking to it.


1 comment:

  1. *update:* I ended up giving up of Eclipse Internet. I was disappointed with the effectiveness of their support. They respond really fast, true, but responding fast is worthless when the issue is not solved. I was also dissapointed that they sell up to 6 IPs in a subnet but do not provide a router able to handle the use case, or not configured to handle to use case, to be more precise. I was much more satisfied with BeThere, but unfortunately they were acquired by Sky, which I prefer to stay very far of.