Linux Server Security: Hack and Defend by Chris Binnie Make Your Servers Invisible!

I'm delighted to be able to offer the first chapter of my new security book, Linux Server Security: Hack and Defend, for free as a downloadable PDF. Chapter One, entitled Invisibility Cloak, walks you through making your servers invisible online to the extent that not only can you successfully hide your services but even your servers themselves aren't discoverable on the network.


Despite removing the most common attack vectors you might be surprised to discover that you can still access your servers over the Linux® command line remotely and continue to run key production services. The chapter walks you through, step by step, the obfuscation of your servers so that their attack surfaces are significantly reduced and they are less prone to attacks. Click the link below to download the free PDF.



Linux Server Security: Hack and Defend by Chris Binnie

Learn how to attack and defend the world’s most popular web server platform


Linux Server Security: Hack and Defend presents a detailed guide for experienced admins, aspiring hackers and other IT professionals seeking a more advanced understanding of Linux security. Written by a 20-year veteran of Linux server deployment this book provides the insight of experience along with highly practical instruction.


The topics range from the theory of past, current, and future attacks, to the mitigation of a variety of online attacks, all the way to empowering you to perform numerous malicious attacks yourself (in the hope that you will learn how to defend against them). By increasing your understanding of a hacker’s tools and mindset you're less likely to be confronted by the all-too-common reality faced by many admins these days: someone else has control of your systems.

The techniques presented apply to almost all Linux distributions including the many Debian and Red Hat derivatives and some other Unix-type systems. Further your career with this intriguing, deeply insightful, must-have technical book. Diverse, broadly-applicable and hands-on practical, Linux Server Security: Hack and Defend is an essential resource which will sit proudly on any techie's bookshelf.


Download: Linux Server Security: Hack and Defend by Chris Binnie - Chapter One - Invisibility Cloak


©2016 Chris Binnie - please do not copy content without permission.

Linux® is the registered trademark of Linus Torvalds in the U.S. and other countries.

Click for DevSecOps articles and free Linux tutorials on DevSecOps.cc


Arrive On Time With NTP

©2016 Chris Binnie

Few services on the Internet can claim to be so critical in nature. Subtle issues which affect the timekeeping of your systems can sometimes take a day or two to be realised and they are almost always very unwelcome when they surface due to the knock-on effects they cause.

Consider as an example that your backup server loses connectivity to your Network Time Protocol (NTP) Server and over a period of a few days introduces several hours of clock skew. Your colleagues arrive at work at 9am as usual only to find the bandwidth-intensive backups consuming all of the network resources meaning that they can barely even log into their workstations to start their day’s work until the backup has finished. From the timestamps on your e-mails to remembering when you clocked in to start your shift at work, so you can go home on time, NTP services are essential to a happy infrastructure.

Let’s explore a brief overview of NTP to help prevent encouraging such disasters as the backup one mentioned..

You might consider that the really important NTP servers (from which other servers pick up their clock data) are at the bottom of an inverted pyramid and referred to as Stratum 1 servers (also known as “primary” servers). These servers speak directly to national time services (known as Stratum 0 which might be devices such as atomic clocks or GPS clocks for example). There are apparently a number of ways of communicating with them securely, via satellite or radio for example.

Somewhat surprisingly it’s reasonably common for even large enterprises to connect to Stratum 2 servers (or “secondary” servers) as opposed to primary servers. Stratum 2 servers as you’d expect synchronise directly with Stratum 1 servers. If you think for a moment that a corporation might have their own onsite NTP Servers (at least two, usually three, for resilience) then you could consider these to be Stratum 3 servers. As a result such a corporation’s Stratum 3 servers would then connect upstream to predefined secondary servers and dutifully pass the time onto its many client and server machines as an accurate reflection of the current time.

A simple design component of NTP is that it works on the premise, thanks to the large geographical distances travelled by Internet traffic, that round-trip times (of when a packet was sent and how many seconds later it was received) are sensibly taken into account before trusting to a time as being entirely accurate. There’s a lot more to setting a computer’s clock than you might at first think, if you don’t believe me then this fascinating webpage is well worth a look at: http://www.ntp.org/ntpfaq/NTP-s-sw-clocks-quality.htm

At the risk of revisiting the point, NTP is so key to making sure that your infrastructure functions as expected that the Stratum servers which your NTP servers connect to in order to fuel your internal timekeeping need to be absolutely trusted and additionally offer redundancy. There’s an informative list of the Stratum 1 servers available at the main NTP site here: http://support.ntp.org/bin/view/Servers/StratumOneTimeServers

As you can see from that list there’s some NTP Stratum 1 servers which run in a “ClosedAccount” state; these servers can’t be used without prior consent. However, so long as you adhere to their usage guidelines, “OpenAccess” servers are indeed open for polling. Any “RestrictedAccess” servers can sometimes be limited due to a maximum number of clients or a minimum poll interval and additionally these are occasionally only available to certain types of organisations, such as academia.

Respect My Authority

On a public NTP Server you are likely to find that the usage guidelines follow a number of rules. Let’s have a look at some of them now.

The “iburst” option involves a client sending a number of packets (eight packets apparently rather than the usual single packet) to an NTP Server should it not respond during at a standard polling interval. If after shouting loudly at the NTP Server, a few times within a short period of time, and a recognised response isn’t forthcoming then the local time isn’t changed.

Unlike “iburst” the “burst” option is not commonly allowed (so don’t use it!) as per the general rules for NTP Servers. That option instead sends numerous packets (eight again apparently) at each polling interval and also when the server is available. If you are continually throwing packets at higher-up Stratum servers even when they are responding normally you may get blacklisted for using the “burst” option.

Clearly how often you connect to a server makes a difference to its load (and the negligible amount of bandwidth used). These settings can be configured locally using the “minpoll” and “maxpoll” options. However to follow the connecting rules on to an NTP Server you shouldn’t generally alter the the defaults of 64 seconds and 1024 seconds respectively.

Another, far from tacit, rule is that clients should always respect any Kiss-Of-Death (KOD) messages generated by those servers that they request the time from. If an NTP Server doesn’t want to respond to a particular request, similar to certain routing and firewalling techniques, then it’s perfectly possible for it to simply discard or blackhole any associated packets.

In other words the recipient server of these unwanted packets takes on no extra load to speak of and simply drops the traffic which it doesn’t think it should serve a response to. As you can imagine however this isn’t always entirely helpful and on occasion it’s better to politely ask the client to cease and desist, rather than ignoring the requests. For this reason there’s a specific packet type called the KOD packet. Should a client be sent an unwelcome KOD packet then it should then remember that particular server as having responded with an access-denied style marker.

If it’s not the first KOD packet received from back the server then the client assumes that there is a rate-limiting condition (or something similar) present on the server. It’s common at this stage for the client to write to its local logs, noting the less-than-satisfactory outcome of the transaction with that particular server, if you ever need to troubleshoot such a scenario.

Something else worth bearing in mind is that for obvious reasons it’s key that your NTP’s infrastructure is dynamic and as a result it’s important not to hard-code IP Addresses into your NTP config. By using DNS names individual servers can fall off the network and the service can still be maintained, IP Address space can be re-allocated and simple load balancing (with a degree of resilience) can be introduced.

Let’s not forget that we also need to consider that the exponential growth of the Internet of Things (IoT), eventually involving billions of new devices, will mean a whole host of equipment will need to keep its wristwatches set to the correct time. Should a hardware vendor inadvertently (or purposely) configure their devices to only communicate with one provider’s NTP Servers (or even a single server) then there can be, and have been in the past, very unwelcome issues. As you might imagine, as more units of hardware are purchased and brought online, the owner of the NTP infrastructure is likely to be less than grateful for the associated fees that they are incurring without any clear gain. This scenario is far from being unique to the realms of fantasy. Ongoing headaches thanks to NTP traffic forcing a provider’s infrastructure to unwittingly creak at the seams has been seen several times in the press over the last few years.

Attacks On NTP

Let’s get our hands a little dirtier now and look at some other NTP options.

One nasty attack which caused the Internet community to sit up and take note of, in order to keep the Internet’s lights on, involved what’s called a Reflection Attack. In essence by spoofing IP Addresses it was possible to ask a third party machine to respond to an unsuspecting victim with a barrage of unwelcome data. Multiplied by thousands of machines it was possible to slow down, or even bring to a halt, some of the largest Internet Service Providers’ (ISPs) and enterprises’ infrastructure. These Distributed Denial of Service (DDoS) attacks forced some rapid patching along with some community hand-holding to keep services up and running whilst hundreds of thousands of machines were patched, reconfigured or upgraded promptly out of necessity.

This particular attack took advantage of a command referred to as “monlist” or “MON_GETLIST”. This command is found in older versions of NTP and allows a machine to query up to the last 600 hosts which have requested the time from that NTP Server. We mentioned Reflection Attacks and clearly the “monlist” functionality is an ideal route to generating a great deal of traffic off of a very small initial request.

Thankfully newer versions of NTP no longer respect these requests and additionally it’s relatively easy, if you have access to the file “/etc/ntp/conf” on your NTP Server, to add the following line to disable it:

disable monitor

Firewalling

There’s little doubt that it is a worthwhile pursuit hardening your NTP Servers. To what extent you go to however is obviously up to you. If you are using the kernel-based Netfilter’s IPtables then you would allow access to your NTP Servers by punching a hole through your firewall as so:

# iptables -A INPUT -p udp --dport 123 -j ACCEPT

# iptables -A OUTPUT -p udp --sport 123 -j ACCEPT

This configuration is not the best solution by any means as any machine can connect to your NTP Servers if you haven’t locked down the relevant NTP configuration. We will look at tightening this up with a little more detail shortly.

Configuration

Fully understanding all of the configuration options which NTP can offer, including authentication, peering and monitoring might be considered a Herculean task but let’s continue with our overview of the subject matter in hand.

It’s probably helpful to distinguish a little between two common tools. The interactive “ntpdc” program is a “special” query tool which can be used to probe an NTP Server about its current configuration and additionally make changes to that config.

With a relatively similar name the “ntpq” tool is however for standard queries, such as looking up any of your upstream machines (like printing a list of “peers” by typing “peer” for example) which you are receiving the latest time from. It tends to be mostly used to monitor your NTP infrastructure to help you keep an eye on its performance as opposed to making changes and special queries.

Back to IPtables for a moment. If you wanted to allow your NTP client, that wishes to look up NTP queries from an upstream server, you could use rules such as these:

# iptables -A OUTPUT -p udp --dport 123 -j ACCEPT

# iptables -A INPUT -p udp --sport 123 -j ACCEPT

On your NTP client you can test which servers you have connected to as so:

# ntpq -p

The results of which can be seen in the example below:  

remote              refid              st t when poll reach   delay   offset    jitter

=====================================================

10.10.10.100    .STEP.          16 u    -  1024    0    0.000    0.000     0.000

clock.ubuntu.   .STEP.          16 u   63 1024   34    0.870   70.168   3.725

time.ntp.orgl     10.1.1.1          3 u   71 1024   37    0.811  -35.751  26.362

As we can see we’re connecting to three NTP Servers for reliability. These are known as our “peers”. The “STEP” entries are “kiss codes” and show that our client has updated (and corrected) its time having connected to the pertinent servers. Under the “st” field we can see the Stratum number allocated to each server.

Restrict

It is the NTP daemon, ntpd, option called “restrict” which offers a way of configuring access controls as to who can set the time on their wristwatches using our server. Be warned that initially at least its syntax may seem a little counterintuitive. If you get lost at any point then thankfully there’s a hillock-sized number of man pages available to you. Simply prepend the “man” command to any of these manuals on the command line, such as “man ntp_mon”, to view them:

ntpd, ntp_auth, ntp_mon, ntp_acc, ntp_clock, ntp_misc

Under the manual for “ntp_acc” we can see a multitude of “restrict” options. Along the lines of traditional Access Control Lists (ACLs) the clever ntpd will allow you to grow a list of IP Addresses and ranges which you want to permit. To fully harden your NTP infrastructure however authentication should be employed which is a subject for another day.

I mentioned that “restrict” was a little counterintuitive as far as options go, here is what ntpd expects to see if we want to open up NTP Server to our localhost (with IP Address “127.0.0.1”). This allows the localhost to have full access without any restrictions and not as you might think restrict its access:

restrict 127.0.0.1

The above assists with IPv4 addressing and the IPv6 equivalent would look like this:

restrict -6 ::1

Conversely when we want to subjugate one or several hosts we add options to the end of that command as so:

restrict 10.10.10.0 mask 255.255.255.0 nomodify notrap

This example above deals with 254 hosts on a Class C subnet. Let’s look at some of the many other options available to the “restrict” option and how they manipulate NTP. It’s worth mentioning that DNS names can also be used instead of IP Addresses, however be warned that poorly maintained Domain Name Services can cause havoc with your NTP infrastructure. In the UK for example it’s recommended to configure your NTP Servers and clients to look upstream to these DNS names inside our “/etc/ntp.conf” file:

server 0.uk.pool.ntp.org
server 1.uk.pool.ntp.org
server 2.uk.pool.ntp.org
server 3.uk.pool.ntp.org

You may well ask why. There’s very good reason as we can see from this DNS lookup, on just the first entry in the list, for public NTP Servers available to the United Kingdom:

# host 0.uk.pool.ntp.org

85.119.80.232

178.18.118.13

217.114.59.3

176.126.242.239

In our abbreivated DNS lookup results even the first DNS name “0.uk.pool.ntp.org” offers us the resilience of four unique IP Addresses; some of which may also be several machines and be hosted on resilient network links too perhaps.

According to the excellent site NTP Pool Project website (http://www.pool.ntp.org), who point users at a “ big virtual cluster of Time Servers providing reliable easy to use NTP service for millions of clients”, there are 182 active servers for the UK pool in the IPv4 space and 99 available to IPv6 at the time of writing. For reference they provide useful historical statistics and graphs to presumably keep an eye on any geographical areas which require more resilience, amongst other things. Figure One shows us a graph of the available servers for the UK.

Linux Server Security

Figure One: The historical number of NTP Servers available in the pool to the UK. Copyright NTP Pool Project and Develooper, found at http://www.pool.ntp.org/zone/uk 

Back to our friendly “restrict” command. It can unilaterally “ignore” everything from hosts or subnets, This should deny absolutely everything, indeed packets of all kinds, including ntpq and ntpdc queries.

We mentioned the Kiss of Death packets earlier and adding the “kod” option to a “restrict” line means that we will send a kiss-o'-death (KoD) packet if we want help reduce unwelcome packets and introduce rate-limiting of some description.

One other point to note is that “limited” option only denies clock updates if a requests comes up against the rate limits established by the “discard” command. The “limited” option doesn’t apply to “ntpq” and “ntpdc” queries which might add more load from a user with nefarious intentions.

A common addition to the “restrict” line is “nomodify”. This denies ntpq and ntpdc queries that might attempt to modify the time on an NTP server. As you would expect any queries which return information only are however allowed.

You are unlikely to avoid seeing this option is one place or another, the “noquery” flag makes certain that you deny all ntpq and ntpdc queries. Be aware however that offering up the correct time is still possible despite this being enabled.

If you want to avoid building relationships with other NTP Servers (unless they are successfully authenticated with you for example) then the “nopeer“ option will allow you to do this. According to the manual “This includes broadcast, symmetric-active and many-cast server packets when a configured association does not exist.”

One of the more simple options is called “noserve”. It dutifully denies all packets from a machine (or range of machines) except for ntpq and ntpdc queries.

As you might expect the “notrust” switch enforces who can connect to your NTP Server. In fact it will deny traffic that aren’t cryptographically authenticated. Again quoting from the manual:

“Note carefully how this flag interacts with the auth option of the enable and disable commands. If auth is enabled, which is the default, authentication is required for all packets that might mobilize  an association. If auth is disabled, but the notrust flag is not present, an association can be mobilized whether or not authenticated. If auth is disabled, but the notrust flag is present, authentication is  required only for the specified address/mask range.”

One final option to pay attention immediately to is called “version”. This option will be certain to disallow traffic that doesn’t match the current NTP version of your server or client. This can clearly be useful for enforcing that up-to-date versions are used and therefore older security issues present less risk. A similar approach appears in OpenSSH where using the legacy “version 1” is far from recommended and therefore it is necessary to explicitly enable that version to avoid falling into a potential mantrap and opening up security holes unnecessarily.

Localised infrastructure

One infrastructure recommendation relates to installing NTP Servers and introducing “peering” yourself for improving resilience and capacity. Implementing a peer-to-peer infrastructure within your Stratum 2 servers means that the those servers within the peer group allow each other to update their clocks and in turn this helps with load balancing and any additional load on their relevant upstream servers. This might also be called distributing the time horizontally.

There are a number of other factors to consider, as with all online architectural challenges. For example it’s of little use keeping time diligently with three upstream servers every few seconds and building a complex internal NTP fabric, upon which you become to heavily rely, if you only have one external internet connection.

Perimeter Lockdown

Earlier we promised to quickly examine how to lock down your firewalling with IPtables a little further, rather than opening up UDP port 123 carte blanche. This might apply to your local NTP client, or if you added such rules to a perimeter firewall all of the clients on your LAN. We’ll look at limiting who you can speak to so that you are always assured that only very select, predefined Time Servers can connect to you.

We’ll use the “uk.pool.ntp.org” example for familiarity but in reality you might lock these rules down to three individual servers and not a pool of servers. This is because although you are assured of the reliability of multitudinous NTP Servers being available to you there’s a chance that you may not trust them all. Due to the high churn rate some might be compromised for example and cause you unwelcome headaches.

Outbound, egress, traffic rules are slightly more sophisticated to what we use in order to allow traffic into our machine or network. This is because we want to allow “NEW” time lookups to be performed and also pick up the response when an NTP Server responds to such a request with what’s called an “ESTABLISHED” connection. You can achieve that as follows:

# iptables -A OUTPUT -o eth0 -p udp -d uk.pool.ntp.org --sport ntp -m state --state NEW,ESTABLISHED -j ACCEPT

A reminder that if this was my config then I would most likely tie these rules to specific IP Addresses.

Conversely to allow an inbound time check to occur we can use this fractionally simpler line with just “ESTABLISHED” connections being allowed through:

# iptables -A INPUT -i eth0 -p udp -s uk.pool.ntp.org --sport ntp -m state --state ESTABLISHED -j ACCEPT

Alternative to NTP

Finally to help clear up any confusion there’s an alternative, of sorts at least, to the venerable Network Time Protocol which is worth a quick mention. It comes in the form of the Simple Network Time Protocol (SNTP). What is interesting is that both timekeeping solutions follow the same packet format and actually, according to RFC 4330 (https://tools.ietf.org/html/rfc4330), “the NTP and SNTP packet formats are the same, and the arithmetic operations to calculate the client time, clock offset, and roundtrip delay are the same.”

So be confused no longer you should actually think of SNTP as being more of a subset of NTP, rather than being a contender. SNTP becomes most useful when a fully-fledged NTP Server infrastructure can’t be justified and for good reason.

An earlier RFC (RFC 2030 https://tools.ietf.org/html/rfc2030) talks specifically about not expecting SNTP to be performant at passing on the time to other clients. It goes on to mention that at the furthest edge of the NTP tree (by that I mean the the leaves, or the highest stratum, as opposed to the branches which then connect to the trunk) is where SNTP implementations are best suited. It also warns that without redundant network links and diverse, resilient configurations it should not be relied upon for serving the time, just receiving it.

Clearly however on lesser-powered devices SNTP can be implemented quite happily in order to use less system resource. It should therefore be applicable to a number of scenarios, specifically where the IoT demands time lookups from otherwise low-powered equipment such as refrigerators or other domestic appliances (which are likely to remain low-powered from a computing perspective for a relatively short period of time I suspect as demands on their functionality grows).

EOF

Although we’ve only run through an overview on keeping your clocks correct with NTP and SNTP hopefully this introduction has explored some of the more important aspects of the subject.

We have looked at the configuration, monitoring and securing of NTP and hopefully extolled its value to the integrity of your infrastructure. You should consider NTP absolutely critical and now appreciate why it should be one of the first services, possibly directly after DNS if that’s at the top of your list, which you need to explore in detail when building infrastructure.

Armed with a smattering of IPtables and where to find listings of public NTP Servers you are now suitably armed to begin that very task should the need arise.

Linux Servers and Security Howtos - Chris Binnie - Linux Server Tutorial

Linux Server Security: Hack and Defend by Chris Binnie Make Your Servers Invisible!

I'm delighted to be able to offer the first chapter of my new security book, Linux Server Security: Hack and Defend, for free as a downloadable PDF. Chapter One, entitled Invisibility Cloak, walks you through making your servers invisible online to the extent that not only can you successfully hide your services but even your servers themselves aren't discoverable on the network.


Despite removing the most common attack vectors you might be surprised to discover that you can still access your servers over the Linux® command line remotely and continue to run key production services. The chapter walks you through, step by step, the obfuscation of your servers so that their attack surfaces are significantly reduced and they are less prone to attacks. Click the link below to download the free PDF.



Linux Server Security: Hack and Defend by Chris Binnie

Learn how to attack and defend the world’s most popular web server platform


Linux Server Security: Hack and Defend presents a detailed guide for experienced admins, aspiring hackers and other IT professionals seeking a more advanced understanding of Linux security. Written by a 20-year veteran of Linux server deployment this book provides the insight of experience along with highly practical instruction.


The topics range from the theory of past, current, and future attacks, to the mitigation of a variety of online attacks, all the way to empowering you to perform numerous malicious attacks yourself (in the hope that you will learn how to defend against them). By increasing your understanding of a hacker’s tools and mindset you're less likely to be confronted by the all-too-common reality faced by many admins these days: someone else has control of your systems.

The techniques presented apply to almost all Linux distributions including the many Debian and Red Hat derivatives and some other Unix-type systems. Further your career with this intriguing, deeply insightful, must-have technical book. Diverse, broadly-applicable and hands-on practical, Linux Server Security: Hack and Defend is an essential resource which will sit proudly on any techie's bookshelf.


Download: Linux Server Security: Hack and Defend by Chris Binnie - Chapter One - Invisibility Cloak


©2016 Chris Binnie - please do not copy content without permission.

Linux® is the registered trademark of Linus Torvalds in the U.S. and other countries.

Click for DevSecOps articles and free Linux tutorials on DevSecOps.cc