A Primer on NFS

NFS was originally designed for trusted environments of a scale that today seems rather modest. The primary design goal was data integrity, not performance, which makes sense as Ethernet of the time ran at 10 Mbps and files and file systems were relatively small. The integrity goal led to a use of synchronous write operations, greatly slowing things down. But simultaneously, there was no lock function in the NFS server itself and multiple clients could write to the same file at the same time, leading to unpredictable corruption.

As for the trust built into the system, there was no real security mechanism beyond access control based on IP addresses plus the assumption that user and group IDs were consistent across all systems on the network.

The NFS server and the added NFS lock server were not required to run on any specific port, so the RPC port mapper was invented. The port mapper did a great job of solving a problem that didn't need to exist. It would start as one of the first network services, then all these other services expected to be listening on unpredictable ports would start and register themselves with the port mapper. "If anyone asks, I'm listening on TCP port 2049." A client would first connect to the port mapper on its predictable TCP port 111 and then ask it how to connect to the service they really wanted.

Like I said, a solution to a problem that shouldn't have been there in the first place. The NFS server was about the only network service that needed the port mapper, and it generally listened on the same port every time.

Evolution of NFS Versions NFS version 2 only allowed access to the first 2 GB of a file. The client made the decision about whether the user could access the file or not. This version supported UDP only, and it limited write operations to 8 kbytes. Ugh. It was a nice start, though.

NFS version 3 supported much larger files. The client could ask the server to make the access decision — although it didn't have to and so we still had to fully trust all client hosts. NFSv3 over UDP could use up to 56 kbyte transactions. There was now support for NFS over TCP, although client implementations varied quite a bit as to details with associated compatibility issues.

NFS version 4 added statefulness, so an NFSv4 client could directly inform the server about its locking, reading, writing the file, and so on. All the NFS application protocols — mount, stat, ACL checking, and the NFS file service itself, run over a single network protocol always running on a predictable TCP port 2049. There is no longer any need for an RPC port mapper to start first, or for separate mount and file locking daemons to run independently. Access control is now done by user name, not possibly inconsistent UID and GID as before, and security could be further enhanced with the use of Kerberos.

NFS version 4.1 adds Parallel NFS (or pNFS), which can stripe data across multiple NFS servers. You can do block-level access and use NFS v4.1 much like Fibre Channel and iSCSI, and object access is meant to be analogous to AWS S3. NFS v4.1 adds some performance enhancements Perhaps more importantly to many current users, NFS v4.1 on Linux better inter-operates with non-Linux NFS servers and clients. Specifically, mounting NFS from Windows servers using Kerberos V5. For all the details see RFC 5661.

Currently, there are two major versions of NFS included in Red Hat Enterprise Linux and that is the distro I will focus on:

• NFS version 3 (NFSv3) supports safe asynchronous writes and is more robust at error handling than the previous NFSv2; it also supports 64-bit file sizes and offsets, allowing clients to access more than 2 GB of file data.
• NFS version 4 (NFSv4) works through firewalls and on the Internet, no longer requires an rpcbind service, supports ACLs, and utilizes stateful operations

NFS version 2 (NFSv2) is no longer supported by Red Hat.

NFS clients attempt to mount using NFSv4.1 by default, and fall back to NFSv4.0 when the server does not support NFSv4.1. The mount later fall back to NFSv3 when server does not support NFSv4.0.

All versions of NFS can use Transmission Control Protocol (TCP) running over an IP network, with NFSv4 requiring it. NFSv3 can use the User Datagram Protocol (UDP) running over an IP network to provide a stateless network connection between the client and server.

Real world application here, I advise my clients who require security over speed to choose NFSv4+ and clients who can sacrifice security in the pursuit of speed and simplicity to go with NFSv3. As every year goes by with NFSv3 ageing and our world becoming more and more security conscious I am seeing less and less implementations of NFSv3. It has less security and authentication overhead built in to it. Good and bad, it merely relies on a simple host access list to authenticate.

Required Services

Red Hat Enterprise Linux uses a combination of kernel-level support and daemon processes to provide NFS file sharing. All NFS versions rely on Remote Procedure Calls (RPC) between clients and servers. RPC services under Red Hat Enterprise Linux 7 are controlled by the rpcbind service. To share or mount NFS file systems, the following services work together depending on which version of NFS is implemented:

Note : The portmap service was used to map RPC program numbers to IP address port number combinations in earlier versions of Red Hat Enterprise Linux. This service is now replaced by rpcbind in Red Hat Enterprise Linux 7 to enable IPv6 support.

nfs

systemctl start nfs starts the NFS server and the appropriate RPC processes to service requests for shared NFS file systems.

nfslock

systemctl start nfs-lock activates a mandatory service that starts the appropriate RPC processes allowing NFS clients to lock files on the server.

Rpcbind

rpcbind accepts port reservations from local RPC services. These ports are then made available (or advertised) so the corresponding remote RPC services can access them. rpcbind responds to requests for RPC services and sets up connections to the requested RPC service. This is not used with NFSv4.

rpc.mountd

When receiving a MNT request from an NFS client, rpc.mountd checks both the pathname and the sender’s IP address against its export table.If the sender  is  permitted  to  access  the requested export, rpc.mountd returns an NFS file handle for the export’s root directory to the client.  The client can then use the root file handle and NFS LOOKUP requests to navigate the directory structure of the export.

The rpc.mountd daemon registers every successful MNT request by adding an entry to the /var/lib/nfs/rmtab file.  When receivng a UMNT request from an  NFS  client,  rpc.mountd  simply removes the matching entry from /var/lib/nfs/rmtab, as long as the access control list for that export allows that sender to access the export.

Clients can discover the list of file systems an NFS server is currently exporting, or the list of other clients that have mounted its exports, by using  the  showmount(8)  command.  showmount(8) uses other procedures in the NFS MOUNT protocol to report information about the server’s exported file systems.

Note, however, that there is little to guarantee that the contents of /var/lib/nfs/rmtab are accurate.  A client may continue accessing an  export

even after invoking UMNT.  If the client reboots without sending a UMNT request, stale entries remain for that client in /var/lib/nfs/rmtab.

 rpc.nfsd

The rpc.nfsd program implements the user level part of the NFS service. The main functionality is handled by the nfsd kernel module. The user space program merely specifies what sort of sockets the kernel service should listen on, what NFS versions it should support, and how many kernel threads it should use..

 It works with the Linux kernel to meet the dynamic demands of NFS clients, such as providing server threads each time an NFS client connects. This process corresponds to the nfs service.

lockd

lockd is a kernel thread which runs on both clients and servers. It implements the Network Lock Manager (NLM) protocol, which allows NFSv3 clients to lock files on the server. It is started automatically whenever the NFS server is run and whenever an NFS file system is mounted. NFSv4 does not need to interact with lockd.

rpc.statd

This process implements the Network Status Monitor (NSM) RPC protocol, which notifies NFS clients when an NFS server is restarted without being gracefully brought down. rpc.statd is started automatically by the nfslock service, and does not require user configuration. This is not used with NFSv4.

rpc.rquotad

This process provides user quota information for remote users. rpc.rquotad is started automatically by the nfs service and does not require user configuration.

Note : An NFS server maintains a table of local physical file systems that are accessible to NFS clients.  Each file system in this table is  referred to as an exported file system, or export, for short.The exportfs command maintains the  current  table  of  exports  for  the  NFS  server.The master  export  table  is  kept  in  a  file  named /var/lib/nfs/etab.  This file is read by rpc.mountd when a client sends an NFS MOUNT request.

Normally  the  master  export table is initialized with the contents of /etc/exports and files under /etc/exports.d by invoking exportfs -a.  However, a system administrator can choose to add or delete exports without modifying  /etc/exports  or  files  under  /etc/exports.d  by  using  the exportfs command.

exportfs and its partner program rpc.mountd work in one of two modes: a legacy mode which applies to 2.4 (RHEL3) and earlier versions of the Linux kernel,

and a new mode which applies to 2.6(starting with RHEL4) and later versions, providing the nfsd virtual filesystem has been mounted at /proc/fs/nfsd  or  /proc/fs/nfs.

On 2.6 kernels, if this filesystem is not mounted, the legacy mode is used.

In  the  new  mode,  exportfs  does not give any information to the kernel, but provides it only to rpc.mountd through the /var/lib/nfs/etab file.

rpc.mountd then manages kernel requests for information about exports, as needed.

In the legacy mode, exports which identify a specific host, rather than a subnet or netgroup, are entered directly into the kernel’s export table, as  well as being written to /var/lib/nfs/etab.  Further, exports listed in /var/lib/nfs/rmtab which match a non host-specific export request will

cause an appropriate export entry for the host given in rmtab to be added to the kernel’s export table.

NFS Server configuration:

Install NFS packages on the system using the following command:

yum install nfs-utils rpcbind

Enable the services at boot time:

systemctl enable nfs-server

systemctl enable rpcbind 

systemctl enable nfs-lock 

In RHEL7.1 (nfs-utils-1.3.0-8.el7) this does not work (No such file or directory). it does not need to be enabled since rpc-statd.service is static.

systemctl enable nfs-idmap

In RHEL7.1 (nfs-utils-1.3.0-8.el7) this does not work (No such file or directory). it does not need to be enabled since nfs-idmapd.service is static.

Start the NFS services:

systemctl start rpcbind

systemctl start nfs-server

systemctl start nfs-lock

systemctl start nfs-idmap

Check the status of NFS service:

systemctl status nfs

Create a shared directory:

mkdir /test

Export the directory:

vi /etc/exports

/test (rw)

or

/test *(rw,sync,no\_root\_squash)**

Exporting the share:

exportfs -r

Restart the NFS service:

systemctl restart nfs-server

NFS Client Side configuration:

Only three steps required on Client side

yum install nfs-utils -y

showmount -e <server ip>

mount -t nfs :/test /mount point

Troubleshooting

So now that we understand NFSv3 vs NFSv4 and how to set it up lets offer some real world tips on troubleshooting when you inevitably run into an issue. Lets cover some of the most commonly occurring NFS issues in Linux and how to resolve them.

1. Slow Performance:

NFS relies on the existing network infrastructure, any glitches on the network may affect the performance of the connection. One of the tools that can be used is nfsstat

yum install nfs-utils

The nfsstat command displays statistical information about the NFS and Remote Procedure Call (RPC) interfaces to the kernel.

On Server Side,

nfsstat -s

Server rpc stats:
calls badcalls badclnt badauth xdrcall
107310012 0 0 0 0

The most important field to check is the badcalls, which represents the total number of calls rejected by the RPC layer. When the badcalls is greater than 0, than the underlying network needs to be checked, as there might be latency.

On NFS Client Side,

nfsstat -c

Client rpc stats:
calls retrans authrefrsh
23158 0 23172

Client nfs v3:
null getattr setattr lookup access readlink
0 0% 7237 31% 7 0% 1443 6% 7874 34% 11 0%
read write create mkdir symlink mknod
578 2% 4548 19% 585 2% 1 0% 0 0% 0 0%
remove rmdir rename link readdir readdirplus
0 0% 0 0% 0 0% 0 0% 0 0% 51 0%
fsstat fsinfo pathconf commit
25 0% 10 0% 5 0% 781 3%

The client is doing well as it has relatively few retransmission requests. If you are encountering excessive retransmissions, you may want to adjust data transfer buffer sizes, which are specified by the mount command options rsize and size.

Check for dropped packets

Check dropped packet by running the following command on both the server and the client:

nfsstat -o net

Client packet stats:
packets udp tcp tcpconn
0 0 0 0

2. Error: “Server Not Responding”

The Network File System (NFS) client and server communicate using Remote Procedure Call (RPC) messages over the network. Both the host->client and client->host communication paths must be functional. Use common tools such as ping, traceroute or tracepath to verify that the client and server machines can reach each other. If not, examine the network interface card (NIC) settings using either ifconfig or ethtool to verify the IP settings.

The NFS file system also reports “server not responding” when a heavy server or network loads cause the RPC message responses to time out. Use the “timeo=N” mount option on the client to increase the timeout. Check “man mount” for more information.

3. Error: “No route to host”

The “no route to host” error can be reported when the client attempts to mount an NFS file system, even if the client can successfully ping the server:

mount NFS-Server:/data /data_remote
mount: mount to NFS server ‘NFS-Server’ failed: System Error: No route to host.

This can be caused by the RPC messages being filtered by either the host firewall, the client firewall, or a network switch. Verify if a firewall is active and if NFS traffic is allowed. Normally nfs is using port 2049. As a quick test one can switch the firewall off by:

service iptables stop

on both the client and the server. Try mounting the NFS directory again. Do not forget to switch it back on and configure it correctly to allow NFS traffic/

4. Error: “mount clntudp_create: RPC: Port mapper failure – RPC: Unable to receive”

The Linux NFS implementation requires that both the NFS service and the portmapper (RPC) service be running on both the client and the server. Check it like this:

rpcinfo -p
program vers proto port
100000 2 tcp 111 portmapper \[portmap service is started.\]
100000 2 udp 111 portmapper
100011 1 udp 881 rquotad
100011 2 udp 881 rquotad

service portmap status
portmap (pid 7428) is running… \[portmap service is started.\]

If not, start it with the commands give below.

chkconfig portmap on

service portmap start

5. Error: “NFS Stale File Handle”

A program uses the open(2) system call to access an NFS file in the same way the application opens a local file. This system call returns a file descriptor, or “handle”, that the program subsequently uses in I/O commands to identify the file to be manipulated.

Unlike traditional Linux file systems that allow an application to access an open file even if the file has been deleted using unlink or rm, NFS does not support this feature. An NFS file is deleted immediately. Any program which attempts to do further I/O on the deleted file will receive the “NFS Stale File Handle” error. For example, if your current working directory is an NFS directory and is deleted, you will see this error at the next shell prompt.

To refresh the client’s state with that of the server you may forcely unmount the mount point:

umount -f /mnt/mount_point

or kill the process, which references the mounted file system:

fuser -k \[mounted-filesystem\]

6. Error: “Access Denied” or “Permission Denied”

Check the export permissions for the NFS file system. You can do this from the client:

showmount -e server_name

or from server:

exportfs -a

If you see unexpected export permissions, check the /etc/exports file on the server. Make sure there is no syntax error such as space between the permitted host and the permissions. There is a significant difference in the line:

/home (ro) and the line: /home (ro)

because the second exports /home read-write to all systems: not what was intended. Note that the line still has correct syntax, so NFS will not complain about it.

7. Error: “rpc mount export: RPC: Timed out”

Error message: Unable to access file system at [NFS SERVER]: rpc mount export: RPC: Timed out

This is caused by DNS name resolution issue. NFS(RPC) needs reverse name resolution. If NFS server or client cannot resolve their name, this error occurs. In case gets the error message, check DNS configuration and /etc/hosts configuration.