You do have a file size limit defined in limit.conf of 102400000 which works out to about 97.6GB, so that still isn't the cause of the problem.
HP-UX 11.11 is pretty old. Not sure if all the suggested NFS mount options are supported in that version, but you definitely need to be using mount options.
First, change your options in Linux /etc/exports options from (rw,insecure,sync,no_wdelay,insecure_locks,no_root_squash) to (rw,all_squash,sync) and reexport it.
Next, add an /etc/fstab line in the HP-UX host:
OWP-15:/home/back /transfer nfs rw,bg,hard,intr,rsize=32768,timeo=600,noac,forcedirectio 0 0
If needed change the /transfer mount point to where your mounting the NFS file system on HP-UX.
Once you've done that, on HP-UX use the command "mount /transfer" to mount the file system. Now try your backup.