ptrace is a system call that allows process to trace systems calls made by their child processes.
When a (parent) process uses ptrace to monitor the system calls of (one of) its child processes, the parent-process gets a notification (call back function) right before the child's system call is executed. The parent can then inspect inspect the values in the CPU-registers and those on the stack and, if desired, change them. When the parent is done inspecting these values, the syscall will be executed (possibly with the changed values). When the syscall returns, the parent process is notified again to have a possibility to check the syscall's return value or other data.
ptrace is not a POSIX system call. Thus, it is not standardized among different UNIX flavours.
On Linux, gdb apparently used ptrace to debug a program.