For tracing system calls of a program, there is a "strace" command in Linux. This program comes with the most of Linux distributions. If it is not present in your system, you can download it from http://sourceforge.net/projects/strace/, build and install.
Lets try run the hcidump and see which system calls it use:
sudo strace -o output hcidump -i hci0
Do some bluetooth related things(search a device) and see the output file
As we can see, hcidump uses socket related system calls(socket, bind, recvmsg and etc...) for getting bluetooth data, ioctl system call(for controlling bluetooth device), write system call (guess why :-) ), poll system call (which used for waiting for a socket IO events).
There is "-e trace=" option, which trace only specified set of system calls, for example
sudo strace -o output -e trace=socket,write,poll hcidump -i hci0
will trace only system calls socket, write and poll.
And at last there is a good syntax highlighter in the "vim" for strace's output
(In the case if vim doesn't automatically enable it -- :setf strace).