Interrupt Top Halves and bottom halves .

For Interrupt handler , a substantial amount of work must be done in response to a device interrupt, but interrupt handlers need to finish up quickly and not keep interrupts blocked for long. These two needs (work and speed) conflict with each other, leaving the driver writer in a bit of a bind.

Linux (along with many other systems) resolves this problem by splitting the interrupt handler into two halves. The so-called top half is the routine that actually responds to the interrupt—the one you register with request_irq. The bottom half is a routine that is scheduled by the top half to be executed later, at a safer time. The big difference between the top-half handler and the bottom half is that all interrupts are enabled during execution of the bottom half—that's why it runs at a safer time. In the typical scenario, the top half saves device data to a device-specific buffer, schedules its bottom half, and exits: this operation is very fast. The bottom half then performs whatever other work is required, such as awakening processes, starting up another I/O operation, and so on. This setup permits the top half to service a new interrupt while the bottom half is still working.

Almost every serious interrupt handler is split this way. For instance, when a network interface reports the arrival of a new packet, the handler just retrieves the data and pushes it up to the protocol layer; actual processing of the packet is performed in a bottom half.
-----------------------
Link: http://www.makelinux.info/ldd3/chp-10-sect-4.shtml
--------------------------------------------
SMP affinity and its effect
to see which cpu is handeling whic IRQ
cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
0: 4865302 5084964 4917705 5017077 IO-APIC-edge timer
1: 132 108 159 113 IO-APIC-edge keyboard
2: 0 0 0 0 XT-PIC cascade
8: 0 1 0 0 IO-APIC-edge rtc
10: 0 0 0 0 IO-APIC-level usb-ohci
------------------------

TO check affininty
cat /proc/irq/24/smp_affinity
--------------------------
SMP IRQ Affinity

Link : http://www.cs.uwaterloo.ca/~brecht/servers/apic/SMP-affinity.txt
------------
WE faced an issue where there was heavy load on one VCPU timer thread was not calling timer callback and application was mis behaving Problem was smp Affinity was not evenly Distributed it was by default all f's .
WE had 6 VCPU s so we set affinity as 1f
echo 0f > /proc/irq/161/smp_affinity

GDB and Strace Attach to running process

Strace -p PID
note it should not run more as it eats lot of resources .

$ gdb --quiet
(gdb) attach pid
(gdb) info proc
process 13601
cmdline = './a.out'
cwd = '/home/amit/ccode'
exe = '/home/amit/ccode/a.out'

(gdb) info functions
---Fir this command in compilation -g flag is mandatory in gcc to insert debug information
(gdb) list main
22           printf("done thread #%ld!\n", tid);
23           pthread_exit(NULL);
24      }
25
26      int main (int argc, char *argv[])
27      {
28           pthread_t threads[NUM_THREADS];
29           int rc;
30           long t;
--------Show Assembler code disass option of gdb------
(gdb)  disass main
Dump of assembler code for function main:
0x0804858f :    push   %ebp
0x08048590 :    mov    %esp,%ebp
0x08048592 :    sub    $0x28,%esp



$gdb -p pid 

Ref : http://www.ibm.com/developerworks/aix/library/au-unix-strace.html