Sometimes programs are written badly, maybe by you, maybe by other people. Sometimes you end up running those programs. Sometimes these programs leave hanging processes, either waiting for user input that will never come or expecting something somewhere to happen that it’s already missed in the grand pipe line of life.

For example, a program is waiting for a socket close, the socket timed out and the program is still waiting for a hard close (for one reason or another). You’ve debugged this far but the program is closed source, black box and from a third party vendor. You could check each day, or even each hour, but whenever it hangs it locks up another of your processes which is waiting for it to finish and your whole script gets paused.

This, strangely enough, happened to me. A process gets hung up for one reason or another and won’t finish, my script copes with the process being killed but until then it’s also hung. I’ve heard of this happening with Apache and badly written scripts too. Here’s my solution.

while [ 1 ]
do
    PROC=`ps -eo pid,etime,comm | awk '$2~/02\:..\:../ && $3~/proc_name/ { print $1 }'`
    if [ "$PROC" = "" ]; then
        #Do nothing!
        sleep 5m
    else
        ps -eo pid,etime,comm | grep proc_name
        echo "Killing $PROC"
        kill $PROC    
        sleep 1h;
    fi
done

Basically this looks for a process that’s two hours (beyond normal process run time) old with the process name you specify and kills it. You could adjust this to be one hour, three hours, ten hours or whatever works best for you. Before killing the process it will print all the processes of that name and the process to be killed so you can review the logs to see what is being seen and killed.

I know it’s very dirty and a bit of a hack, but it saves some administrative time for more important things.

Good luck!