Share via


Console keyboard hook not getting called

Question

Wednesday, February 19, 2014 9:12 PM

I am trying to install a keyboard hook to trap key down/up events in a console app.  I started with this example:

http://blogs.msdn.com/b/toub/archive/2006/05/03/589423.aspx

The code compiles, loads, and the hook appears to install fine (all functions return expected values, no failures, etc.) however my hook function is never called.

My hook code is in a DLL that is loaded by the application that also opens the console window, and within my DLL methods I can use Console.WriteLine() and have the text output correctly to the console.  I can also successfully use PostMessage() to inject messages back to the console, so everything is working except the hook.

How can I figure out what is happening here and why my hook function is not being called?

Thanks,

Matthew

All replies (10)

Thursday, February 20, 2014 9:31 PM âś…Answered | 1 vote

"Obviously, Windows does handle the message loop ..."

Getting people to admit / realize that is near impossible.

Getting people to admit/realize that is near impossible to do what? I merely stated a fact. :)

Right, I understand that.  However, I know there *is* a message pump somewhere under a console app and the SetWindowsHookEx looked like a way to gain access to it.  But since it is not working in my context I am getting confused and frustrated with it.

Yes! of course there is a message pump, windows receives and sends messages all the time, everything within windows is based on messages but when you create a console application and you want to listen to the messages you need to get an access to the message loop.

Basically that's what you need.

namespace ConsoleApplication7
{
    using System;
    using System.Runtime.InteropServices;

    using Keyboard;

    public class NativeMethods
    {
        [Serializable]
        public struct MSG
        {
            public IntPtr hwnd;

            public IntPtr lParam;

            public int message;

            public int pt_x;

            public int pt_y;

            public int time;

            public IntPtr wParam;
        }

        [DllImport("user32.dll")]
        public static extern bool GetMessage(out MSG lpMsg, IntPtr hWnd, uint wMsgFilterMin, uint wMsgFilterMax);

        [DllImport("user32.dll")]
        public static extern bool TranslateMessage([In] ref MSG lpMsg);

        [DllImport("user32.dll")]
        public static extern IntPtr DispatchMessage([In] ref MSG lpmsg);
    }

    internal class Program
    {
        private static void Main(string[] args)
        {
            Keyboard.set_hook();

            NativeMethods.MSG msg;

            while ((!NativeMethods.GetMessage(out msg, IntPtr.Zero, 0, 0)))
            {
                NativeMethods.TranslateMessage(ref msg);
                NativeMethods.DispatchMessage(ref msg);
            }
        }
    }
}

Regards, Eyal Shilony


Wednesday, February 19, 2014 10:04 PM

Hello,

Well, to understand why it doesn't work you will need to provide the code.

However, one of the most common issues with hooking is that the pointer to the callback (delegate) gets disposed so you might want to check it out.

If it's a console application and you don't use Application.Run you may also want to check that you handled the message loop correctly.

Regards, Eyal Shilony


Wednesday, February 19, 2014 11:48 PM

This is an extension for the Microsoft SmallBasic language, and the InterruptRead() function works fine, and when InKey() is called for the first time the console displays:

Hooked: 20842909

The code is compiled as a .NET 3.5 DLL and loaded by the app (SmallBasic) at startup.  The SmallBasic app is responsible for creating the console window.  I am able to post messages to the console (InterruptRead() works fine), and I can use the Console.* functions as well.  The hook seems to go in without problems, which is why I don't understand why it is not being called (the Console.Write() is never executed in the hook function and keycode is never changed.)

using System;
using System.Runtime.InteropServices;
using System.Diagnostics;
using Microsoft.SmallBasic.Library;

namespace Keyboard
{
    [SmallBasicType]
    public static class Keyboard
    {
        [DllImport("User32.Dll", EntryPoint = "PostMessageA")]
        private static extern bool PostMessage(IntPtr hWnd, uint msg, int wParam, int lParam);

        [DllImport("kernel32.dll", CharSet = CharSet.Auto, SetLastError = true)]
        private static extern IntPtr LoadLibrary(string librayName);

        [DllImport("user32.dll", CharSet = CharSet.Auto, SetLastError = true)]
        private static extern IntPtr SetWindowsHookEx(int idHook, LowLevelKeyboardProc lpfn, IntPtr hMod, uint dwThreadId);

        [DllImport("user32.dll", CharSet = CharSet.Auto, SetLastError = true)]
        private static extern IntPtr CallNextHookEx(IntPtr hhk, int nCode, IntPtr wParam, IntPtr lParam);

        [DllImport("kernel32.dll", CharSet = CharSet.Auto, SetLastError = true)]
        private static extern IntPtr GetModuleHandle(string lpModuleName);

        private delegate IntPtr LowLevelKeyboardProc(int nCode, IntPtr wParam, IntPtr lParam);

        private static int hook_installed = 0;
        private static LowLevelKeyboardProc hook_proc = hook_callback;
        private static IntPtr hook_id = IntPtr.Zero;

        private static int keycode = 0;

        private const int WH_KEYBOARD_LL = 13;
        private const int VK_RETURN = 0x0D;
        private const int WM_KEYDOWN = 0x100;
        private const int WM_KEYUP = 0x101;


        private static IntPtr hook_callback(int nCode, IntPtr wParam, IntPtr lParam)
        {
            Console.Write("e: " + nCode);

            if (nCode >= 0)
            {
                if (wParam == (IntPtr)WM_KEYDOWN)
                    keycode = Marshal.ReadInt32(lParam);
                else if (wParam == (IntPtr)WM_KEYUP)
                    keycode = -1;
            }

            return CallNextHookEx(hook_id, nCode, wParam, lParam);
        }


        private static void set_hook()
        {
            IntPtr h = LoadLibrary("SmallBasic Extension.dll");

            if (h == null)
                Console.WriteLine("No module handle!");
            else
            {
                hook_id = SetWindowsHookEx(WH_KEYBOARD_LL, hook_proc, h, 0);
                if (hook_id == null)
                    Console.WriteLine("FAILED HOOK!");
                else
                    Console.Write("Hooked: " + hook_id);
            }

            hook_installed = 1;
        }

        /// <summary>
        /// Check if a key is down.
        /// </summary>
        /// <param name="keycode">The code of the key to check.</param>
        /// <returns>1 if the key is down, otherwise 0.</returns>
        public static Primitive IsKeyDown(Primitive keycode)
        {
            int down;
            down = 0;
            
            if (hook_installed == 0) set_hook();
            if ( keycode != -1 ) down = keycode;
            return (Primitive)down;
        }

        /// <summary>
        /// Interrupts a Read() or ReadNumber() function.
        /// </summary>
        /// <returns></returns>
        public static Primitive InterruptRead()
        {
            IntPtr hWnd = System.Diagnostics.Process.GetCurrentProcess().MainWindowHandle;
            PostMessage(hWnd, WM_KEYDOWN, VK_RETURN, 0);
            return (Primitive)null;
        }
    }
}

Thursday, February 20, 2014 1:19 PM

Well, I tried it and it works.

If another application creates the console it's possible that it doesn't call to CallNextHook and as a result the console never gets the input.

Regards, Eyal Shilony


Thursday, February 20, 2014 1:39 PM

Tried it in what context?  As a SmallBasic extension, or by loading it as a DLL from a console app?

I'm not sure about Application.Run, does a console have Application.Run buried inside it somewhere?  It must receive and pump messages, even if it does abstract the messages from the programmer/console application.

In this case my DLL is loaded by Miscrsoft's SmallBasic application which is creating the console window as well.  However, the fact that I can use Console.WriteLine to put text on the console window and that the InterruptRead() function using PostMessage works leads me to believe that the everything is loaded and running properly, other than the keyboard hook callback.

It is possible that SetWindowsHookEx is creating a new thread with the specified function, and thus the callback function that is actually being executed is no longer associated with the SmallBasic process?  That would explain the reason my test Console.WriteLine in the hook function does not appear to be working, but this is just a hunch.


Thursday, February 20, 2014 2:29 PM

I didn't tried it inside SmallBasic I made a simple console application and tried to use the class you made and it captures the input to make sure it's working.

No a console application does not handle the message loop at all, you need to do it yourself or reference a library that already implements it such as System.Windows.Forms that by calling Application.Run handles the message loop and allows you to listen to the messages.

There's a major difference between executing a function to post a message to a window and listen to messages that are sent because the running application can stop the loop and prevent the window that it creates or farther processes from getting these messages, I'm not saying that this is definitely the case just saying that it's a possibility.

SetWindowsHookEx itself does not create any threads.

Regards, Eyal Shilony


Thursday, February 20, 2014 2:41 PM

Try specifying IntPtr.Zero instead of h in SetWindowsHookEx. Note that WH_KEYBOARD_LL hooking has a global scope.


Thursday, February 20, 2014 6:49 PM

"No a console application does not handle the message loop at all, "

If that is the case, how does the function with PostMessage() work?  A console app must have a message pump, even if it is not exposed to the developer.

"... I made a simple console application and tried to use the class you made and it captures the input ..."

So if it works in your console app, then why not in the console created by the SmallBasic app?  Did you have to include System.Windows.Forms and call Application.Run in the console app you set up?  Can you post the code you tested with?

@Viorel\_ : I have tried every possible combination of parameters to the SetWindowsHookEx function.  Setting the 3rd parameter to 0 didn't help, neither did passing a handle to the DLL, or a handle to the main process.

I do realize that WH_KEYBOARD_LL is global scope, and I don't really need that, so I also tried WH_KEYBOARD, but that did not work either.  I tried a few other options (WH_CALLWNDPROC) that could have thread-specific scope, and I tried passing the thread ID for the 4th parameter, but all my attempts failed to register the hook, or simply did not work.


Thursday, February 20, 2014 8:05 PM

"No a console application does not handle the message loop at all, "

If that is the case, how does the function with PostMessage() work?  A console app must have a message pump, even if it is not exposed to the developer.

Obviously, Windows does handle the message loop and using PostMessage is pure Win32 programming it has nothing to do with .NET so when you use it and tell it to put a message on the queue it will do so, no question about it but a console application does not understand how to automagically get the messages you post, meaning, you can't just create a console application and expect it to listen to messages you need to write code to do it unlike a WinForms application which comes with an infrastructure that you can use to do it because it was built on top of it.

So if it works in your console app, then why not in the console created by the SmallBasic app?  Did you have to include System.Windows.Forms and call Application.Run in the console app you set up?  Can you post the code you tested with?

Sure,

namespace Keyboard
{
    using System;
    using System.Runtime.InteropServices;

    public static class Keyboard
    {
        [DllImport("User32.Dll", EntryPoint = "PostMessageA")]
        private static extern bool PostMessage(IntPtr hWnd, uint msg, int wParam, int lParam);

        [DllImport("kernel32.dll", CharSet = CharSet.Auto, SetLastError = true)]
        private static extern IntPtr LoadLibrary(string librayName);

        [DllImport("user32.dll", CharSet = CharSet.Auto, SetLastError = true)]
        private static extern IntPtr SetWindowsHookEx(int idHook, LowLevelKeyboardProc lpfn, IntPtr hMod, uint dwThreadId);

        [DllImport("user32.dll", CharSet = CharSet.Auto, SetLastError = true)]
        private static extern IntPtr CallNextHookEx(IntPtr hhk, int nCode, IntPtr wParam, IntPtr lParam);

        [DllImport("kernel32.dll", CharSet = CharSet.Auto, SetLastError = true)]
        private static extern IntPtr GetModuleHandle(string lpModuleName);

        private delegate IntPtr LowLevelKeyboardProc(int nCode, IntPtr wParam, IntPtr lParam);

        private static int hook_installed = 0;
        private static LowLevelKeyboardProc hook_proc = hook_callback;
        private static IntPtr hook_id = IntPtr.Zero;

        private static int keycode = 0;

        private const int WH_KEYBOARD_LL = 13;
        private const int VK_RETURN = 0x0D;
        private const int WM_KEYDOWN = 0x100;
        private const int WM_KEYUP = 0x101;


        private static IntPtr hook_callback(int nCode, IntPtr wParam, IntPtr lParam)
        {
            Console.Write("e: " + nCode);

            if (nCode >= 0)
            {
                if (wParam == (IntPtr)WM_KEYDOWN)
                    keycode = Marshal.ReadInt32(lParam);
                else if (wParam == (IntPtr)WM_KEYUP)
                    keycode = -1;
            }

            Console.WriteLine("e: " + keycode);


            return CallNextHookEx(hook_id, nCode, wParam, lParam);
        }


        public static void set_hook()
        {
            IntPtr h = LoadLibrary("SmallBasic Extension.dll");

            if (h == null)
                Console.WriteLine("No module handle!");
            else
            {
                hook_id = SetWindowsHookEx(WH_KEYBOARD_LL, hook_proc, h, 0);
                if (hook_id == null)
                    Console.WriteLine("FAILED HOOK!");
                else
                    Console.Write("Hooked: " + hook_id);
            }

            hook_installed = 1;
        }
    }
}

I've changed the set_hook() function to public so I can call and test whether it's working.

      private static void Main(string[] args)
        {
            Keyboard.Keyboard.set_hook();

            Application.Run();
        }

Regards, Eyal Shilony


Thursday, February 20, 2014 8:42 PM

"Obviously, Windows does handle the message loop ..."

Getting people to admit / realize that is near impossible.

"... but a console application does not understand how to automagically get the messages you post, meaning, you can't just create a console application and expect it to listen to messages you need to write code to do it unlike a WinForms application ..."

Right, I understand that.  However, I know there *is* a message pump somewhere under a console app and the SetWindowsHookEx looked like a way to gain access to it.  But since it is not working in my context I am getting confused and frustrated with it.

This might all be moot anyway, since I was just introduced to the user32 GetKeyState() function.  Too many functions in Windows...  Looks to do exactly what I was ultimately trying to achieve with my hook.  I would still like to know why the hook did not work though, otherwise I walk away still frustrated instead of having learned something.