83

If I load a kernel module and list the loaded modules with lsmod, I can get the "use count" of the module (number of other modules with a reference to the module). Is there a way to figure out what is using a module, though?

The issue is that a module I am developing insists its use count is 1 and thus I cannot use rmmod to unload it, but its "by" column is empty. This means that every time I want to re-compile and re-load the module, I have to reboot the machine (or, at least, I can't figure out any other way to unload it).

4
  • "what" in which terms? what code? what module? what user? what program? tho i slightly get the feeling this is not programming related :) interesting none-the-less Jan 16, 2009 at 1:18
  • 2
    Well, it is programming related, since I'm asking because I'm writing a kernel module.
    – mipadi
    Jan 16, 2009 at 1:21
  • please clarify the question to show the programming problem you are trying to solve. Jan 16, 2009 at 1:23
  • 15
    The question is pretty clear to me, Norman: how can he find out what's keeping rmmod from removing his experimental module?; how can avoid having to reboot every time he compiles a new version? Jan 16, 2009 at 3:29

7 Answers 7

54

Actually, there seems to be a way to list processes that claim a module/driver - however, I haven't seen it advertised (outside of Linux kernel documentation), so I'll jot down my notes here:

First of all, many thanks for @haggai_e's answer; the pointer to the functions try_module_get and try_module_put as those responsible for managing the use count (refcount) was the key that allowed me to track down the procedure.

Looking further for this online, I somehow stumbled upon the post Linux-Kernel Archive: [PATCH 1/2] tracing: Reduce overhead of module tracepoints; which finally pointed to a facility present in the kernel, known as (I guess) "tracing"; the documentation for this is in the directory Documentation/trace - Linux kernel source tree. In particular, two files explain the tracing facility, events.txt and ftrace.txt.

But, there is also a short "tracing mini-HOWTO" on a running Linux system in /sys/kernel/debug/tracing/README (see also I'm really really tired of people saying that there's no documentation…); note that in the kernel source tree, this file is actually generated by the file kernel/trace/trace.c. I've tested this on Ubuntu natty, and note that since /sys is owned by root, you have to use sudo to read this file, as in sudo cat or

sudo less /sys/kernel/debug/tracing/README

... and that goes for pretty much all other operations under /sys which will be described here.


First of all, here is a simple minimal module/driver code (which I put together from the referred resources), which simply creates a /proc/testmod-sample file node, which returns the string "This is testmod." when it is being read; this is testmod.c:

/*
https://github.com/spotify/linux/blob/master/samples/tracepoints/tracepoint-sample.c
https://www.linux.com/learn/linux-training/37985-the-kernel-newbie-corner-kernel-debugging-using-proc-qsequenceq-files-part-1
*/

#include <linux/module.h>
#include <linux/sched.h>
#include <linux/proc_fs.h>
#include <linux/seq_file.h> // for sequence files

struct proc_dir_entry *pentry_sample;

char *defaultOutput = "This is testmod.";


static int my_show(struct seq_file *m, void *v)
{
  seq_printf(m, "%s\n", defaultOutput);
  return 0;
}

static int my_open(struct inode *inode, struct file *file)
{
  return single_open(file, my_show, NULL);
}

static const struct file_operations mark_ops = {
  .owner    = THIS_MODULE,
  .open = my_open,
  .read = seq_read,
  .llseek   = seq_lseek,
  .release  = single_release,
};


static int __init sample_init(void)
{
  printk(KERN_ALERT "sample init\n");
  pentry_sample = proc_create(
    "testmod-sample", 0444, NULL, &mark_ops);
  if (!pentry_sample)
    return -EPERM;
  return 0;
}

static void __exit sample_exit(void)
{
    printk(KERN_ALERT "sample exit\n");
    remove_proc_entry("testmod-sample", NULL);
}

module_init(sample_init);
module_exit(sample_exit);

MODULE_LICENSE("GPL");
MODULE_AUTHOR("Mathieu Desnoyers et al.");
MODULE_DESCRIPTION("based on Tracepoint sample");

This module can be built with the following Makefile (just have it placed in the same directory as testmod.c, and then run make in that same directory):

CONFIG_MODULE_FORCE_UNLOAD=y
# for oprofile
DEBUG_INFO=y
EXTRA_CFLAGS=-g -O0

obj-m += testmod.o

# mind the tab characters needed at start here:
all:
    make -C /lib/modules/$(shell uname -r)/build M=$(PWD) modules

clean:
    make -C /lib/modules/$(shell uname -r)/build M=$(PWD) clean

When this module/driver is built, the output is a kernel object file, testmod.ko.


At this point, we can prepare the event tracing related to try_module_get and try_module_put; those are in /sys/kernel/debug/tracing/events/module:

$ sudo ls /sys/kernel/debug/tracing/events/module
enable  filter  module_free  module_get  module_load  module_put  module_request

Note that on my system, tracing is by default enabled:

$ sudo cat /sys/kernel/debug/tracing/tracing_enabled
1

... however, the module tracing (specifically) is not:

$ sudo cat /sys/kernel/debug/tracing/events/module/enable
0

Now, we should first make a filter, that will react on the module_get, module_put etc events, but only for the testmod module. To do that, we should first check the format of the event:

$ sudo cat /sys/kernel/debug/tracing/events/module/module_put/format
name: module_put
ID: 312
format:
...
    field:__data_loc char[] name;   offset:20;  size:4; signed:1;

print fmt: "%s call_site=%pf refcnt=%d", __get_str(name), (void *)REC->ip, REC->refcnt

Here we can see that there is a field called name, which holds the driver name, which we can filter against. To create a filter, we simply echo the filter string into the corresponding file:

sudo bash -c "echo name == testmod > /sys/kernel/debug/tracing/events/module/filter"

Here, first note that since we have to call sudo, we have to wrap the whole echo redirection as an argument command of a sudo-ed bash. Second, note that since we wrote to the "parent" module/filter, not the specific events (which would be module/module_put/filter etc), this filter will be applied to all events listed as "children" of module directory.

Finally, we enable tracing for module:

sudo bash -c "echo 1 > /sys/kernel/debug/tracing/events/module/enable"

From this point on, we can read the trace log file; for me, reading the blocking, "piped" version of the trace file worked - like this:

sudo cat /sys/kernel/debug/tracing/trace_pipe | tee tracelog.txt

At this point, we will not see anything in the log - so it is time to load (and utilize, and remove) the driver (in a different terminal from where trace_pipe is being read):

$ sudo insmod ./testmod.ko
$ cat /proc/testmod-sample 
This is testmod.
$ sudo rmmod testmod

If we go back to the terminal where trace_pipe is being read, we should see something like:

# tracer: nop
#
#           TASK-PID    CPU#    TIMESTAMP  FUNCTION
#              | |       |          |         |
          insmod-21137 [001] 28038.101509: module_load: testmod
          insmod-21137 [001] 28038.103904: module_put: testmod call_site=sys_init_module refcnt=2
           rmmod-21354 [000] 28080.244448: module_free: testmod

That is pretty much all we will obtain for our testmod driver - the refcount changes only when the driver is loaded (insmod) or unloaded (rmmod), not when we do a read through cat. So we can simply interrupt the read from trace_pipe with CTRL+C in that terminal; and to stop the tracing altogether:

sudo bash -c "echo 0 > /sys/kernel/debug/tracing/tracing_enabled"

Here, note that most examples refer to reading the file /sys/kernel/debug/tracing/trace instead of trace_pipe as here. However, one problem is that this file is not meant to be "piped" (so you shouldn't run a tail -f on this trace file); but instead you should re-read the trace after each operation. After the first insmod, we would obtain the same output from cat-ing both trace and trace_pipe; however, after the rmmod, reading the trace file would give:

   <...>-21137 [001] 28038.101509: module_load: testmod
   <...>-21137 [001] 28038.103904: module_put: testmod call_site=sys_init_module refcnt=2
   rmmod-21354 [000] 28080.244448: module_free: testmod

... that is: at this point, the insmod had already been exited for long, and so it doesn't exist anymore in the process list - and therefore cannot be found via the recorded process ID (PID) at the time - thus we get a blank <...> as process name. Therefore, it is better to log (via tee) a running output from trace_pipe in this case. Also, note that in order to clear/reset/erase the trace file, one simply writes a 0 to it:

sudo bash -c "echo 0 > /sys/kernel/debug/tracing/trace"

If this seems counterintuitive, note that trace is a special file, and will always report a file size of zero anyways:

$ sudo ls -la /sys/kernel/debug/tracing/trace
-rw-r--r-- 1 root root 0 2013-03-19 06:39 /sys/kernel/debug/tracing/trace

... even if it is "full".

Finally, note that if we didn't implement a filter, we would have obtained a log of all module calls on the running system - which would log any call (also background) to grep and such, as those use the binfmt_misc module:

...
  tr-6232  [001] 25149.815373: module_put: binfmt_misc call_site=search_binary_handler refcnt=133194
..
  grep-6231  [001] 25149.816923: module_put: binfmt_misc call_site=search_binary_handler refcnt=133196
..
  cut-6233  [000] 25149.817842: module_put: binfmt_misc call_site=search_binary_handler refcnt=129669
..
  sudo-6234  [001] 25150.289519: module_put: binfmt_misc call_site=search_binary_handler refcnt=133198
..
  tail-6235  [000] 25150.316002: module_put: binfmt_misc call_site=search_binary_handler refcnt=129671

... which adds quite a bit of overhead (in both log data ammount, and processing time required to generate it).


While looking this up, I stumbled upon Debugging Linux Kernel by Ftrace PDF, which refers to a tool trace-cmd, which pretty much does the similar as above - but through an easier command line interface. There is also a "front-end reader" GUI for trace-cmd called KernelShark; both of these are also in Debian/Ubuntu repositories via sudo apt-get install trace-cmd kernelshark. These tools could be an alternative to the procedure described above.

Finally, I'd just note that, while the above testmod example doesn't really show use in context of multiple claims, I have used the same tracing procedure to discover that an USB module I'm coding, was repeatedly claimed by pulseaudio as soon as the USB device was plugged in - so the procedure seems to work for such use cases.

2
  • 5
    Thanks for the comment, @RichardHansen - the question is "Is there a way to figure out what is using a module"; and you can see in the module trace that for instance rmmod-21354, or tr-6232 (process name - process ID) are the ones doing the module_put, that is, changing the refcount of the module - that is, those processes are "using" the module; so I'd argue it answers exactly what the OP asked for... Cheers!
    – sdaau
    Apr 16, 2013 at 11:35
  • 1
    The "I'm really really tired" link is broken
    – Hi-Angel
    Mar 11, 2020 at 12:38
8

It says on the Linux Kernel Module Programming Guide that the use count of a module is controlled by the functions try_module_get and module_put. Perhaps you can find where these functions are called for your module.

More info: https://www.kernel.org/doc/htmldocs/kernel-hacking/routines-module-use-counters.html

4

All you get are a list of which modules depend on which other modules (the Used by column in lsmod). You can't write a program to tell why the module was loaded, if it is still needed for anything, or what might break if you unload it and everything that depends on it.

3

You might try lsof or fuser.

2
  • 1
    How exactly can I use lsof for that? Apr 26, 2017 at 11:25
  • 2
    It's not a universal solution, but in case the driver creates a device in /dev it can be listed by lsof.
    – SergeyM
    Nov 15, 2019 at 22:53
2

If you use rmmod WITHOUT the --force option, it will tell you what is using a module. Example:

$ lsmod | grep firewire
firewire_ohci          24695  0 
firewire_core          50151  1 firewire_ohci
crc_itu_t               1717  1 firewire_core

$ sudo modprobe -r firewire-core
FATAL: Module firewire_core is in use.

$ sudo rmmod firewire_core
ERROR: Module firewire_core is in use by firewire_ohci

$ sudo modprobe -r firewire-ohci
$ sudo modprobe -r firewire-core
$ lsmod | grep firewire
$
2
  • 5
    Well, that is not correct in general: I have on my machine: $ lsmod | grep snd snd_seq 47263 1 snd_timer 19130 1 snd_seq snd_seq_device 5100 1 snd_seq ... ; so snd_seq is claimed by something (refcount is 1), but one cannot tell why, as the column after it is empty and so no other module claims it specifically (but maybe if one ftraces the kernel already from the start of the boot process, one could find out, I guess).
    – sdaau
    Apr 14, 2014 at 20:21
  • 6
    This only works in the situation where lsmod also displays something in the "used by" column; rmmod doesn't have any more logic than lsmod does in terms of displaying dependencies.
    – dannysauer
    Sep 27, 2016 at 14:37
0

try kgdb and set breakpoint to your module

-4

For anyone desperate to figure out why they can't reload modules, I was able to work around this problem by

  • Getting the path of the currently used module using "modinfo"
  • rm -rfing it
  • Copying the new module I wanted to load to the path it was in
  • Typing "modprobe DRIVER_NAME.ko".
1
  • 2
    This answer does not actually answer the question asked.
    – kelnos
    Oct 3, 2017 at 21:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.