1

I am trying to load an eBPF program via the bpf syscall in Go but am seeing an error returned from the syscall. In order to restrict the problem I am using the following minimal eBPF program, which does nothing:

struct task_group {};    

The important parts of the Go program are as follows:

b, err := ioutil.ReadFile("bpf/bbf_tty.o")
if err != nil {
    fmt.Print(err)
}

progType := BPF_PROG_TYPE_KPROBE
insns := unsafe.Pointer(&b)
insnCnt := len(b)

lba := struct {
    progType    uint32
    pad0        [4]byte
    insnCnt     uint32
    pad1        [4]byte
    insns       uint64
    license     uint64
    logLevel    uint32
    pad2        [4]byte
    logSize     uint32
    pad3        [4]byte
    logBuf      uint64
    kernVersion uint32
    pad4        [4]byte
}{
    progType:    uint32(progType),
    insns:       uint64(uintptr(insns)),
    insnCnt:     uint32(insnCnt),
    license:     uint64(uintptr(0)),
    logBuf:      uint64(uintptr(0)),
    logSize:     uint32(0),
    logLevel:    uint32(0),
    kernVersion: uint32(4),
}

ret, _, err := unix.Syscall(
    unix.SYS_BPF,
    bpf.BPF_PROG_LOAD,
    uintptr(unsafe.Pointer(&lba)),
    unsafe.Sizeof(lba),
)

if ret != 0 || err != 0 {
    return fmt.Errorf("Unable to load program: %s", err)
}

However the error that's getting returned is Unable to load program: argument list too long. Why is this? Or better yet, how can I get a more verbose output to find out the root cause of the issue?

From here there are only three places that E2BIG (argument list too long) gets returned from the bpf syscall, but none of them seem to fit.

I can provide a more complete version of my code if needed, I just tried to strip out the irrelevant parts for brevity.

To help with recreating this issue, I have included my full BPF program below. The full repo is here:

#include <node_config.h>
#include <netdev_config.h>
#include <filter_config.h>

#include <bpf/api.h>

#include <stdint.h>
#include <stdio.h>

#include <linux/bpf.h>
#include <linux/if_ether.h>

#include "lib/utils.h"
#include "lib/common.h"
#include "lib/maps.h"
#include "lib/xdp.h"
#include "lib/eps.h"
#include "lib/events.h"

// define structures
enum pid_type
{
    PIDTYPE_PID,
    PIDTYPE_PGID,
    PIDTYPE_SID,
    PIDTYPE_MAX,
    // only valid to __task_pid_nr_ns() 
    __PIDTYPE_TGID
};
struct upid {
  int nr;
};
struct pid
{
  struct upid numbers[1];
};
struct pid_link
{
  struct pid *pid;
};
struct task_group {
};
struct task_struct {
  struct task_struct *group_leader;
  struct pid_link           pids[PIDTYPE_MAX];
};
struct sid_t {
    int sid;
};

#define BUFSIZE 256
struct tty_write_t {
    int count;
    char buf[BUFSIZE];
    unsigned int sessionid;
};

// define maps
struct bpf_elf_map __section_maps active_sids = {
    .type       = BPF_MAP_TYPE_HASH,
    .size_key   = sizeof(struct sid_t),
    .size_value = sizeof(uint64_t),
};

struct bpf_elf_map __section_maps tty_writes = {
    .type       = BPF_MAP_TYPE_PERF_EVENT_ARRAY,
};

// save_sid saves a sessionid generated from a call
// to setsid to the active_sids map
int save_sid(struct pt_regs *ctx) {

    struct sid_t sid_struct = {};
    int sid = PT_REGS_RC(ctx);
    uint64_t time_ns = bpf_ktime_get_ns();

    sid_struct.sid = sid;

    bpf_map_update(&sid_struct, &time_ns);

    return 0;

}

//int kprobe__tty_write(struct pt_regs *ctx, struct file *file, const char __user *buf, size_t count)
int kprobe__tty_write(struct pt_regs *ctx, struct file *file, const char *buf, size_t count)
{
    struct task_struct *task;
    struct pid_link pid_link;
    struct pid pid;
    int sessionid;

    // get current sessionid
    task = (struct task_struct *)bpf_get_current_task();
    bpf_probe_read(&pid_link, sizeof(pid_link), (void *)&task->group_leader->pids[PIDTYPE_SID]);
    bpf_probe_read(&pid, sizeof(pid), (void *)pid_link.pid);
    sessionid = pid.numbers[0].nr;

    // build session struct key
    struct sid_t sid_key;
    sid_key.sid = sessionid;

    // if sid does not exist in our map then return
    //u64 *time_ns = active_sids.lookup(&sid_key);
    //if (!time_ns) {
    //    return 0;
    //}

    // bpf_probe_read() can only use a fixed size, so truncate to count
    // in user space:
    struct tty_write_t tty_write = {};
    bpf_probe_read(&tty_write.buf, BUFSIZE, (void *)buf);
    if (count > BUFSIZE) {
        tty_write.count = BUFSIZE;
    } else {
        tty_write.count = count;
    }

    // add sessionid to tty_write structure and submit
    tty_write.sessionid = sessionid;
    bpf_perf_event_output(ctx, &tty_write, sizeof(tty_write));

    return 0;
}
9
  • Is this a GO or C code problem? You should take the tour, read How to Ask and minimal reproducible example.
    – jwdonahue
    Feb 6, 2018 at 23:19
  • Strictly Go - I tagged C as I'm importing the C package. It's not really a problem in any language, it's a error returned from the bpf syscall
    – dippynark
    Feb 6, 2018 at 23:26
  • But I will read those things, I guess I haven't included enough detail for it to be answered properly
    – dippynark
    Feb 6, 2018 at 23:26
  • @dippynark Could you add the C code of the BPF program (or the bytecode if you didn't write it in C)? This error usually comes from an incorrect use of BPF helpers.
    – pchaigno
    Feb 7, 2018 at 6:59
  • @jwdonahue I've added back the C tag as it uses both C and Go and I'm not sure where the root cause is
    – dippynark
    Feb 7, 2018 at 9:30

2 Answers 2

2

Your problem here is the way you try to load the BPF bytecode.

b, err := ioutil.ReadFile("bpf/bbf_tty.o")

I have never used Go, but from what I understand this reads all the bytes from the ELF object file, without any specific processing, and feed them to the bpf() syscall later in your code.

The thing is, this is not how things work: when it compiles into eBPF, clang puts your program into one particular section (by default, .text, but you could specify another name). In addition, if you use eBPF maps, some magic happens (“map relocation”) so that your ELF file can embed map info, and your userspace program calling to bpf() can retrieve it and send it to the kernel.

So when you load the whole file to send it to bpf(), you load your actual bytecode, plus all ELF sections and header. The kernel probably does not like it much. I don't know how to fix it in Go, but here are some pointers that might be helpful:

  • libbpf, a C library that can load eBPF programs from the ELF files: located in the kernel tree.
  • Gobpf, some framework to use eBPF programs with Go (link). I've never used it, but surely they would have some code to load programs from object files?
2
  • I've got my program working with Gobpf but had issues when moving it into a container (I want to run distributed tracing with Kubernetes) so tried to go back to basics and implement everything myself through the syscall - thanks for pointing that out though, I'll try those things asap
    – dippynark
    Feb 7, 2018 at 11:17
  • started using gobpf's elf package instead of the bcc package and it works nicely without having to compile the BPF program on the fly :)
    – dippynark
    Feb 11, 2018 at 22:11
1

See @Qeole's answer for the actual cause of this error message.

You need a non-empty BPF program. Otherwise, you will fail the following precondition in bpf_prog_load:

if (attr->insn_cnt == 0 || attr->insn_cnt > BPF_MAXINSNS)
    return -E2BIG;

Your current compiled BPF program appears to be empty since it does not contain any function. Therefore, attr->insn_cnt is null.


Details I've checked that attr->insn_cnt is actually null:

$ cat tmp.c 
struct task_group {};
$ clang -O2 -target bpf -c tmp.c -o tmp.o
$ ls -lh tmp.o 
-rw-rw-r-- 1 paul paul 368 févr.  7 11:21 tmp.o
$ readelf -x .text tmp.o

Section '.text' has no data to dump.

The object file is not empty but its .text section, which should contains the BPF instructions, is. If I run readelf -x .text tmp.o on one of my own programs I get a hexdump, as expected.

15
  • @pchaigo isnt_cnt is not non-zero; even with just a task_struct definition it is over 300 bytes big. I've tried with a full program of about 1KB in size with the same issue
    – dippynark
    Feb 7, 2018 at 10:14
  • Hm, I'll run some more checks. What is 300B big? The bpf/bbf_tty.o file?
    – pchaigno
    Feb 7, 2018 at 10:16
  • Yeah, even with just that task_group struct above. I guess there's come default overhead that's added - the details of C compilation is not one of my strong points
    – dippynark
    Feb 7, 2018 at 10:19
  • I'm using CLANG for the compilation of the BPF program in the same way as Cilium do here: github.com/cilium/cilium/blob/master/bpf/Makefile
    – dippynark
    Feb 7, 2018 at 10:20
  • @pchaigo have you managed to recreate the error at all? I realise there's a lot of stuff around the edges that's missing, just didn't want to clog the question
    – dippynark
    Feb 7, 2018 at 10:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.