Thursday, July 28, 2011

The curious case of pthread_atfork on PowerPC

Recently I was asked to take a look at an issue on PowerPC whereby the symbol pthread_atfork seems to be missing from the 64-bit libpthread.so.0 library because the following message was output when trying to run an application that was linked with -lpthread.
undefined symbol: pthread_atfork
(Note: while this problem was encountered on PowerPC, the general concept is relevant to all platforms.)

If we compare the symbols exported in /lib/libpthread.so.0 (32-bit shared object) to those in /lib64/libpthread.so.0 (64-bit shared object) we see that there are no exported symbols for pthread_atfork in the 64-bit shared object:

nm /lib/libpthread.so.0 | grep atfork
0000c900 t __dyn_pthread_atfork
0000c900 T pthread_atfork@GLIBC_2.0
         U __register_atfork@@GLIBC_2.3.2
nm /lib64/libpthread.so.0 | grep atfork
<nothing>
In the 32-bit library notice that the addresses for __dyn_pthread_atfork and pthread_atfork@GLIBC_2.0 are the same!  This indicates that pthread_atfork@GLIBC_2.0 is a version tagged symbol which is not the same as a non-versioned pthread_atfork symbol.  This is telling us that there is no dynamically exported pthread_atfork in either the 32-bit or 64-bit libpthread.so.0.

So what's going on?  Since pthread_atfork hasn't been deprecated from POSIX where is it?

Let's take a look at some code from GLIBC that may help clarify things:

nptl/old_pthread_atfork.c:

/* Copyright (C) 2002 Free Software Foundation, Inc.
   This file is part of the GNU C Library.
   Contributed by Ulrich Drepper <drepper@redhat.com>, 2002.

   The GNU C Library is free software; you can redistribute it and/or
   modify it under the terms of the GNU Lesser General Public
   License as published by the Free Software Foundation; either
   version 2.1 of the License, or (at your option) any later version.

   The GNU C Library is distributed in the hope that it will be useful,
   but WITHOUT ANY WARRANTY; without even the implied warranty of
   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
   Lesser General Public License for more details.

   You should have received a copy of the GNU Lesser General Public
   License along with the GNU C Library; if not, write to the Free
   Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
   02111-1307 USA. */

#include <shlib-compat.h>

#if SHLIB_COMPAT (libpthread, GLIBC_2_0, GLIBC_2_3)
# define __pthread_atfork __dyn_pthread_atfork
# include "pthread_atfork.c"
# undef __pthread_atfork
compat_symbol (libpthread, __dyn_pthread_atfork, pthread_atfork, GLIBC_2_0);
#endif
nptl/pthread_atfork.c:
/* Copyright (C) 2002, 2006 Free Software Foundation, Inc.
   This file is part of the GNU C Library.
   Contributed by Ulrich Drepper <drepper@redhat.com>, 2002.

   The GNU C Library is free software; you can redistribute it and/or
   modify it under the terms of the GNU Lesser General Public
   License as published by the Free Software Foundation; either
   version 2.1 of the License, or (at your option) any later version.

   In addition to the permissions in the GNU Lesser General Public
   License, the Free Software Foundation gives you unlimited
   permission to link the compiled version of this file with other
   programs, and to distribute those programs without any restriction
   coming from the use of this file. (The GNU Lesser General Public
   License restrictions do apply in other respects; for example, they
   cover modification of the file, and distribution when not linked
   into another program.)

   Note that people who make modified versions of this file are not
   obligated to grant this special exception for their modified
   versions; it is their choice whether to do so. The GNU Lesser
   General Public License gives permission to release a modified
   version without this exception; this exception also makes it
   possible to release a modified version which carries forward this
   exception.

   The GNU C Library is distributed in the hope that it will be useful,
   but WITHOUT ANY WARRANTY; without even the implied warranty of
   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
   Lesser General Public License for more details.

   You should have received a copy of the GNU Lesser General Public
   License along with the GNU C Library; if not, write to the Free
   Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
   02111-1307 USA. */

#include "pthreadP.h"
#include <fork.h>

/* This is defined by newer gcc version unique for each module. */
extern void *__dso_handle __attribute__ ((__weak__,
                                          __visibility__ ("hidden")));


/* Hide the symbol so that no definition but the one locally in the
executable or DSO is used. */
int
#ifndef __pthread_atfork
/* Don't mark the compatibility function as hidden. */
attribute_hidden
#endif
__pthread_atfork (prepare, parent, child)
     void (*prepare) (void);
     void (*parent) (void);
     void (*child) (void);
{
return __register_atfork (prepare, parent, child,
                            &__dso_handle == NULL ? NULL : __dso_handle);
}
#ifndef __pthread_atfork
extern int pthread_atfork (void (*prepare) (void), void (*parent) (void),
                           void (*child) (void)) attribute_hidden;
strong_alias (__pthread_atfork, pthread_atfork)
#endif
Here we see that __dyn_pthread_atfork is a compat symbol for an older pthread_atfork implementation bound by alias to pthread_atfork@GLIBC_2.0, a versioned symbol.  Older applications that were linked against GLIBC 2.0 and pthread_atfork will be bound to __dyn_pthread_atfork when run against newer versions of GLIBC.

Looking back at our nm output, what we don't see is a non-versioned pthread_atfork symbol being exported by /lib/libpthread.so.0 (32-bit) and there is neither that nor a versioned symbol for /lib64/libpthread.so.0 (64-bit); why?
       
To answer the second question first; It's because PPC64 wasn't supported as a platform in GLIBC until after pthread_atfork was aliased to __dyn_pthread_atfork. So on PPC64 there isn't a dynamically exported version of pthread_atfork@GLIBC_2.0.

To get to the bottom of the first question takes a bit more analysis. The guard, #ifndef __pthread_atfork, is only true if we're building the compat version of the symbol, i.e. building "old_pthread_atfork.c".    
       
Otherwise, we're building pthread_atfork.c directly, in which case __pthread_atfork is marked hidden, and a strong_alias pthread_atfork is bound to it.  Curiously this non-versioned pthread_atfork alias is also marked hidden!  This implies that this will not be exported from libpthread.so.0.

We then see that the nptl/Makefile explicitly omits a non-versioned pthread_atfork symbol from being exported by the shared object file libpthread.so.0 by simply excluding the pthread_atfork relocatable file from libpthread.so.0. this is done using eliding in the nptl/Makefile:   
libpthread-static-only-routines = pthread_atfork
Any relocatable files elided via a lib*-static-only-routines variable are left out of the lib*.so.* shared object and built into a lib*_nonshared.a archive instead.

In general the functions in a lib*_nonshared.a archive are kept hidden, hence the reason for the curious hidden attribute on the pthread_atfork strong_alias.

If we look at libpthread_nonshared.a we will indeed see the pthread_atfork symbol.
nm /usr/lib/libpthread_nonshared.a

pthread_atfork.oS:
         w __dso_handle
00000000 T __pthread_atfork
         U __register_atfork
00000000 T pthread_atfork

nm /usr/lib64/libpthread_nonshared.a

pthread_atfork.oS:
                 w __dso_handle
0000000000000000 D __pthread_atfork
                 U __register_atfork
0000000000000000 D pthread_atfork
The libraries in GLIBC that need to use pthread_atfork are each statically linked against the libpthread_nonshared.a static archive.  The pthread_atfork symbol becomes statically linked into each individual GLIBC library and need not be exported.

So how does a user application use pthread_atfork?  By linking against libpthread.so, which is in fact a linker script that directs the linker to dynamically link against libpthread.so.0 and statically link in libpthread_nonshared.a:

cat /usr/lib64/libpthread.so
/* GNU ld script
Use the shared library, but some functions are only in
the static library, so try that secondarily. */
OUTPUT_FORMAT(elf32-powerpc)
GROUP ( /lib/libpthread.so.0 /usr/lib/libpthread_nonshared.a )

cat /usr/lib64/libpthread.so
/* GNU ld script
Use the shared library, but some functions are only in
the static library, so try that secondarily. */
OUTPUT_FORMAT(elf64-powerpc)
GROUP ( /lib64/libpthread.so.0 /usr/lib64/libpthread_nonshared.a )
This will automatically statically link libpthread_nonshared.a into an application at compilation and link time.  As of 2002-11-26 the pthread_atfork function is no longer a dynamically exported symbol in GLIBC.

So why are we getting undefined symbol: pthread_atfork even when we're linking with -lpthread?  The reason most likely has to due with the order of linking.

Let's say we have a relocatable object foo.o which includes a reference to pthread_atfork and we link foo.o into a shared object in the following way:

gcc -shared -o libfoo-0.0.1.so -lpthread ./foo.o

This seems like a perfectly reasonable way to link a shared object, and if we were only requesting dynamically exported symbols from libpthread.so.0 it would probably work just fine.  The problem comes in when we need to use a symbol provided by libpthread_nonshared.a, a static archive.

The link editor (static linker) is generally quite smart, but it links files in sequential order as it encounters them on the command line.  When it examines libpthread.so and sees that it needs to statically link in libpthread_nonshared.a it searches for outstanding undefined symbol references that this archive might satisfy.  If it finds one it links it, and its dependencies into the archive shared object.  Otherwise it purges the symbols it encounters in libpthread_nonshared.a.

So when -lpthread is added before foo.o on the linker invocation, libpthread_nonshared.a is linked and pthread_atfork is purged as unneeded before foo.o is processed.  So to solve this we need to make sure that the reference to pthread_atfork by foo.o is registered before libpthread_nonshared.a is linked.  This can be accomplished by any one of the following methods.
  1. ... -o foo-0.0.1.so ./foo.o -lpthread
  2. ... -o foo-0.0.1.so -lpthread ./foo.o -Wl,-u,pthread_atfork
  3. ... -o foo-0.0.1.so -Wl,--whole-archive -lpthread -Wl,--no-whole-archive ./foo.o
  4. ... -o foo-0.0.1.so -\( -lpthread ./foo.o \)-
The first method simply makes sure we link libpthread_nonshared.a after foo.o.

The second method explicitly tells the linker to not purge the pthread_atfork symbol.

The third method is heavy handed and tells the linker to include all of the symbols in libpthread_nonshared.a and libpthread.so.0 in the final link.

The fourth method tells the linker to reprocess all of the parenthesis enclosed libraries each time a new relocatable object or shared object is processed.  This means that the pthread libraries would be processed twice, once before and once after foo.o is processed.  This can increase link time significantly.

Monday, April 25, 2011

GDB Python Script To Get The Start Address Of A Loaded Library


The following GDB Python convenience function (as defined by the script) will search through the output of (gdb) info proc mappings for the library you've specified in the function parameter and return the start address of the .text segment of that library or executable.

Here is an example output of (gdb) info proc mapping with the entry for libc.so highlighted in red.  The Start Addr is what we'll have the convenience function fetch:
   Start Addr        End Addr       Size     Offset objfile
     0x100000        0x103000     0x3000   0x100000
    0x8000000       0x8029000    0x29000          0   /home/user/glibc64_power6/elf/ld.so
    0x8039000       0x803a000     0x1000    0x29000   /home/user/glibc64_power6/elf/ld.so
    0x803a000       0x803e000     0x4000    0x2a000   /home/user/glibc64_power6/elf/ld.so
   0x10000000      0x10002000     0x2000          0   /home/user/libdfp64_power6/test-isinf
   0x10011000      0x10012000     0x1000     0x1000   /home/user/libdfp64_power6/test-isinf
0x40000000000   0x40000002000     0x2000          0
0x40000002000   0x40000195000   0x193000          0   /home/user/glibc64_power6/libc.so
0x40000195000   0x400001a4000     0xf000   0x193000   /home/user/glibc64_power6/libc.so
0x400001a4000   0x400001a8000     0x4000   0x192000   /home/user/glibc64_power6/libc.so
0x400001a8000   0x400001bf000    0x17000   0x196000   /home/user/glibc64_power6/libc.so
0x400001bf000   0x400001c3000     0x4000   0x1bf000
0x400001c3000   0x4000026b000    0xa8000          0   /home/user/glibc64_power6/math/libm.so
0x4000026b000   0x4000027a000     0xf000    0xa8000   /home/user/glibc64_power6/math/libm.so
0x4000027a000   0x4000027b000     0x1000    0xa7000   /home/user/glibc64_power6/math/libm.so
0x4000027b000   0x40000285000     0xa000    0xa8000   /home/user/glibc64_power6/math/libm.so
0x40000285000   0x40000286000     0x1000   0x285000
0x40000286000   0x400002a3000    0x1d000          0   /home/user/glibc64_power6/nptl/libpthread.so
0x400002a3000   0x400002b2000     0xf000    0x1d000   /home/user/glibc64_power6/nptl/libpthread.so
0x400002b2000   0x400002b3000     0x1000    0x1c000   /home/user/glibc64_power6/nptl/libpthread.so
0x400002b3000   0x400002b5000     0x2000    0x1d000   /home/user/glibc64_power6/nptl/libpthread.so
0x400002b5000   0x400002ba000     0x5000   0x2b5000
0x400002ba000   0x4000031a000    0x60000          0   /home/user/libdfp64_power6/libdfp-1.0.7.so
0x4000031a000   0x40000329000     0xf000    0x60000   /home/user/libdfp64_power6/libdfp-1.0.7.so
0x40000329000   0x40000332000     0x9000    0x5f000   /home/user/libdfp64_power6/libdfp-1.0.7.so
0x40000332000   0x40000334000     0x2000   0x332000
0xffffff9d000   0xffffffb3000    0x16000 0xfff9d000               [stack]

Add the following to your ~.gdbinit file, or source it directly in your .gdb script:
source /path/to/gdb_start_address.py
gdb_start_address.py:
# gdb_start_address.py created by Ryan S. Arnold, April 2011.  No
# attribution required or necessary to use/reuse/copy/modify this
# function/script.

import re
class StartAddress(gdb.Function):
    """Returns the start address of a library or executable from info
    proc mappings."""

    def __init__(self):
        super (StartAddress, self).__init__ ("start_address")

    def invoke(self, library):
        mappings = gdb.execute("info proc mappings", to_string=True)
        lines = mappings.split('\n')
        to_mappings = ""
        for line in lines:
            if (to_mappings != "true"):
                if (re.search("Start Addr", line)):
                    to_mappings = "true"
                continue
            else:

                # The first match is the .text segment. Make sure to
                # match on "/libdfp-1.0.7.so" when passed "libdfp" and
                # not on "/libdfp" in the following example:
                # /home/ryanarn/libdfp/build64_power6/libdfp-1.0.7.so
                if (re.search("/" + library.string() + "[^/]*$", line)):
                    chunks = line.split()
                    return int(str(chunks[0]),16)
        return 0x0

StartAddress ()
There is currently a bug in GDB which crashes GDB after a direct assignment from this user function, e.g., (gdb) set $foo = $start_address("libc"). To use the function to set a variable do the following:
(gdb) p $start_address("libc")
$3 = 4398046760960
(gdb) set $foo = $
(gdb) p $foo
$4 = 4398046760960
I created this function to use in the Libdfp Makefile.gdb GNU make script.  Libdfp's Make system will automatically generate .gdb scripts for each of the test cases in the test-suite.  I use this function to find the load address of the library so that I can programmaticaly load the symbol file in the correct location, as in the following snippet of test-isinf.gdb:
source /home/ryanarn/eglibc/eglibc/libdfp/trunk/tests/gdb_start_address.py
...
p/x $start_address("libdfp")
set $libdfp_start = $
set $libdfp_text = 0x0af40
set $libdfp_addr = $libdfp_start + $libdfp_text
add-symbol-file ./libdfp.so.1 $libdfp_addr

Power.org Published the Power Architecture 32-bit ELF ABI Supplement

Power.org has published the Power Architecture® 32-bit Application Binary Interface Supplement 1.0 - Linux® & Embedded.
http://www.power.org/resources/downloads/Power-Arch-32-bit-ABI-supp-1.0-Unified.pdf
This is a specification document that I worked on for four years as part of the PowerABI TSC, under Power.org.

It defines the 32-bit Power Architecture processor specific ELF ABI for the GNU/Linux Operating System and Embedded environments.

I served as the document owner, style-sheet maintainer, and TSC Chair (for the last year).  It's very rewarding (relieving) to see this finally published.  It was a pleasure to work with everyone on the TSC.  Their expertise and contributions were invaluable.


There are three versions of the document: the Linux & Embedded (Unified), the Linux, and the Embedded.  These are all generated from the same parent document and are separated for convenience of the readers.

http://www.power.org/resources/downloads/Power-Arch-32-bit-ABI-supp-1.0-Unified.pdf
http://www.power.org/resources/downloads/Power-Arch-32-bit-ABI-supp-1.0-Linux.pdf
http://www.power.org/resources/downloads/Power-Arch-32-bit-ABI-supp-1.0-Embedded.pdf