View source for Chapter:C libraries


= Introduction =

Let's say you're writing code that manages books. You use the code in a server and would like to use some the same code in a text client and why not in a GUI. So we would like to share code between programs. We could do this in some different ways:
# compile the same source code files and use the object files 
# create an archive and let the programs linked against that
# create a shared library and let the programs linked against that

==Compile the same source code files==  
The first one is ok, well it could be kind of ok, if all our code (server, client and GUI) were located in the same place. But what would happen if we decide to split the project into some smaller sub projects? It will not be as easy to manage the code base shared. Wouldn't it be easier if the sub project writing the shared code could produce a deliverable that in itself could be shared. Fact is, they can. They can produce (create) an archive or a shared library.

==Create an archive==
An archive is a collection of object files (compiler c files). So instead of delivering a some object files as they are, [[tar]]ed, or [[zip]]ped we could use a format is suited, actually written, for this purpose. This is an archive. It is a collection of object files that can be linked the same way as the objects files could. The functions (and variables..) you would like to use are linked in to the program when the program is being created (linked).

==Create a shared library==
A shared library is also a collection of object files. The functions (and variables..) you would like to use are linked in to the program when the program is being executed. This kind of linking is called dynamic linking and the library loaded dynamically is called a dynamic library. The same library can be loaded (at runtime) to many programs, which might save disk space. If the team responsible for the shared code delivers a new version all the other teams have to is replace the shared library with the new one - no compilation or linked needed (apart from run-time linking of course).

= Videos = 

[https://vimeo.com/couchmode/channels/1377050 C libraries (Full playlist)] | [https://vimeo.com/271436372 C - libraries 1/2] | [https://vimeo.com/271436366 C - libraries 2/2] | [[:Media:Libraries.pdf|Libraries (PDF)]]

= Library bascis =
Let's assume that we want to create a library from the files <code>book.c</code> and <code>author.c</code>.They both have coresponding header files <code>book.h</code> and <code>author.h</code>. We create a simple test program (<code>test-book.c</code>) that creates one book and prints it out - yeah, in a true academic style. 

You can find the source code here: [https://github.com/progund/programming-with-c/tree/master/libraries/book-example github.com/progund/programming-with-c/tree/master/libraries/book-example]. In this directory you'll find a bash script called <code>build.sh</code>. Invoke it to build as in the following section:
<source lang="bash">
$ ./build.sh
Using no libraries
---------------------------
gcc -Ilib lib/*.c test/test-book.c -o test-prog
Executing program:
Pendulum (Umberto Eco, umberto@thegreatone.com)

Using archive
---------------------------
cd lib && gcc -c *.c
ar rcu lib/libbook.a lib/book.o lib/author.o
gcc -Ilib test/test-book.c -Llib -lbook -o test-prog
Executing program:
./test-prog: error while loading shared libraries: libbook.so: cannot open shared object file: No such file or directory

Using shared library
---------------------------
rm lib/lib*.a
cd lib && gcc -fPIC -c *.c
gcc -shared  -o lib/libbook.so lib/book.o lib/author.o
gcc -Ilib test/test-book.c -Llib -lbook -o test-prog
Executing program:
Pendulum (Umberto Eco, umberto@thegreatone.com)
</source>

Below we will go through what happened when invoking the script.

== Without libraries ==

Without libraries you would compile as follows:
<source lang="c">
$ rm lib/*.o
$ gcc -Ilib lib/*.c test/test-book.c -o test-prog
</source>
Explanation (line by line):
# remove all objects files in <code>lib</code>
# compile all c files in <code>lib</code> and <code>test/test-book.c</code> and call the program <code>test-prog</code>
=== Create and use a static archive ===

<source lang="c">
$ rm lib/*.o
$ cd lib && gcc -c *.c
$ ar rcu lib/libbook.a lib/book.o lib/author.o
$ gcc -Ilib test/test-book.c -Llib -lbook -o test-prog
</source>
Explanation (line by line):
# remove all objects files in <code>lib</code>
# change to directory <code>lib</code>
#: compile all c files
# create an archive, <code>lib/libbook.a</code> from these object files
# compile <code>test/test-book.c</code> and link with the archive just created, and finally call the program <code>test-prog</code>

== Create and use a shared library ==
<source lang="c">
$ rm lib/*.o lib/*.a
$ cd lib && gcc -fPIC -c *.c
$ gcc -shared  -o lib/libbook.so lib/book.o lib/author.o
$ gcc -Ilib test/test-book.c -Llib -lbook -o test-prog
</source>
Explanation (line by line):
# remove all objects files and archives in <code>lib</code>
# change to directory <code>lib</code>
#: compile all c files (with the flag <code>-fPIC</code> to produce so called position independent code).
# create a shared library, <code>lib/libbook.so</code> from these object files 
# compile <code>test/test-book.c</code> and link with the shared library just created, and finally call the program <code>test-prog</code>

= Using multiple files to create a program =

Let's leave the book code above and look at some other example. This time we will look at some code which can be used to store names. As wo go along using this code and a small test program we will look into:
* compiling all c files at once (not using libraries)
* creating an archive and use it
* creating a shared library and use it
* examining the content of object files (be it object files, archives, shared libraries and binaries)
* examining the library dependency of a program
* check the size of object files (be it object files, archives, shared libraries and binaries)

== Preparations ==

Set some compiler flags to make the command listing slightly easier to read:
<source lang="bash">
export CFLAGS="-pedantic -Wconversion -Wall -Werror -Wextra -Wstrict-prototypes"
</source>
When having set this variable we can compile like this
<source lang="bash">
gcc ${CFLAGS} ......
</source>
This way we don't have to write all the compiler flags every time we invoke gcc on this page. 

== Introducing the name library==
We've written a small c with functions that can be used to store names dynamically. The names are stored in a struct called <code>name_list</code>:
<source  lang="c">
typedef struct name_list_
{
  char** names;
  unsigned int size;
} name_list;
</source>
Together with this struct we have written a couple of functions. 
<div class="mw-collapsible mw-collapsed">
Expand using link to the right to see a list of the functions.
<div class="mw-collapsible-content">
<source lang="C">
/**
 * @brief Returns a pointer to a name_list struct (dynamically allocated).
 * @return A pointer to the allocated memory. NULL is returned if memory allocation fails
 */
name_list* name_list_new(void);

/**
 * @brief Adds name name_list struct. The string is copied and memory has been created for it using the alloc family.
 * @param list the list to add name to
 * @param name the name to add 
 * @return NAME_OK (0) on success. 
 */
int name_list_add(name_list *list, char *name);

/**
 * @brief Removes the first occurance of name in the list. The string
 * is freed and the names afterwards are copied down. The size of the list is alos adjusted (with -1) on success.
 * @param list the list to remove name from
 * @param name to remove
 * @return NAME_OK (0) on success.
 */
int name_list_remove(name_list *list, char *name);

/**
 * @brief Removes the name at a given position in the list. The string
 * is freed and the names afterwards are copied down. The size of the list is alos adjusted (with -1) on success.
 * @param list the list to remove name from
 * @param pos the position of the name to remove
 * @return NAME_OK (0) on success.
 */
int name_list_remove_at(name_list *list, unsigned int pos);

/**
 * @brief Returns the position of the name in the list
 * @param list the list to get the position of name from
 * @param name the name to search for
 * @return NAME_OK (0) on success.
 */
int name_list_positon(name_list *list, char *name);

/**
 * @brief Prints (to stream) the entire list of names.
 * @param list the list to get the position of name from
 * @param stream where to print
 * to add @return NAME_OK (0) on success.
 */
int name_list_print(name_list *list, FILE* stream);

/**
 * @brief Frees all the memory used by the list
 * @param list the list to free
 * @return NAME_OK (0) on success.
 */
int name_list_free(name_list *list);

/**
 * @brief Frees all the memory used by the list
 * @param list the list to free
 * @return NAME_OK (0) on success.
 */
int name_list_size(name_list *list);

/**
 * @brief Returns the name at position in the list
 * @param list the list to get the name (at position) from
 * @param pos the position to get the name
 * @return NAME_OK (0) on success.
 */
char* name_list_at_position(name_list *list, unsigned int pos);
</source>
</div></div>


The source code can be found at github: [https://github.com/progund/programming-with-c/tree/master/libraries/src github.com/progund/programming-with-c/libraries/src]

== Without archive ==

===Compile name.c===
<source lang="bash">
$ gcc $CFLAGS -c  name.c -o name.o
</source>

===Examine name.o===
<source lang="bash">
$ nm name.o
                 U calloc
                 U fprintf
                 U free
                 U _GLOBAL_OFFSET_TABLE_
0000000000000045 T name_list_add
00000000000001d7 T name_list_at_position
0000000000000490 T name_list_free
0000000000000000 T name_list_new
0000000000000375 T name_list_positon
00000000000003fd T name_list_print
000000000000017c T name_list_remove
0000000000000212 T name_list_remove_at
0000000000000530 T name_list_size
                 U realloc
0000000000000004 C RET_VAL
                 U stderr
                 U strdup
                 U strlen
                 U strncmp
</source>

This might need a bit of explanation. The lines with ''U'' are lines with functions not defined (undefined) in this object files. For example, the line <code>U strlen</code> means that <code>strlen</code> is used, but not defined in <code>name.o</code>. Check the lines with '''T'''. T means ''The symbol is in the text (code) section.'' so this is what we're looking for. Let's use grep to filter a bit.
<source lang="bash">
$ nm name.o | grep " [Tt] name"
0000000000000045 T name_list_add
00000000000001d7 T name_list_at_position
0000000000000490 T name_list_free
0000000000000000 T name_list_new
0000000000000375 T name_list_positon
00000000000003fd T name_list_print
000000000000017c T name_list_remove
0000000000000212 T name_list_remove_at
0000000000000530 T name_list_size
</source>
This means that the functions above are defined in the file <code>name.o</code>. 

===Use the object file / link the program===
<source lang="bash">
$ gcc $CFLAGS -I. test/test-name.c name.o -o test-prog
</source>

===Execute the program===
Run the test program
<source lang="bash">
$ ./test-prog 
realloc 0x14ce260, new size: 1
Assign 0x14ce280, ..
.....
</source>

===Examine the program===
<source lang="bash">
$ nm test-prog 
                 U __assert_fail@@GLIBC_2.2.5
0000000000602064 B __bss_start
                 U calloc@@GLIBC_2.2.5
... snip
00000000004007a7 T main
0000000000400fd1 T name_list_add
0000000000401163 T name_list_at_position
000000000040141c T name_list_free
.... snip
</source>
''Note: we have cut away rather much from the printout. 
<div class="mw-collapsible mw-collapsed">
Expand using link to the right to see a complete listing from the nm command above.
<div class="mw-collapsible-content">
<source lang="bash">
                 U __assert_fail@@GLIBC_2.2.5
0000000000602064 B __bss_start
                 U calloc@@GLIBC_2.2.5
0000000000602064 b completed.6973
0000000000602060 D __data_start
0000000000602060 W data_start
0000000000400700 t deregister_tm_clones
0000000000400770 t __do_global_dtors_aux
0000000000601e08 t __do_global_dtors_aux_fini_array_entry
0000000000401568 R __dso_handle
0000000000601e10 d _DYNAMIC
0000000000602064 D _edata
0000000000602070 B _end
0000000000401554 T _fini
                 U fprintf@@GLIBC_2.2.5
00000000004007a0 t frame_dummy
0000000000601e00 t __frame_dummy_init_array_entry
0000000000401a94 r __FRAME_END__
                 U free@@GLIBC_2.2.5
0000000000602000 d _GLOBAL_OFFSET_TABLE_
                 w __gmon_start__
0000000000401804 r __GNU_EH_FRAME_HDR
0000000000400610 T _init
0000000000601e08 t __init_array_end
0000000000601e00 t __init_array_start
0000000000401560 R _IO_stdin_used
0000000000401550 T __libc_csu_fini
00000000004014e0 T __libc_csu_init
                 U __libc_start_main@@GLIBC_2.2.5
00000000004007a7 T main
0000000000400fd1 T name_list_add
0000000000401163 T name_list_at_position
000000000040141c T name_list_free
0000000000400f8c T name_list_new
0000000000401301 T name_list_positon
0000000000401389 T name_list_print
0000000000401108 T name_list_remove
000000000040119e T name_list_remove_at
00000000004014bc T name_list_size
00000000004017a4 r __PRETTY_FUNCTION__.2478
                 U realloc@@GLIBC_2.2.5
0000000000400730 t register_tm_clones
0000000000602068 B RET_VAL
00000000004006d0 T _start
                 U stderr@@GLIBC_2.2.5
                 U stdout@@GLIBC_2.2.5
                 U strcmp@@GLIBC_2.2.5
                 U strdup@@GLIBC_2.2.5
                 U strlen@@GLIBC_2.2.5
                 U strncmp@@GLIBC_2.2.5
0000000000602068 D __TMC_END__
</source>
</div></div>

Let's use grep to filter a bit.
<source lang="bash">
$ nm test-prog | grep " [Tt] name"
0000000000400fd1 T name_list_add
0000000000401163 T name_list_at_position
000000000040141c T name_list_free
0000000000400f8c T name_list_new
0000000000401301 T name_list_positon
0000000000401389 T name_list_print
0000000000401108 T name_list_remove
000000000040119e T name_list_remove_at
00000000004014bc T name_list_size
</source>
We can see that the program (<code>test-prog</code>) contains the functions we wrote in the files <code>name.c</code> and <code>test/test-name.c</code>. This should hopefully not come as a surprise for you.

===Size matters===

Let's look at the size of the compiled c file (<code>name.c</code>) and the program

<source lang="bash">
$ du -sb name.o test-prog 
4624	name.o
13256	test-prog 
</source>

== Archive ==

=== Compile name.c and create an archive ===
Compile name.c and create an archive
<source lang="bash">
$ gcc $CFLAGS -c -I. -I.. name.c -o name.o
$ ar rcu  libname.a name.o
</source>

=== Examine the archive===
Get someinformation about the archive
<source lang="bash">
$ file libname.a 
libname.a: current ar archive
</source>
Ah.. it is an archive. Great. Wouldn't it be great if we could see what functions are defined in the archive? Well, we can using mn. Let's use grep to filter a bit.
<source lang="bash">
$ nm libname.a | grep " [tT] " 
0000000000000045 T name_list_add
00000000000001d7 T name_list_at_position
0000000000000490 T name_list_free
0000000000000000 T name_list_new
0000000000000375 T name_list_positon
00000000000003fd T name_list_print
000000000000017c T name_list_remove
0000000000000212 T name_list_remove_at
0000000000000530 T name_list_size
</source>
So the functions <code>name_list_add</code>, <code>name_list_at_position</code>, <code>name_list_free</code>, <code>name_list_new</code>, <code>name_list_positon</code>, <code>name_list_print</code>, <code>name_list_remove</code>, <code>name_list_remove_at</code>, <code>name_list_size</code> are defined in the archive.

''Note: If you're interested, we used the command: <pre>nm libname.a | grep " [tT] " | awk '{ printf "<code>%s</code>, ", $3}'; echo</pre> to get the list above printed in mediawiki format.''

=== Use the archive / link the program ===
Compile test/test-name.c and create a binary using the archive 
<source lang="bash">
$ gcc $CFLAGS -I. test/test-name.c -L. -lname -o test-prog
</source>

=== Execute the program (linked with the archive) ===
Run the test program
<source lang="bash">
$ ./test-prog 
realloc 0x14ce260, new size: 1
Assign 0x14ce280, ..
.....
</source>


=== Examine the program (linked with the archive) ===

Let's look at what functions are defined in the program
<source lang="bash">
$ nm test-prog  | grep " [Tt] "
0000000000400700 t deregister_tm_clones
0000000000400770 t __do_global_dtors_aux
0000000000601e08 t __do_global_dtors_aux_fini_array_entry
0000000000401554 T _fini
00000000004007a0 t frame_dummy
0000000000601e00 t __frame_dummy_init_array_entry
0000000000400610 T _init
0000000000601e08 t __init_array_end
0000000000601e00 t __init_array_start
0000000000401550 T __libc_csu_fini
00000000004014e0 T __libc_csu_init
00000000004007a7 T main
0000000000400fd1 T name_list_add
0000000000401163 T name_list_at_position
000000000040141c T name_list_free
0000000000400f8c T name_list_new
0000000000401301 T name_list_positon
0000000000401389 T name_list_print
0000000000401108 T name_list_remove
000000000040119e T name_list_remove_at
00000000004014bc T name_list_size
0000000000400730 t register_tm_clones
00000000004006d0 T _start
</source>

Lots of internal things, so let's focus on our "name" things.
<source lang="bash">
$ nm test-prog  | grep " [Tt] name"
0000000000400fd1 T name_list_add
0000000000401163 T name_list_at_position
000000000040141c T name_list_free
0000000000400f8c T name_list_new
0000000000401301 T name_list_positon
0000000000401389 T name_list_print
0000000000401108 T name_list_remove
000000000040119e T name_list_remove_at
00000000004014bc T name_list_size
</source>
This means that the functions above are now defined in the program. They have been included (statically) to the program via the archive (<code>libname.a</code>) which was build from <code>name.o</code>.

<source lang="bash">
$ file test-prog 
test-prog: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=01b9d852e2d40905432a1e1d30e40bbd3c91c6cf, not stripped
</source>
So, the test program is actually using some shared libraries. Let's check if it is using our library as a dynamic library or as an archive. Time to get some extra information about the test program
<source lang="bash">
$ ldd test-prog 
	linux-vdso.so.1 (0x00007fff5177d000)
	libc.so.6 => /lib64/libc.so.6 (0x00007fa14abf3000)
	/lib64/ld-linux-x86-64.so.2 (0x00007fa14afd8000)
</source>
So, we can see that test-prog is '''not''' using the library <code>name</code> dynamically.

===Size matters===

Let's look at the size of the compiled c file (<code>name.c</code>), the archive and the program

<source lang="bash">
$ du -sb name.o libname.a test-prog 
4624	name.o
4956	libname.a
13256	test-prog
</source>
The object file (<code>name.o</code>) and the archive (<code>libname.a</code>) more or less have the same size. A good guess is that the archive has some additional information. The program has size '''13256'''. Let's compare this to when using the object files directly (previous section). Then the size was '''13256'''. Oh what a coincidence (''co-inky-dinky'') :).

== Shared library ==

===Compile name.c and create a shared library===

Compile name.c and create an shared library:
<source lang="bash">
$ gcc -c -fPIC ${CFLAGS} -I. -I.. name.c -o name.o
$ gcc -shared -o libname.so name.o
</source>

===Examine the shared library===
Get someinformation about the shared library
<source lang="bash">
$ file libname.so 
libname.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=c4d19ab6e8c97cc5623bc112524c1ec2e11a0380, not stripped
</source>
The words <code>shared object</code> tell us that this is a shared library.

Let's analys it the same way we did with the archive:
<source lang="bash">
$ $ nm libname.so | grep " [tT] name" 
0000000000000a3f T name_list_add
0000000000000bd1 T name_list_at_position
0000000000000e8a T name_list_free
00000000000009fa T name_list_new
0000000000000d6f T name_list_positon
0000000000000df7 T name_list_print
0000000000000b76 T name_list_remove
0000000000000c0c T name_list_remove_at
0000000000000f2a T name_list_size
</source>

===Use the archive / link the program===
<source lang="bash">
$ gcc -fPIC ${CFLAGS} -I. -I.. test/test-name.c -L. -lname -o test-prog
</source>

Let's see what libraries this program depends on:
<source lang="bash">
$ ldd test-prog 
	linux-vdso.so.1 (0x00007ffc37357000)
	libname.so => not found
	libc.so.6 => /lib64/libc.so.6 (0x00007f83ba538000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f83ba91d000)
</source>
The programs uses four dynamic libraries. This may vary depending on your operating system. The important thing to note here is that we can see <code>libname.so</code> listed there. Why "not found".... you'll find out in the next section.

If we were to examine a program linked with the archive the same way we'd get something like this:
<source lang="bash">
$ ldd test-prog 
	linux-vdso.so.1 (0x00007ffdb1fbe000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f5da05a3000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f5da0988000)
</source>
We can see that <code>libname.so</code> is not listed.

===Execute the program (linked with the shared library)===
Let's execute our program.
<source lang="bash">
$ ./test-prog 
./test-prog: error while loading shared libraries: libname.so: cannot open shared object file: No such file or directory
</source>
Uh oh. Kind of reminds about when bash can not find the program to start ([[FAQ#xxx:_command_not_found|Command not found]]). This time bash can find the program <code>test-prog</code> but it can't find the shared library. So, just as we need to tell bash that it can search for programs in the PATH variable we need to give the dynamic linker (a tool responsible for loading dynamic/shared libraries at run-time). We do this by using the <code>LD_LIBRARY_PATH</code> variable.

<source lang="bash">
$ export LD_LIBRARY_PATH=.
$ ./test-prog 
</source>
Et voila. It works :)

This could be done as a one-liner:
<source lang="bash">
$ LD_LIBRARY_PATH=. ./test-prog 
</source>
''Note: the command above only sets the variable once (before executing the program).''

===Examine the program (linked with the shared library)===
Let's look at what functions are defined in the program
<source lang="bash">
$ nm test-prog  | grep name_
                 U name_list_add
                 U name_list_at_position
                 U name_list_free
                 U name_list_new
                 U name_list_print
                 U name_list_remove
                 U name_list_remove_at
                 U name_list_size
</source>
Now the functions in the file <code>name.c</code> are not defined in the program. They are used and need to be defined elsewhere (in the shared library).
 
===Size matter===
Let's look at the size of the compiled c file (<code>name.c</code>), the shared library and the program

<source lang="bash">
$ du -sb name.o libname.so test-prog
4624	name.o
12736	libname.so
12840	test-prog
</source>
The object file (<code>name.o</code>) and the archive (<code>libname.a</code>) have different sizes, fact is that the shared library is roughly 3 times as big. The program has size '''12840'''. Let's compare this to when using the object files directly (previous section) and when using an archive. Then the size, in both cases, was '''13256'''. So using shared libraries with only one program using the shared library does not give us any advantage when it comes to saving disk space ;)

= Links =
[https://en.wikipedia.org/wiki/Library_(computing) Library] (wikipedia)

[https://en.wikipedia.org/wiki/Static_library Static_library] (wikipedia)

Source code: [https://github.com/progund/programming-with-c/tree/master/libraries https://github.com/progund/programming-with-c/tree/master/libraries] (Our Github)

='''TODO:''' (Possibly)=
Other useful tools:
* readelf (-h | main -d | ...)
* ldd -d test-prog (report missing objects)
* ldd -r test-prog (report missing objects and functions)
* ltrace?
* strace
* truss? Just kidding