无法在 ubuntu 14.04.1 上安装 fluentd redshift 插件找不到 PostgreSQL 客户端库 (libpq)

Failed to Install fluentd redshift plugin on ubuntu 14.04.1 Can't find the PostgreSQL client library (libpq)

这是一个全新的 AWS ubuntu 14.04.1 VM。

启动后,尝试安装 fluentd redshift 插件。

sudo su 
curl -L http://toolbelt.treasuredata.com/sh/install-ubuntu-trusty-td-agent2.sh | sh 
apt-get install libpq-dev
apt-get install ruby ruby-dev
apt-get install make
gem install pg
/usr/sbin/td-agent-gem install fluent-plugin-redshift

应该很容易重现。请帮忙。非常感谢。

root@ip-172-30-0-131:/home/ubuntu# sudo /usr/sbin/td-agent-gem install fluent-plugin-redshift
sudo: unable to resolve host ip-172-30-0-131
Building native extensions.  This could take a while...
ERROR:  Error installing fluent-plugin-redshift:
    ERROR: Failed to build gem native extension.

    /opt/td-agent/embedded/bin/ruby extconf.rb
checking for pg_config... yes
Using config values from /usr/bin/pg_config
checking for libpq-fe.h... yes
checking for libpq/libpq-fs.h... yes
checking for pg_config_manual.h... yes
checking for PQconnectdb() in -lpq... no
checking for PQconnectdb() in -llibpq... no
checking for PQconnectdb() in -lms/libpq... no
Can't find the PostgreSQL client library (libpq)
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
need configuration options.

Provided configuration options:
    --with-opt-dir
    --with-opt-include
    --without-opt-include=${opt-dir}/include
    --with-opt-lib
    --without-opt-lib=${opt-dir}/lib
    --with-make-prog
    --without-make-prog
    --srcdir=.
    --curdir
    --ruby=/opt/td-agent/embedded/bin/ruby
    --with-pg
    --without-pg
    --with-pg-config
    --without-pg-config
    --with-pg_config
    --without-pg_config
    --with-pg-dir
    --without-pg-dir
    --with-pg-include
    --without-pg-include=${pg-dir}/include
    --with-pg-lib
    --without-pg-lib=${pg-dir}/lib
    --with-pqlib
    --without-pqlib
    --with-libpqlib
    --without-libpqlib
    --with-ms/libpqlib
    --without-ms/libpqlib

extconf failed, exit code 1

Gem files will remain installed in /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/pg-0.17.2.pre.546 for inspection.
Results logged to /opt/td-agent/embedded/lib/ruby/gems/2.1.0/extensions/x86_64-linux/2.1.0/pg-0.17.2.pre.546/gem_make.out

mkft.log 文件包含以下内容:

find_executable: checking for pg_config... -------------------- yes

--------------------

find_header: checking for libpq-fe.h... -------------------- yes

"gcc -o conftest -I/opt/td-agent/embedded/include/ruby-2.1.0/x86_64-linux -I/opt/td-agent/embedded/include/ruby-2.1.0/ruby/backward -I/opt/td-agent/embedded/include/ruby-2.1.0 -I. -I/usr/include/postgresql  -I/opt/td-agent/embedded/include   -I/opt/td-agent/embedded/include -O3 -g -pipe -fPIC conftest.c  -L. -L/opt/td-agent/embedded/lib -Wl,-R/opt/td-agent/embedded/lib -L/usr/lib -Wl,-R/usr/lib -L/opt/td-agent/embedded/lib -Wl,-R/opt/td-agent/embedded/lib -L. -Wl,-rpath,/opt/td-agent/embedded/lib -fstack-protector -L/opt/td-agent/embedded/lib -rdynamic -Wl,-export-dynamic -L/opt/td-agent/embedded/lib  -Wl,-R/opt/td-agent/embedded/lib      -Wl,-R -Wl,/opt/td-agent/embedded/lib -L/opt/td-agent/embedded/lib -lruby  -lpthread -ldl -lcrypt -lm   -lc"
checked program was:
/* begin */
1: #include "ruby.h"
2: 
3: int main(int argc, char **argv)
4: {
5:   return 0;
6: }
/* end */

"gcc -E -I/opt/td-agent/embedded/include/ruby-2.1.0/x86_64-linux -I/opt/td-agent/embedded/include/ruby-2.1.0/ruby/backward -I/opt/td-agent/embedded/include/ruby-2.1.0 -I. -I/usr/include/postgresql  -I/opt/td-agent/embedded/include   -I/opt/td-agent/embedded/include -O3 -g -pipe -fPIC  conftest.c -o conftest.i"
checked program was:
/* begin */
1: #include "ruby.h"
2: 
3: #include <libpq-fe.h>
/* end */

--------------------

find_header: checking for libpq/libpq-fs.h... -------------------- yes

"gcc -E -I/opt/td-agent/embedded/include/ruby-2.1.0/x86_64-linux -I/opt/td-agent/embedded/include/ruby-2.1.0/ruby/backward -I/opt/td-agent/embedded/include/ruby-2.1.0 -I. -I/usr/include/postgresql  -I/opt/td-agent/embedded/include   -I/opt/td-agent/embedded/include -O3 -g -pipe -fPIC  conftest.c -o conftest.i"
checked program was:
/* begin */
1: #include "ruby.h"
2: 
3: #include <libpq/libpq-fs.h>
/* end */

--------------------

find_header: checking for pg_config_manual.h... -------------------- yes

"gcc -E -I/opt/td-agent/embedded/include/ruby-2.1.0/x86_64-linux -I/opt/td-agent/embedded/include/ruby-2.1.0/ruby/backward -I/opt/td-agent/embedded/include/ruby-2.1.0 -I. -I/usr/include/postgresql  -I/opt/td-agent/embedded/include   -I/opt/td-agent/embedded/include -O3 -g -pipe -fPIC  conftest.c -o conftest.i"
checked program was:
/* begin */
1: #include "ruby.h"
2: 
3: #include <pg_config_manual.h>
/* end */

--------------------

have_library: checking for PQconnectdb() in -lpq... -------------------- no

"gcc -o conftest -I/opt/td-agent/embedded/include/ruby-2.1.0/x86_64-linux -I/opt/td-agent/embedded/include/ruby-2.1.0/ruby/backward -I/opt/td-agent/embedded/include/ruby-2.1.0 -I. -I/usr/include/postgresql  -I/opt/td-agent/embedded/include   -I/opt/td-agent/embedded/include -O3 -g -pipe -fPIC conftest.c  -L. -L/opt/td-agent/embedded/lib -Wl,-R/opt/td-agent/embedded/lib -L/usr/lib -Wl,-R/usr/lib -L/opt/td-agent/embedded/lib -Wl,-R/opt/td-agent/embedded/lib -L. -Wl,-rpath,/opt/td-agent/embedded/lib -fstack-protector -L/opt/td-agent/embedded/lib -rdynamic -Wl,-export-dynamic -L/opt/td-agent/embedded/lib  -Wl,-R/opt/td-agent/embedded/lib      -Wl,-R -Wl,/opt/td-agent/embedded/lib -L/opt/td-agent/embedded/lib -lruby -lpq  -lpthread -ldl -lcrypt -lm   -lc"
/usr/lib/libpq.so: undefined reference to `SSL_CTX_use_certificate_chain_file@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_write@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_set_fd@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_use_PrivateKey_file@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `CRYPTO_set_locking_callback@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `X509_NAME_get_text_by_NID@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_connect@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `ENGINE_init@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `X509_STORE_load_locations@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_CTX_get_cert_store@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_ctrl@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_free@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_library_init@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_CTX_ctrl@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `ERR_get_error@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_pending@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `ENGINE_free@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `X509_get_subject_name@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_use_certificate_file@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_check_private_key@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_load_error_strings@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `ENGINE_by_id@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_get_peer_certificate@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_CTX_new@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `CRYPTO_num_locks@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `ENGINE_load_private_key@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `ENGINE_finish@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_set_verify@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `X509_free@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `CRYPTO_set_id_callback@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_get_error@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_new@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_shutdown@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_use_PrivateKey@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `TLSv1_method@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `X509_STORE_set_flags@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_read@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `OPENSSL_config@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_CTX_load_verify_locations@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `ERR_reason_error_string@OPENSSL_1.0.0'
/usr/lib/libpq.so: undefined reference to `SSL_set_ex_data@OPENSSL_1.0.0'
collect2: error: ld returned 1 exit status
checked program was:
/* begin */
 1: #include "ruby.h"
 2: 
 3: #include <libpq-fe.h>
 4: 
 5: /*top*/
 6: extern int t(void);
 7: int main(int argc, char **argv)
 8: {
 9:   if (argc > 1000000) {
10:     printf("%p", &t);
11:   }
12: 
13:   return 0;
14: }
15: int t(void) { void ((*volatile p)()); p = (void ((*)()))PQconnectdb; return 0; }
/* end */

"gcc -o conftest -I/opt/td-agent/embedded/include/ruby-2.1.0/x86_64-linux -I/opt/td-agent/embedded/include/ruby-2.1.0/ruby/backward -I/opt/td-agent/embedded/include/ruby-2.1.0 -I. -I/usr/include/postgresql  -I/opt/td-agent/embedded/include   -I/opt/td-agent/embedded/include -O3 -g -pipe -fPIC conftest.c  -L. -L/opt/td-agent/embedded/lib -Wl,-R/opt/td-agent/embedded/lib -L/usr/lib -Wl,-R/usr/lib -L/opt/td-agent/embedded/lib -Wl,-R/opt/td-agent/embedded/lib -L. -Wl,-rpath,/opt/td-agent/embedded/lib -fstack-protector -L/opt/td-agent/embedded/lib -rdynamic -Wl,-export-dynamic -L/opt/td-agent/embedded/lib  -Wl,-R/opt/td-agent/embedded/lib      -Wl,-R -Wl,/opt/td-agent/embedded/lib -L/opt/td-agent/embedded/lib -lruby -lpq  -lpthread -ldl -lcrypt -lm   -lc"
conftest.c: In function ‘t’:
conftest.c:15:1: error: too few arguments to function ‘PQconnectdb’
 int t(void) { PQconnectdb(); return 0; }
 ^
In file included from conftest.c:3:0:
/usr/include/postgresql/libpq-fe.h:250:16: note: declared here
 extern PGconn *PQconnectdb(const char *conninfo);
                ^
checked program was:
/* begin */
 1: #include "ruby.h"
 2: 
 3: #include <libpq-fe.h>
 4: 
 5: /*top*/
 6: extern int t(void);
 7: int main(int argc, char **argv)
 8: {
 9:   if (argc > 1000000) {
10:     printf("%p", &t);
11:   }
12: 
13:   return 0;
14: }
15: int t(void) { PQconnectdb(); return 0; }
/* end */

--------------------

have_library: checking for PQconnectdb() in -llibpq... -------------------- no

"gcc -o conftest -I/opt/td-agent/embedded/include/ruby-2.1.0/x86_64-linux -I/opt/td-agent/embedded/include/ruby-2.1.0/ruby/backward -I/opt/td-agent/embedded/include/ruby-2.1.0 -I. -I/usr/include/postgresql  -I/opt/td-agent/embedded/include   -I/opt/td-agent/embedded/include -O3 -g -pipe -fPIC conftest.c  -L. -L/opt/td-agent/embedded/lib -Wl,-R/opt/td-agent/embedded/lib -L/usr/lib -Wl,-R/usr/lib -L/opt/td-agent/embedded/lib -Wl,-R/opt/td-agent/embedded/lib -L. -Wl,-rpath,/opt/td-agent/embedded/lib -fstack-protector -L/opt/td-agent/embedded/lib -rdynamic -Wl,-export-dynamic -L/opt/td-agent/embedded/lib  -Wl,-R/opt/td-agent/embedded/lib      -Wl,-R -Wl,/opt/td-agent/embedded/lib -L/opt/td-agent/embedded/lib -lruby -llibpq  -lpthread -ldl -lcrypt -lm   -lc"
/usr/bin/ld: cannot find -llibpq
collect2: error: ld returned 1 exit status
checked program was:
/* begin */
 1: #include "ruby.h"
 2: 
 3: #include <libpq-fe.h>
 4: 
 5: /*top*/
 6: extern int t(void);
 7: int main(int argc, char **argv)
 8: {
 9:   if (argc > 1000000) {
10:     printf("%p", &t);
11:   }
12: 
13:   return 0;
14: }
15: int t(void) { void ((*volatile p)()); p = (void ((*)()))PQconnectdb; return 0; }
/* end */

"gcc -o conftest -I/opt/td-agent/embedded/include/ruby-2.1.0/x86_64-linux -I/opt/td-agent/embedded/include/ruby-2.1.0/ruby/backward -I/opt/td-agent/embedded/include/ruby-2.1.0 -I. -I/usr/include/postgresql  -I/opt/td-agent/embedded/include   -I/opt/td-agent/embedded/include -O3 -g -pipe -fPIC conftest.c  -L. -L/opt/td-agent/embedded/lib -Wl,-R/opt/td-agent/embedded/lib -L/usr/lib -Wl,-R/usr/lib -L/opt/td-agent/embedded/lib -Wl,-R/opt/td-agent/embedded/lib -L. -Wl,-rpath,/opt/td-agent/embedded/lib -fstack-protector -L/opt/td-agent/embedded/lib -rdynamic -Wl,-export-dynamic -L/opt/td-agent/embedded/lib  -Wl,-R/opt/td-agent/embedded/lib      -Wl,-R -Wl,/opt/td-agent/embedded/lib -L/opt/td-agent/embedded/lib -lruby -llibpq  -lpthread -ldl -lcrypt -lm   -lc"
conftest.c: In function ‘t’:
conftest.c:15:1: error: too few arguments to function ‘PQconnectdb’
 int t(void) { PQconnectdb(); return 0; }
 ^
In file included from conftest.c:3:0:
/usr/include/postgresql/libpq-fe.h:250:16: note: declared here
 extern PGconn *PQconnectdb(const char *conninfo);
                ^
checked program was:
/* begin */
 1: #include "ruby.h"
 2: 
 3: #include <libpq-fe.h>
 4: 
 5: /*top*/
 6: extern int t(void);
 7: int main(int argc, char **argv)
 8: {
 9:   if (argc > 1000000) {
10:     printf("%p", &t);
11:   }
12: 
13:   return 0;
14: }
15: int t(void) { PQconnectdb(); return 0; }
/* end */

--------------------

have_library: checking for PQconnectdb() in -lms/libpq... -------------------- no

"gcc -o conftest -I/opt/td-agent/embedded/include/ruby-2.1.0/x86_64-linux -I/opt/td-agent/embedded/include/ruby-2.1.0/ruby/backward -I/opt/td-agent/embedded/include/ruby-2.1.0 -I. -I/usr/include/postgresql  -I/opt/td-agent/embedded/include   -I/opt/td-agent/embedded/include -O3 -g -pipe -fPIC conftest.c  -L. -L/opt/td-agent/embedded/lib -Wl,-R/opt/td-agent/embedded/lib -L/usr/lib -Wl,-R/usr/lib -L/opt/td-agent/embedded/lib -Wl,-R/opt/td-agent/embedded/lib -L. -Wl,-rpath,/opt/td-agent/embedded/lib -fstack-protector -L/opt/td-agent/embedded/lib -rdynamic -Wl,-export-dynamic -L/opt/td-agent/embedded/lib  -Wl,-R/opt/td-agent/embedded/lib      -Wl,-R -Wl,/opt/td-agent/embedded/lib -L/opt/td-agent/embedded/lib -lruby -lms/libpq  -lpthread -ldl -lcrypt -lm   -lc"
/usr/bin/ld: cannot find -lms/libpq
collect2: error: ld returned 1 exit status
checked program was:
/* begin */
 1: #include "ruby.h"
 2: 
 3: #include <libpq-fe.h>
 4: 
 5: /*top*/
 6: extern int t(void);
 7: int main(int argc, char **argv)
 8: {
 9:   if (argc > 1000000) {
10:     printf("%p", &t);
11:   }
12: 
13:   return 0;
14: }
15: int t(void) { void ((*volatile p)()); p = (void ((*)()))PQconnectdb; return 0; }
/* end */

"gcc -o conftest -I/opt/td-agent/embedded/include/ruby-2.1.0/x86_64-linux -I/opt/td-agent/embedded/include/ruby-2.1.0/ruby/backward -I/opt/td-agent/embedded/include/ruby-2.1.0 -I. -I/usr/include/postgresql  -I/opt/td-agent/embedded/include   -I/opt/td-agent/embedded/include -O3 -g -pipe -fPIC conftest.c  -L. -L/opt/td-agent/embedded/lib -Wl,-R/opt/td-agent/embedded/lib -L/usr/lib -Wl,-R/usr/lib -L/opt/td-agent/embedded/lib -Wl,-R/opt/td-agent/embedded/lib -L. -Wl,-rpath,/opt/td-agent/embedded/lib -fstack-protector -L/opt/td-agent/embedded/lib -rdynamic -Wl,-export-dynamic -L/opt/td-agent/embedded/lib  -Wl,-R/opt/td-agent/embedded/lib      -Wl,-R -Wl,/opt/td-agent/embedded/lib -L/opt/td-agent/embedded/lib -lruby -lms/libpq  -lpthread -ldl -lcrypt -lm   -lc"
conftest.c: In function ‘t’:
conftest.c:15:1: error: too few arguments to function ‘PQconnectdb’
 int t(void) { PQconnectdb(); return 0; }
 ^
In file included from conftest.c:3:0:
/usr/include/postgresql/libpq-fe.h:250:16: note: declared here
 extern PGconn *PQconnectdb(const char *conninfo);
                ^
checked program was:
/* begin */
 1: #include "ruby.h"
 2: 
 3: #include <libpq-fe.h>
 4: 
 5: /*top*/
 6: extern int t(void);
 7: int main(int argc, char **argv)
 8: {
 9:   if (argc > 1000000) {
10:     printf("%p", &t);
11:   }
12: 
13:   return 0;
14: }
15: int t(void) { PQconnectdb(); return 0; }
/* end */

--------------------

这是因为omnibus 依赖于自己的OpenSSL 库,而libpq 需要系统OpenSSL。 这是当前基于综合包的普遍限制。 看到这个问题:https://bitbucket.org/ged/ruby-pg/issue/178/pgs-way-of-linking-to-openssl-is-not

有人用这个脚本安装pg gem: https://gist.github.com/repeatedly/d4c9f9084f321f5a95e3
fluent-plugin-redshift 需要 v0.17,因此需要在最后一行添加“-v 0.17.1”。

我们现在正在考虑通过包含 postgresql 或更改包依赖性来避免此问题。