ATS crashes when Metalink Enabled in plugins.config

[Degreane]

Hi
I downloaded several versions of ATS that are available (4.1.x ~ 4.2.x ) and all compiled with --enable-experimental-plugins as described in the manual ./configure --help shows

nevertheless whenever i enable the plugin metalink traffic_server crashes and the manager keeps restarting it

tried it with different versions of linux. (ubuntu 13.10 - fedora 18,19,20) same issue

Much Regards

[Jan 27 18:47:51.693] Manager {0xb7609780} NOTE: [ClusterCom::ClusterCom] Node running on OS: 'Linux' Release: '3.12.8-300.fc20.i686+PAE'
[Jan 27 18:47:51.693] Manager {0xb7609780} NOTE: [LocalManager::listenForProxy] Listening on port: 8080
[Jan 27 18:47:51.693] Manager {0xb7609780} NOTE: [TrafficManager] Setup complete
[Jan 27 18:47:52.712] Manager {0xb7609780} NOTE: [LocalManager::startProxy] Launching ts process
[Jan 27 18:47:52.717] Manager {0xb7609780} NOTE: [LocalManager::pollMgmtProcessServer] New process connecting fd '13'
[Jan 27 18:47:52.717] Manager {0xb7609780} NOTE: [Alarms::signalAlarm] Server Process born

[Jan 27 18:48:35.991] Manager {0xb7609780} FATAL: [LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
[Jan 27 18:48:35.991] Manager {0xb7609780} NOTE: [LocalManager::mgmtShutdown] Executing shutdown request.
[Jan 27 18:48:35.991] Manager {0xb7609780} NOTE: [LocalManager::processShutdown] Executing process shutdown request.
[Jan 27 18:48:35.991] Manager {0xb7609780} ERROR: [LocalManager::sendMgmtMsgToProcesses] Error writing message
[Jan 27 18:48:35.991] Manager {0xb7609780} ERROR: (last system error 32: Broken pipe)

hopefully you may check it out and get it fixed ..

[antbryan]

from @jablko

I pushed the updated plugin to GitHub (hopefully it will also be distributed with the next Traffic Server release).
To build it,

1. download the updated metalink.cc file
2. replace the plugins/experimental/metalink/metalink.cc file in your Traffic Server source tree with the updated file,
3. and rebuild Traffic Server by rerunning "make".

[Degreane]

excuse me but when did you update it ???

much regards

     Faysal Banna

Meteorological Services
Rafic Harriri International Airport
Beirut - Lebanon
Mob: +961-3-258043

On 02/07/2014 10:50 PM, antbryan wrote:

from @jablko https://github.com/jablko

I pushed the updated plugin to GitHub (hopefully it will also be
distributed with the next Traffic Server release).
To build it,

1. download the updated metalink.cc file
2. replace the plugins/experimental/metalink/metalink.cc file in your
   Traffic Server source tree with the updated file,
3. and rebuild Traffic Server by rerunning "make".

—
Reply to this email directly or view it on GitHub
#1 (comment).

[antbryan]

if you look at the commit log, you can see that @jablko updated it a few days ago on 4 & 5 Feb.

have you tried the new code since then?

it's also been included upstream, so hopefully should be available w/ the next ATS release: https://git-wip-us.apache.org/repos/asf?p=trafficserver.git

[Degreane]

I already used this metalink.cc with ATS 5.0.x and 4.1.2 and both give this error below when i activate the metalink plugin...

did you compile it and test it yourself ?

anyhow the error is as shows
[TrafficServer] using root directory '/usr/local'
FATAL: InkAPI.cc:6248: failed assert sdk_sanity_check_iocore_structure(connp) == TS_SUCCESS
/usr/local/bin/traffic_server - STACK TRACE:
/usr/local/lib/libtsutil.so.4(+0xf36b)[0x4001136b]
/usr/local/lib/libtsutil.so.4(+0xe23d)[0x4001023d]
/usr/local/bin/traffic_server(TSVConnWrite+0x2c9)[0x80f4489]
/usr/local/libexec/trafficserver/metalink.so(+0x1d0f)[0x40047d0f]
/usr/local/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x15f)[0x83087bf]
/usr/local/bin/traffic_server(_ZN7EThread7executeEv+0x7a3)[0x83093c3]
/usr/local/bin/traffic_server(main+0xc6a)[0x80bc04a]
/lib/libc.so.6(__libc_start_main+0xf3)[0x47794b73]
/usr/local/bin/traffic_server[0x80c134d]
[TrafficServer] using root directory '/usr/local'
FATAL: InkAPI.cc:6248: failed assert sdk_sanity_check_iocore_structure(connp) == TS_SUCCESS
/usr/local/bin/traffic_serverFATAL: InkAPI.cc:6248: failed assert sdk_sanity_check_iocore_structure(connp) == TS_SUCCESS

• STACK TRACE:
  /usr/local/bin/traffic_server - STACK TRACE:
  /usr/local/lib/libtsutil.so.4(+0xf36b)[0x4001136b]
  /usr/local/lib/libtsutil.so.4(+0xe23d)[0x4001023d]
  /usr/local/bin/traffic_server(TSVConnWrite+0x2c9)[0x80f4489]
  /usr/local/libexec/trafficserver/metalink.so(+0x1d0f)[0x40047d0f]
  /usr/local/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x15f)[0x83087bf]
  /usr/local/bin/traffic_server(_ZN7EThread7executeEv+0x7a3)[0x83093c3]
  /usr/local/bin/traffic_server[0x830830b]
  /lib/libpthread.so.0[0x47948d8a]
  /lib/libc.so.6(clone+0x5e)[0x47872a0e]
  [TrafficServer] using root directory '/usr/local'
  FATAL: InkAPI.cc:6248: failed assert sdk_sanity_check_iocore_structure(connp) == TS_SUCCESS
  /usr/local/bin/traffic_server - STACK TRACE:
  /usr/local/lib/libtsutil.so.4(+0xf36b)[0x4001136b]
  /usr/local/lib/libtsutil.so.4(+0xe23d)[0x4001023d]
  /usr/local/bin/traffic_server(TSVConnWrite+0x2c9)[0x80f4489]
  /usr/local/libexec/trafficserver/metalink.so(+0x1d0f)[0x4004cd0f]
  /usr/local/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x15f)[0x83087bf]
  FATAL: InkAPI.cc:6248: failed assert sdk_sanity_check_iocore_structure(connp) == TS_SUCCESS
  /usr/local/bin/traffic_server - STACK TRACE:
  /usr/local/bin/traffic_server(_ZN7EThread7executeEv+0x7a3)[0x83093c3]
  /usr/local/lib/libtsutil.so.4(+0xf36b)[0x4001136b]
  /usr/local/lib/libtsutil.so.4(+0xe23d)[0x4001023d]
  /usr/local/bin/traffic_server(TSVConnWrite+0x2c9)[0x80f4489]
  /usr/local/libexec/trafficserver/metalink.so(+0x1d0f)[0x4004cd0f]
  /usr/local/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x15f)[0x83087bf]
  /usr/local/bin/traffic_server(_ZN7EThread7executeEv+0x7a3)[0x83093c3]
  /usr/local/bin/traffic_server[0x830830b]
  /lib/libpthread.so.0[0x47948d8a]
  /lib/libc.so.6(clone+0x5e)[0x47872a0e]
  [TrafficServer] using root directory '/usr/local'
  [TrafficServer] using root directory '/usr/local'
  FATAL: InkAPI.cc:6248: failed assert sdk_sanity_check_iocore_structure(connp) == TS_SUCCESS
  /usr/local/bin/traffic_server - STACK TRACE:
  /usr/local/lib/libtsutil.so.4(+0xf36b)[0x4001136b]
  /usr/local/lib/libtsutil.so.4(+0xe23d)[0x4001023d]
  /usr/local/bin/traffic_server(TSVConnWrite+0x2c9)[0x80f4489]
  /usr/local/libexec/trafficserver/metalink.so(+0x1d0f)[0x40047d0f]
  /usr/local/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x15f)[0x83087bf]
  /usr/local/bin/traffic_server(_ZN7EThread7executeEv+0x7a3)[0x83093c3]
  /usr/local/bin/traffic_server[0x830830b]
  /lib/libpthread.so.0[0x47948d8a]
  /lib/libc.so.6(clone+0x5e)[0x47872a0e]
  [TrafficServer] using root directory '/usr/local'
  FATAL: InkAPI.cc:6248: failed assert sdk_sanity_check_iocore_structure(connp) == TS_SUCCESS
  /usr/local/bin/traffic_server - STACK TRACE:
  /usr/local/lib/libtsutil.so.4(+0xf36b)[0x4001136b]
  /usr/local/lib/libtsutil.so.4(+0xe23d)[0x4001023d]
  /usr/local/bin/traffic_server(TSVConnWrite+0x2c9)[0x80f4489]
  /usr/local/libexec/trafficserver/metalink.so(+0x1d0f)[0x40047d0f]
  /usr/local/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x15f)[0x83087bf]
  /usr/local/bin/traffic_server(_ZN7EThread7executeEv+0x7a3)[0x83093c3]
  /usr/local/bin/traffic_server(main+0xc6a)[0x80bc04a]
  /lib/libc.so.6(__libc_start_main+0xf3)[0x47794b73]
  /usr/local/bin/traffic_server[0x80c134d]
  [TrafficServer] using root directory '/usr/local'

[Degreane]

try to compile it yourself and when you get it running and tested please post your procedure here so we may get it running ....

tested with ATS 4.1.2 and 5.0.x and still same error

what debug procedure you wish me to do so we can ease up the issue of fixing it up ?

[kieronrob]

Hi,

I get these errors when the plugin is enabled on ATS 4.2.1:

FATAL: InkAPI.cc:6261: failed assert sdk_sanity_check_iocore_structure(connp) == TS_SUCCESS
/usr/local/bin/traffic_server - STACK TRACE:
/usr/local/lib/libtsutil.so.4(+0x1f5d8)[0x2b82d6c3c5d8]
/usr/local/lib/libtsutil.so.4(+0x1da3f)[0x2b82d6c3aa3f]
/usr/local/bin/traffic_server(TSVConnWrite+0x11f)[0x4b507f]
/usr/local/libexec/trafficserver/metalink.so(+0x1ffc)[0x2b82de5f4ffc]
/usr/local/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x8f)[0x6ab7cf]
/usr/local/bin/traffic_server(_ZN7EThread7executeEv+0x61b)[0x6ac2fb]
/usr/local/bin/traffic_server(main+0xe1b)[0x4c511b]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x3d2921ed1d]
/usr/local/bin/traffic_server[0x487e69]

Please let me know if you need any additional info.

[jablko]

Hi, are you able to recompile Traffic Server?
If so, can you please try the following,

1. download the latest metalink.cc code [1]
2. replace the plugins/experimental/metalink/metalink.cc in your Traffic Server source tree with the latest metalink.cc code
3. and recompile Traffic Server by running "make"

[1] https://raw.github.com/jablko/dedup/master/metalink.cc

After recompiling and restarting Traffic Server, do you still get these errors?

[kieronrob]

Hi Jack,

Recompiled ATS with the updated metalink.cc and the erors are no longer coming up.

Thank you for the swift reply and resolution as this is a feature I really think is valuable.

Regards,

Kieron

[jablko]

Great!
If you encounter any more problems, please let me know.
The latest metalink.cc code should be included in the next major Traffic Server release (5.X)

[kieronrob]

Hi Jack,

I have setup a new box with ATS and included the new metalink.cc in the
/plugins/experimental/metalink directory before running ./configure
--enable--experimental-plugins.

I am still getting the same error:

FATAL: InkAPI.cc:4534: failed assert sdk_sanity_check_txn(txnp) == TS_SUCCESS
/usr/local/bin/traffic_server - STACK TRACE:
/usr/local/lib/libtsutil.so.4(+0x1a9d4)[0x4001c9d4]
/usr/local/lib/libtsutil.so.4(+0x191dd)[0x4001b1dd]
/usr/local/bin/traffic_server(TSHttpTxnClientReqGet+0x32)[0x80df972]
/usr/local/libexec/trafficserver/metalink.so(+0x18d6)[0x400348d6]
/usr/local/bin/traffic_server(_ZN7CacheVC8callcontEi+0x5a)[0x821ac9a]
/usr/local/bin/traffic_server(_ZN7CacheVC19openWriteStartBeginEiP5Event+0x288)[0x8297b38]
/usr/local/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x8c)[0x82dbd9c]
/usr/local/bin/traffic_server(_ZN7EThread7executeEv+0x5ec)[0x82dc87c]
/usr/local/bin/traffic_server[0x82db17c]
/lib/libpthread.so.0[0xbd1b39]
/lib/libc.so.6(clone+0x5e)[0xb0dd8e]

Are there any other things I can try as I would like to make use of the
plugin.

I am also using the cacheurl plugin and can send you a copy of my
records.config if it will help.

Kind regards,

Kieron

On Sun, May 11, 2014 at 11:12 PM, Jack Bates notifications@github.comwrote:

Great!
If you encounter any more problems, please let me know.
The latest metalink.cc code should be included in the next major Traffic
Server release (5.X)

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/1#issuecomment-42783451
.

Kieron Robertson

Man and Machine

Mail: kieron@manandmachine.biz
Cell: 083 309 3538

[jablko]

Hmm, first I'm curious if this crash is specific to the version of Traffic Server or not.
If you can recompile Traffic Server, can you please reproduce the crash with the latest code from the Traffic Server repository?

The following commands will check out and build the latest code from the repository:

$ git clone https://git-wip-us.apache.org/repos/asf/trafficserver.git
$ cd trafficserver
$ autoreconf -i
$ ./configure --enable-experimental-plugins
$ make

You don't need to replace the metalink.cc file in this case because the repository already contains the latest metalink.cc code.

Next, can you please run Traffic Server with the "http" debug tag and send me the output when it crashes?
Run traffic_server with the "-Thttp" switch or run traffic_manager with the "-debug http" option.

Thanks for reporting this crash. With the extra details above, I will work on fixing it.
If you have any questions or problems, please let me know.

[kieronrob]

Hi Jack,

Sorry for the long delay, but been a bit hectic. I have recompiled and
reinstalled Traffic Server as per your suggestions above and am now running
it with the trafficserver start -Thttp switch.

I will come back to you if it crashes and give you the logs.

Regards,

Kieron

On Fri, May 23, 2014 at 6:52 PM, Jack Bates notifications@github.com
wrote:

Hmm, first I'm curious if this crash is specific to the version of Traffic
Server or not.
If you can recompile Traffic Server, can you please reproduce the crash
with the latest code from the Traffic Server repository?

The following commands will check out and build the latest code from the
repository:

$ git clone https://git-wip-us.apache.org/repos/asf/trafficserver.git
$ cd trafficserver
$ autoreconf -i
$ ./configure --enable-experimental-plugins
$ make

You don't need to replace the metalink.cc file in this case because the
repository already contains the latest metalink.cc code.

Next, can you please run Traffic Server with the "http" debug tag and send
me the output when it crashes?
Run traffic_server with the "-Thttp" switch or run traffic_manager with
the "-debug http" option.

Thanks for reporting this crash. With the extra details above, I will work
on fixing it.
If you have any questions or problems, please let me know.

—
Reply to this email directly or view it on GitHub
#1 (comment).

Kieron Robertson

Man and Machine

Mail: kieron@manandmachine.biz
Cell: 083 309 3538

[kieronrob]

Hi,

OK - got a crash. Here is the output on the CLI:

[root@ats ~]# traffic_manager -debug http
[E. Mgmt] log ==> [TrafficManager] using root directory '/usr/local'
[Jun 3 10:14:50.803] Manager {0x7f4b7a9cb7e0} DEBUG: (http_tproxy) Listen
port 8080 inbound transparency enabled.
[TrafficServer] using root directory '/usr/local'
FATAL: InkAPI.cc:4535: failed assert sdk_sanity_check_txn(txnp) == TS_SUCCESS
/usr/local/bin/traffic_server - STACK TRACE:
/usr/local/lib/libtsutil.so.5(+0x1f5b8)[0x2b486fd7e5b8]
/usr/local/lib/libtsutil.so.5(+0x1da7f)[0x2b486fd7ca7f]
/usr/local/bin/traffic_server(TSHttpTxnClientReqGet+0x26)[0x4b6526]
/usr/local/libexec/trafficserver/metalink.so(+0x1fd2)[0x2b487c203fd2]
/usr/local/bin/traffic_server(_ZN7CacheVC8callcontEi+0x53)[0x669583]
/usr/local/bin/traffic_server(_ZN7CacheVC19openWriteStartBeginEiP5Event+0x2d8)[0x6e2f08]
/usr/local/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x8f)[0x7296bf]
/usr/local/bin/traffic_server(_ZN7EThread7executeEv+0x663)[0x72a233]
/usr/local/bin/traffic_server[0x728a6a]
/lib64/libpthread.so.0(+0x38a88079d1)[0x2b48705079d1]
/lib64/libc.so.6(clone+0x6d)[0x38a84e8b6d]

And the second SSH session I had open:

[connect] ERROR (main_socket_fd 3): Connection refused
traffic_top:
stats.h:251: void Stats::getStats(): Assertion `TSRecordGetInt(item.name,
&value) == TS_ERR_OKAY' failed.
Aborted (core dumped)
[root@ats ~]#

Manager Log:

[Jun 3 11:02:16.279] Manager {0x7f4b7a9cb7e0} FATAL:
[LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
[Jun 3 11:02:16.279] Manager {0x7f4b7a9cb7e0} NOTE:
[LocalManager::mgmtShutdown] Executing shutdown request.
[Jun 3 11:02:16.279] Manager {0x7f4b7a9cb7e0} NOTE:
[LocalManager::processShutdown] Executing process shutdown request.
[Jun 3 11:02:16.280] Manager {0x7f4b7a9cb7e0} ERROR:
[LocalManager::sendMgmtMsgToProcesses] Error writing message
[Jun 3 11:02:16.280] Manager {0x7f4b7a9cb7e0} ERROR: (last system error
32: Broken pipe)

Please let me know if you need anything else.

Regards,

Kieron

On Tue, Jun 3, 2014 at 10:12 AM, Kieron Robertson kieron@manandmachine.biz
wrote:

Hi Jack,

Sorry for the long delay, but been a bit hectic. I have recompiled and
reinstalled Traffic Server as per your suggestions above and am now running
it with the trafficserver start -Thttp switch.

I will come back to you if it crashes and give you the logs.

Regards,

Kieron

On Fri, May 23, 2014 at 6:52 PM, Jack Bates notifications@github.com
wrote:

Hmm, first I'm curious if this crash is specific to the version of
Traffic Server or not.
If you can recompile Traffic Server, can you please reproduce the crash
with the latest code from the Traffic Server repository?

The following commands will check out and build the latest code from the
repository:

$ git clone https://git-wip-us.apache.org/repos/asf/trafficserver.git
$ cd trafficserver
$ autoreconf -i
$ ./configure --enable-experimental-plugins
$ make

You don't need to replace the metalink.cc file in this case because the
repository already contains the latest metalink.cc code.

Next, can you please run Traffic Server with the "http" debug tag and
send me the output when it crashes?
Run traffic_server with the "-Thttp" switch or run traffic_manager with
the "-debug http" option.

Thanks for reporting this crash. With the extra details above, I will
work on fixing it.
If you have any questions or problems, please let me know.

—
Reply to this email directly or view it on GitHub
#1 (comment).

Kieron Robertson

Man and Machine

Mail: kieron@manandmachine.biz
Cell: 083 309 3538

Kieron Robertson

Man and Machine

Mail: kieron@manandmachine.biz
Cell: 083 309 3538

[jablko]

No sweat, thanks for reproducing the crash with the latest Traffic Server code.
I was wrong about how to turn on the "http" debug tag, sorry!
Can you please run either "trafficserver start" or traffic_manager with the following two environment variables and send me the output again?

PROXY_CONFIG_DIAGS_DEBUG_ENABLED=1 PROXY_CONFIG_DIAGS_DEBUG_TAGS=http

Thanks for your help tracking down this crash. If you have any questions or problems, please let me know.

[kieronrob]

Hi Jack,

Just an update - I haven't had another crash since the last one, so looking
good for the re-compiled ats from source.

I will use this method on another test appliance and give you feedback if I
see the crash again.

Kind regards,

Kieron

On Tue, Jun 3, 2014 at 10:21 PM, Jack Bates notifications@github.com
wrote:

No sweat, thanks for reproducing the crash with the latest Traffic Server
code.
I was wrong about how to turn on the "http" debug tag, sorry!
Can you please run either "trafficserver start" or traffic_manager with
the following two environment variables and send me the output again?

PROXY_CONFIG_DIAGS_DEBUG_ENABLED=1 PROXY_CONFIG_DIAGS_DEBUG_TAGS=http

Thanks for your help tracking down this crash. If you have any questions
or problems, please let me know.

—
Reply to this email directly or view it on GitHub
#1 (comment).

Kieron Robertson

Man and Machine

Mail: kieron@manandmachine.biz
Cell: 083 309 3538