Jon Schutz Technical Notes and Recommendations

Scribe is a log aggregator, developed at Facebook and released as open source. Scribe is built on Thrift, a cross-language RPC type platform, and therefore it is possible to use scribe with any of the Thrift-supported languages. Whilst Perl is one of the supported languages, there is little in the way of working examples, so here’s how I did it:

1. Install Thrift.
2. Build and install FB303 perl modules

     cd thrift/contrib/fb303
     # Edit if/fb303.thrift and add the line 'namespace perl Facebook.FB303' after the other namespace declarations
     thrift --gen perl if/fb303.thrift
     sudo cp -a gen-perl/ /usr/local/lib/perl5/site_perl/5.10.0 # or wherever you keep your site perl
   

   This creates the modules Facebook::FB303::Constants, Facebook::FB303::FacebookService and Facebook::FB303::Types.

3. Install Scribe.
4. Build and install Scribe perl modules

     cd scribe
     # Edit if/scribe.thrift and add 'namespace perl Scribe.Thrift' after the other namespace declarations
     thrift -I /path/to/thrift/contrib/ --gen perl scribe.thrift
     sudo cp -a gen-perl/Scribe /usr/local/lib/perl5/site_perl/5.10.0/ # or wherever
   

This creates the modules Scribe::Thrift::Constants, Scribe::Thrift::scribe, Scribe::Thrift::Types.

Here is an example program that uses the client (reading one line at a time from stdin and sending to a scribe instance running locally on port 1465):

#! /usr/bin/perl

use Scribe::Thrift::scribe;
use Thrift::Socket;
use Thrift::FramedTransport;
use Thrift::BinaryProtocol;
use strict;
use warnings;

my $host = 'localhost';
my $port = 1465;
my $cat = $ARGV[0] || 'test';

my $socket = Thrift::Socket->new($host, $port);
my $transport = Thrift::FramedTransport->new($socket);
my $proto = Thrift::BinaryProtocol->new($transport);

my $client = Scribe::Thrift::scribeClient->new($proto, $proto);
my $le = Scribe::Thrift::LogEntry->new({ category => $cat });

$transport->open();

while (my $line = <>) {
    $le->message($line);
    my $result = $client->Log([ $le ]);
    if ($result == Scribe::Thrift::ResultCode::TRY_LATER) {
	print STDERR "TRY_LATER\n";
    }
    elsif ($result != Scribe::Thrift::ResultCode::OK) {
	print STDERR "Unknown result code: $result\n";
    }
}

$transport->close();

UPDATE Log::Dispatch::Scribe is now available on CPAN. Also works with Log::Log4perl. Note though, you still need to install Thrift and Scribe perl modules as described above.