Configuring Scribe

port=1463
max_msg_per_second=2000000
max_queue_size=10000000
check_interval=3

Store Configuration

Scribe Server determines how to log messages based on the Stores defined in the configuration. Every store must specify what message category it handles with two exceptions:
default store: The ‘default’ category handles any category that is not handled by any other store. There can only be one default store.

• category=default

prefix stores: If the specified category ends in a *, the store will handle all categories that begin with the specified prefix.

• category=web*

Store Configuration Variables

category: Determines which messages are handled by this store
type:

• Currently used types (defined in Store::createStore)
  1. file
  2. buffer
  3. network
  4. bucket
  5. thriftfile
  6. null
  7. multi

target_write_size: 16,384 bytes by default

• determines how large to let the message queue grow for a given category before processing the messages

max_write_interval: 10 seconds by default

• determines how long to let the messages queue for a given category before processing the messages

Example:

<store>
category=statistics
type=file
target_write_size=20480
max_write_interval=2

...

</store>

File Store Configuration

File Stores write messages to a file.

file_path: defaults to “/tmp”
base_filename: defaults to category name
rotate_period: “hourly”, “daily”, or “never”; “never” by default

• determines how often to create new files

rotate_hour: 0-23, 1 by default

• if rotation_period is daily, determines what hour of day to rotate

rotate_minute 0-59, 15 by default

• if rotation_period is daily or hourly, determines how many minutes after the hour to rotate

max_size: 1,000,000,000 bytes by default

• determines approximately how large to let a file grow before rotating to a new file

write_meta: “yes” or anything else; false by default

• whether to log the following metadata in each file:
  1. 1. the length of each message is prepended to the message as an unsigned integer
     2. if the file was rotated, the last line will contain “scribe_meta<new_logfile>: ” followed by the next filename

fs_type: currently only “std” is supported; “std” by default
chunk_size: 0 by default

• if a chunk size is specified, no messages within the file will cross chunk boundaries unless there are messages larger than the chunk size

add_newlines: 0 or 1, 0 by default

• if set to 1, will write a newline after every message

create_symlink: “yes” or anything else; “yes” by default

• if true, will maintain a symlink that points to the most recently written file

Example:

<store>
category=sprockets
type=file
file_path=/tmp/sprockets
base_filename=sprockets_log
max_size=1000000
add_newlines=1
rotate_period=daily
rotate_hour=0
rotate_minute=10
</store>

Network Store Configuration

Network Stores forward messages to other Scribe Servers.

remote_host: name or ip of remote host to forward messages
remote_port: port number on remote host
timeout: socket timeout, in MS; defaults to DEFAULT_SOCKET_TIMEOUT_MS, which is set to 5000 in store.h
use_conn_pool: “yes” or anything else; defaults to false

• whether to use connection pooling instead of opening up multiple connections to each remote host

Example:

<store>
category=default
type=network
remote_host=hal
remote_port=1465
</store>

Buffer Store Configuration

Buffer Stores must have two sub-stores named “primary” and “secondary”. Buffer Stores will first attempt to Log messages to the primary store and only log to the secondary if the primary is not available. Once the primary store comes back online, a Buffer store will read messages out of the secondary store and send them to the primary store. Only stores that are readable (store that implement the readOldest() method) may be used as secondary store. Currently, the only readable stores are File Stores and Null Stores.

max_queue_length: 2,000,000 messages by default

• if the number of messages in the queue exceeds this value, the buffer store will switch to writing to the secondary store

buffer_send_rate: 1 by default

• determines, for each check_interval, how many times to read a group of messages from the secondary store and send them to the primary store

retry_interval: 300 seconds by default

• how long to wait to retry sending to the primary store after failing to write to the primary store

retry_interval_range: 60 seconds by default

• will randomly pick a retry interval that is within this range of the specified retry_interval

Example:

<store>
category=default
type=buffer
buffer_send_rate=1
retry_interval=30
retry_interval_range=10

<primary>
type=network
remote_host=wopr
remote_port=1456
</primary>

<secondary>
type=file
file_path=/tmp
base_filename=thisisoverwritten
max_size=10000000
</secondary>
</store>

Note! When the network connection is re-established, the messages from the secondary store are sent one whole file at a time. Thus max_size determines not only the size of the file that triggers rotation, but also the size of the network messages; if this is too large, the receiver may not be able to handle it. Best to keep it to a number that can be comfortably handled in memory. max_size does not limit the total number of messages that can be buffered (presumably that’s limited by the amount of space available on the filesystem).

Bucket Store Configuration

Bucket Stores will hash messages to multiple files using a prefix of each message as the key.
a Bucket Store must have a substore named “bucket” that is either a File Store or ThriftFile Store.

num_buckets: defaults to 1

• number of buckets to hash into
• messages that cannot be hashed into any bucket will be put into a special bucket number 0

bucket_type: “key_hash” or “key_modulo”
delimiter: must be an ascii code between 0 and 255; if 0, uses DEFAULT_DELIMITER (set in common.h)

• The message prefix up to(but not including) the first occurrence of the delimiter will be used as the key to do the hash/modulo

bucket_subdir: the name of each subdirectory will be this name followed by the bucket number
Example:

<store>
category=bucket_me
type=bucket
num_buckets=5
bucket_subdir=bucket
bucket_type=key_hash
delimiter=58

<bucket>
type=file
fs_type=std
file_path=/tmp/scribetest
base_filename=bucket_me
</bucket>
</store>

Null Store Configuration

Null Stores can be used to tell Scribe to ignore all messages of a given category.

(no configuration parameters)
Example:

<store>
category=tps_report*
type=null
</store>

Multi Store Configuration

A Multi Store is a store that will forward all messages to multiple sub-stores.

A Multi Store may have any number of substores named “store0″, “store1″, “store2″, etc
report_success: “all” or “any”, defaults to “all”

• whether all substores or any substores must succeed in logging a message in order for the Multi Store to report the message logging as successful

Example:

<store>
category=default
type=multi
target_write_size=20480
max_write_interval=1

<store0>
type=file
file_path=/tmp/store0
</store0>

<store1>
type=file
file_path=/tmp/store1
</store1>
</store>
<store>

Thriftfile Store Configuration

A Thriftfile store is similar to a File store except that it stores messages in a Thrift TFileTransport file.

file_path: defaults to “/tmp”
base_filename: defaults to category name
rotate_period: “hourly”, “daily”, or “never”; “never” by default

• determines how often to create new files

rotate_hour: 0-23, 1 by default

• if rotation_period is daily, determines what hour of day to rotate

rotate_minute 0-59, 15 by default

• if rotation_period is daily or hourly, determines how many minutes after the hour to rotate

max_size: 1,000,000,000 bytes by default

• determines approximately how large to let a file grow before rotating to a new file

write_meta: “yes” or anything else; false by default

• whether to log the following metadata in each file:
  1. 1. the length of each message is prepended to the message as an unsigned integer
     2. if the file was rotated, the last line will contain “scribe_meta<new_logfile>: ” followed by the next filename

fs_type: currently only “std” is supported; “std” by default
chunk_size: 0 by default

• if a chunk size is specified, no messages within the file will cross chunk boundaries unless there are messages larger than the chunk size

create_symlink: “yes” or anything else; “yes” by default

• if true, will maintain a symlink that points to the most recently written file

flush_frequency_ms: milliseconds, will use TFileTransport default of 3000ms if not specified

• determines how frequently to sync the Thrift file to disk

msg_buffer_size: in bytes, will use TFileTransport default of 0 if not specified

• if non-zero, store will reject any writes larger than this size

Example:

<store>
category=sprockets
type=thriftfile
file_path=/tmp/sprockets
base_filename=sprockets_log
max_size=1000000
flush_frequency_ms=2000
</store>