HTrace
HTrace is a tracing framework intended for use with distributed systems written in java.
The project is hosted at http://github.com/cloudera/htrace.
The project is available in Maven Central with groupId: org.htrace, and name: htrace.
(It was formally at groupId: org.cloudera.htrace, and name: htrace).
API
Using HTrace requires some instrumentation to your application.
Before we get into that we have to review our terminology. HTrace
borrows Dapper's
terminology.
Span: The basic unit of work. For example, sending an RPC is a
new span, as is sending a response to an RPC.
Span's are identified by a unique 64-bit ID for the span and another
64-bit ID for the trace the span is a part of. Spans also have other
data, such as descriptions, key-value annotations, the ID of the span
that caused them, and process ID's (normally IP address).
Spans are started and stopped, and they keep track of their timing
information. Once you create a span, you must stop it at some point
in the future.
Trace: A set of spans forming a tree-like structure. For example, if you are running a distributed big-data store, a trace might be formed by a put request.
To instrument your system you must:
1. Attach additional information to your RPC's.
In order to create the causal links necessary for a trace, HTrace
needs to know about the causal
relationships between spans. The only information you need to add to
your RPC's is two 64-bit longs. If tracing is enabled (Trace.isTracing()
returns true) when you send an RPC, attach the ID of the current span
and the ID of the current trace to the message.
On the receiving end of the RPC, check to see if the message has the
additional tracing information above. If it does, start a new span
with the information given (more on that in a bit).
2. Wrap your thread changes.
HTrace stores span information in java's ThreadLocals, which causes
the trace to be "lost" on thread changes. The only way to prevent
this is to "wrap" your thread changes. For example, if your code looks
like this:
Thread t1 = new Thread(new MyRunnable());
...
Just change it to look this:
Thread t1 = new Thread(Trace.wrap(new MyRunnable()));
That's it! Trace.wrap()
takes a single argument (a runnable or a
callable) and if the current thread is a part of a trace, returns a
wrapped version of the argument. The wrapped version of a callable
and runnable just knows about the span that created it and will start
a new span in the new thread that is the child of the span that
created the runnable/callable. There may be situations in which a
simple Trace.wrap()
does not suffice. In these cases all you need
to do is keep a reference to the "parent span" (the span before the
thread change) and once you're in the new thread start a new span that
is the "child" of the parent span you stored.
For example:
Say you have some object representing a "put" operation. When the
client does a "put," the put is first added to a list so another
thread can batch together the puts. In this situation, you
might want to add another field to the Put class that could store the
current span at the time the put was created. Then when the put is
pulled out of the list to be processed, you can start a new span as
the child of the span stored in the Put.
3. Add custom spans and annotations.
Once you've augmented your RPC's and wrapped the necessary thread
changes, you can add more spans and annotations wherever you want.
For example, you might do some expensive computation that you want to
see on your traces. In this case, you could start a new span before
the computation that you then stop after the computation has
finished. It might look like this:
Span computationSpan = Trace.startSpan("Expensive computation.");
try {
//expensive computation here
} finally {
computationSpan.stop();
}
HTrace also supports key-value annotations on a per-trace basis.
Example:
Trace.currentTrace().addAnnotation("faultyRecordCounter".getBytes(), "1".getBytes());
Trace.currentTrace()
will not return null
if the current thread is
not tracing, but instead it will return a NullSpan
, which does
nothing on any of its method calls. The takeaway here is you can call
methods on the currentTrace()
without fear of NullPointerExceptions.
Samplers
Sampler
is an interface that defines one function:
boolean next(T info);
All of the Trace.startSpan()
methods can take an optional sampler.
A new span is only created if the sampler's next function returns
true. If the Sampler returns false, the NullSpan
is returned from
startSpan()
, so it's safe to call stop()
or addAnnotation()
on it.
As you may have noticed from the next()
method signature, Sampler is
parameterized. The argument to next()
is whatever piece of
information you might need for sampling. See Sampler.java
for an
example of this. If you do not require any additional information,
then just ignore the parameter.
HTrace includes a sampler that always returns true, a
sampler that always returns false and a sampler returns true some
percentage of the time (you pass in the percentage as a decimal at construction).
Trace.startSpan()
There is a single method to create and start spans: startSpan()
.
For the startSpan()
methods that do not take an explicit Sampler, the
default Sampler is used. The default sampler returns true if and only
if tracing is already on in the current thread. That means that
calling startSpan()
with no explicit Sampler is a good idea when you
have information that you would like to add to a trace if it's already
occurring, but is not something you would want to start a whole new
trace for.
If you are using a sampler that makes use of the T info
parameter to
next()
, just pass in the object as the last argument. If you leave it
out, HTrace will pass null
for you (so make sure your Samplers can
handle null
).
Aside from whether or not you pass in an explicit Sampler
, there are
other options you have when calling startSpan()
.
For the next section I am assuming you are familiar with the options
for passing in Samplers
and info
parameters, so when I say "no
arguments," I mean no additional arguments other than whatever
Sampler
/info
parameters you deem necessary.
You can call startSpan()
with no additional arguments.
In this case, Trace.java
will start a span if the sampler (explicit
or default) returns true. If the current span is not the NullSpan
, the span
returned will be a child of the current span, otherwise it will start
a new trace in the current thread (it will be a
ProcessRootMilliSpan
). All of the other startSpan()
methods take some
parameter describing the parent span of the span to be created. The
versions that take a TraceInfo
or a long traceId
and long
parentId
will mostly be used when continuing a trace over RPC. The
receiver of the RPC will check the message for the additional two
longs
and will call startSpan()
if they are attached. The last
startSpan()
takes a Span parent
. The result of parent.child()
will be used for the new span. Span.child()
simply returns a span
that is a child of this
.
Testing Information
The test that creates a sample trace (TestHTrace) takes a command line argument telling it where to write span information. Run mvn test -DspanFile="FILE_PATH" to write span information to FILE_PATH. If no file is specified, span information will be written to standard out. If span information is written to a file, you can use the included graphDrawer python script in tools/ to create a simple visualization of the trace. Or you could write some javascript to make a better visualization, and send a pull request if you do :).