How To Connect Zeek To Python

I was recently asked how to send data from Zeek to Python. After flipping through the Zeek Broker documentation I couldn’t find a good example to reference, so here is my example.

The code for this demo is available here:

https://github.com/keithjjones/zeek-python-broker-demo

The first piece of our source code is the Python program here:

https://github.com/keithjjones/zeek-python-broker-demo/blob/master/broker-test.py

There is a prerequisite for this Python script, the Zeek Python Broker bindings must be downloaded and installed. You can find the installation instructions here:

https://docs.zeek.org/projects/broker/en/master/python.html#installation-in-a-virtual-environment

The trick is you must install the version of broker that corresponds to your Zeek’s version. Be sure to read that last sentence again. I figured this out the hard way.

Your version of Broker can be looked up by visiting the Zeek repository and selecting your Zeek version in the branch dropdown, such as v5.0.4. Then, go into the “auxil” directory to find Broker:

https://github.com/zeek/zeek/tree/v5.0.4/auxil

If you click into the Broker directory you will enter its repository at the version used in Zeek v5.0.4. If you click on “VERSION”, you will find version 2.3.5 of Broker is used in Zeek v5.0.4:

https://github.com/zeek/broker/blob/a7d55f8da2c47bf8b3de2524b24e1d8bfec1c2ce/VERSION

You can download Broker v2.3.5 here:

https://github.com/zeek/broker/releases

Once you download Broker and install the Python bindings using the instructions linked above, you can finally execute the Python script. Here is the script’s content:

import sys
import broker

# Setup endpoint and connect to Zeek.
with broker.Endpoint() as ep, \
    ep.make_subscriber("/topic/test") as sub:

    ep.listen("127.0.0.1", 60000)

    while True:
        (t, d) = sub.get()
        pong = broker.zeek.Event(d)
        print("received {}  --   {}".format(pong.name(), pong.args()))

        python_results = broker.zeek.Event("python_results", pong.args()[0]);
        ep.publish("/topic/test", python_results);

The script is relatively simple. The following lines set up a Zeek Broker listener on 127.0.0.1:60000 via Python:

https://github.com/keithjjones/zeek-python-broker-demo/blob/master/broker-test.py#L5-L8

Then, the Python script runs in an infinite loop pulling messages from Broker via the “get” function:

https://github.com/keithjjones/zeek-python-broker-demo/blob/master/broker-test.py#L11

Here “t” is the message’s topic, and “d” is the message’s raw data. Line 12 then translates the data pulled from Broker into a Python event object called “pong”. The name “pong” is not important, I wanted to show that this object could be named anything because the “.name()” function will tell you the true event name.

The last two lines in the script then create an event object and publishes it through Broker so that the “python_results” event will fire back in Zeek. The event will have the same argument that was originally passed to the Python process from Zeek (“pong.args()[0]”). Here is example output from the Python script:

received some_test_event  --   [(IPv4Address('172.20.32.121'), 47808/udp, IPv4Address('172.20.32.255'), 47808/udp)]
received some_test_event  --   [(IPv4Address('172.20.32.109'), 47808/udp, IPv4Address('172.20.32.255'), 47808/udp)]
received some_test_event  --   [(IPv4Address('172.20.32.200'), 47808/udp, IPv4Address('172.20.32.115'), 47808/udp)]
received some_test_event  --   [(IPv4Address('172.20.32.200'), 47808/udp, IPv4Address('172.20.32.112'), 47808/udp)]
received some_test_event  --   [(IPv4Address('172.20.32.200'), 47808/udp, IPv4Address('172.20.32.110'), 47808/udp)]
received some_test_event  --   [(IPv4Address('172.20.32.200'), 47808/udp, IPv4Address('172.20.32.121'), 47808/udp)]
received some_test_event  --   [(IPv4Address('172.20.32.200'), 47808/udp, IPv4Address('172.20.32.109'), 47808/udp)]

In the Zeek script content below you will find the event “python_results” is handled here:

https://github.com/keithjjones/zeek-python-broker-demo/blob/master/broker-test.zeek#L6

It just prints the arguments to the event. In “connection_state_remove” the script sends the “c$id” conn_id record to Python:

https://github.com/keithjjones/zeek-python-broker-demo/blob/master/broker-test.zeek#L11-L14

Therefore, this is the same data that will be printed when it is returned via “python_results”.

global test_topic = "/topic/test";

global some_test_event: event(c_id: conn_id);

event python_results(c_id: conn_id)
{
	print(cat("Got Python Results: ", c_id));
}

event connection_state_remove(c: connection)
{
    Broker::publish(test_topic, some_test_event, c$id);
}

event zeek_init()
{
	Broker::peer(addr_to_uri(127.0.0.1), 60000/tcp);
	Broker::subscribe(test_topic);
}

The two lines in “zeek_init” take care of connecting to the Python process before publishing the results to the “/topic/test” topic in Zeek’s Broker.

When running the Zeek script on a PCAP while the Python script is executing in another window, you will see the connection ID records printed from the Zeek and the Python processes in their respective windows as they are processed. This will prove that the data was transferred from your PCAP through Zeek, into Python over Zeek’s Broker, and back to Zeek again over Broker. This is example output from the Zeek script:

Got Python Results: [orig_h=172.20.32.200, orig_p=47808/udp, resp_h=172.20.32.115, resp_p=47808/udp]
Got Python Results: [orig_h=172.20.32.200, orig_p=47808/udp, resp_h=172.20.32.112, resp_p=47808/udp]
Got Python Results: [orig_h=172.20.32.200, orig_p=47808/udp, resp_h=172.20.32.110, resp_p=47808/udp]
Got Python Results: [orig_h=172.20.32.200, orig_p=47808/udp, resp_h=172.20.32.121, resp_p=47808/udp]
Got Python Results: [orig_h=172.20.32.200, orig_p=47808/udp, resp_h=172.20.32.109, resp_p=47808/udp]

Now imagine all the new types of processing you can do in Python that may have been more difficult with Zeek alone!