Skip to main content

Getting Started with Apache Flume NG: Flows, Agents and Syslog-to-HDFS Examples

By Anonymous
Struggling with delivery, architecture alignment, or platform stability?

I help teams fix systemic engineering issues: processes, architecture, and clarity.
→ See how I work with teams.


Apache Flume NG replaced the original master/collector architecture with lightweight agents that can be wired together to form flexible data flows. This guide explains what changed with Flume NG, how the agent–channel–sink model works, and walks through simple configurations for syslog ingestion to a console logger and to HDFS. It’s aimed at engineers who still operate Flume in legacy estates or need to understand it for migrations.

From Flume to Flume NG

Apache Flume is a distributed log and event collection service. With Flume NG, the project moved away from the original master/client and node/collector design and adopted a simpler, more robust architecture based on standalone agents.

Key changes introduced by Flume NG:

  • No external coordination service required for basic operation.
  • No master/client or node/collector roles—only agents.
  • Agents can be chained together to build arbitrary flows and fan-in/fan-out patterns.
  • Lightweight runtime; small heap sizes are sufficient for simple pipelines.
  • General-purpose exec source instead of dedicated tail/tailDir sources.

Requirements

To build Flume from source you need:

  • A JDK (e.g. 1.6+ in the original context; use a supported JDK for modern builds).
  • Maven 3.x.
  • Git or Subversion to fetch the source code.

Building Flume NG from Source

You can check out and build Flume using Git and Maven: 

git clone git://git.apache.org/flume.git
cd flume
git checkout trunk
mvn clean
mvn package -DskipTests

After a successful build, the distribution artifacts are located under: 

flume-ng-dist/target

Copy the desired distribution archive to the host where you want to run Flume, unpack it and you are ready to start configuring agents.

What Is a Flow in Flume NG?

A flow describes the full path of events from Source to Channel to Sink. Sinks can also feed into other agents, effectively becoming sources for downstream flows.

Conceptually, flows can look like this:

source                -> source => channel => sink
    \                /
     \-> channel => sink

source                -> channel => source => channel => sink

Flume NG runs one or more agents. Each agent hosts its own configured sources, channels and sinks.

The Configuration Model

Flume NG configuration is text-based and follows a logical pattern. For each agent, you declare:

  • A list of sources.
  • A list of channels.
  • A list of sinks.

The naming scheme is: 

<agentName>.sources
<agentName>.channels
<agentName>.sinks

and for each component: 

<agentName>.sources.<sourceName>.property = value
<agentName>.channels.<channelName>.property = value
<agentName>.sinks.<sinkName>.property = value

You are free to choose meaningful names for sources, channels and sinks; those names become the identifiers you wire together.

Example 1: Syslog to Console Logger

The following configuration (syslog-agent.cnf) defines a simple flow:

  • Source: receives syslog over TCP.
  • Channel: in-memory channel.
  • Sink: logger sink, prints events to stdout for debugging.
syslog-agent.sources  = Syslog
syslog-agent.channels = MemoryChannel-1
syslog-agent.sinks    = Console

# Source definition
syslog-agent.sources.Syslog.type = syslogTcp
syslog-agent.sources.Syslog.port = 5140

# Wiring
syslog-agent.sources.Syslog.channels = MemoryChannel-1
syslog-agent.sinks.Console.channel   = MemoryChannel-1

# Sink definition
syslog-agent.sinks.Console.type = logger

# Channel definition
syslog-agent.channels.MemoryChannel-1.type = memory

In this example, the agent syslog-agent listens on TCP port 5140 for syslog messages and writes every event to the console via the logger sink.

Example 2: Syslog to HDFS

To persist events into HDFS instead of logging to stdout, you can swap the sink to an HDFS sink: 

syslog-agent.sources  = Syslog
syslog-agent.channels = MemoryChannel-1
syslog-agent.sinks    = HDFS-LAB

# Source definition
syslog-agent.sources.Syslog.type = syslogTcp
syslog-agent.sources.Syslog.port = 5140

# Wiring
syslog-agent.sources.Syslog.channels   = MemoryChannel-1
syslog-agent.sinks.HDFS-LAB.channel    = MemoryChannel-1

# HDFS sink definition
syslog-agent.sinks.HDFS-LAB.type            = hdfs
syslog-agent.sinks.HDFS-LAB.hdfs.path       = hdfs://NN.URI:PORT/flumetest/%{host}
syslog-agent.sinks.HDFS-LAB.hdfs.filePrefix = syslogfiles
syslog-agent.sinks.HDFS-LAB.hdfs.rollInterval = 60
syslog-agent.sinks.HDFS-LAB.hdfs.fileType   = SequenceFile

# Channel definition
syslog-agent.channels.MemoryChannel-1.type = memory

This configuration listens for syslog events and writes them into HDFS, rolling files every 60 seconds with the prefix syslogfiles.

Starting an Agent

Flume NG runs one agent per process. To start an agent with a specific configuration file: 

bin/flume-ng agent -n YOUR_AGENT_NAME -f YOUR_CONFIG_FILE

For the syslog example: 

bin/flume-ng agent -n syslog-agent -f conf/syslog-agent.cnf

Once started, the agent will bind to the configured syslog port and begin routing events through the defined channel and sink.

Further Reading

Related guides:

If you need help with distributed systems, backend engineering, or data platforms, check my Services.

Labels:
Location: Stuttgart, Germany

Most read articles

Building a Model-Agnostic Multi-Agent System with OpenClaw

By
Over one week we rebuilt our AI stack around OpenClaw’s multi-agent architecture to avoid provider lock-in and stop wasting premium tokens. By aligning models to tasks, diversifying fallbacks across providers, enforcing minimal tool access, and switching to memory-first workflows with ephemeral sessions, we reduced token usage per task by about 70% and cut our monthly bill by 77% while improving operational resilience. How We Achieved 77% Cost Reduction and Provider Independence Over the past week, we rebuilt our AI infrastructure around OpenClaw’s multi-agent architecture. The result was a 77% cost reduction , provider independence , and a delegation system that routes work to the most cost-effective model for each job. Below is the technical journey of optimizing a 7-agent squad with OpenClaw. The Challenge: Model Provider Lock-In We started with a simple problem: our entire squad defaulted to a single model provider. This created three issues: Cost inefficiency beca...
Read more

BacNet => MQTT in Production: The Real Cost of Bridging BACnet to MQTT at Scale

By
bacnet2mqtt looks simple in a README and expensive in production. Once BACnet polling, reconnection behavior, stale state, and MQTT publishing collide, teams discover they are not deploying a lightweight adapter but operating infrastructure. This article breaks down where bacnet2mqtt works, where it becomes a bottleneck, and which production patterns reduce the operational damage before incidents, backlogs, and silent data loss turn a building integration into a long-running engineering problem. I inherited a building controls integration problem 18 months ago. Three office floors. 217 BACnet sensors covering temperature, occupancy, and HVAC actuators. The data was trapped inside the building automation network while the business wanted analytics, reporting, and compliance visibility in the data platform. The obvious answer looked easy enough: deploy bacnet2mqtt, bridge BACnet into MQTT, and push the stream into the lakehouse stack. The repository made it sound like a w...
Read more

Get Apache Flume 1.3.x running on Windows

By Anonymous
Since we found an increasing interest in the flume community to get Apache Flume running on Windows systems again, I spent some time to figure out how we can reach that. Finally, the good news - Apache Flume runs on Windows. You need some tweaks to get them running. Prerequisites Build system: maven 3x, git, jdk1.6.x, WinRAR (or similar program) Apache Flume agent: jdk1.6.x, WinRAR (or similar program), Ultraedit++ or similar texteditor Tweak the Windows build box 1. Download and install JDK 1.6x from Oracle 2. Set the environment variables    => Start - type " env " into the search box, select " E dit system environment variables ", click Environment Variables, Select " New " from the " Systems variables " box, type " JAVA_HOME " into " variable name " and the path to your JDK installation into "Variable value" (Example:  C:\Program Files (x86)\Java\jdk1.6.0_33 ) 3. Download maven from Apache 4. Set...
Read more