Patman is a tiny command-line tool designed for processing and manipulating raw text data. It excels in tasks like log filtering and aggregation, offering a range of operators for various annoying text operations.
Its reason for existence is in all those cases where grep and sed are not enough, but a dedicated script language is overkill or too slow.
Have you ever tried to parse GBs of logs while debugging a production incident?
Quick installation on Linux, macOS, or FreeBSD:
curl -sSfL https://raw.githubusercontent.com/lucagez/patman/main/install.sh | shQuick installation on Windows (powershell):
irm https://raw.githubusercontent.com/lucagez/patman/main/install.ps1 | iexDownload pre-built binaries from the releases page.
If you have Go installed:
go install github.com/lucagez/patman/cmd/patman@latestgit clone https://github.com/lucagez/patman.git
cd patman
go build -o patman ./cmd/patman
sudo mv patman /usr/local/bin/Patman can be extended with custom operators by implementing the Operator interface. The following example shows how to implement a uppercase operator.
package main
import (
"strings"
"github.com/lucagez/patman"
)
func Upper(line, arg string) string {
return strings.ToUpper(line)
}
func main() {
patman.Register("upper", patman.OperatorEntry{
Operator: Upper,
Usage: "converts line to uppercase",
Example: "echo 'hello' | patman 'upper(/)' # HELLO",
})
patman.Run()
}The operator can then be used as follows:
echo hello | patman 'upper(/)' # HELLOThe new operator will then be available also in the patman command help message.
The basic structure of a Patman command is:
patman [options] | '[operator1] |> [operator2] |> ...'The patman command takes in a list of operators and applies them to the input data. The |> symbol is used to pipe the output of one operator to the next. The patman command can be used in a standard unix pipeline with other commands.
Let's use as an example a log file containing the following lines:
2018-01-01 00:00:00 ERROR: Something went wrong.
2018-01-01 00:00:00 INFO: Something went right.
2018-01-01 00:00:00 ERROR: Something went wrong again.Match all lines containing the word "ERROR" and replace it with "WARNING":
cat logs.txt | patman 'matchline(ERROR) |> replace(WARNING)'Match all error lines and output a csv a timestamp and message columns:
cat logs.txt | patman 'matchline(ERROR)' | patman -format csv \
'split(,/1) |> name(timestamp)' \
'split(: /1) |> name(message)'This will output a csv file with the following contents:
timestamp,message
2018-01-01 00:00:00,Something went wrong.
2018-01-01 00:00:00,Something went wrong again.-help,-h: Show help message.-file: Specify the input file (default:stdin).-index: Define the index property for log aggregation.-format: Set the output format (default:stdout). One ofstdout,csv,jsonor a custom formatted string.-mem: Buffer size in MB for parsing larger file chunks.-delimiter: Custom delimiter for splitting input lines.-join: Custom delimiter for joining output (default:\n).-buffer: Size of the stdout buffer when flushing (default:1).
Patman includes a variety of operators for text manipulation:
Assigns a name to the output of an operator, useful for log aggregation and naming columns in csv or json formats. Usage:
echo something | patman 'name(output_name)'Matches the first instance of a regex expression. Usage:
echo hello | patman 'match(e(.*))' # elloMatches all instances of a regex expression. Usage:
echo hello | patman 'matchall(l)' # llReplaces text matching a regex expression with a specified string. Usage:
echo hello | patman 'replace(e/a)' # halloPerforms regex replacement using named capture groups. Usage:
echo hello | patman 'named_replace(e(?P<first>l)(?P<second>l)o/%second%first)' # ohellMatches entire lines that satisfy a regex expression. Usage:
cat test.txt | patman 'matchline(hello)' # ... matching linesReturns lines that do not match a regex expression. Usage:
cat test.txt | patman 'notmatchline(hello)' # ... non-matching linesSplits a line by a specified delimiter and selects a part based on index. Usage:
echo 'a b c' | patman 'split(\s/1)' # bFilters lines containing a specified substring. Way faster than grep for large files. Usage:
cat logs.txt | patman 'filter(hello)' # ... matching linesSplits a line by delimiter and selects field(s) by index or range. Usage:
echo 'a:b:c' | patman 'cut(:/0-1)' # a:b
echo 'a:b:c' | patman 'cut(:/1)' # bConverts line to uppercase. Usage:
echo 'hello' | patman 'uppercase()' # HELLOConverts line to lowercase. Usage:
echo 'HELLO' | patman 'lowercase()' # helloRemoves duplicate lines (keeps first occurrence). Usage:
cat logs.txt | patman 'ml(error) |> uniq(_)'Filters lines that are numerically greater than the provided number. Usage:
echo 101 | patman 'gt(100)' # 101Filters lines that are numerically greater than or equal to the provided number. Usage:
echo 100 | patman 'gte(100)' # 100Filters lines that are numerically less than the provided number. Usage:
echo 99 | patman 'lt(100)' # 99Filters lines that are numerically less than or equal to the provided number. Usage:
echo 100 | patman 'lte(100)' # 100Filters lines that are numerically equal to the provided number. Usage:
echo 100 | patman 'eq(100)' # 100Executes a JavaScript expression, passing x as the argument.
Usage:
echo something | patman 'js(x.toUpperCase())' # SOMETHING
echo hello | patman 'js(x + 123)' # hello123Splits a line by a specified delimiter and joins resulting parts with a newline character. Usage:
echo 'a b c' | patman 'explode(\s)'
# a
# b
# c
# Split by any character
echo something | patman 'explode(\.*/)'
# s
# o
# m
# e
# t
# h
# i
# n
# gThe MIT License (MIT)